Statistical Paradigm

Transcription

1 EE 688 Overvew of Sascal Models for Vdeo Indexng Prof. ShhFu Chang Columba Unversy TA: Erc Zavesky Fall 7, Lecure 4 Course web se: hp:// Sascal Paradgm Many problems can be posed as paern recognon Image classfcaon: ndoor vs. oudoor? Face? sho boundary deecon, sory segmenaon Is he curren pon a boundary? Sascal models o handle uncerany and provde flexbly Image processng ools avalable E.g., homework # Rch ools for learnng and predcon See course web se Increasng daa avalable NIST TREC Vdeo: 3+ hours Consumer and youube vdeos

2 A Very HghLevel Sa. Paern Recog. Archecure (From Jan, Dun, and Mao, SPR Revew, 99) Imporan ssues () Image/vdeo processng Wha s he adequae qualy, resoluon, ec? Feaure exracon Color, exure, moon, regon, shape, neres pons, audo, speech, ex, ec Feaure represenaon Hsogram, bag, graph ec Invarance o scale, roaon, ranslaon, vew, llumnaon, How o reduce dmensons?

3 Imporan ssues () Dsance measuremen How o measure smlary beween mages/vdeos? L, L, Mahalanobs, Earh Mover s dsance, vecor/graph machng Classfcaon models Generave vs. dscrmnave Mulmodal fuson, early fuson vs. lae fuson E.g., how o use jon audovsual feaures o deec evens (dancng, weddng ) Effcency ssues how o speed up ranng and esng processes? How o rapdly buld a model for new domans Valdaon and evaluaon How o measure performance? Are models generalzable o new domans? Three relaed problems Rereval, Rankng Gven a query mage, fnd relevan ones May apply rank hreshold o decde relevance Classfcaon, caegorzaon, deecon Gven an mage x, predc class label y Cluserng, groupng Group mages/vdeos no clusers of dsnc arbues 3

4 An example News sory segmenaon usng mulmodal, mulscale feaures Frs Undersand Daa Types and Explore Unque Characerscs (3.%) (8.8%) Percenage of conen (a): regular anchor segmen (5.%) (b) dfferen anchor (.3%) (c): mulsory n an anchor seg. (d): con. spors brefngs (e): con. shor brefngs (f) sep. by musc or anm. (g): weaher repor (h): anchor leadn before comm. (): comm. afer spors : sory : weaher : commercal : msc./anmaon : vsual anchors 4

5 News Sory Segmenaon Objecve: a sory boundary a me τ k? τ = { sho boundares or sgnfcan pauses} k observaon τ k {vdeo, audo} me τ τ k k + An anchor face? moon changes? change from musc o speech? speech segmen? {cue words} appear {cue words} j appear Modaly Vdeo Audo Tex Msc. Need o decde how o formulae feaures Challenge: dverse feaures Raw Feaures moon sho boundary face commercal pause pch jump sgnfcan pause musc./spch. dsc. spch seg./rapdy ASR cue erms VOCR cue erms ex seg. score combnaoral spors Daa Type segmen pon segmen segmen pon pon pon segmen segmen pon pon pon pon segmen Value connuous bnary connuous bnary connuous connuous connuous bnary connuous bnary bnary connuous bnary bnary ex seg. score musc comm. sgpas face sho moon canddae pon One way s o use bnary predcae: f x > hreshold, hen predc segmen boundary (b=) 5

6 6 Example Predcaes The surroundng observaon wndow has a pause wh he duraon larger han.5 second. Pause A speech segmen sars n he surroundng observaon wndow Speech segmen 6 A commercal sars n 5 o seconds afer he canddae pon. Commercal 7 A speech segmen ends afer he canddae pon Speech segmen 8 A speech segmen before he canddae pon Speech segmen 5 An anchor face segmen occupes a leas % of nex wndow Anchor face 9 An audo pause wh he duraon larger han. second appears afer he boundary pon. Pause 3 An anchor face segmen jus sars afer he canddae pon Anchor Face Sgnfcan pause Sgnfcan pause & noncommercal raw feaure se The surroundng observaon wndow has a sgnfcan pause wh he pch jump nensy larger han he normalzed pch hreshold. and he pause duraon larger han.5 second. 4 A sgnfcan pause whn he noncommercal secon appears n he surroundng observaon wndow. Predcaes no Collec Feaures from Tranng Samples b One ranng sample Each row represens one predcae Face Moon Sgnfcan Pause Speech segmen Commercal Tex segmenaon score ASR cue erms f

7 Choose Model Maxmum enropy model For example λ λ λ ( = ) = /( + ) λ λ λ ( = ) = /( + ) qλ ( b x) = e Z ( x) λ f ( xb, ) where f ( x, b), b {,} predcae f = ' anchor face' f = ' sgnfcan pause' f curren observaon: face = YES pause = NO qb YES x e e e qb NO x e e e λ Classfcaon: f q(b=yes x) >.5, hen predc YES. Background: Enropy Enropy (bs) H = KullbackLebler (KL) Dsance A measure of dsance beween dsrbuons D KL D KL, and = ( p( x), q( x) ) ff p m = p log p = or = ( ) = q( ) x q( x) q( x)log p( x) q( x) q( x)log dx p( x) No necessarly symmerc, may no sasfy rangular nequaly 7

8 How o Deermne he Weghs n he Model? Esmae q ( b x) from ranng daa T = {( x λ k, bk)} by mnmzng KullbackLebler dvergence, defned as pb ( x) D( p qλ ) = p ( b, x)log p x b qλ ( b x) = pxb (, )log q( b x) + consan( p ) Fnd o maxmze he enropy L ( ) (, )log ( ) p q p λ x b qλ b x Ieravely fnd λ = λ + Δλ λ x x b b λ pxb (, ) f(, ) log, xb xb Δ λ = M pxq ( ) ( b x) f(, ) xb, λ xb The objecve funcon s convex. So he erave process can reach he opmum. λ emprcal dsrbuon from daa D( p q) q esmaed model The same model used o selec feaures Inpu: collecon of canddae feaures, ranng samples, and he desred model sze Oupu: opmal subse of feaures and her correspondng exponenal weghs Curren model q augmened wh feaure h wh wegh α, α h( x, b) e q( b x) qα, h( b x) = Zα ( x) Selec he canddae whch mproves curren model he mos, n each eraon; { { α, h }} α { { Lp qα, h Lp q }} * h = D p q D p q h C arg max sup ( ) ( ) = arg max sup ( ) ( ) h C α q Reducon of dvergence Increase of loglkelhood 8

9 Opmal Feaures (from CNN news vdeo) * The frs A+V feaures auomacally dscovered for he CNN channel no raw feaure se Anchor Face Sgnfcan pause & noncommercal gan λ nerpreaon An anchor face segmen jus sars afer he canddae pon A sgnfcan pause whn he noncommercal secon appears n he surroundng observaon wndow. 3 Pause An audo pause wh he duraon larger han. second appears afer he boundary pon. 4 Sgnfcan pause The surroundng observaon wndow has a sgnfcan pause wh he pch jump nensy larger han he normalzed pch hreshold. and he pause duraon larger han.5 second. 5 Speech segmen A speech segmen before he canddae pon 6 Speech segmen A speech segmen sars n he surroundng observaon wndow 7 Commercal.5.78 A commercal sars n 5 o seconds afer he canddae pon. 8 Speech segmen..47 A speech segmen ends afer he canddae pon 9 Anchor face.6.75 An anchor face segmen occupes a leas % of nex wndow Pause The surroundng observaon wndow has a pause wh he duraon larger han.5 second. every modaly helps : anchor face, prosody, and speech segmen Issues of hs model (Dscusson) Feaures Bnary predcaes reasonable? Does capure he unque characerscs? qλ ( b x) = e Z ( x) Models Exponenal models wh lnear weghs adequae? How abou he learnng algorhm? Enough daa o learn he probably models? Speed and complexy bnary predcae: f x > hreshold, hen predc segmen boundary (b=) λ λ f( xb, ) 9

10 A Broader Perspecve: Classfcaon Paradgms Lkelhood P(x C=) > P(x C=)? Class Class hreshold Generave x feaures x Decson f(x) > Boundary f(x) < x f(x) dscrmnan funcon Dscrmnave Whch one does he prevous model fall no? Generave Models Gaussan Mxure Model

11 One common ssue s o learn probably models Gaussan dsrbuon p( x) = e πσ Gven he same mean and varance, Gaussan has he max enropy Sum of a large number of small, ndependen random varables approaches Gaussan ( x μ ) σ Pr[ x μ σ ].68 Pr[ x μ σ ].95 Pr[ x μ 3σ ].997 Mahalanobs dsance from x o μ r = x μ / σ Enropy of Gaussan : H gau =.5 + log ( πσ ) Comparson w. unform Mulvarae Gaussan Mulvarae Gaussan, T ( x μ) Σ ( x μ) N( μ, Σ) p( x) = e D / ( π ) Σ where x, μ are Ddmensonal vecors Σ : D D marx Σ s he deermnan of Σ x x σ Σ = x σ General Σ ( σ ) =Σ (, ) = cov( ( ), ( )) j j x x j = E[( x ( ) μ( ))( x( j) μ( j))] x

12 Effec of Lnear Transformaon Lnear ransformaon of Gaussan y= A x y N( A μ, AΣA) y: k, A: d k, x: d Whenng ransform Σ=ΦΛΦ (SVD, Egenvecors) Φ : [ φ φ... φ ] columns are orhogonal ev Λ: dag. marx of egenvalues A Σ A= AΦΛΦ A= I d Whenng Trans. also PCA Transform y = A x N( A μ, I) w w A w =ΦΛ / Mahalanobs Dsance Mahalanobs ds n D p( x) = e πσ ( x μ ) σ Mahalanobs dsance from x o μ r = x μ / σ Pr[ x μ σ] = Pr[ r ].68 Pr[ x μ σ] = Pr[ r ].95 Pr[ x μ 3 σ] = Pr[ r 3].997 Mul Dmensonal case p( x) = exp( ( x μ) Σ ( x μ) ) D / ( π ) Σ = exp( / ( x μ) ΦΛ Φ ( x μ) ) D ( π ) Σ = exp( ( A / w ( x μ) ) ( Aw ( x μ) )) D π Σ ( ) r s he Mahalanobs dsance s also he Eucldean ds n he PCA space r

13 Malalanobs dsance from pon x o he mean of a Gaussan x u Gaussan Used In Classfcaon ( j ) ( ) x C, f p C x p C x, when j j MAP classfer p ML classfcaon: ( Cj x) pror lkelhood pxc ( j) pc ( j) = px ( ) ( j ) evdence f unform p C C = arg max p( x C) pxc ( ) can be modeled by Gaussans j C 3

14 How o Esmae Gaussan Model Parameers? l = ln P( xk θ ) = ln( πσ ) ( x ) k μ σ (ln Px ( k θ )) ( xk μ ) μ θ θl = = ( x ) (ln Px ( )) k μ k θ θ σ ( σ ) xk n θ ln Px ( k θ ) = k ˆ μ = ˆ σ = ( x ˆ k μ) n n k = θ = (μ, σ ) k MulDmensonal θ= ( μ, Σ) ˆ (/ ) ˆ μ = n x Σ= (/ n) ( x μ)( x μ) k k k k k ML esmaor: mean > sample mean, varance >based sample varance Mxure Of Gaussans Real dsrbuons seldom follow a sngle Gaussan mxure of Gaussans p(x) π π p( x) = p( x, z) = pzpx z ( ) ( z ) Gven daa x,, x N, defne loglkelhood: N ( π μ π μ ) l= log( p( x )) = log N( x, Σ ) + N( x, Σ ) n responsbly of ( componen ) z z ( μ, ) = π N x Σ z z z = Z z= π z ( π ) T ( x μz) Σz ( x μz) n n n n= Poseror probably of x beng generaed by a specfc componen D / ( ) poserers = τ = p z = x, θ, θ = { μ, Σ, μ, Σ } Σ z e x 4

15 EM Opmzaon Mehod log lkelhood Maxmzaon of l(θ) drecly s hard due o log_of_sum Insead, look a Δl( θ ) = l( θ ) l( θ ), θ : curren esmaon of θ Jensen s Inequaly If f s concave, e. g., f log If f s convex, f N n= ( x n, z ) l( θ ) = log p θ f ( E{ x} ) E( f{ x} ) f ( E{ g( x) }) E( f{ g( x) }) ( x) = log( x) ( E{} x ) E( f{ x} ) z px p log ( x ), Log(x) p(x) where x p = Auxlary Funcon n EM Δ l( θ ) = l( θ) l( θ ) = log p( x θ ) log p( x θ ) N p x = log p x N N ( n θ ) ( θ ) n= n N = log p x p z x, log ( θ ) n n n= n= = log ( n, z θ ) (, z θ ) n n= z p xn N ( θ ) ( θ ) N p( xn, z θ ) p( x θ ) ( θ ) ( θ ) n= z n p x, z p x, z p x p x, z n n n= z n n N = ( θ ) log p z x, ( n, z θ ) (, z θ ) p x n n= z p xn ( ) = Q θ θ margnalzaon Jensen s nequaly Noe here s no log_of_sum. So akng dervave s easer 5

16 EM mproves lkelhood Auxlary funcon derved based on Jensen s Inequaly, N Now esmae θ + by maxmzng Q θ+ = arg maxq( θ θ ) θ ( z x, θ ) log p( x, z ) + cons Q( θ θ ) p z n n θ = n= expecaon over z wh curren θ jon lkelhood of observed & hdden z So n he expecaon sep, compue τ n, he responsbly of componen z for sample x n In he maxmzaon sep, ake dervave of Q over θ, and fnd he new esmae for θ (Noe only sum_of_log s nvolved) EM Always Improves Lkelhood Why does EM always mprove l(θ)? Δ l( θ ) = l( θ ) l( θ ) Q( θ θ ) Q( θ + θ ) = max Q( θ θ ) Q( θ θ ) θ N ( z x, θ ) log p( x, z ) + cons Q( θ θ ) p z n n θ = General seps of EM: n= expecaon over jon lkelhood of z wh curren θ observed & hdden = l θ + Δ ( ) Defne lkelhood model wh parameers θ Idenfy hdden varables z Derve he auxlary funcon and he E and M equaons In each eraon, esmae he poserors of hdden varables Reesmae he model parameers. Repea unl sop 6

17 ExpecaonMaxmzaon (EM) Soluon of GMM τ EM for esmang θ and. N ( z x, θ ) log p( x, z ) + cons Q( θ θ ) p z n n θ = n= expecaon over z wh curren θ jon lkelhoodof observed & hdden Follow dvde and conquer prncple. In eraon sep : ( ) ( ) ( ) () π N( xn μ, Σ ) Expecaon : τ n = ( ) () () Wegh from componen π N( xn μ j, Σ j ) j T ( ) τ n x () () () n τ n ( xn μ )( xn μ ) ( + ) n Maxmaon : μ = ( + ) n () Σ = τ () n τ n n n () τ n ( ) n Dvde daa o each group, π + = Compue mean and varance N from each group Dscrmnave Models 7

18 Smple Dscrmnave Classfer Fnd mos opporunsc dmenson n each sep Selecon creron Enropy Varance before / afer Sop creron Avod overfng x() TH o o o o + ++ o o o o + TH x() x(3) x()>th Y N C + x()>th Y N C o C + Paramerc Dscrmnan Analyss Example of dscrmnan funcon Lnear and quadrac dscrmnan x() + o o + o + Decson o boundary o + x() g( x) = ax + bx x() gx ( ) > o o Decson o o o o o boundary o + o o o gx ( ) < x() g( x) = ax + bx + cx x 8

19 Lnear Dscrmnan Classfers g( x) = w x+ w Augmened Vecor x y = = x x d fnd wegh w a w w w w w d = = map y o class ω f g( y)>, oherwse class ω Desgn Objecve ay ay >b, f class ω, < b, f class ω, y, b> Each y defnes a half plane n he wegh space (a). Noe we search wegh soluons n he aspace. and bas wo g( x) = g( y) = a y aspace Suppor Vecor Machne (uoral by Burges 98) Look for separaon plane wh he hghes margn Decson boundary : H wx+ b= Lnearly separable wx + b + x n class ω.e. y =+ wx + b x n class ω.e. y = Inequaly consrans : ( label ) ( wx + b), Two parallel hyperplanes defnng he margn hyperplane H( H ) : + wx+ b=+ hyperplane H( H ) : wx+ b= Margn: sum of dsances of he closes pons o he separaon plane margn = / w Bes plane defned by w and b 9

20 Fndng he maxmal margn mnmze w Use he Lagrange mulpler echnque for he consraned op. problem mnmze L wr... w and b p L = ( y ( + b) ) l p w α w x = α dl l p = w= αyx dw = dl l p = αy = db = subjec o nequaly consrans y ( wx + b) =,, l ; y s label maxmze L wr... w and b L D j j j = = j= = l l l = α αα y y x x wh condons : l α α y = D Quadrac Programmng Prmal Problem Dual Problem Prme and Dual have he same soluons of w and b KKT condons l * w = αyx = α > Wegh sum from posve class = Wegh sum from negave class Drecon of w: roughly from negave suppor vecors o posve ones w α = f α >, x s on H+ or H and s a suppor vecor

21 Nonseparable: no every sample can be correcly classfed Add slack varables ξ f ξ >, hen x s msclassfed (.e. ranng error) Lagrange mulpler: mnmze New objecve funcon Ensure posvy KKT Condons for nonseparable Soluons If < α < C, hen ξ = : x s on H or H If α = C, hen ξ > : x s nsde he margn or on he wrong sde or ξ = : x s on H or H

22 All he pons locaed n he margn gap or he wrong sde wll ge α = C Wha f C ncreases? b and ξ boh < α < C α = C afer C ncreases When C ncreases, ncorrec samples ge more weghs ry o mnmze ncorrec samples beer ranng accuracy, bu smaller margn less generalzaon performance Generalzed Lnear Dscrmnan Funcons Include more han jus he lnear erms d d d g( x ) = w + wx + w xx = w + w x + x Wx In general Example j j = = j= Shape of decson boundary ellpsod, hyperhyperbolod, lnes ec d g( x) = aφ ( x) = aφ = gx ( ) = a+ ax+ ax 3 = [ a a a3] x x g( x) = a x + a x + a x x 3 = [ a a a ][ x x x ] 3 Daa become separable n hgherdmensonal space learnng parameers n hgh dmenson s hard (curse of dmensonaly) nsead, ry o maxmze margns SVM

23 NonLnear Space Map o a hgh dmensonal space, o make he daa separable Fnd he SVM n he hghdm space N s g( x) = α yφ( s ) Φ ( x) + b = w Luckly, we don have o fnd We can use he same mehod o maxmze L D o fnd l l l L = α αα y y Φ( x ) Φ( x ) D j j j = = j= l l l α αα jyyk j ( x, x j) = = j= = Φ( s ) nor α y Φ( s ) l = Insead, we defne kernel K ( s, x) = Φ( s ) Φ( x) N s g( x) = α yk( s, x) + b = α Some popular kernels polynomal Gaussan Radal Bass Funcon (RBF) sgmodal neural nework separable Cubc polynomal nonseparable 3

24 See he SVM demos (Erc) Evaluaon V n = " Relevan" " Irrelevan " n = N N Tes Image K deeced resuls K R = A/( A + C ) Deecon A = V Recall n n K False Alarms B = ( V n n ) Precson = P = A/( A + B) N Msses C = ( V A n n) = Fallou N Correc Dsmssals D = ( ( V B F = B /( B + D) n n)) = Combned Reurned D B A C Relevan Ground Truh P R F = ( P + R) / 4

25 Evaluaon Measures Precson Recall Curve P R. Recever Operang Characersc (ROC Curve) A (h) A vs B B(false) Evaluaon Merc: Average Precson S Ranked ls of daa n response o a query D5 D8 D63 D Ds Ground ruh Precson / / /3 3/4 3/5 3/6 3/7 s R j Average precson: AP = I j, R j= j R : oal number of relevan daa AP measures he average of precson values a R relevan daa pons Precson. P 3 R j j j 5

26 Evaluaon Merc: Average Precson () AP depends on he rankngs of relevan daa and he sze of he relevan daa se. E.g., R= Case I: Pre: AP= Case II: + Pre: / / / / / / / / / / AP=/ Case III: Pre: / / / AP~.3 Example: SVM for News Sory Segmenaon Precson SVMbased Maxmum Enropy SVM wh 95 bnary feaures performs he bes SVM has excellen feaure fuson capably Predcae bnarzaon shelds nose n he feaure BST Recall 6

27 Tranng / Valdaon / Tesng x() Tranng Use hs o opmze parameers x() x() Valdaon Selec opmal models hrough valdaon x() x() Tesng Evaluae performance over es daa x() Approprae f he same dsrbuons are followed over dfferen ses Tranng / Valdaon / Tesng (con.) Cross valdaon, leaveoneou K Tranng Tesng Roae he choce of he es se and average he performance over runs 7

28 Curse of Dmensonaly and Overranng x() overfng A case of overfng x() es ranng es performance Very rough rule of humb # of ranng samples per class feaure dmenson > 8