EE 6885 Statistical Pattern Recognition

Transcription

1 EE 6885 Sascal Paer Recogo Fall 005 Prof. Shh-Fu Chag hp://.ee.columba.edu/~sfchag Reve: Fal Exam (//005) Reve-Fal- Fal Exam Dec. 6 h Frday :0-3 pm, Mudd Rm 644 Reve Fal-

2 Chap 5: Lear Dscrma Fucos Reve Fal-3 Lear Dscrma Classfers g( x) = x Augmeed Vecor 0 y fd egh ad bas o x x x d = = a 0 d 0 = = g( x) = g( y) = a y map y o class ω f g( y)>0, oherse class ω r [, x] Normalzao y class ω, y ( y ) y-space Desg Obecve a y >b, y a-space Reve Fal-4

3 Mmal Squared-Error Soluo y Y y = y rag sample marx dmeso: x (d) s ay b = defe J = ( ) = ajs Y ( Ya b) = 0 a= YY Yb= Yb ( ) ( ) Y = Y Y Y Obecve: a y =b, y Example rag samples: class ω: (, ),(,0) class ω : (3,),(,3) 0 * Y = b = fd Y, he compue a = Y b 3 3 Reve Fal-5 = Ya b = ( Ya b) ( Ya b) pseudo-verse : (d) x Vecor Dervave (Grade) ad Cha Rule Cosder scalar fuco of vecor pu: J ( x) Vecor dervave (grade) xj( x)=[ J / x, J / x,, J / x ] d = er produc J ab= akbk = a ab Marx-vecor mulplcao Herma J x Ax x A x b = = k b ab= ba b = a Geeralzed cha rule o cosder x= Ax, e.. x = Ax δx J J x = x x δx x A x x = A x A x J = A x J J = Ab b b = A δx / δ x = A HW#5 P. Reve Fal-6 3

4 Chap. 5. ad Burges 98 paper: Suppor Vecor Mache Reve Fal-7 Suppor Vecor Mache (uoral by Burges 98) Look for separao plae h he hghes marg Decso boudary : H 0 HW#5 P. 0 x b= Learly separable x b x class ω.e. y = x b x class ω.e. y = Iequaly cosras : y ( x b) 0, o parallel hyperplaes defg he marg hyperplae H( H ) : x b= hyperplae H( H ) : x b= Marg: sum of dsaces of he closes pos o he separao plae marg = / Bes plae defed by ad b Reve Fal-8 4

5 Fdg he maxmal marg mmze Use he Lagrage mulpler echque for he cosraed op. problem mmze L r... ad b p L = ( y ( b) ) l p α x = α 0 dl l p = 0 = αyx d = dl l p = 0 αy = 0 db = subec o equaly cosras y ( x b) 0 =,, l maxmze L r... ad b L D = = = = l l l = α αα y y x x h codos : l α 0 α y = 0 D Quadrac Programmg Prmal Problem Dual Problem Reve Fal-9 HW#6 P. KK codos for separable case l * = αyx = α > 0 α = 0 Ho o compue ad b? Ho o classfy e daa? f α > 0, x s o H or H ad s a suppor vecor Reve Fal-0 5

6 No-separable Add slack varables ξ f ξ >, he x s msclassfed (.e. rag error) Lagrage mulpler: mmze Ne obecve fuco Esure posvy Reve Fal- All he pos locaed he marg gap or he rog sde ll ge α = C 0 < α C α = C afer C creases Whe C creases, samples h errors ge more eghs beer rag accuracy, bu smaller marg less geeralzao performace Reve Fal- 6

7 Mappg o Hgher-Dmeso Space Map o a hgh dmesoal space, o make he daa separable Fd he SVM he hgh-dm space (embeddg space) N s g( x) = α yφ( s ) Φ ( x) b = defe kerel K ( s, x) = Φ( s ) Φ( x) N s = g( x) = α yk( s, x) b We ca use he same mehod (Dual Problem) o maxmze L D o fd l l l L = α αα y y Φ( x ) Φ( x ) D = = = l l l α αα yyk ( x, x ) = = = = HW#5 P. α Reve Fal-3 Chap. 9 : Aalyss of Learg Algorhms Reve Fal-4 7

8 Bas vs. varace for esmaor Assume F s a quay hose value s o be esmaed daa source Radomly dra samples D = x, x, x { } expeced esmao error: E D gd F = E ( g ) F E g E ( g ) D D D D D D g D Lear o esmae F Repea mulple mes Bas Varace Reve Fal-5 Bas vs. varace for classfcao Groud ruh: D Gaussa Complex models have smaller bases, more varaces ha smple models Icreasg rag pool sze helps reduce he varace Occam s Razor prcple Reve Fal-6 8

9 Boosg For each compoe classfer, use he subse of daa ha s mos formave gve he curre se of compoe classfers rag daa D = { x, x, x} Radomly dra a subse of samples D Use he mos formave subse D from remag se Weak classfer C Weak classfer C classfer C k Classfer Fuso Reve Fal-7 HW#7 P. As AdaBoos Ref. Reve Fal-8 9

10 Fal Classfer h f Whe ll he fal classfer be correc? Suppose c()=0, he h f () s correc f I geeral amely (log β ) h( ) log( β ) = = h () / β β = = h β β h ( ) c( ) f( ) s correc f = = D β h ( ) c( ) () β D () = = Reve Fal-9 / D() / h ( ) / β β = = D ( ) β =? / hf( ) s correc f D( ) β N N D ( ), h () c (), h () c () = f heorem Ref. N N ( ( β)( E)) = = N N ( E) = = f β = / / N ( E) = = = E β =?? / E ( E ) / β = ( E ( E )) = = = / Ref. E β = E Fll deals o complee HW7 P. Reve Fal-0 0

11 AdaBoos Learg he frs o feaures afer feaure seleco Reve Fal- Cascade classfer for effcecy Break a large classfer o cascade of smaller classfers E.g., 00 feaures o {, 0, 5, 50, 50} Adus hreshold early sage so ha reecs ulkely regos quckly Desg radeoffs Number of feaures each classfer hreshold uses each classfer Number of classfers Add sages ul obecve P-R s me P R Reve Fal-

12 Mxure of Expers Each compoe classfer s reaed as a exper he predcos from each exper are pooled ad fused by a gag subsysem k P( y x, Θ ) = P( r x, θ0 ) P( y x, θr ) r= here x s he pu paer, y s he oupu Pr ( x, θ ) Deerme 0,.e., mxure prors? Maxmze daa lkelhood grade dece or EM k ld (, Θ ) = l Pr ( x, θ ) P( y x, θ ) r= 0 r Reve Fal-3 Chap. 0 : feaure dmeso reduco ad cluserg Reve Fal-4

13 PCA for feaure dmeso reduco Approxmae daa h reduced dmesos -D approxmao xˆ = mae, m: mea J ˆ () e = x k x k = ( m a e k ) x k k= k= ak e ake ( xk m) xk m = ( ) e xk m xk m k= k= k= k= k= e ( k )( k ) k x m x m e x m = ese xk m k= k= k= Approxmao Error = = S: scaer marx Opmal e mmzg error J = ( ) sample covarace -- egevecor of S h he larges egevalue Mul-Dm. approxmao d x m a e ha are he opmal e? = = Reve Fal-5 Idepede Compoe Aalyss Seek mos depede drecos, sead of mmze represeao errors (sum-squared-error) as PCA Bld source separao speech mxure f : sgmod f( x)=/( e x ) Reve Fal-6 3

14 Fd he bes eghs o make he oupu compoes depede Ho o measure depedece? Lear combao of radom varables leads o Normal dsrbuo Use he hgh-order sascs o measure No-Gaussay Grade Dece o eghs for dscoverg each compoe (from Ells) FasICA Malab package : hp://.cs.hu.f/proecs/ca/fasca/ PCA Reve Fal-7 ICA LDA: Lear Dscrma Aalyss Gve a se of daa x, x,, x, ad her class labels Fd he bes proeco dmeso, y = x so ha are mos separable y m = x= m x D y Y m : sample meas s = ( y m ) s s : sample meas of proeced pos m : h-class scaer PCA LDA maxmzes crero fuco: ( ) = m J m s s s s Reve Fal-8 m m y 4

15 LDA Scaer Marces before proeco: S = ( x m )( x m ) x D s S s s afer proeco: = = ( S S ) = S S = S S : h-class scaer marx Smlarly, beee-class scaer marx S = ( )( ) B m m m m SB J ( ) = S op = arg max J ( ) Recall he Gaussa Cases =Σ ( μ μ ) = S ( m m ) Mea dfferece vecor he PCA space P.5-8 of Chap 0 Reve Fal-9 Mul-Dmesoal Scalg (MDS) Vsualze he daa pos a loer-dm space Ho o preserve he orgal srucure (e.g., dsace)? Opmzao Crero J ee = < ( d δ ) < δ J ff d δ = ( ) δ < Grade Dece o fd e locaos y k y y J ( ) k ee = dk δk δ d < k Somemes rak order s more mpora k Reve Fal-30 5

16 Classfcao vs. Cluserg x Decso Boudary x x x Daa h labels Supervsed Fd decso boudares Daa hou labels Usupervsed Fd daa srucures ad clusers Reve Fal-3 Reve: Mxure Of Gaussas Model daa dsrbuos as GMM px ( ) = pzpx ( ) ( z) Gve daa x,, x N, log-lkelhood: l = N = log z z ( μ, ) = π N x Σ z z z ( π N( x μ, Σ ) π N( μ, Σ ) x ) (, θ ) poserers = τ = p z = x Reve Fal-3 p(x) π 0 π D / Poseror probably of x beg geeraed by a cluser = Z z= π z ( π ) 0 0 z Σ z x e z Σz z ( x μ ) ( x μ ) parameer : θ = { μ, Σ, μ, Σ } Opmzao fd { μ, Σ, μ, Σ} ad mxure prors π o max. lkelhood 0 0 6

17 GMM for Cluserg Gve he esmaed GMM model, compue he probably ha x s geeraed by cluser ( ) poserers = τ = p z = x, θ, θ = { μ, Σ, μ, Σ, π } Expecao : τ () = π ( ) N ( ) π ( ) ( ) ( x μ, Σ ) () () N ( x μ, Σ ) Each sample s assged o every cluser h a sof decso. Reve Fal-33 Comparso: K-Mea Cluserg K-mea cluserg Fx K values Choose al represeave of each cluser Map each sample o s closes cluser for =,,...,N, Re-compue he ceers Ca be used o alze he EM for GMM x() C C o o o oo oo C K x Ck, f Ds(x, Ck) < Ds(x, Ck' ), k k ' Hard decso ed C 3 x() Reve Fal-34 7

18 Herarchcal Cluserg Add herarchcal srucures o clusers may real-orld problems have such herarchcal srucures e.g., bologcal, semac axoomy Agglomerave vs. Dvsve Dedrogram Use large gap of smlary o fd a suable umber of clusers cluserg valdy Reve Fal-35 dsaces or smlary for mergg d m ( D, D ) = m x x dmax ( D, D) = max x x x D, x D x D, x D Neares eghbor algorhm, mmal algorhm Mergg resuls he m. dsace spag ree Bu sesve o ose/ouler Farhes eghbor algorhm, maxmum algorhm Use dsace hreshold o avod large-dameer clusers Dscourage formg elogaed clusers HW#8 P. Reve Fal-36 8