EE 6885 Statistical Pattern Recognition

Transcription

1 EE 6885 Sascal Paer Recogo Fall 005 Prof. Shh-Fu Chag hp:// Lecure 5 (9//05 4- Readg Model Parameer Esmao ML Esmao, Chap. 3. Mure of Gaussa ad EM Referece Boo, HTF Chap. 8.5 Teboo, DHS 3.9 Homewor # due , Wed o class/offce hours e Moday,

2 Mul-varae Gaussa p( ( π D / e Σ T ( μ Σ ( μ Geeral Σ Bayesa Classfers Decso Boudares for Gaussas w ( 0 0 w Σ ( μ μ l ( / ( ( P ω P ω ( 0 μ + μ μ μ ( μ μ Σ ( μ μ Mssg Feaures by Margalao [ g, b ], g : good feaures, b : bad feaures p( w, g compue P( w g pw (, g, b d p(, w p( w d b p ( p ( p (, d g Jo prob. of (ω ι, g, b margaled over b g g b b g b b 5-3 Parameer Esmao Paramerc form of dsrbuo e.g., p ( w μ (, Σ p ( w p ( w, How o esmae? lear from daa samples D {,... } Lelhood l( p( D p ( assume,..., depede Fd ˆ arg ma pd ( argma l p ( Use grade operao arg ma p ( /... / p l( 0 5-4

3 Case I: Gaussa: μ uow P μ π μ μ Σ Σ d l ( l ( ( ( ad μ l P( μ Σ ( μ l P ( μ 0 μ Σ ( μ 0 ˆ μ ML esmaor of he Gaussa mea s he sample mea 5-5 Case II: Gaussa Case: uow μ ad σ (D l l P( l( πσ ( μ σ (l P ( ( μ μ l ( (l P ( μ σ ( σ l P ( 0 ˆ μ ˆ σ ( ˆ μ Ep( ˆ σ σ σ f he Ep( ˆ σ σ Asympocally ubased (μ, σ Mul-Dmesoal ( μ, Σ ˆ (/ ˆ μ Σ (/ ( μ( μ ML esmaor: mea -> sample mea, varace ->based sample varace 5-6 3

4 Mure Of Gaussas Real dsrbuos seldom follow a sgle Gaussa mure of Gaussas p( π 0 π p( p(, pp ( ( Gve daa,,, defe log-lelhood: l log resposbly compoe of ( μ, π Σ ( π ( μ, Σ + π ( μ, Σ Poseror probably of beg geeraed by a specfc compoe poserers τ p Z π ( π (,, { μ, Σ, μ, Σ } D / Σ e T ( μ Σ ( μ Dervao of he E-M soluo log lelhood Mamao of l( drecly s hard due o log_of_sum Isead, loo a Δl( l( l(, : curre esmao of Jese s Iequaly If f s cocave, e. g., f log If f s cove, f (, l( log p f ( E{ } E( f{ } f ( E{ g( } E( f{ g( } ( log( ( E{} E( f{ } p p log (, Log( p( where p 5-8 4

5 Aulary Fuco E-M Δ l( l( l( log p( log p( p log p ( ( log p p, log ( log (, (, p ( ( p(, p( ( ( p, p, p p, ( log p, (, (, p p ( Q oe here s o log_of_sum. So ag dervave s easer 5-9 margalao Jese s equaly E-M mproves lelhood Aulary fuco derved based o Jese s Iequaly, ow esmae + by mamg Q + arg maq( (, log p(, + cos Q( p epecao over wh curre o lelhood of observed & hdde So he epecao sep, compue τ, he resposbly of compoe for sample I he mamao sep, ae dervave of Q o, ad fd he ew esmae for (oe oly sum_of_log s volved 5-0 5

6 EM Always Improves Lelhood Why does EM always mprove l(? Δ l( l( l( Q( Q( + ma Q( Q( 0 (, log p(, + cos Q( p Geeral seps of EM: epecao over o lelhood of wh curre observed & hdde l + Δ ( 0 Defe lelhood model wh parameers Idefy hdde varables Derve he aulary fuco ad he E ad M equaos I each erao, esmae he poserors of hdde varables Re-esmae he model parameers. Repea ul sop 5- Epecao-Mamao (E-M Soluo of GMM τ EM for esmag ad. Follow dvde ad coquer prcple. I erao sep : ( ( ( ( π ( μ, Σ Epecao : τ ( ( ( π ( μ, Σ ( τ ( + Mamao : μ ( + ( Σ τ Dvde daa o each group, Compue mea ad varace from each group (+ π Wegh from compoe ( ( ( ( ( τ μ μ τ τ ( T 5-6

7 GMM for Cluserg Gve he esmaed GMM model, compue p( π0 π (,, { μ, Σ,, } poserers τ p μ 0 0 Σ Esmae he probably ha s geeraed by cluser Epecao : τ ( π ( ( π ( ( ( μ, Σ ( ( ( μ, Σ Each sample s assged o every cluser wh a sof decso. 5-3 Comparso: K-Mea Cluserg Trag daa { } + { label(? } Usupervsed learg K-mea cluserg F K values Iale he represeave of each cluser Map samples o closes cluser (hard decso Re-compue he ceers ( C ++ + C + ++ o o ++ + o oo oo C K C 3 (,,..., samples for,,...,, C, f Ds(, C < Ds(, C', ' ed 5-4 7