GMM paraeer esaon Xaoye Lu M290c Fnal rojec
GMM nroducon Gaussan ure Model obnaon of several gaussan coponens Noaon: For each Gaussan dsrbuon:, s he ean and covarance ar. A GMM h ures(coponens): p ( 2π ) ( ) p( ) (, L,,,,, ) p ( ) ep d 2 2 2 T ( ) ( ), 2 2 L
EM algorh EM s ofen used n GMM paraeer esaon 0.45 0.7 0.5 0.45 0.4 0.6 0.4 0.35 0.35 0.5 0.3 0.3 0.4 0.25 0.25 0.2 0.3 0.2 0.5 0.5 0.2 0. 0. 0. 0.05 0.05 0-3 -2-0 2 3 4
EM algorh (Bad Inals)
EM algorh EM s an algorh o aze he lkelhood by fng he oher paraeer n each eraon Lke K-Mean Effcen hen gven good nal sengs Local opu algorh No good onlne verson for GMM esaon
EM updang equaon Updang Weghs Updang Mean Updang ovarance N M N p p N N p p N N p p
KM fraeork A fraeork of onlne learnng algorh: Fnd he updae radng off he dvergence n he paraeer doan h he dvergence n he labellng doan. Dvergence Loss Facor To nze:, ( Loss ) U ( ) (, ) Loss( )
Iplc vs Eplc Too hard o solve plc updang equaon: U ( ) (, ) Loss( ) Eplc Equaon U, Loss( )
GMM Jon Enropy Updang Usng jon enropy as he dvergence ln (, ) ˆ (, ) Usng negave log lkelhood as he loss funcon ln ( ) ln( )
GMM JE updang Lagrange Mehod o ge he Mnu h consran Loss V, ˆ λ 0 ln, ˆ λ ϖ ϖ ϖ V 0 ln, ˆ V 0 ln, ˆ V
GMM JE updang To avod fuure dervave n he loss funcon, JE uses Taylor Epanson: Loss ( ) Loss( ) ( ) ( ) Loss Loss ( ) Loss( )
GMM JE updang Dervaves of dvergence Dervaves of Loss funcon here λ ϖ T r d 2 2 2 ln 2 ln, ˆ, ˆ 2 2, ˆ β ln α ln 2 ln T I α α β
Updang Equaons On-Lne Verson ϖ ϖ ep Z j ( β ) Z ϖ ep β β β ( ) ( )( ) T I ( )( ) Bach Verson ϖ ϖ ep β Z j ϖ ep Z β β β ( ) T I ( )( )
Iplc Updang Usng fuure dervaves for loss funcon Z ep I,,,, 0 T
Le ( ) ξ Weghs updae ( ) ep ξ earch for ξ nzng he U funcon U funcon s conve (Log s concave) Usng bnary search, hch s fas. Z
Mean and ovarance F For quadrac equaon of, Usng: Roo(s) of here Need o check he valdy of α β I,,,, 0 T I 0 Q B A [ ] T B ± 2 AQ B B T T 4
Learnng Rae For Onlne: Learnng rae should ge closer o 0 as he daa nubers ncreases. () ( 0) 0.0 ( 0) < 00 > 00 For Bach: Learnng rae s fed. For EM, e don have learnng rae o adjus.
Onlne Verson Resuls I
Eperens I Toy Daa: 2 ures ( densonal) Weghs: [0.5, 0.5] Mean: [-, ] ovarance [0.5, 0.5] W_In [0.2; 0.8]; Mu_In [-0.5; 0.5]; _In [/3; /3]; ea0.05;
0.45 0.4 0.35 0.3 0.25 0.2 0.5 0. EM localzaon Fnal Approaon, Hsogra 5000 Daa Hsogra True Dsrbuon Densy go by In Value e Densy go by In Value e Densy go by In Value e eng : W_In [0.; 0.9]; Mu_In [-0.; 0.]; _In [0.; 0.]; eng : W_In [0.2; 0.8]; Mu_In [-0.5; 0.5]; _In [0.4; 0.4]; eng : W_In [0.4; 0.6]; Mu_In [-0.9; 0.9]; _In [0.4; 0.4]; 0.05 0-3 -2-0 2 3
0 4 Bach MyTral - -.5-2 -2.5-3 ea.0 ea.05 ea. ea.5 ea 2 ea 3 2 4 6 8 0 2 4 6 8 20
Bach JE -400-4200 -4300 ea.0 ea.05 ea. ea.5 ea 2 ea 3-4400 -4500-4600 -4700-4800 -4900 Faled 3 es, because he becae negave soes. -5000-500 2 3 4 5 6 7 8 9 0
Bach JE vs MyTral Densy afer 50 eraons Densy afer 3 eraons 0.45 0.4 Fnal Approaon, Hsogra 0.5 0.45 5000 Daa Hsogra True densy JE MyTral Ieraon 3 0.35 0.4 0.3 0.35 0.3 0.25 0.25 0.2 0.2 0.5 0.5 0. 0.05 5000 Daa Hsogra True Dsrbuon Densy go by JEBach Densy go by TralBach 0. 0.05 0-3 -2-0 2 3 0-3 -2-0 2 3
On-Lne JE vs MyTral Densy afer 00 daa 0.45 JE Densy a eraon 00 Tral Densy a eraon 00 Densy afer 300 daa 0.45 0.4 0.4 0.35 0.35 0.3 0.25 0.2 0.5 0.3 0.25 0.2 0.5 0. 0.05 0. 0.05 JE Densy a eraon 300 Tral Densy a eraon 300 0-3 -2-0 2 3 Inal eng 0.45 0.4 0-3 -2-0 2 3 Fnal Approaon, Hsogra hsogra rue densy JE a eraon MyTral a eraon 0.35 0.3 0.25 0.2 0.5 0. 0.05 0-3 -2-0 2 3
Wha I learned MyTral s uch ore sable hen JE, snce JE ll generae non posve defne ovarance ar. JE and MyTral depend less on he nal seng, hle EM does.