GMM parameter estimation. Xiaoye Lu CMPS290c Final Project

Transcription

1 GMM paraeer esaon Xaoye Lu M290c Fnal rojec

2 GMM nroducon Gaussan ure Model obnaon of several gaussan coponens Noaon: For each Gaussan dsrbuon:, s he ean and covarance ar. A GMM h ures(coponens): p ( 2π ) ( ) p( ) (, L,,,,, ) p ( ) ep d T ( ) ( ), 2 2 L

3 EM algorh EM s ofen used n GMM paraeer esaon

4 EM algorh (Bad Inals)

5 EM algorh EM s an algorh o aze he lkelhood by fng he oher paraeer n each eraon Lke K-Mean Effcen hen gven good nal sengs Local opu algorh No good onlne verson for GMM esaon

6 EM updang equaon Updang Weghs Updang Mean Updang ovarance N M N p p N N p p N N p p

7 KM fraeork A fraeork of onlne learnng algorh: Fnd he updae radng off he dvergence n he paraeer doan h he dvergence n he labellng doan. Dvergence Loss Facor To nze:, ( Loss ) U ( ) (, ) Loss( )

8 Iplc vs Eplc Too hard o solve plc updang equaon: U ( ) (, ) Loss( ) Eplc Equaon U, Loss( )

9 GMM Jon Enropy Updang Usng jon enropy as he dvergence ln (, ) ˆ (, ) Usng negave log lkelhood as he loss funcon ln ( ) ln( )

10 GMM JE updang Lagrange Mehod o ge he Mnu h consran Loss V, ˆ λ 0 ln, ˆ λ ϖ ϖ ϖ V 0 ln, ˆ V 0 ln, ˆ V

11 GMM JE updang To avod fuure dervave n he loss funcon, JE uses Taylor Epanson: Loss ( ) Loss( ) ( ) ( ) Loss Loss ( ) Loss( )

12 GMM JE updang Dervaves of dvergence Dervaves of Loss funcon here λ ϖ T r d ln 2 ln, ˆ, ˆ 2 2, ˆ β ln α ln 2 ln T I α α β

13 Updang Equaons On-Lne Verson ϖ ϖ ep Z j ( β ) Z ϖ ep β β β ( ) ( )( ) T I ( )( ) Bach Verson ϖ ϖ ep β Z j ϖ ep Z β β β ( ) T I ( )( )

14 Iplc Updang Usng fuure dervaves for loss funcon Z ep I,,,, 0 T

15 Le ( ) ξ Weghs updae ( ) ep ξ earch for ξ nzng he U funcon U funcon s conve (Log s concave) Usng bnary search, hch s fas. Z

16 Mean and ovarance F For quadrac equaon of, Usng: Roo(s) of here Need o check he valdy of α β I,,,, 0 T I 0 Q B A [ ] T B ± 2 AQ B B T T 4

17 Learnng Rae For Onlne: Learnng rae should ge closer o 0 as he daa nubers ncreases. () ( 0) 0.0 ( 0) < 00 > 00 For Bach: Learnng rae s fed. For EM, e don have learnng rae o adjus.

18 Onlne Verson Resuls I

19 Eperens I Toy Daa: 2 ures ( densonal) Weghs: [0.5, 0.5] Mean: [-, ] ovarance [0.5, 0.5] W_In [0.2; 0.8]; Mu_In [-0.5; 0.5]; _In [/3; /3]; ea0.05;

20 EM localzaon Fnal Approaon, Hsogra 5000 Daa Hsogra True Dsrbuon Densy go by In Value e Densy go by In Value e Densy go by In Value e eng : W_In [0.; 0.9]; Mu_In [-0.; 0.]; _In [0.; 0.]; eng : W_In [0.2; 0.8]; Mu_In [-0.5; 0.5]; _In [0.4; 0.4]; eng : W_In [0.4; 0.6]; Mu_In [-0.9; 0.9]; _In [0.4; 0.4];

21 0 4 Bach MyTral ea.0 ea.05 ea. ea.5 ea 2 ea

22 Bach JE ea.0 ea.05 ea. ea.5 ea 2 ea Faled 3 es, because he becae negave soes

23 Bach JE vs MyTral Densy afer 50 eraons Densy afer 3 eraons Fnal Approaon, Hsogra Daa Hsogra True densy JE MyTral Ieraon Daa Hsogra True Dsrbuon Densy go by JEBach Densy go by TralBach

24 On-Lne JE vs MyTral Densy afer 00 daa 0.45 JE Densy a eraon 00 Tral Densy a eraon 00 Densy afer 300 daa JE Densy a eraon 300 Tral Densy a eraon Inal eng Fnal Approaon, Hsogra hsogra rue densy JE a eraon MyTral a eraon

25 Wha I learned MyTral s uch ore sable hen JE, snce JE ll generae non posve defne ovarance ar. JE and MyTral depend less on he nal seng, hle EM does.