Tomoki Toda. Augmented Human Communication Laboratory Graduate School of Information Science

Transcription

1 Seuetial Data Modelig d class Basics of seuetial data odelig ooki oda Augeted Hua Couicatio Laboratory Graduate School of Iforatio Sciece

2 Basic Aroaches How to efficietly odel joit robability of high diesioal data (i.e. seuetial data) Markov rocess 3 4 Latet variables State sace odel e.g. HMM LDS

3 owards Uderstadig Latet Variables I this class we focus o a iture odel usig a discrete latet variable. We assue that the legth of seuetial data is costat. e.g. if the legth of each data is 5 Data : { } Data : { } each data sale is rereseted as a 5 diesioal vector. We lear how to odel joit robability desity i such a highdiesioal sace usig the iture odel. P( 3 4) 3 4 P ) P( ) P( ) P( ) P( ) (

4 Effectiveess of Miture Model P( X Y ) # of robs: = 0 0 X Y # of robs: = 3 ( ) P ( ) 0.4 P ( ) 0. 4 P ( 3 ) 0. Prob. High P X ) P( Y ) P X ) P( Y ) P X ) P( Y ) ( ( P X ) P X ) P X ) ( ( ( 3 Low ( 3 3 P Y ) P Y ) P( Y 3) ( ( 3

5 Miture Model Gaussia Miture Model

6 he Gaussia Distributio oral distributio or Gaussia distributio ; e / Paraeters Mea : Variace : Stadard d deviatio: Precisio: / Coditios to be satisfied: ; 0 ; d ; - 4

7 Multivariate Gaussia Distributio Gaussia distributio over a D diesioal vector of cotiuous variables Paraeters Mea vector : Covariace atri : Eale of diesioal case: s s s s Use of diagoal covariace atri s 0 0 s Use of full covariace atri s s s s 5

8 Maiu Likelihood Estiatio (MLE) Observatio data set ( observatios of variable ) X Ideedet ad idetically distributed: i.i.d. Data oits draw ideedetly fro the sae distributio Likelihood fuctio X Because the data set X is i.i.d. X is give by. ; Deterie araeters usig a observatio data set by aiiig likelihood fuctio! 6

9 Sufficiet Statistics Log scaled likelihood fuctio: CAB BCA ABC tr tr tr D l l X CAB BCA ABC tr tr tr D l l tr l tr l # of sales : Su of sales : Su of sales : Su of suared sales: 7

10 ML Estiates Deterie odel araeters by aiiig likelihood fuctio t l X arg a l tr l arg a ML estiate of ea vector: 0 X l Sale ea vector 0 X l ML estiate of covariace atri: 0 S l i Sale covariace 8

11 Miture Model Probability desity fuctio (.d.f.) give by argial.d.f. of joit.d.f. over latet variables Latet variables ( of M): M M use st cooet use d cooet use 3 rd cooet 3 3 9

12 Gaussia Miture Model (GMM) Miture of ultile Gaussia distributios M Prior robability of th iture cooet ; 0 M.d.f. of th iture cooet ; 3 0

13 Eale of.d.f. of GMM GMM with iture cooets o diesioal sace st i co. : d i co. : GMM: ;3 ;3 ; ;3 0.3

14 Effectiveess of GMM Caable of odelig iter diesioal correlatio eve if usig diagoal covariace atrices If usig a sigle Gaussia distributio w/ diagoal covariace atri If usig twogaussia distributios w/ diagoal covariace atrices Caable of odelig iter diesioal correlatio!

15 How to Ileet MLE? Log scaled likelihood of a iture odel: l X l l M l ; Because of the logarith of the suatio its derivatives with resect to odel dlaraeters are ot rereseted as liear euatios he gradiet ethods will be ecessary Ca we ush the logarith iside the suatio? 3

16 Eectatio ti Maiiatio (EM) Algorith

17 EM Algorith Iteratively udate lower boud of likelihood fuctio through two stes: Eectatio ste (E ste) Lower boud Maiiatio ste (M ste) Likelihood fuctio Eable closed for l X solutio of ML estiates Guaratee covergece Coverge to local aiu E ste L L L ew old old M ste l X ew old 4

18 Scheatic Iage of EM Algorith L l X Log scaled likelihood fuctio l X ( ) ( ) i i ( i) ( i) L ( ) L i (i) L (i) ( i ) 3. E ste. E ste: deterie lower boud based o curret odel araeters 0. Curret odel araeter set. M ste: udate odel araeters based o the lower boud 5

19 Lower Boud of Likelihood Fuctio Derivatio of lower boud of log scaled likelihood fuctio Log scaled likelihood fuctio: X l l Probability distributio fuctio of latet variables l J i li l Jese s ieuality L Lower boud: Lower boud: l L 6

20 Jese s Ieuality l l subject to 0 Outut of iterolated iut value Iterolated outut value le E l l l l E Iterolated t diut value 7

21 Lower Boud as Fuctioal of Lower boud: l L l l l l l l KL l X L b d l L b d l l dlik lih d l dlik lih d KLdi KL di KL divergece Log scaled likelihood Lower boud = log Lower boud = log scaled likelihood scaled likelihood KL divergece KL divergece 8

22 KL Divergece Discrete variable case: Cotiuous variable case: KL l KL l d l l KL l l Log scaled likelihood of true distributio d Log scaled likelihood of aother distributio Aroiated usig sales radoly geerated fro Differece of log scaled likelihoods KL 0 If = KL 0 9

23 Lower Boud as Fuctio of Lower boud: L l l l l Auiliary fuctio Costat Lower boud Auiliary fuctio 0

24 EM: Maiiatio of Lower Boud Lower boud: fuctioal of ad fuctio of E ste: Maiie lower boud with resect to while fiig : L l X KL M ste: Maiie lower boud with resect to while fiig KL L l L l X * eed to set iitial odel araeters

25 E Ste: Udate Set KL divergece to 0 uder the fied odel araeters old KL 0 old Cl Calculate lt osterior robabilities biliti of lt latet tvariables ibl for each sale old KL 0 old M M k k ; ; old k old k old old L old l X old

26 M Ste: Udate Maiie auiliary fuctio with resect to odel araeters ew L l KL Auiliary fuctio Q ew old M l old l l ; ew L ew l X ew 3

27 Sufficiet Statistics Auiliary fuctio: Q M l l ; ew old M l l M tr For each iture cooet # of sales : Su of sales : Su of suared sales: 4

28 ML Estiates Auiliary fuctio: M old ew l l Q M tr old ew M Q For each iture cooet 0 old ew Q M Miture weights 0 old ew Q Mea vectors 0 old ew Q Covariace 0 atrices 5

29 Eale

30 Eale: EM Algorith for GMM Five -D diesioal sales give as traiig data Iitial odel ; st iture cooet w/ a diagoal covariace atri: 0 0 d iture cooet w/ a diagoal covariace atri: = = 6

31 E Ste. Calculate joit robabilities for idividual iture cooets. Calculate osterior robabilities (i.e. eected # ofsales) (as ; I ) ; e d d d M k k ; k k = = = =

32 Calculatio of Sufficiet Statistics 3. Calculate sufficiet statistics # of sales Su of sales Su of suared sales * * * *

33 M Ste 4. Udate odel araeters Miture M weights Mea vectors Covariace atrices Udate = 3 4 = = = 9

34 Icrease of Likelihood 5. Calculate likelihood ad coare it before ad after udate ; Before udate: After udate: M = = P( ) = = P( ) l l