Fitting a Conditional Linear Gaussian Distribution

Transcription

1 Fng a Condonal Lnear Gaussan Dsrbuon Kevn P. Murphy 28 Ocober 1998 Revsed 29 January Inroducon We consder he problem of fndng he maxmum lkelhood ML esmaes of he parameers of a condonal Gaussan varable Y wh connuous paren X and dscree paren Q,.e., py x, Q = = c Σ 1 2 exp 1 2 y B x µ Σ 1 y B x µ where c = 2π d/2 s a consan and y = d. The j h row of B s he regresn vecor for he j h componen of y gven ha Q =. We consder yng and varous consrans on he covarance marx n order o reduce he number of free parameers. We wll allow any of he varables o be hdden we wll replace observed values wh expeced values condoned on evdence, as n EM. We express all he esmaes n erms of expeced suffcen sascs, whose sze s ndependen of he number of samples. Ths s dfferen from he usual presenaon, whch gve he formulas n erms of he raw daa marx. The resulng formulas can be used n he M sep of all of he followng common models, whch use specal cases of he above equaon: Facor analyss. Q does no exs, Σ s assumed dagonal, X s hdden and Y s observed. The emporal vern of hs s he Kalman fler. Mxure of Gaussans. X does no exs, Q s hdden, and Y s observed. The emporal vern of hs s an HMM wh MOG oupus. Mxure of facor analyzers. Σ s dagonal, Q and X are hdden, Y s observed. The emporal vern of hs s a swchng Kalman fler. We assume ha we have..d. ranng cases {e }, he complee-daa log-lkelhood s log Q [Pry x, Q =, e ] q =1 =1 where q = 1 f Q has value n he h complee case, and 0 oherwse. Snce Q, X and Y may all be unobserved, we compue he expeced complee-daa log lkelhood as follows droppng erms whch are ndependen of he parameers of Y [ ] L = 1 2 E q log Σ + qy B x µ Σ 1 y B x µ e By he chan rule, we can wre E[q x x e ] = E[q e ]E[x x Q =, e ] = w E [XX ] 1

2 where he weghs w = PrQ = e are poseror probables responsbles, and E [XX ] s a condonal second momen; we can rewre he oher momens smlarly. In hs new noaon, he expeced complee-daa log-lkelhood becomes L = 1 2 w log Σ 1 2 we [ y B x µ Σ 1 y B x µ ] 1 To smplfy fuure equaons, we nroduce he followng expeced suffcen sascs: w = w S Y Y, = w E [Y Y ] S Y Y, = w E [Y Y ] S Y, = w E [Y ] S XX, = w E [XX ] S X, = w E [X] S XY, = w E [XY ] S Y X, = w E [Y X ] Obvously w =, S XY, = S XY,, ec. The goal s o derve he equaons we can mplemen a funcon of he form µ, Σ, B = Msep-clgw, S Y Y,, S Y Y,, S Y,, S XX,, S X,, S XY, For he no regresn case, where X does no exs, we can smplfy hs o µ, Σ = Msep-cond-gaussw, S Y Y,, S Y Y,, S Y, 2 Esmang he regresn marx 2.1 Uned Usng he followng deny see e.g., [Row99],[Jor03, ch.13] Xa + b CXa + b X = C + C Xa + ba 2 where X = B, a = x, b = y µ, C = Σ 1, we have B L = 1 2 { w 2Σ 1 E [y B x µ x ] = Σ 1 we [Y X ] B we [XX ] µ = Σ 1 { SY X, B S XX, µ S X,} w E [X ] } 2

3 Seng B L = 0 yelds If we se µ = 0, we recognze hs as he weghed normal equaons: ˆB = S Y X, µ S X, S 1 XX, 3 ˆB = S Y X,S 1 XX, 2.2 Ted The dervaon s smlar o he above. B L = Σ 1 { SY X, BS XX, µ S X,} Unforunaely, hs s hard o lve. So we wll assume he covarance s al ed, leadng o { SY X, BS XX, µ S X,} and hence B L = Σ 1 ˆB = S Y X, 1 µ S X, S XX, 4 3 Esmang he mean 3.1 Uned We can esmae µ smlarly o B. Usng Equaon 2 where X = µ, a = 1, b = y B x, C = Σ 1, we have µ L = 1 2 { w 2Σ 1 E [y B x µ ] = Σ 1 w E [Y ] B = Σ 1 {S Y, B S X, µ w } w E [X] µ w 1 } Seng µ L = 0 yelds ˆµ = S Y, B S X, w o regresn If B = 0, hs yelds he famlar specal case ˆµ = S Y, w = w E Y w Ted So µ L = Σ 1 ˆµ = {S Y, B S X, µ w } S Y, B S X, 7 3

4 3.2.1 o regresn If B = 0, hs yelds ˆµ = S Y, 8 4 Esmang he regresn marx and he mean smulaneously Snce he equaon for B depends on µ and vce versa, f hey are boh o be esmaed as opposed o beng clamped o fxed values, we mus esmae hem jonly. We can do hs by appendng µ as he las column o B o creae A, and appendng a 1 o he las componen of X o creae Z. Then he lkelhood becomes py x, Q = = c Σ 1 2 exp 1 2 y A z Σ 1 y A z We use he equaons from Secon 2, wh µ = 0 and replacng S XX, wh S ZZ, and S Y X, wh S ZY,, ned below. Specfcally,  = S Y Z,S 1 ZZ, 9 The subsuons are Al, E ZZ = E X 1 X 1 = E S ZZ, = E ZY = E X 1 S ZY, = SXX, S X, S X, w XX X X 1 Y = E XY Y SXY, S Y, 5 Esmang a full covarance marx We assume he mean wheher esmaed or clamped s appended o A, and ha a 1 s appended o Z, o smplfy noaon. 5.1 Uned Usng he denes we have Al, usng he deny ln X X = X 1 and ln X = ln X 1 Σ 1 ln Σ = where a = b = y A z and X = Σ 1, we have Σ 1 L = 1 2 = 0 Σ 1 a Xb X ln Σ 1 = Σ = ab w Σ 1 2 w E y A z y A z 4

5 Hence ˆΣ = 1 w w E Y Y Y Z A A ZY + A ZZ A = 1 w S Y Y, S Y Z,A A S ZY, + A S ZZ,A 10 If A =  n Equaon 9, we have A S ZZ,A = S Y Z,S 1 ZZ, S ZZ, S 1 ZZ, S ZY, = S Y Z,A he above smplfes furher o ˆΣ = 1 w S Y Y, A S ZY, o regresn If A = µ, Equaon 10 smplfes o ˆΣ = 1 SY Y, S Y, µ µ S Y, + µ µ w If n addon µ = ˆµ = SY, w from Equaon 6, hen ˆΣ = S Y Y, w µ µ Ted We have Hence If A = Â, hen Σ 1 L = 1 2 w Σ 1 2 w E y A z y A z ˆΣ = 1 S Y Y, S Y Z,A A S ZY, + A S ZZ,A 13 ˆΣ = 1 S Y Y, A S ZY, o regresn If A = µ, Equaon 13 smplfes o ˆΣ = 1 SY Y, S Y, µ µ S Y, + µ µ If n addon µ = ˆµ = SY, w from Equaon 6, hen ˆΣ = S Y Y, µ µ 15 6 Esmang a dagonal covarance marx Proceed as n esmang a full marx, bu hen se all off-dagonal enres o 0. 5

6 7 Esmang a sphercal covarance marx 7.1 Uned If we have he consran ha Σ = σ 2 I s ropc, he condonal densy of Y becomes py x, Q = = cσ d exp 1 2 σ 2 y A z 2 Hence L = d σ L = d w E [log σ 1 2 σ 2 y A z 2 ] w σ 1 + σ 3 w E y A z 2 = 0 and ow σ 2 = 1/d w w E y A x 2 y A z 2 = y A z y A z = y y + z A A z 2y A z To compue he expeced value of hs dsance, we use he fac ha x Ay = rx Ay = rayx, E[x Ay] = rae[yx ]. Hence we y A z y A z = r ow ra + rb = ra + B, o regresn w E Y Y + r w A A E ZZ 2r w A E ZY σ 2 = 1/d w r S Y Y, + A A S ZZ, 2A S ZY, 16 If A = µ and z = 1, we y A z y A z = If µ = ˆµ = EY w as n Equaon 5, hs becomes σ 2 = 1 d SY Y, w w E Y Y + µ µ 2Y µ µ µ Ted If σ 2 s ed, we ge σ 2 = 1/d r S Y Y, + A A S ZZ, 2A S ZY, o regresn For he ed case, we ge σ 2 = 1 S Y Y, + d w µ µ 19 6

7 8 MAP esmaes You may encouner numercal problems when esmang CLG dsrbuons, especally wh small daa ses or wh mxure componens ha have low reponsbly and hence lle daa assgned o hem. A smple luon o hs s o pu a pror on he parameers, and compue maxmum a poseror MAP esmaes nsead of maxmum lkelhood ML esmaes. Mnka [Mn00] dscusses conjugae prors for he case of lnear regresn ncludng rdge regresn, ec. To exend hese formulas o he curren case, would be necessary o derve he condonng on Q, and o consder he parally observed case. Mos of he formulas needed for he no regresn case have been derved n [HC95]; We summarze he resuls for he full-covarance, uned case below. 1 The ed and dagonal cases are smlar. The deals for he sphercal case are no gven, snce regularzaon of a sngle scalar parameer s less mporan. We pu a ormal-wshar pror on each Gaussan mxure componen where P µ, Σ = P µ P Σ µ = µ ; m, τ 1 I d WΣ µ ; Λ, α WΣ µ ; Λ, α Σ α d/2 exp 1 2 rλ Σ = α Λ 1. Eher of hese can be used The mode of he Wshar s Σ 1 = α dλ 1, and he mean s Σ 1 as nal esmaes for Σ. We can compue he MAP esmaes by seng he dervave of he unnormalzed log poseror o zero: [ ] L MAP = w µ E log Y ; µ, Σ + log µ ; m, τ µ µ [ ] = w E Σ 1 Y µ τ Σ 1 µ m ˆµ MAP = τ m + w E Y τ + w Smlarly, L MAP Σ 1 = = [ w E Σ 1 log Y ; µ, Σ = τ m + S Y, τ + w 20 ] + Σ 1 log WΣ [ ] [ w E 1 2 Σ Y µ Y µ α d + 2 Σ τ ] 2 µ m µ m 1 2 Λ ˆΣ MAP = Λ + τ µ m µ m + w E Y µ Y µ α d + w = Λ + τ µ m µ m + S Y Y, w µ µ α d + w 21 If we don pu a pror on µ by seng he precn τ = 0, hs smplfes o ˆΣ MAP = Λ + S Y Y, w µ µ 22 α d + w 1 We use slghly dfferen noaon. Specfcally, we use µ as a parameer and m as a hyperparameer, whereas hey use he oppose; we use he covarance marx Σ nsead of he precn marx r 1, and denoe he pror covarance u by Λ. oe ha τ s an nverse varance scalar. Al, we use he expeced value of Y. 7

8 A smple choce of hyper-parameers s Λ = s I d and α = d, where s s me scalng facor, e.g., Ths can be mplemened by smply replacng S Y Y, wh S Y Y, + Λ n all he equaons above. Essenally hs jus regularzes he covarance esmae and avods problems wh sngular marces. 9 Deermnsc annealng EM s noorous for geng suck n local opma. One approach s o use deermnsc annealng [Ros98, U98], slowly lowerng a emperaure parameer. Brand [Bra99b, Bra99a] suggess opmzng θ MAP = arg max log P D θ ZHθ θ where Hθ s a mnmum enropy pror, Z = T 0 T, and T s a emperaure; he calls hs pror balancng. In he case of uncondonal Gaussans, hs becomes ˆΣ MAP = S Y Y / + Z Inally, T s a large posve number, Z s a large negave number, whch nflaes ˆΣ MAP. We reduce he emperaure unl Z = 1, he mnmum enropy luon. Z = 0 corresponds o maxmum lkelhood, and Z = 1 corresponds o maxmum enropy. A smlar approach can be appled o he more convenonal Wshar pror: we sar wh s large, forcng all covarances o be broad, and hence all mxure componens o receve a lo of suppor; hen we gradually reduce he nose level. 10 Acknowledgmens Thanks o Raner Devener for careful proof-readng. References [Bra99a] M. Brand. Paern dscovery va enropy mnmzaon. In AI/Sas, uncerany99.mcrof.com/brand.hm. [Bra99b] M. Brand. Srucure learnng n condonal probably models va an enropc pror and parameer exncon. eural Compuaon, 11: , [HC95] [Jor03] Qang Huo and Chorkn Chan. Bayesan adapve learnng of he parameers of hdden Markov model for speech recognon. IEEE Transacons on Speech and Audo Processng, 35S: , M. I. Jordan. An nroducon o probablsc graphcal models, In preparaon. [Mn00] T. Mnka. Bayesan lnear regresn. Techncal repor, MIT, [Ros98] K. Rose. Deermnsc annealng for cluserng, compresn, classfcaon, regresn, and relaed opmzaon problems. Proc. IEEE, 80: , ovember [Row99] S. Rowes. Marx denes. Techncal repor, U. Torono, rowes/noes/marxd.pdf. [U98]. Ueda and R. akano. Deermnsc annealng EM algorhm. eural eworks, 11: ,