LCUR :FACOR ANALYSIS Rta Osadchy Based on Lecture Notes by A. Ng
Motvaton Dstrbuton coes fro MoG Have suffcent aount of data: >>n denson Use M to ft Mture of Gaussans nu. of tranng ponts If <<n dffcult to odel a sngle Gaussan uch less a ture of Gaussan
Motvaton data ponts span only a low-densonal n subspace of ML estator of Gaussan paraeters: More generally unless eceeds n by soe reasonable aount the au lkelhood estates of the ean and covarance ay be qute poor. Sngular Can t copute Gaussan Densty
Restrcton on Goal: Ft a reasonable Gaussan odel to the data when <<n. Possble solutons: Lt the nuber of paraeters assue s dagonal. Lt I where s the paraeter under our control.
Contours of a Gaussan Densty General Dagonal Contours are as algned I
Correlaton n the data Restrctng to be dagonal eans odellng the dfferent coordnates of the data as beng uncorrelated and ndependent. Often we would lke to capture soe nterestng correlaton structure n the data.
Modelng Correlaton he odel we wll see today
Factor Analyss Model k Assue a latent rando varable k n ~ N0 I he paraeters of the odel ~ N n nk nn s dagonal quvalently and are ndependent. ~ N0
aple of the generatve odel of p 0 3 4 7 ~ N0 0 0 7 0 0 ~ N0
Generatve process n hgher densons We assue that each data pont s generated by saplng a k-denson ultvarate Gaussan. hen t s apped to a k-densonal n affne space of by coputng Lastly s generated by addng covarance nose to.
Defntons Suppose Suppose Parttoned vector s r.v. where ~ N where r s rs Here r s rr rs and Under our assuptons and are jontly ultvarate Gaussan.
Margnal dstrbuton of By defnton of the jont covarance of and Cov. ] [ Cov Margnal dstrbutons of Gaussans are theselves Gaussan hence ~ N d p p
Condtonal dstrbuton of gven p p p N N Referrng to the defnton of the ultvarate Gaussan dstrbuton t can be shown that ~ N where
Fndng the Paraeters of FA odel Assue and have a jont Gaussan dstrbuton: We want to fnd ~ N and [ ] 0 snce ~ N0 I [ ] [ ] [ ] [ ]. 0 k n
Fndng We need to calculate upper left block [ upper-rght block [ [ ]] [ ]] lower-rght block [ ]] [ ]] [ [ ]] [ ]] ] ] ] Cov ~ N0 I I
Fndng [ [ ] [ ] ] [ ] =0 [ ] [ ndependent Cov [ ] [ ] 0 ]
Fndng Slarly [ [ ] [ ] ] [ ] [ ] [ ] [ ]
Fndng the paraeters cont. Puttng everythng together we have that I N 0 ~ We also see that the argnal dstrbuton of s gven by ~ N hus gven a tranng set log lkelhood of the paraeters s: } { n l / ep log
Fndng the paraeters cont. l log ep n/ o perfor au lkelhood estaton we would lke to ae ths quantty wth respect to the paraeters. But ang ths forula eplctly s hard and we are aware of no algorth that does so n closed-for. So we wll nstead use the M algorth.
M for Factor Analyss -step: p M-step: arg a p log ; d
-step M for FA We need to copute p ; Usng a condtonal dstrbuton of a Gaussan we fnd that ~ N I 0 I / ep k
M-step M for FA Mae: wth respect to the paraeters log p ; d We wll work out the optaton wth respect to Dervatons of the updates for Do t! s an eercse
Update for Λ d p ; log d p p ] log log ; [log ] log log ; [log ~ p p pectaton wth respect to drawn fro
Update for Λ cont. ] log log ; [log ~ p p Reeber that We want to ae ths epresson wth respect to Λ ] ; [log ~ p ep log / / n log log n Do not depend on Λ ~ N
Update for Λ cont. ake dervatve wth respect to Λ ; tr a a a scalar tr tr tr Splfy:
tr tr tr tr BA AB Update for Λ cont. C A B C A B C ABA A tr tr tr
Update for Λ cont. Settng ths to ero and splfyng we get: ~ ~ Solvng for Λ we obtan: ~ ~ Snce s Gaussan wth ean and covarance ~ ] [ ~ ] [ ] [ ] [ ] [ Y Y YY Y Cov ] [ ] [ ] [ Y Cov Y Y YY hence
Update for Λ cont. ~ ~ ~ ] [ ~ ] [ substtute
M-step updates for μ and Ψ ; p Doesn t depend on hence can be coputed once for all the teratons. he dagonal contans only dagonal entrees
Probablstc PCA Probablstc generatve vew of data D M
Copare Probablstc PCA FA sphercal dagonal as-algned
Probablstc PCA he coluns of W are the prncple coponents. Can be found usng ML n closed for M ore effcent when only few egenvectors are requred avods evaluaton of data covarance atr Other advantages see Bshop Ch..