LECTURE :FACTOR ANALYSIS

Transcription

1 LCUR :FACOR ANALYSIS Rta Osadchy Based on Lecture Notes by A. Ng

2 Motvaton Dstrbuton coes fro MoG Have suffcent aount of data: >>n denson Use M to ft Mture of Gaussans nu. of tranng ponts If <<n dffcult to odel a sngle Gaussan uch less a ture of Gaussan

3 Motvaton data ponts span only a low-densonal n subspace of ML estator of Gaussan paraeters: More generally unless eceeds n by soe reasonable aount the au lkelhood estates of the ean and covarance ay be qute poor. Sngular Can t copute Gaussan Densty

4 Restrcton on Goal: Ft a reasonable Gaussan odel to the data when <<n. Possble solutons: Lt the nuber of paraeters assue s dagonal. Lt I where s the paraeter under our control.

5 Contours of a Gaussan Densty General Dagonal Contours are as algned I

6 Correlaton n the data Restrctng to be dagonal eans odellng the dfferent coordnates of the data as beng uncorrelated and ndependent. Often we would lke to capture soe nterestng correlaton structure n the data.

7 Modelng Correlaton he odel we wll see today

8 Factor Analyss Model k Assue a latent rando varable k n ~ N0 I he paraeters of the odel ~ N n nk nn s dagonal quvalently and are ndependent. ~ N0

9 aple of the generatve odel of p ~ N ~ N0

10 Generatve process n hgher densons We assue that each data pont s generated by saplng a k-denson ultvarate Gaussan. hen t s apped to a k-densonal n affne space of by coputng Lastly s generated by addng covarance nose to.

11 Defntons Suppose Suppose Parttoned vector s r.v. where ~ N where r s rs Here r s rr rs and Under our assuptons and are jontly ultvarate Gaussan.

12 Margnal dstrbuton of By defnton of the jont covarance of and Cov. ] [ Cov Margnal dstrbutons of Gaussans are theselves Gaussan hence ~ N d p p

13 Condtonal dstrbuton of gven p p p N N Referrng to the defnton of the ultvarate Gaussan dstrbuton t can be shown that ~ N where

14 Fndng the Paraeters of FA odel Assue and have a jont Gaussan dstrbuton: We want to fnd ~ N and [ ] 0 snce ~ N0 I [ ] [ ] [ ] [ ]. 0 k n

15 Fndng We need to calculate upper left block [ upper-rght block [ [ ]] [ ]] lower-rght block [ ]] [ ]] [ [ ]] [ ]] ] ] ] Cov ~ N0 I I

16 Fndng [ [ ] [ ] ] [ ] =0 [ ] [ ndependent Cov [ ] [ ] 0 ]

17 Fndng Slarly [ [ ] [ ] ] [ ] [ ] [ ] [ ]

18 Fndng the paraeters cont. Puttng everythng together we have that I N 0 ~ We also see that the argnal dstrbuton of s gven by ~ N hus gven a tranng set log lkelhood of the paraeters s: } { n l / ep log

19 Fndng the paraeters cont. l log ep n/ o perfor au lkelhood estaton we would lke to ae ths quantty wth respect to the paraeters. But ang ths forula eplctly s hard and we are aware of no algorth that does so n closed-for. So we wll nstead use the M algorth.

20 M for Factor Analyss -step: p M-step: arg a p log ; d

21 -step M for FA We need to copute p ; Usng a condtonal dstrbuton of a Gaussan we fnd that ~ N I 0 I / ep k

22 M-step M for FA Mae: wth respect to the paraeters log p ; d We wll work out the optaton wth respect to Dervatons of the updates for Do t! s an eercse

23 Update for Λ d p ; log d p p ] log log ; [log ] log log ; [log ~ p p pectaton wth respect to drawn fro

24 Update for Λ cont. ] log log ; [log ~ p p Reeber that We want to ae ths epresson wth respect to Λ ] ; [log ~ p ep log / / n log log n Do not depend on Λ ~ N

25 Update for Λ cont. ake dervatve wth respect to Λ ; tr a a a scalar tr tr tr Splfy:

26 tr tr tr tr BA AB Update for Λ cont. C A B C A B C ABA A tr tr tr

27 Update for Λ cont. Settng ths to ero and splfyng we get: ~ ~ Solvng for Λ we obtan: ~ ~ Snce s Gaussan wth ean and covarance ~ ] [ ~ ] [ ] [ ] [ ] [ Y Y YY Y Cov ] [ ] [ ] [ Y Cov Y Y YY hence

28 Update for Λ cont. ~ ~ ~ ] [ ~ ] [ substtute

29 M-step updates for μ and Ψ ; p Doesn t depend on hence can be coputed once for all the teratons. he dagonal contans only dagonal entrees

30 Probablstc PCA Probablstc generatve vew of data D M

31 Copare Probablstc PCA FA sphercal dagonal as-algned

32 Probablstc PCA he coluns of W are the prncple coponents. Can be found usng ML n closed for M ore effcent when only few egenvectors are requred avods evaluaton of data covarance atr Other advantages see Bshop Ch..