Data Analysis and Dimension Reduction

Transcription

1 Data Aalss ad Dmeso Reducto - PCA, LDA ad LA Berl Che Departmet of Computer cece & Iformato Egeerg Natoal aa Normal Uverst Refereces:. Itroducto to Mache Learg, Chapter 6. Data Mg: Cocepts, Models, Methods ad Algorthms, Chapter 3

2 Itroducto (/3) Goal: dscover sgfcat patters or features from the put data alet feature selecto or dmesoalt reducto Iput space Netork Feature space Compute a put-output mappg based o some desrable propertes tatstcs-

3 Itroducto (/3) Prcpal Compoet Aalss (PCA) Lear Dscrmat Aalss (LDA) Latet ematc Aalss (LA) Heteroscedastc Dscrmat Aalss (HDA) Probablstc Latet ematc Aalss (PLA) tatstcs-3

4 Itroducto (3/3) Formulato for feature etracto ad dmeso reducto Model-free (oparametrc) thout pror formato: e.g., PCA th pror formato: e.g., LDA Model-depedet (parametrc), e.g., HLDA th Gaussa cluster dstrbutos PLA th multomal latet cluster dstrbutos tatstcs-4

5 Prcpal Compoet Aalss (PCA) (/) Pearso, 90 Ko as Karhue-Loẻve rasform (947, 963) Or Hotellg rasform (933) A stadard techque commol used for data reducto statstcal patter recogto ad sgal processg A trasform b hch the data set ca be represeted b reduced umber of effectve features ad stll reta the most trsc formato cotet A small set of features to be foud to represet the data samples accuratel Also called ubspace Decomposto, Factor Aalss.. tatstcs-5

6 Prcpal Compoet Aalss (PCA) (/) he patters sho a sgfcat dfferece from each other oe of the trasformed aes tatstcs-6

7 PCA Dervatos (/3) uppose s a -dmesoal zero mea radom vector, μ E { } 0 If s ot zero mea, e ca subtract the mea before processg the follog aalss ca be represeted thout error b the summato of learl depedet vectors φ Φ [.. ] he -th compoet the feature (mapped) space here [ φ. ] Φ φ. φ he bass vectors tatstcs-7

8 tatstcs-8 PCA Dervatos (/3) (,3) (0,) (,0) (,) (-,) (5/,/ ) orthogoal bass sets ubspace Decomposto

9 PCA Dervatos (3/3) Further assume the colum (bass) vectors of the matr form a orthoormal set φ φ Φ 0 uch that s equal to the proecto of o φ ϕ ϕ f f ϕ ϕ cos θ ϕ φ, here φ ϕ tatstcs-9

10 PCA Dervatos (4/3) Further assume the colum (bass) vectors of the matr Φ form a orthoormal set Σ E ( μ )( μ ) also has the follog propertes N μμ Its mea s zero, too N E{ } E{ ϕ } ϕ E{ } ϕ 0 0 R E{ } Its varace s { } [ { }] { } E E E{ ϕ ϕ } ϕ { } σ E E ϕ Rϕ [ R s the(auto-)corelato matr of ] { } ϕ N 0 he correlato betee to proectos ad s E { } E ( φ )( φ ) φ { } E{ φ φ } E { } φ φ Rφ tatstcs-0

11 tatstcs- PCA Dervatos (5/3) Mmum Mea-quared Error Crtero e at to choose ol m of that e stll ca appromate ell mea-squared error crtero s ' φ + + m m φ φ φ ( ) m m ˆ φ ( ) ( ) { } { } m m m k m m k k k m k k m m m R φ φ φ φ φ φ σ ε E E E ˆ E k k k f 0 f φ Q φ { } { } { } [ ] { } 0 E E E E σ e should dscard the bases here the proectos have loer varaces orgal vector recostructed vector

12 PCA Dervatos (6/3) Mmum Mea-quared Error Crtero If the orthoormal (bass) set s selected to be the egevectors of the correlato matr R, assocated th egevalues λ ' s he ll have the propert that: R φ λ φ φ ' s s real ad smmetrc, therefore ts egevectors R form a orthoormal set R s postve defte ( R > 0 ) > all egevalues are postve uch that the mea-squared error metoed above ll be ε ( m ) m + m + σ φ R φ m + φ λ φ m + λ tatstcs-

13 tatstcs-3 PCA Dervatos (7/3) Mmum Mea-quared Error Crtero If the egevectors are retaed assocated th the m largest egevalues, the mea-squared error ll be A to proectos ad ll be mutuall ucorrelated Good es for most statstcal modelg approaches Gaussas ad dagoal matrces ( ) ( ) here + m m ege m λ λ λ λ ε { } ( )( ) { } { } { } 0 E E E E φ φ R φ φ φ φ φ φ φ φ λ σ σ σ σ σ σ σ σ ( )( ) { } N N E N E R μμ μ μ Σ ] [

14 PCA Dervatos (8/3) A o-dmesoal Eample of Prcple Compoet Aalss tatstcs-4

15 PCA Dervatos (9/3) Mmum Mea-quared Error Crtero ( m) Obectve fucto ε ege It ca be proved that s the optmal soluto uder the mea-squared error crtero f k Defe: Partal Dfferetato R φ RΦ J m+ ( φ φ δ ) m+ Rφ Φ mu ( here Φ m [ φm+... φ ]) [ m+... φ ] Φ m[ um+... u ] Φ U ( here U [ u... u ]) m o be mmzed φ m+ J φ m Rφ Rφ m Costrats u k m+ k m+ k m+ u k m U m m+ λ R m+ φ k 0 m+ k ( [ ]) here u u... u k m+ Have a partcular soluto f s a dagoal matr ad ts dagoal elemets s the egevalues of ad φ... φ s ther correspodg egevectors λ... δ k 0 f k ϕ Rϕ Rϕ ϕ tatstcs-5

16 tatstcs-6 PCA Dervatos (0/3) Gve a put vector th dmeso m r to costruct a lear trasform Φ (Φ s a m matr m<) such that the trucato result, Φ, s optmal measquared error crtero Ecoder.. m.. Decoder ˆ.. ˆ ˆ ˆ ( )( ) ( ) - - E ˆ ˆ mmze Φ [ ] m ϕ ϕ ϕ.. here Φ Φ Φ ˆ Φ m..

17 PCA Dervatos (/3) Data compresso commucato PCA s a optmal trasform for sgal represetato ad dmesoal reducto, but ot ecessar for classfcato tasks, such as speech recogto? (o be dscussed later o) PCA eeds o pror formato (e.g. class dstrbutos of output formato) of the sample patters tatstcs-7

18 cree Graph PCA Dervatos (/3) he plot of varace as a fucto of the umber of egevectors kept λ + λ + L+ λm hreshold elect m such that λ + λ + L+ λ + L+ λ m Or select those egevectors th egevalues larger tha the average put varace (average evgevalue) λ λ m tatstcs-8

19 tatstcs-9 PCA Dervatos (3/3) PCA fds a lear trasform such that the sum of average betee-class varato ad average thclass varato s mamal ( ) b b J + + ~ ~ ~ ~ b b ~ N N Σ ( )( ) b N N? ( )( ) N sample de class de b + : r tosho that

20 PCA Eamples: Data Aalss Eample : prcpal compoets of some data pots tatstcs-0

21 PCA Eamples: Feature rasformato Eample : feature trasformato ad selecto Correlato matr for old feature dmesos Ne feature dmesos threshold for formato cotet reserved tatstcs-

22 PCA Eamples: Image Codg (/) Eample 3: Image Codg tatstcs-

23 PCA Eamples: Image Codg (/) Eample 3: Image Codg (cot.) (feature reducto) (value reducto) tatstcs-3

24 PCA Eamples: Egeface (/4) Eample 4: Egeface face recogto (urk ad Petlad, 99) Cosder a dvdual mage to be a lear combato of a small umber of face compoets or egefaces derved from a set of referece mages,, L, teps...,.,...,.... Covert each of the L referece mages to a vector of floatg pot umbers represetg lght test each pel Calculate the coverace/correlato matr betee these referece vectors Appl Prcpal Compoet Aalss (PCA) fd the egevectors of the matr: the egefaces Besdes, the vector obtaed b averagg all mages are called egeface 0. he other egefaces from egeface oards model the varatos from ths average face L L.. L.. tatstcs-4

25 PCA Eamples: Egeface (/4) Eample 4: Egeface face recogto (cot.) teps he the faces are the represeted as egevoce 0 plus a lear combato of the rema K (K L) egefaces he Egeface approach perssts the mmum mea-squared error crtero Icdetall, the egefaces are ot ol themselves usuall plausble faces, but also drectos of varatos betee faces ˆ + ( ) + e( ) e( K ) [,,,..., ],, e,,, K, K Feature vector of a perso tatstcs-5

26 PCA Eamples: Egeface (3/4) Face mages as the trag set he averaged face tatstcs-6

27 PCA Eamples: Egeface (4/4) eve egefaces derved from the trag set A proected face mage? (Idcate drectos of varatos betee faces ) tatstcs-7

28 PCA Eamples: Egevoce (/3) Eample 5: Egevoce speaker adaptato (PL, 000) teps Cocateatg the regarded parameters for each speaker r to form a huge vector a (r) ( a supervectors) D HMM model mea parameters (μ) peaker Data Model rag peaker R Data Model rag I HMM Each e speaker s represeted b a pot P K-space ( 0 ) + e( ) + e( ) + + e( K) P e,,..., K peaker HMM peaker R HMM I HMM model D (M.) Egevoce space costructo Prcpal Compoet Aalss tatstcs-8

29 PCA Eamples: Egevoce (/3) Eample 4: Egevoce speaker adaptato (cot.) tatstcs-9

30 PCA Eamples: Egevoce (3/3) Eample 5: Egevoce speaker adaptato (cot.) Dmeso (egevoce ): Correlate th ptch or se Dmeso (egevoce ): Correlate th ampltude Dmeso 3 (egevoce 3): Correlate th secod-format movemet Note that: Egeface performs o feature space hle egevoce performs o model space tatstcs-30

31 Lear Dscrmat Aalss (LDA) (/) Also called Fsher s Lear Dscrmat Aalss, Fsher-Rao Lear Dscrmat Aalss Fsher (936): troduced t for to-class classfcato Rao (965): eteded t to hadle multple-class classfcato tatstcs-3

32 Lear Dscrmat Aalss (LDA) (/) Gve a set of sample vectors th labeled (class) formato, tr to fd a lear trasform such that the rato of average betee-class varato over average th-class varato s mamal th-class dstrbutos are assumed here to be Gaussas th equal varace the to-dmesoal sample space tatstcs-3

33 LDA Dervatos (/4) uppose there are N sample vectors th dmesoalt, each of them s belogs to oe of the J classes g,,,..., J, g N he sample mea s: he class sample meas are: he class sample covaraces are: he average th-class varato before trasform N Σ N he average betee-class varato before trasform b ( ) { } ( ) s class de N N N Σ ( )( ) N g ( ) ( ) N g ( )( ) tatstcs-33

34 tatstcs-34 LDA Dervatos (/4) If the trasform s appled he sample vectors ll be he sample mea ll be he class sample meas ll be he average th-class varato ll be [ ] m... N N N N ( ) g N ( ) ( ) ( ) ( ) ( ) Σ g g g N N N N N N N ~ ~ b b ~

35 LDA Dervatos (3/4) [ ] If the trasform... s appled mlarl, the average betee-class varato ll be ~ b b r to fd optmal such that the follog obectve fucto s mamzed ~ b b J ( ) ~ A closed-form soluto: the colum vectors of a optmal matr are the geeralzed egevectors correspodg to the largest egevalues ' s hat s, are the egevectors correspodg to the largest egevalues of m b b b λ λ tatstcs-35

36 tatstcs-36 LDA Dervatos (4/4) Proof: ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) b b b b b b b b b b b b J λ λ λ λ λ λ λ : soluto optmal form has qradtc he : that fd at to e of for each colum vector Or equvaletl, Q Q, arg ma ~ ~ arg ma arg ma ˆ ˆ ˆ ˆ G G F F G G F ( ) C C C + d d ) ( determat

37 LDA Eamples: Feature rasformato (/) Eample: Epermets o peech gal Processg Covarace Matr of the 8-Mel-flter-bak vectors Covarace Matr of the 8-cepstral vectors Calculated usg Year-99 s 547 fles Calculated usg Year-99 s 547 fles Σ N ( )( ) Σ N ( )( ) After Cose rasform tatstcs-37

38 LDA Eamples: Feature rasformato (/) Eample: Epermets o peech gal Processg (cot.) Covarace Matr of the 8-PCA-cepstral vectors Covarace Matr of the 8-LDA-cepstral vectors Calculated usg Year-99 s 547 fles Calculated usg Year-99 s 547 fles After PCA rasform MFCC LDA- LDA- Character Error Rate C G After LDA rasform tatstcs-38

39 PCA vs. LDA (/) PCA LDA tatstcs-39

40 Heteroscedastc Dscrmat Aalss (HDA) HDA: Heteroscedastc Dscrmat Aalss he dfferece the proectos obtaed from LDA ad HDA for -class case Clearl, the HDA provdes a much loer classfcato error tha LDA theoretcall Hoever, most statstcal modelg assume data samples are Gaussa ad have dagoal covarace matrces tatstcs-40

41 H: Feature rasformato (/4) Gve to data sets (MaleData, Female Data) hch each ro s a sample th 39 features, please perform the follog operatos:. Merge these to data sets ad fd/plot the covarace matr for the merged data set.. Appl PCA ad LDA trasformatos to the merged data set, respectvel. Also, fd/plot the covarace matrces for trasformatos, respectvel. Descrbe the pheomea that ou have observed. 3. Use the frst to prcpal compoets of PCA as ell as the frst to egevectors of LDA to represet the merged data set. electvel plot portos of samples from MaleData ad FemaleData, respectvel. Descrbe the pheomea that ou have observed. tatstcs-4

42 H: Feature rasformato (/4) tatstcs-4

43 H: Feature rasformato (3/4) Plot Covarace Matr CoVar[ ; ; ; ]; colormap('default'); surf(covar); Ege Decomposto %LDA IIv(I); AII*BE; %PCA ABE+I; % h?? ( Prove t! ) BE[ ]; I[ ]; ; ; ; ; ; ; [V,D]eg(A); [V,D]egs(A,3); fdfope('bass',''); for :3 % feature vector legth for :3 % bass umber fprtf(fd,'%0.0f ',V(,)); ed fprtf(fd,'\'); ed fclose(fd); tatstcs-43

44 Eamples H: Feature rasformato (4/4) 筆原始資料經 PCA 轉換後分布圖 筆原始資料經 LDA 轉換後分布圖 feature feature feature feature tatstcs-44

45 Latet ematc Aalss (LA) (/7) Also called Latet ematc Ideg (LI), Latet ematc Mappg (LM) A techque orgall proposed for Iformato Retreval (IR), hch proects queres ad docs to a space th latet sematc dmesos Co-occurrg terms are proected oto the same dmesos I the latet sematc space (th feer dmesos), a quer ad doc ca have hgh cose smlart eve f the do ot share a terms Dmesos of the reduced space correspod to the aes of greatest varato Closel related to Prcpal Compoet Aalss (PCA) tatstcs-45

46 LA (/7) Dmeso Reducto ad Feature Etracto PCA X φ X feature space Y k k φ Xˆ VD ( LA) φ φ k orthoormal bass φ m φ k Xˆ X for a gve k Σ V latet sematc k space A U A m k mr latet sematc space kk rr r r m(m,) m A A F for a gve m k tatstcs-46

47 LA (3/7) gular Value Decomposto (VD) used for the orddocumet matr A least-squares method for dmeso reducto Proecto of a Vector : ϕ φ ϕ ϕ cos θ ϕ φ, here φ tatstcs-47

48 LA (4/7) Frameorks to crcumvet vocabular msmatch Doc terms structure model Quer doc epaso lteral term matchg quer epaso terms latet sematc structure retreval structure model tatstcs-48

49 LA (5/7) tatstcs-49

50 LA (6/7) Quer: huma computer teracto A OOV ord tatstcs-50

51 m m LA (7/7) gular Value Decomposto (VD) d d d d d Σ r V rr A U mr m d m mr d d Σ k V kk A U mk m m mk d d rm r m(m,) km d r d k Ro A R Col A R m Both U ad V has orthoormal colum vectors U UI rxr V VI rxr K r A F A F m a F Docs ad queres are represeted a k-dmesoal space. he quattes of the aes ca be properl eghted accordg to the assocated dagoal values of Σ k A tatstcs-5

52 LA Dervatos (/7) gular Value Decomposto (VD) A A s smmetrc matr All egevalues λ are oegatve real umbers λ λ... Defe sgular values: λ All egevectors v are orthoormal ( R ) v v ( V V I ) [ v v v ] V... As the square roots of the egevalues of A A As the legths of the vectors Av, Av,., Av 0 ( λ λ,..., ) Σ dag, λ sgma σ λ,,..., For λ 0,, r, {Av, Av,., Av r } s a orthogoal bass of Col A σ σ... Av Av Av Av v A σ Av v λ v λ tatstcs-5

53 LA Dervatos (/7) {Av, Av,., Av r } s a orthogoal bass of Col A u: also a orthoormal matr (mr) Av ( Av ) Av v A Av λ v v 0 Av uppose that A (or A A) has rak r Defe a orthoormal bass {u, u,., u r } for Col A Eted to a orthoormal bass {u, u,, u m } of R m λ λ... λ r > 0, λ r + λ r +... λ u Av Av σ u Av σ [ u u... u ] Σ A[ v v v ] r r [ u u u... u ] Σ A[ v v... v v ] UΣ AV A UΣV r m UΣV Σ m Σr 0 0( ) ( ) ( ) m r r 0 m r r r r AVV I r ( r )? Av A A F 0 V : a orthoormal matr (r) Ko advace m a F σ + σ + + σ?... r tatstcs-53

54 LA Dervatos (3/7) v spas the ro space of A u spas the ro space of A A m V R R m U V U U V AX 0 UΣV U ( U U ) AV V A Σ V Σ 0 U Σ 0 V 0 V AV tatstcs-54

55 Addtoal Eplaatos LA Dervatos (4/7) Each ro of U s related to the proecto of a correspodg ro of A oto the bass formed b colums of V A UΣV AV UΣV V UΣ UΣ AV the -th etr of a ro of U s related to the proecto of a correspodg ro of A oto the -th colum of V Each ro of V s related to the proecto of a correspodg ro of A oto the bass formed b U A UΣV A U VΣ A ( UΣV ) U U VΣU VΣ the -th etr of a ro of s related to the proecto of a correspodg ro of A V oto the -th colum of U U tatstcs-55

56 LA Dervatos (5/7) Fudametal comparsos based o VD he orgal ord-documet matr (A) m d d A d m compare to terms dot product of to ros of A or a etr AA compare to docs dot product of to colums of A or a etr A A compare a term ad a doc each dvdual etr of A U U mk Σ Σ k V V k he e ord-documet matr (A ) compare to terms dot product of to ros of U Σ Irr compare to docs d dot product of to ros of V Σ s d k compare a quer ord ad a doc each dvdual etr of A A A (U Σ V ) (U Σ V ) U Σ V V Σ U (U Σ )(U Σ ) For stretchg or shrkg A A (U Σ V ) (U Σ V ) V Σ U U Σ V (V Σ )(V Σ ) tatstcs-56

57 LA Dervatos (6/7) Fold-: fd represetatos for pesudo-docs q For obects (e queres or docs) that dd ot appear the orgal aalss Fold- a e m quer (or doc) vector ( ) q U ˆ k m m k Σ k k q Just lke a ro of V Quer represeted b the eghted sum of t costtuet term vectors he separate dmesos are dfferetall eghted Cose measure betee the quer ad doc vectors the latet sematc space sm ( ) qˆ, dˆ coe ( qˆ Σ, dˆ Σ ) qˆ Σ qˆ Σ dˆ dˆ Σ ro vectors tatstcs-57

58 LA Dervatos (7/7) Fold- a e X term vector ˆ k t V k Σ k k t tatstcs-58

59 Epermetal results LA Eample HMM s cosstetl better tha VM at all recall levels LA s better tha VM at hgher recall levels Recall-Precso curve at stadard recall levels evaluated o D-3 D collecto. (Usg ord-level deg terms) tatstcs-59

60 Advatages LA: Coclusos A clea formal frameork ad a clearl defed optmzato crtero (least-squares) Coceptual smplct ad clart Hadle som problems ( heterogeeous vocabular ) Good results for hgh-recall search ake term co-occurrece to accout Dsadvatages Hgh computatoal complet LA offers ol a partal soluto to polsem E.g. bak, bass, tatstcs-60

61 LA oolkt: VDLIBC (/5) Doug Rohde's VD C Lbrar verso.3 s based o the VDPACKC lbrar Doload t at tatstcs-6

62 LA oolkt: VDLIBC (/5) Gve a sparse term-doc matr E.g., 4 terms ad 3 docs Doc erm Each etr s eghted b FIDF score Ro #em Col. Nozero # Doc etres ozero etres at Col 0 Col 0, Ro 0 Col 0, Ro ozero etr at Col Col, Ro 3 ozero etr at Col Col, Ro 0 Col, Ro Col, Ro Perform VD to obta correspodg term ad doc vectors represeted the latet sematc space Evaluate the formato retreval capablt of the LA approach b usg varg szes (e.g., 00, 00,..,600 etc.) of LA dmesoalt tatstcs-6

63 LA oolkt: VDLIBC (3/5) Eample: term-docmatr Ideg Nozero erm o. Doc o. etres VD commad (IR_svd.bat) svd -r st -o LA00 -d 00 erm-doc-matr sparse matr put pref of output fles No. of reserved egevectors output ame of sparse matr put LA00-Ut LA00- LA00-Vt tatstcs-63

64 LA00-Ut LA oolkt: VDLIBC (4/5) ords LA00- ord vector (u ): 00 LA00-Vt 65 docs egevalues doc vector (v ): 00 tatstcs-64

65 LA oolkt: VDLIBC (5/5) Fold- a e m quer vector ( ) q U ˆ k m m k Σ k k q Just lke a ro of V FIDF eghted beforehad Quer represeted b the eghted sum of t costtuet term vectors he separate dmesos are dfferetall eghted Cose measure betee the quer ad doc vectors the latet sematc space sm ( ) qˆ, dˆ coe ( qˆ Σ, dˆ Σ ) qˆ Σ qˆ Σ dˆ dˆ Σ tatstcs-65