Principal Component Analysis (PCA)

Transcription

1 BBM0 - Itroduc0o to ML Sprg 0 Prcpa Compoet Aayss PCA Mo0va0o PCA agorthms Appca0os PCA shortcomgs ere PCA oday Ayut Erdem Dept. of Computer Egeerg HaceDepe Uversty Sdes adopted from Barabás Póczos ar Boosh om Mtche Ro Parr Rta Osadchy Mo0va0o PCA agorthms Appca0os PCA Shortcomgs ere PCA oday Data Vsuaza0o Data Compresso Nose Reduc0o Learg Aomay detec0o PCA Appca0os 3 3

2 Eampe: Data Vsuaza0o Gve 53 bood ad ure sampes features from 5 peope. How ca we vsuaze the measuremets? Istaces Data Vsuaza0o Matr format 553 H-WBC H-RBC H-Hgb H-Hct H-MCV H-MCH H-MCHC A A A A A A A A A Features 5 5 Dffcut to see the correatos betwee the features... Data Vsuaza0o Data Vsuaza0o Spectra format 5 curves oe for each perso Measuremet measuremet 50 0 Dffcut to compare the dfferet patets... Vaue 7 7 Spectra format 53 pctures oe for each feature H-Bads Perso Dffcut to see the correatos betwee the features...

3 Data Vsuaza0o Data Vsuaza0o C-LDH B-varate C-rgycerdes M-EPI C-LDH r-varate C-rgycerdes Is there a represeta0o beder tha the coordate aes? Is t reay ecessary to show a the 53 dmesos? -... what f there are strog correa0os betwee the features? How coud we fd the smaest subspace of the 53- D space that eeps the most formado about the orga data? How ca we vsuaze the other varabes??? dffcut to see or hgher dmesoa spaces A sou0o: Prcpa Compoet Aayss oday Prcpa Compoet Aayss Mo0va0o PCA agorthms Appca0os PCA Shortcomgs ere PCA PCA: Orthogoa proec0o of the of data the data oto a oto ow a ower- dmeso ear space that... mamzes varace of proected data purpe e mmzes mea squared dstace betwee - data pot ad - proec0os sum of bue es

4 Prcpa Compoet Aayss Idea: Gve data pots a d- dmesoa space proect them to a ower dmesoa space whe preservg as much forma0o as possbe. - Fd best paar approma0o to 3D data - Fd best - D approma0o to 0 - D data I par0cuar choose proec0o that mmzes squared error recostruc0g the orga data. Prcpa Compoet Aayss PCA Vectors orgate from the ceter of mass. Prcpa compoet #: pots the drec0o of the argest varace. Each subsequet prcpa compoet - s orthogoa to the prevous oes ad - pots the drec0os of the argest varace of the resdua subspace 3 3 D Gaussa dataset st PCA as 5 5

5 d PCA as PCA agorthm I seque0a Gve the cetered data { m } compute the prcpa vectors: arg ma m w { w } w m st PCA vector We mamze the varace of proecto of w m arg ma {[ w ww ] } w m w PCA recostructo th PCA vector We mamze the varace of the proecto the resdua subspace - w w w w 7 7 w PCA agorthm I seque0a Gve w w - we cacuate w prcpa vector as before: Mamze the varace of proecto of arg ma m w m {[ w w w ] } th PCA vector PCA recostructo PCA agorthm II sampe covarace matr Gve data { m } compute covarace matr m m where m m We mamze the varace of the proecto the resdua subspace w w w w PCA bass vectors the egevectors of w w w w w +w w 9 9 Larger egevaue more mportat egevectors 0 9 0

6 PCA agorthm II sampe covarace matr PCA agorthmx : top egevaues/egevectors % X N m data matr % each data pot coum vector..m m m X subtract mea from each coum vector X X X covarace matr of X { u }..N egevectors/egevaues of N PCA agorthm III SVD SVD of of the the data matr Sguar Vaue Decomposto of the cetered data matr X. X features sampes USV X U S sgfcat ose V sg. sgfcat ose ose Retur { u }.. % top PCA compoets sampes 3 PCA agorthm III oday Coums of U the prcpa vectors { u u } orthogoa ad has ut orm so U U I Ca recostruct the data usg ear combatos of { u u } Matr S Dagoa Shows mportace of each egevector Mo0va0o PCA agorthms Appca0os PCA Shortcomgs ere PCA Coums of V he coeffcets for recostructg the sampes 3 3

7 Mo0va0o PCA agorthms Appca0os - Face Recog0o - Image Compresso - Nose Fterg PCA Shortcomgs ere PCA oday Face Recog0o Wat to de0fy specfc perso based o faca mage Robust to gasses gh0g - Ca t ust use the gve 5 5 pes 5 5 Appyg PCA: Egefaces Computa0oa Compety Method A: Bud a PCA subspace for each perso ad chec whch subspace ca recostruct the test mage the best Method B: Bud oe PCA database for the whoe dataset ad the cassfy based o the weghts. X m 5 5 rea vaues Eampe data set: Images of faces Famous Egeface approach [ur & Petad] [Srovch & rby] Each face s 5 5 vaues umace at ocato 5 5 vew as dm vector Form X [ m ] cetered data mt Compute XX Suppose m staces each of sze N Egefaces: m500 faces each of sze N Gve N N covarace matr ca compute a N egevectors/egevaues ON 3 frst egevectors/egevaues O N But f N EXPENSIVE m faces Probem: s HUGE 7 7 7

8 A Cever Woraroud Prcpe Compoets Method B Note that m<< Use LX X stead of XX If v s egevector of L the Xv s egevector of Proof: L v v X X v v X X X v X v Xv XX X v Xv Xv Xv X m m faces 5 5 rea vaues Prcpe Compoets Method B Shortcomgs Requres carefuy cotroed data: - A faces cetered frame - Same sze - Some ses0vty to age faster f tra wth - oy peope w/out gasses - same gh0g cod0os Method s competey owedge free - some0mes ths s good - Does t ow that faces are wrapped aroud 3D obects heads - Maes o effort to preserve cass ds0c0os

9 Happess subspace method A Dsgust subspace method A Faca Epresso Recog0o Moves 3 Faca Epresso Recog0o Moves

10 Faca Epresso Recog0o Moves Mo0va0o PCA agorthms Appca0os - Face Recog0o - Image Compresso - Nose Fterg PCA Shortcomgs ere PCA oday Orga Image L error ad PCA dm de the orga 379 mage to patches: Dvde the orga 379 mage to patches: - Each patch s a stace Vew each as a - D vector

11 PCA compresso: D > 0D PCA compresso: D > D most mportat egevectors PCA compresso: D > D PCA compresso: D D

12 most mportat egevectors PCA compresso: D > 3D most mportat egevectors PCA compresso: D > D

13 0 most mportat egevectors D Dscrete Cose Bass Loos e the dscrete cose bases of JPG Mo0va0o PCA agorthms Appca0os - Face Recog0o - Image Compresso - Nose Fterg PCA Shortcomgs ere PCA oday Nose Fterg U

14 Nosy mage Deosed mage usg 5 PCA compoets oday Probema0c Data Set for PCA Mo0va0o PCA agorthms Appca0os PCA Shortcomgs ere PCA PCA does t ow abes

15 PCA vs. Fsher Lear Dscrmat Probema0c Data Set for PCA PCA caot capture NON- LINEAR structure PCA mamzes varace depedet of cass mageta FLD attempts to separate casses gree e PCA Cocusos PCA - Fds orthoorma bass for data - Sorts dmesos order of mportace - Dscard ow sgfcace dmesos Uses: - Get compact descrp0o - Igore ose - Improve cassfca0o hopefuy Not magc: - Does t ow cass abes - Ca oy capture ear vara0os Mo0va0o PCA agorthms Appca0os PCA Shortcomgs ere PCA oday Oe of may trcs to reduce dmesoaty

16 Dmesoaty Reduc0o he magc of hgh dmesos Data represeta0o - Iputs are rea- vaued vectors a hgh dmesoa space. Lear structure - Does the date ve a ow dmesoa subspace? Noear structure - Does the data ve o a ow dmesoa submafod? PCA Gve some probem how do we ow what casses of fuc0os are capabe of sovg that probem? VC Vap- Chervoes theory tes us that oqe mappgs whch tae us to a hgher dmesoa space tha the dmeso of the put space provde us wth greater cassfca0o power. Eampe R Eampe: Hgh- Dmesoa Mappg hese casses are eary separabe the put space. We ca mae the probem eary separabe by a smpe mappg Φ : R a + R 3 3 3

17 ere rc Popuar eres Hgh- dmesoa mappg ca serousy crease computa0o 0me. Ca we get aroud ths probem ad s0 get the beeft of hgh- D? Yes ere rc rc Gve ay agorthm that ca be epressed soey terms of dot products ths trc aows us to costruct dfferet oear versos of t. 5 5 ere Prcpe Compoet Aayss PCA Eteds cove0oa prcpa compoet aayss PCA to a hgh dmesoa feature space usg the ere trc. Ca etract up to umber of sampes oear prcpa compoets wthout epesve computa0os. Mag PCA No- Lear Suppose that stead of usg thepots we woud frst map them to some oear feature space - E.g. usg poar coordates stead of cartesa coordates woud hep us dea wth the crce. Etract prcpa compoet that space PCA he resut w be o- ear the orga data space 7 7

18 Derva0o Suppose that the mea of the data the feature space s Covarace: Egevectors µ 0 C Cv λv Derva0o cot d. Egevectors ca be epressed as ear comba0o of features: Proof: thus Cv v λv v v v v λ λ Showg that v v Showg that v v

19 Derva0o cot d. So from before we had ths meas that a sou0os v wth λ 0 e the spa of...e. Fdg the egevectors s equvaet to fdg the coeffcets 73 from before we had v v v λ λ ust a scaar v 73 Derva0o cot d. By subs0tu0g ths bac to the equa0o we get: We ca rewrte t as Mu0pe ths by from the eq: 7 λ λ λ 7 Derva0o cot d. By puggg the ere ad rearragg we get: We ca remove a factor of from both sdes of the matr ths w oy affects the egevectors wth zero egevaue whch w ot be a prcpe compoet ayway: We have a ormaza0o cod0o for vectors: 75 λ λ v v 75 Derva0o cot d. By mu0pyg λ by ad usg the ormaza0o cod0o we get: For a ew pot ts proec0o oto the prcpa compoets s: 7 λ v 7

20 Normazg the feature space I geera may ot be zero mea. Cetered features: he correspodg ere s: 77 ~ + ~ ~ ~ 77 Normazg the feature space cot d. I a matr form where s a matr wth a eemets /. 7 + ~ / / / - ~ + / 7 Summary of ere PCA Pc a ere Costruct the ormazed ere matr of the data dmeso m m: Sove a egevaue probem: For ay data pot ew or od we ca represet t as 79 / / / - ~ + vaue probem: / / / - + ~ λ d y.. 79 Iput pots before ere PCA 0 0

21 Output aqer ere PCA Eampe: De- osg mages he three groups are dstgushabe usg the frst compoet oy Proper0es of PCA ere PCA ca gve a good re- ecodg of the data whe t es aog a o- ear mafod. he ere matr s so ere PCA w have dffcu0es f we have ots of data pots. 3 3