Principal Component Analysis (PCA)

Transcription

1 BBM406 - Itroduc0o to ML Sprg 204 Prcpal Compoet Aalyss PCA Aykut Erdem Dept. of Computer Egeerg HaceDepe Uversty

2 Today Mo0va0o PCA algorthms Applca0os PCA shortcomgs Kerel PCA Sldes adopted from Barabás Póczos, Kar Booksh, Tom Mtchell, Ro Parr, Rta Osadchy 2

3 Today Mo0va0o PCA algorthms Applca0os PCA Shortcomgs Kerel PCA 3

4 PCA Applca0os Data Vsualza0o Data Compresso Nose Reduc0o Learg Aomaly detec0o 4

5 Data Vsualza0o Eample:! Gve 53 blood ad ure samples features from 65 people.! How ca we vsualze the measuremets? 5

6 Data Vsualza0o Matr format 6553 Istaces H-WBC H-RBC H-Hgb H-Hct H-MCV H-MCH H-MCHC A A A A A A A A A Features Dffcult to see the correlatos betwee the features... 6

7 Data Vsualza0o Spectral format 65 curves, oe for each perso measuremet Measuremet Dffcult to compare the dfferet patets... Value 7

8 Data Vsualza0o Spectral format 53 pctures, oe for each feature H-Bads Perso Dffcult to see the correlatos betwee the features... 8

9 Data Vsualza0o B-varate Tr-varate C-LDH C-Trglycerdes M-EPI C-LDH C-Trglycerdes How ca we vsualze the other varables??? dffcult to see 4 or hgher dmesoal spaces

10 Data Vsualza0o Is there a represeta0o beder tha the coordate aes? Is t really ecessary to show all the 53 dmesos? -... what f there are strog correla0os betwee the features? How could we fd the smallest subspace of the 53- D space that keeps the most formado about the orgal data? A solu0o: Prcpal Compoet Aalyss 0

11 Today Mo0va0o PCA algorthms Applca0os PCA Shortcomgs Kerel PCA

12 Prcpal Compoet Aalyss PCA:! Orthogoal projec0o of the of data the data oto a oto low a lower- dmeso lear space that... mamzes varace of projected data purple le mmzes mea squared dstace betwee - data pot ad - projec0os sum of blue les 2

13 Prcpal Compoet Aalyss Idea: Gve data pots a d- dmesoal space, project them to a lower dmesoal space whle preservg as much forma0o as possble. - Fd best plaar approma0o to 3D data - Fd best 2- D approma0o to D data I par0cular, choose projec0o that mmzes squared error recostruc0g the orgal data. 3

14 Prcpal Compoet Aalyss PCA Vectors orgate from the ceter of mass. Prcpal compoet #: pots the drec0o of the largest varace. Each subsequet prcpal compoet - - s orthogoal to the prevous oes, ad pots the drec0os of the largest varace of the resdual subspace 4

15 2D Gaussa dataset 5

16 st PCA as 6

17 2 d PCA as 7

18 PCA algorthm I seque0al Gve the cetered data {,, m }, compute the prcpal vectors: m 2 arg ma T w { w } w m st PCA vector We mamze the varace of projecto of w m T T 2 2 arg ma {[ w ww ] } w m We mamze the varace of the projecto the resdual subspace w w - PCA recostructo k th PCA vector =w w T w 8

19 PCA algorthm I seque0al Gve w,, w k-, we calculate w k prcpal vector as before: Mamze the varace of projecto of w k arg ma w m m {[ w T k j w j w T j ] 2 } k th PCA vector PCA recostructo We mamze the varace of the projecto the resdual subspace w 2 w 2T w 2 w w w T w =w w T +w 2 w 2T 9 9

20 PCA algorthm II sample covarace matr Gve data {,, m }, compute covarace matr m m T where m m PCA bass vectors = the egevectors of Larger egevalue more mportat egevectors 20

21 PCA algorthm II sample covarace matr PCA algorthmx, k: top k egevalues/egevectors % X = N m data matr, % each data pot = colum vector, =..m m m X subtract mea from each colum vector X X X T covarace matr of X {, u } =..N = egevectors/egevalues of 2 N Retur {, u } =.. k % top k PCA compoets 2

22 PCA algorthm III SVD of the data matr SVD of the data matr Sgular Value Decomposto of the cetered data matr X. X features samples = USV T X = U S V T sg. sgfcat sgfcat ose ose ose samples 23 22

23 PCA algorthm III Colums of U the prcpal vectors, { u,, u k } orthogoal ad has ut orm so U T U = I Ca recostruct the data usg lear combatos of { u,, u k } Matr S Dagoal Shows mportace of each egevector Colums of V T The coeffcets for recostructg the samples 23

24 Today Mo0va0o PCA algorthms Applca0os PCA Shortcomgs Kerel PCA 24

25 Today Mo0va0o PCA algorthms Applca0os Face Recog0o Image Compresso Nose Flterg PCA Shortcomgs Kerel PCA 25

26 Face Recog0o Wat to de0fy specfc perso, based o facal mage Robust to glasses, lgh0g, - Ca t just use the gve pels 26

27 Applyg PCA: Egefaces Method A: Buld a PCA subspace for each perso ad check whch subspace ca recostruct the test mage the best Method B: Buld oe PCA database for the whole dataset ad the classfy based o the weghts. X =,, m m faces real values Eample data set: Images of faces Famous Egeface approach [Turk & Petlad], [Srovch & Krby] Each face s values lumace at locato vew as 64K dm vector Form X = [,, m ] cetered data mt Compute = XX T Problem: s 64K 64K HUGE!!! 27 27

28 Computa0oal Complety Suppose m staces, each of sze N Egefaces: m=500 faces, each of sze N=64K Gve N N covarace matr ca compute all N egevectors/egevalues ON 3 frst k egevectors/egevalues Ok N 2 But f N=64K, EXPENSIVE! 28

29 A Clever Workaroud Note that m<<64k Use L=X T X stead of =XX T If v s egevector of L the Xv s egevector of Proof: L v = v X T X v = v X X T X v = X v = Xv XX T X v = Xv Xv = Xv X =,, m m faces real values 29

30 Prcple Compoets Method B 30

31 Prcple Compoets Method B faster f tra wth - - oly people w/out glasses same lgh0g cod0os 3

32 Shortcomgs Requres carefully cotrolled data: All faces cetered frame Same sze Some ses0vty to agle! Method s completely kowledge free - - some0mes ths s good! Does t kow that faces are wrapped aroud 3D objects heads - Makes o effort to preserve class ds0c0os 32

33 Happess subspace method A 33

34 Dsgust subspace method A 34

35 Facal Epresso Recog0o Moves 35

36 Facal Epresso Recog0o Moves 36

37 Facal Epresso Recog0o Moves Moves 37

38 Today Mo0va0o PCA algorthms Applca0os Face Recog0o Image Compresso Nose Flterg PCA Shortcomgs Kerel PCA 38

39 Orgal Image de the orgal mage to patches: Dvde the orgal mage to patches: - Each patch s a stace Vew each as a 44- D vector 39

40 L 2 error ad PCA dm 40

41 PCA compresso: 44D => 60D 4

42 PCA compresso: 44D => 6D 42

43 6 most mportat egevectors

44 PCA compresso: 44D => 6D 44

45 6 most mportat egevectors

46 PCA compresso: 44D => 3D 46

47 3 most mportat egevectors

48 PCA compresso: 44D => D 48

49 60 most mportat egevectors Looks lke the dscrete cose bases of JPG! 49

50 2D Dscrete Cose Bass 50

51 Today Mo0va0o PCA algorthms Applca0os Face Recog0o Image Compresso Nose Flterg PCA Shortcomgs Kerel PCA 5

52 Nose Flterg U 52

53 Nosy mage 53

54 Deosed mage usg 5 PCA compoets 54

55 Today Mo0va0o PCA algorthms Applca0os PCA Shortcomgs Kerel PCA 55

56 Problema0c Data Set for PCA PCA does t kow labels! 56

57 PCA vs. Fsher Lear Dscrmat PCA mamzes varace, depedet of class mageta FLD attempts to separate classes gree le 57

58 Problema0c Data Set for PCA PCA caot capture NON- LINEAR structure! 58

59 PCA Coclusos PCA Fds orthoormal bass for data Sorts dmesos order of mportace Dscard low sgfcace dmesos Uses: Get compact descrp0o Igore ose Improve classfca0o hopefully Not magc: - - Does t kow class labels Ca oly capture lear vara0os Oe of may trcks to reduce dmesoalty! 59

60 Today Mo0va0o PCA algorthms Applca0os PCA Shortcomgs Kerel PCA 60

61 Dmesoalty Reduc0o Data represeta0o - Iputs are real- valued vectors a hgh dmesoal space. Lear structure PCA - Does the date lve a low dmesoal subspace? Nolear structure - Does the data lve o a low dmesoal submafold? 6

62 The magc of hgh dmesos Gve some problem, how do we kow what classes of fuc0os are capable of solvg that problem?! VC Vapk- Chervoeks theory tells us that oqe mappgs whch take us to a hgher dmesoal space tha the dmeso of the put space provde us wth greater classfca0o power. 62

63 Eample R 2 These classes are learly separable the put space. 63

64 Eample: Hgh- Dmesoal Mappg We ca make the problem learly separable by a smple mappg Φ : R 2 R 3 2, a,,

65 Kerel Trck Hgh- dmesoal mappg ca serously crease computa0o 0me. Ca we get aroud ths problem ad s0ll get the beeft of hgh- D? Yes! Kerel Trck!! rck K = φ T φ, j j Gve ay algorthm that ca be epressed solely terms of dot products, ths trck allows us to costruct dfferet olear versos of t. 65

66 Popular Kerels 66

67 Kerel Prcple Compoet Aalyss KPCA Eteds cove0oal prcpal compoet aalyss PCA to a hgh dmesoal feature space usg the kerel trck.! Ca etract up to umber of samples olear prcpal compoets wthout epesve computa0os. 67

68 Makg PCA No- Lear Suppose that stead of usg thepots we would frst map them to some olear feature space φ - E.g. usg polar coordates stead of cartesa coordates would help us deal wth the crcle.! Etract prcpal compoet that space PCA! The result wll be o- lear the orgal data space! 68

69 Derva0o Suppose that the mea of the data the feature space s!! Covarace:! µ C = = = φ = 0 φ φ = T! Egevectors Cv = λv 69

70 Derva0o cot d. Egevectors ca be epressed as lear comba0o of features:!!! Proof: thus 70 T T v v v φ φ λ φ φ λ = = = = v v v T λ φ = φ = = C v φ α = =

71 Showg that T v = v T 7

72 Showg that T v = v T 72

73 Derva0o cot d. So, from before we had,!!! ths meas that all solu0os v wth λ = 0 le the spa of φ,..,φ,.e.,!!! Fdg the egevectors s equvalet to fdg the coeffcets α 73 from before we had, T T v v v φ φ λ φ φ λ = = = = just a scalar v φ α = =

74 Derva0o cot d. By subs0tu0g ths back to the equa0o we get:!!! We ca rewrte t as!!! Mul0ple ths by φ k from the leq: 74 = = = = l l jl j l l jl T λ φ α φ α φ φ = = = = l l jl j l l jl λ K, φ α α φ = = = = l l T k jl j l l jl T k λ K, φ φ α α φ φ

75 Derva0o cot d. By pluggg the kerel ad rearragg we get: K 2 α We ca remove a factor of K from both sdes of the matr ths wll oly affects the egevectors wth zero egevalue, whch wll ot be a prcple compoet ayway: j = λ j K α j Kα = λ α j j j We have a ormalza0o cod0o for α j vectors: v T j v j = k = l= T α α φ φ = α Kα = jl jk l k j j T 75

76 Derva0o cot d. By mul0plyg Kαj = λjαj by αj ad usg the ormalza0o cod0o we get:!! λ j α T j α j =, j! For a ew pot, ts projec0o oto the prcpal compoets s: T T = j = = φ v = α φ φ α j j K, 76

77 Normalzg the feature space I geeral, φ may ot be zero mea. Cetered features:!! The correspodg kerel s: 77 = = k k k ~ φ φ φ = = = = = + = = = k l k l k k j k k j k k j T k k j T j K K K K K, 2,,,, ~ ~, ~ φ φ φ φ φ φ

78 Normalzg the feature space cot d. I a matr form where s a matr wth all elemets /. 78 = = = + = k l k l k k j k k j j K K K K K, 2,,,,, ~ / / / K K K - 2 K ~ + = /

79 Summary of Kerel PCA Pck a kerel Costruct the ormalzed kerel matr of the data dmeso m m:!! Solve a egevalue problem:! K ~ = K K K / / value problem: K ~ α = λ α! For ay data pot ew or old, we ca represet / t as y j = = α j K,, j =,.., d 79

80 Iput pots before kerel PCA 80

81 Output aqer kerel PCA The three groups are dstgushable usg the frst compoet oly 8

82 Eample: De- osg mages 82

83 Proper0es of KPCA Kerel PCA ca gve a good re- ecodg of the data whe t les alog a o- lear mafold.! The kerel matr s, so kerel PCA wll have dffcul0es f we have lots of data pots. 83