Machine Learning for Data Science (CS4786) Lecture 9

Size: px

Start display at page:

Download "Machine Learning for Data Science (CS4786) Lecture 9"

Laurel Jenkins
6 years ago
Views:

1 Machie Learig for Data Sciece (CS4786) Lecture 9 Pricipal Compoet Aalysis Course Webpage :

2 DIM REDUCTION: LINEAR TRANSFORMATION x > y > Pick a low imesioal subspace X Project liearly to this subspace W = Subspace retais as much iformatio Y x > y > i = x > i W K y > K

3 Example: Stuets i classroom z y x

4 PCA: VARIANCE MAXIMIZATION

5 PCA: VARIANCE MAXIMIZATION

6 PCA: VARIANCE MAXIMIZATION

7 DIM REDUCTION: LINEAR TRANSFORMATION Prelue: reucig to imesio Pick a low imesioal subspace w Project liearly to this subspace x x 2 Subspace retais as much iformatio y = w > x = kx k cos (\wx ) x 4 x 3 0

8 PCA: VARIANCE MAXIMIZATION Pick irectios alog which ata varies the most First pricipal compoet: Variace = X = = = = y t X s= y s! 2! X X 2 w > x t w > x s s= X w > x t w > X w > (x t µ) 2!! X 2 x s s= = average square ier prouct

9 Which Directio? O average parallel O average orthogoal X w > (x t µ) 2 = X kx t µk 2 cosie(w, x t µ)

10 PCA: VARIANCE MAXIMIZATION Pick irectios alog which ata varies the most First pricipal compoet: w = arg max w w 2 = = arg max w w 2 = = arg max w w 2 = = arg max w w 2 = is the covariace matrix w x t w (x t µ) 2 2 w x t w (x t µ)(x t µ) w w w

11 Review Review covariace Review Eige vectors

12 PCA: VARIANCE MAXIMIZATION Covariace matrix: = (x t µ)(x t µ) Its a matrix, [i, j] measures covariace of features i a j [i, j] = (x t [i] µ[i])(x t [j] µ[j])

13 What are Eige Vectors?

14 What are Eige Vectors?.5 x 7! Ax

15 What are Eige Vectors?.5 x 7! Ax Ax = x

16 Which Directio? O average parallel O average orthogoal Top Eigevector of covariace matrix

17 What if we wat more tha oe umber for each ata poit? That is we wat to reuce to K > imesios? z y x

18 PCA: VARIANCE MAXIMIZATION How o we fi the K compoets? We are lookig for orthogoal irectios that maximize total sprea i each irectio Fi orthoormal W that maximizes K j= y t [j] 2 y t [j] = = K j= w j x t K w j w j j= 2 x t This solutios is give by W = Top K eigevectors of

19 PCA: VARIANCE MAXIMIZATION How o we fi the K compoets? As: Maximize sum of sprea i the K irectios We are lookig for orthogoal irectios that maximize total sprea i each irectio Fi orthoormal W that maximizes K j= y t [j] 2 y t [j] = = K j= w j x t K w j w j j= 2 x t This solutios is give by W = Top K eigevectors of

20 PCA: VARIANCE MAXIMIZATION How o we fi the K compoets? We are lookig for orthogoal irectios that maximize total sprea i each irectio Fi orthoormal W that maximizes K j= y t [j] 2 y t [j] = = K j= w j x t K w j w j j= X w i [k]w j [k] =0 & k= x t X w i [k] = k= 2 This solutios is give by W = Top K eigevectors of

21 ! PRINCIPAL COMPONENT ANALYSIS Eigevectors of the covariace matrix are the pricipal compoets. =cov X Top K pricipal compoets are the eigevectors with K largest eigevalues 2. W = eigs,k Projectio = Data Top Keigevectors ( ) Recostructio = Projectio Traspose of top K eigevectors X µ Iepeetly iscovere by Pearso i 90 a Hotellig i Y = W

22 A Alterative View of PCA

23 PCA: MINIMIZING RECONSTRUCTION ERROR x ˆx X kˆx t x t k 2

24 Maximize Sprea Miimize Recostructio Error

25 ORTHONORMAL PROJECTIONS Thik of w,...,w K as cooriate system for PCA (i a K imesioal subspace) y values provie coefficiets i this system Without loss of geerality, w,...,w K ca be orthoormal, i.e. w i w j & w i =. 2 Recostructio: ˆx t = kw i k 2 2 = X K y t [j]w j If we take all,...,w, the x t = j= y t[j]w j. To reuce w i? w j ) w i [k]w j [k] =0 imesioality we oly cosier first K vectors of the basis k= X j= k= w i [k] 2

26 CENTERING DATA 0 Compressig these ata poits

27 CENTERING DATA 0 is same as compressig these.

28 ORTHONORMAL PROJECTIONS (Cetere) Data-poits as liear combiatio of some orthoormal basis, i.e. x t = µ + y t [j]w j j= where w,...,w R are the orthoormal basis a µ = x t. Represet ata as liear combiatio of just K orthoormal basis, ˆx t = µ + K y t [j]w j + µ j=

29 PCA: MINIMIZING RECONSTRUCTION ERROR Goal: fi the basis that miimizes recostructio error, ˆx t x t 2 2 = = = = = K y t [j]w j + µ x t j= K y t [j]w j + µ j= j=k+ y t [j]w j j=k+ 2 2 y t [j] 2 w j y t [j]w j µ j= 2 2 (but a + b 2 2 = a2 2 + b a b) j=k+ i=j+ y t [j]y t [i]w j w i y t [j] 2 w j 2 2 (last step because w j w i )

30 PCA: MINIMIZING RECONSTRUCTION ERROR ˆx t x t 2 2 = = = = = y t [j] 2 w j 2 2 (but w j = ) y t [j] 2 (ow y j = w j (x t µ)) (w j (x t µ)) 2 w j w j w j (x t µ)(x t µ) w j

31 PCA: MINIMIZING RECONSTRUCTION ERROR ˆx t x t 2 2 = = = = = y t [j] 2 w j 2 2 (but w j = ) y t [j] 2 (ow y j = w j (x t µ)) (w j (x t µ)) 2 w j w j w j (x t µ)(x t µ) w j

32 PCA: MINIMIZING RECONSTRUCTION ERROR ˆx t x t 2 2 = = = = = y t [j] 2 w j 2 2 (but w j = ) y t [j] 2 (ow y j = w j (x t µ)) (w j (x t µ)) 2 w j w j w j (x t µ)(x t µ) w j

33 PCA: MINIMIZING RECONSTRUCTION ERROR ˆx t x t 2 2 = = = = = y t [j] 2 w j 2 2 (but w j = ) y t [j] 2 (ow y j = w j (x t µ)) (w j (x t µ)) 2 w j w j w j (x t µ)(x t µ) w j

34 PCA: MINIMIZING RECONSTRUCTION ERROR ˆx t x t 2 2 = = = = = y t [j] 2 w j 2 2 (but w j = ) y t [j] 2 (ow y j = w j (x t µ)) (w j (x t µ)) 2 w j w j w j (x t µ)(x t µ) w j

35 PCA: MINIMIZING RECONSTRUCTION ERROR Miimize w.r.t. w,...,w K s that are orthoormal, argmi j, w j 2 = w j w j Solutio, (iscar) w K+,...,w are bottom K eigevectors Hece w,...,w K are the top K eigevectors

36 ! PRINCIPAL COMPONENT ANALYSIS Eigevectors of the covariace matrix are the pricipal compoets. =cov X Top K pricipal compoets are the eigevectors with K largest eigevalues 2. W = eigs,k Projectio = Data Top Keigevectors ( ) Recostructio = Projectio Traspose of top K eigevectors X µ Iepeetly iscovere by Pearso i 90 a Hotellig i Y = W

37 RECONSTRUCTION 4. bx W > +µ = Y

Machine Learning for Data Science (CS4786) Lecture 4

Machine Learning for Data Science (CS4786) Lecture 4 Machie Learig for Data Sciece (CS4786) Lecture 4 Caoical Correlatio Aalysis (CCA) Course Webpage : http://www.cs.corell.edu/courses/cs4786/2016fa/ Aoucemet We are gradig HW0 ad you will be added to cms