MLCC 2015 Dimensionality Reduction and PCA

Transcription

1 MLCC 2015 Dimensionality Reduction and PCA Lorenzo Rosasco UNIGE-MIT-IIT June 25, 2015

2 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component PCA and Singular Value Decomposition Kernel PCA MLCC

3 Dimensionality Reduction In many practical applications it is of interest to reduce the dimensionality of the data: data visualization data exploration: for investigating the effective dimensionality of the data MLCC

4 Dimensionality Reduction (cont.) This problem of dimensionality reduction can be seen as the problem of defining a map M : X = R D R k, k D, according to some suitable criterion. MLCC

5 Dimensionality Reduction (cont.) This problem of dimensionality reduction can be seen as the problem of defining a map M : X = R D R k, k D, according to some suitable criterion. In the following data reconstruction will be our guiding principle. MLCC

6 Principal Component Analysis PCA is arguably the most popular dimensionality reduction procedure. MLCC

7 Principal Component Analysis PCA is arguably the most popular dimensionality reduction procedure. It is a data driven procedure that given an unsupervised sample S = (x 1,..., x n ) derive a dimensionality reduction defined by a linear map M. MLCC

8 Principal Component Analysis PCA is arguably the most popular dimensionality reduction procedure. It is a data driven procedure that given an unsupervised sample S = (x 1,..., x n ) derive a dimensionality reduction defined by a linear map M. PCA can be derived from several prospective and here we give a geometric derivation. MLCC

9 Dimensionality Reduction by Reconstruction Recall that, if w R D, w = 1, then (w T x)w is the orthogonal projection of x on w MLCC

10 Dimensionality Reduction by Reconstruction Recall that, if w R D, w = 1, then (w T x)w is the orthogonal projection of x on w MLCC

11 Dimensionality Reduction by Reconstruction (cont.) First, consider k = 1. The associated reconstruction error is x (w T x)w 2 (that is how much we lose by projecting x along the direction w) MLCC

12 Dimensionality Reduction by Reconstruction (cont.) First, consider k = 1. The associated reconstruction error is x (w T x)w 2 (that is how much we lose by projecting x along the direction w) Problem: Find the direction p allowing the best reconstruction of the training set MLCC

13 Dimensionality Reduction by Reconstruction (cont.) Let S D 1 = {w R D w = 1} is the sphere in D dimensions. Consider the empirical reconstruction minimization problem, 1 min w S D 1 n n x i (w T x i )w 2. The solution p to the above problem is called the first principal component of the data MLCC

14 An Equivalent Formulation A direct computation shows that x i (w T x i )w 2 = x i (w T x i ) 2 MLCC

15 An Equivalent Formulation A direct computation shows that x i (w T x i )w 2 = x i (w T x i ) 2 Then, problem is equivalent to 1 min w S D 1 n n x i (w T x i )w 2 1 max w S D 1 n n (w T x i ) 2 MLCC

16 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component PCA and Singular Value Decomposition Kernel PCA MLCC

17 Reconstruction and Variance Assume the data to be centered, x = 1 n x i = 0, then we can interpret the term (w T x) 2 as the variance of x in the direction w. MLCC

18 Reconstruction and Variance Assume the data to be centered, x = 1 n x i = 0, then we can interpret the term (w T x) 2 as the variance of x in the direction w. The first PC can be seen as the direction along which the data have maximum variance. 1 n max (w T x i ) 2 w S D 1 n MLCC

19 Centering If the data are not centered, we should consider 1 max w S D 1 n n (w T (x i x)) 2 (1) equivalent to with x c = x x. 1 max w S D 1 n n (w T x c i) 2 MLCC

20 Centering and Reconstruction If we consider the effect of centering to reconstruction it is easy to see that we get where 1 min w,b S D 1 n n x i ((w T (x i b))w + b) 2 ((w T (x i b))w + b is an affine (rather than an orthogonal) projection MLCC

21 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component PCA and Singular Value Decomposition Kernel PCA MLCC

22 PCA as an Eigenproblem A further manipulation shows that PCA corresponds to an eigenvalue problem. MLCC

23 PCA as an Eigenproblem A further manipulation shows that PCA corresponds to an eigenvalue problem. Using the symmetry of the inner product, 1 n n (w T x i ) 2 = 1 n n w T x i w T x i = 1 n n w T x i x T i w = w T ( 1 n n x i x T i )w MLCC

24 PCA as an Eigenproblem A further manipulation shows that PCA corresponds to an eigenvalue problem. Using the symmetry of the inner product, 1 n n (w T x i ) 2 = 1 n n w T x i w T x i = 1 n n w T x i x T i w = w T ( 1 n n x i x T i )w Then, we can consider the problem max w T C n w, w S D 1 C n = 1 n n x i x T i MLCC

25 PCA as an Eigenproblem (cont.) We make two observations: The ( covariance ) matrix C n = 1 n n XT n X n is symmetric and positive semi-definite. MLCC

26 PCA as an Eigenproblem (cont.) We make two observations: The ( covariance ) matrix C n = 1 n n XT n X n is symmetric and positive semi-definite. The objective function of PCA can be written as the so called Rayleigh quotient. w T C n w w T w MLCC

27 PCA as an Eigenproblem (cont.) We make two observations: The ( covariance ) matrix C n = 1 n n XT n X n is symmetric and positive semi-definite. The objective function of PCA can be written as the so called Rayleigh quotient. w T C n w w T w Note that, if C n u = λu then ut C nu u T u = λ, since u is normalized. MLCC

28 PCA as an Eigenproblem (cont.) We make two observations: The ( covariance ) matrix C n = 1 n n XT n X n is symmetric and positive semi-definite. The objective function of PCA can be written as the so called Rayleigh quotient. w T C n w w T w Note that, if C n u = λu then ut C nu u T u = λ, since u is normalized. Indeed, it is possible to show that the Rayleigh quotient achieves its maximum at a vector corresponding to the maximum eigenvalue of C n MLCC

29 PCA as an Eigenproblem (cont.) Computing the first principal component of the data reduces to computing the biggest eigenvalue of the covariance and the corresponding eigenvector. C n u = λu, C n = 1 n n Xn T X n MLCC

30 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component PCA and Singular Value Decomposition Kernel PCA MLCC

31 Beyond the First Principal Component We discuss how to consider more than one principle component (k > 1) M : X = R D R k, k D The idea is simply to iterate the previous reasoning MLCC

32 Residual Reconstruction The idea is to consider the one dimensional projection that can best reconstruct the residuals r i = x i (p T x i )p i MLCC

33 Residual Reconstruction The idea is to consider the one dimensional projection that can best reconstruct the residuals r i = x i (p T x i )p i An associated minimization problem is given by 1 min w S D 1,w p n (note: the constraint w p) n r i (w T r i )w 2. MLCC

34 Residual Reconstruction (cont.) Note that for all i = 1,..., n, since w p r i (w T r i )w 2 = r i 2 (w T r i ) 2 = r i 2 (w T x i ) 2 MLCC

35 Residual Reconstruction (cont.) Note that for all i = 1,..., n, since w p r i (w T r i )w 2 = r i 2 (w T r i ) 2 = r i 2 (w T x i ) 2 Then, we can consider the following equivalent problem 1 max w S D 1,w p n n (w T x i ) 2 = w T C n w. MLCC

36 PCA as an Eigenproblem 1 max w S D 1,w p n n (w T x i ) 2 = w T C n w. Again, we have to minimize the Rayleigh quotient of the covariance matrix with the extra constraint w p MLCC

37 PCA as an Eigenproblem 1 max w S D 1,w p n n (w T x i ) 2 = w T C n w. Again, we have to minimize the Rayleigh quotient of the covariance matrix with the extra constraint w p Similarly to before, it can be proved that the solution of the above problem is given by the second eigenvector of C n, and the corresponding eigenvalue. MLCC

38 PCA as an Eigenproblem (cont.) C n u = λu, C n = 1 n n x i x T i The reasoning generalizes to more than two components: computation of k principal components reduces to finding k eigenvalues and eigenvectors of C n. MLCC

39 Remarks Computational complexity roughly O(kD 2 ) (complexity of forming C n is O(nD 2 )). If we have n points in D dimensions and n D can we compute PCA in less than O(nD 2 )? MLCC

40 Remarks Computational complexity roughly O(kD 2 ) (complexity of forming C n is O(nD 2 )). If we have n points in D dimensions and n D can we compute PCA in less than O(nD 2 )? The dimensionality reduction induced by PCA is a linear projection. Can we generalize PCA to non linear dimensionality reduction? MLCC

41 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component PCA and Singular Value Decomposition Kernel PCA MLCC

42 Singular Value Decomposition Consider the data matrix X n, its singular value decomposition is given by where: X n = UΣV T U is a n by k orthogonal matrix, V is a D by k orthogonal matrix, Σ is a diagonal matrix such that Σ i,i = λ i, i = 1,..., k and k min{n, D}. The columns of U and the columns of V are the left and right singular vectors and the diagonal entries of Σ the singular values. MLCC

43 Singular Value Decomposition (cont.) The SVD can be equivalently described by the equations 1 C n p j = λ j p j, n K nu j = λ j u j, X n p j = 1 λ j u j, n XT n u j = λ j p j, for j = 1,..., d and where C n = 1 n XT n X n and 1 n K n = 1 n X nx T n MLCC

44 PCA and Singular Value Decomposition If n p we can consider the following procedure: form the matrix K n, which is O(Dn 2 ) find the first k eigenvectors of K n, which is O(kn 2 ) compute the principal components using p j = 1 λj X T n u j = 1 λj n x i u i j, j = 1,..., d where u = (u 1,..., u n ), This is O(knD) if we consider k principal components. MLCC

45 Outline PCA & Reconstruction PCA and Maximum Variance PCA and Associated Eigenproblem Beyond the First Principal Component PCA and Singular Value Decomposition Kernel PCA MLCC

46 Beyond Linear Dimensionality Reduction? By considering PCA we are implicitly assuming the data to lie on a linear subspace... MLCC

47 Beyond Linear Dimensionality Reduction? By considering PCA we are implicitly assuming the data to lie on a linear subspace......it is easy to think of situations where this assumption might violated. MLCC

48 Beyond Linear Dimensionality Reduction? By considering PCA we are implicitly assuming the data to lie on a linear subspace......it is easy to think of situations where this assumption might violated. Can we use kernels to obtain non linear generalization of PCA? MLCC

49 From SVD to KPCA Using SVD the projection of a point x on a principal component p j, for j = 1,..., d, is (M(x)) j = x T p j = 1 λj x T X T n u j = 1 λj n x T x i u i j, Recall 1 C n p j = λ j p j, n K nu j = λ j u j, X n p j = 1 λ j u j, n XT n u j = λ j p j, MLCC

50 PCA and Feature Maps (M(x)) j = 1 λj What if consider a non linear feature-map Φ : X F, before performing PCA? n x T x i u i j, MLCC

51 PCA and Feature Maps (M(x)) j = 1 λj What if consider a non linear feature-map Φ : X F, before performing PCA? n x T x i u i j, (M(x)) j = Φ(x) T p j = 1 λj n where K n σ j = σ j u j and (K n ) i,j = Φ(x) T Φ(x j ). Φ(x) T Φ(x i )u i j, MLCC

52 Kernel PCA (M(x)) j = Φ(x) T p j = 1 λj n Φ(x) T Φ(x i )u i j, If the feature map is defined by a positive definite kernel K : X X R, then (M(x)) j = 1 λj n where K n σ j = σ j u j and (K n ) i,j = K(x i, x j ). K(x, x i )u i j, MLCC

53 Wrapping Up In this class we introduced PCA as a basic tool for dimensionality reduction. We discussed computational aspect and extensions to non linear dimensionality reduction (KPCA) MLCC

54 Next Class In the next class, beyond dimensionality reduction, we ask how we can devise interpretable data models, and discuss a class of methods based on the concept of sparsity. MLCC