Last time: PCA. Statistical Data Mining and Machine Learning Hilary Term Singular Value Decomposition (SVD) Eigendecomposition and PCA

Transcription

1 Last time: PCA Statistical Data Mining and Machine Learning Hilary Term 2016 Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: PCA Find an orthogonal basis {v 1, v 2,..., v p } for the data space such that: The first principal component (PC) v 1 is the direction of greatest variance of data. The j-th PC v j is the direction orthogonal to v 1, v 2,..., v j 1 of greatest variance, for j = 2,..., p. Eigendecomposition of the sample covariance matrix S = 1 n 1 n i=1 x ix i. S = VΛV. Λ is a diagonal matrix with eigenvalues (variances along each principal component) λ 1 λ 2 λ p 0 V is a p p orthogonal matrix whose columns are the p eigenvectors of S, i.e. the principal components v 1,..., v p Dimensionality reduction by projecting x i R p onto first k principal components: z i = [ v 1 x i,..., v k x i ] R k. Eigendecomposition and PCA Singular Value Decomposition (SVD) S = 1 n 1 n i=1 x i x i = 1 n 1 X X. S is a real and symmetric matrix, so there exist p eigenvectors v 1,..., v p that are pairwise orthogonal and p associated eigenvalues λ 1,..., λ p which satisfy the eigenvalue equation Sv i = λ i v i. In particular, V is an orthogonal matrix: VV = V V = I p. S is a positive-semidefinite matrix, so the eigenvalues are non-negative: λ i 0, i. Why is S symmetric? Why is S positive-semidefinite? Reminder: A symmetric p p matrix R is said to be positive-semidefinite if a R p, a Ra 0. SVD Any real-valued n p matrix X can be written as X = UDV where U is an n n orthogonal matrix: UU = U U = I n D is a n p matrix with decreasing non-negative elements on the diagonal (the singular values) and zero off-diagonal elements. V is a p p orthogonal matrix: VV = V V = I p SVD always exists, even for non-square matrices. Fast and numerically stable algorithms for SVD are available in most packages.the relevant R command is svd.

2 SVD and PCA Let X = UDV be the SVD of the n p data matrix X. Note that (n 1)S = X X = (UDV ) (UDV ) = VD U UDV = VD DV, using orthogonality (U U = I n ) of U. The eigenvalues of S are thus the diagonal entries of Λ = 1 n 1 D D. We also have XX = (UDV )(UDV ) = UDV VD U = UDD U, using orthogonality (V V = I p ) of V. Gram matrix B = XX, B ij = x i x j is called the Gram matrix of dataset X. B and (n 1)S = X X have the same nonzero eigenvalues, equal to the non-zero squared singular values of X. > biplot(crabs.pca,scale=1) CWCL BD FL RW PCA plots show the data items (rows of X) in the space spanned by PCs. allow us to visualize the original variables X (1),..., X (p) (corresponding to columns of X) in the same plot. Iris Data Recall that X = [X (1),..., X (p) ] and X = UDV is the SVD of the data matrix. The full PC projection of x i is the i-th row of UD: z i = V x i = D U i, equivalently: XV = UD. The j-th unit vector e j R p points in the direction of the original variable X (j). Its PC projection η j is: η j = V e j = V j (the j-th row of V) The projection of e j indicates the weighting each PC gives to the original variable X (j). Dot products between these projections give entries of the data matrix: 50 samples from each of the 3 species of iris: setosa, versicolor, and virginica Each measuring the length and widths of both sepal and petals Collected by E. Anderson (135) and analysed by R.A. Fisher (136) x ij = min{n,p} k=1 U ik D kk V jk = D U i, V j = z i, η j. focus on the first two PCs and the quality depends on the proportion of variance explained by the first two PCs.

3 Iris Data Iris data biplot > data(iris) > iris[sample(150,20),] Sepal.Length Sepal.Width Petal.Length Petal.Width Species versicolor setosa setosa versicolor virginica setosa versicolor versicolor setosa versicolor virginica versicolor setosa setosa versicolor versicolor versicolor virginica setosa setosa > iris.pca<-princomp(iris[,-5],cor=true) > loadings(iris.pca) Comp.3 Comp.4 Sepal.Length Sepal.Width Petal.Length Petal.Width > biplot(iris.pca,scale=0) Sepal.Width Petal.Length Petal.Width Sepal.Length Iris Data biplot - scaled There are other projections we can consider for biplots (assuming p < n to simplify notation): p x ij = U ik D kk V jk = D 1:p,1:pUi,1:p, Vj = D 1 α 1:p,1:p U i,1:p, D α 1:p,1:pVj. k=1 where 0 α 1, i.e., we change representation to z i = D 1 α 1:p,1:p U i,1:p, η j = D α 1:p,1:pV j case α = 1: Sample covariance of the projected points is: Ĉov ( Z ) = 1 n 1 U 1:n,1:pU 1:n,1:p = 1 n 1 I p. Projected points are uncorrelated and dimensions are equi-variance. Sample covariance between X (i) and X (j) is: Ê(X (i) X (j) ) = 1 ( VD DV ) = 1 n 1 i,j n 1 D 1:p,1:pVi, D 1:p,1:p Vj The angle between the projected variables corresponds to their correlation. >?biplot... scale: The variables are scaled by lambda ^ scale and the observations are scaled by lambda ^ (1-scale) where lambda are the singular values as computed by princomp. (default=1)... > biplot(iris.pca,scale=1) Sepal.Width Petal.Len Petal.Wid Sepal.Length

4 Crabs Data biplots US Arrests Data > biplot(crabs.pca,scale=0) > biplot(crabs.pca,scale=1) CW CL BD FL RW CWCL BD FL RW This data set contains statistics, in arrests per 100,000 residents for assault, murder, and rape in each of the 50 US states in 173. Also given is the percent of the population living in urban areas. pairs(usarrests) usarrests.pca <- princomp(usarrests,cor=t) plot(usarrests.pca) pairs(predict(usarrests.pca)) biplot(usarrests.pca) US Arrests Data Pairs Plot US Arrests Data Biplot > pairs(usarrests) > biplot(usarrests.pca) Murder Assault UrbanPop Rape Mississippi North Carolina South Carolina West VirginiaVermont Georgia Alabama Alaska Arkansas Kentucky Murder Louisiana Tennessee South Dakota North Dakota Montana Maryland Assault Wyoming Maine Virginia Idaho New Mexico Florida New Hampshire Michigan Iowa Indiana Nebraska Missouri Kansas Rape Delaware Oklahoma Texas Oregon Pennsylvania Minnesota Wisconsin Illinois Nevada Arizona Ohio New York Colorado Washington Connecticut New Jersey Massachusetts Utah Rhode Island California Hawaii UrbanPop

5 Suppose there are n points X in R p, but we are only given the n n matrix D of inter-point distances. Can we reconstruct X? Rigid transformations (translations, rotations and reflections) do not change inter-point distances so cannot recover X exactly. However X can be recovered up to these transformations! Let d ij = x i x j 2 be the distance between points x i and x j. d 2 ij = x i x j 2 2 = (x i x j ) (x i x j ) = x i x i + x j x j 2x i x j Let B = XX be the n n matrix of dot-products, b ij = x i x j. The above shows that D can be computed from B. Some algebraic exercise shows that B can be recovered from D if we assume n i=1 x i = 0. US City Flight Distances If we knew X, then SVD gives X = UDV. As X has rank at most r = min(n, p), we have at most r non-zero singular values in D and we can assume U R n r, D R r r and V R r p. The eigendecomposition of B is then: B = XX = UD 2 U = UΛU. This eigendecomposition can be obtained from B without knowledge of X! Let x i = U i Λ 1 2 R r. If r < p, pad x i with 0s so that it has length p. Then, x i x j = U i ΛU j = b ij = x i x j and we have found a set of vectors with dot-products given by B, as desired. The vectors x i differs from x i only via the orthogonal matrix V (recall that xi = U i DV = x i V ) so are equivalent up to rotation and reflections. We present a table of flying mileages between 10 American cities, distances calculated from our 2-dimensional world. Using D as the starting point, metric MDS finds a configuration with the same distance matrix. ATLA CHIG DENV HOUS LA MIAM NY SF SEAT DC

6 US City Flight Distances US City Flight Distances library(mass) us <- read.csv(" ## use classical MDS to find lower dimensional views of the data ## recover X in 2 dimensions us.classical <- cmdscale(d=us,k=2) plot(us.classical) text(us.classical,labels=names(us)) MIAMI NY DC ATLANTA CHICAGO HOUSTON DENVER LA SF SEATTLE Lower-dimensional Reconstructions Crabs Data In classical MDS derivation, we used all eigenvalues in the eigendecomposition of B to reconstruct x i = U i Λ 1 2. library(mass) crabs$spsex=paste(crabs$sp,crabs$sex,sep="") varnames<-c( FL, RW, CL, CW, BD ) Crabs <- crabs[,varnames] Crabs.class <- factor(crabs$spsex) crabsmds <- cmdscale(d= dist(crabs),k=2) plot(crabsmds, pch=20, cex=2,col=unclass(crabs.class)) We can use only the largest k < min(n, p) eigenvalues and eigenvectors in the reconstruction, giving the best k-dimensional view of the data. This is analogous to PCA, where only the largest eigenvalues of X X are used, and the smallest ones effectively suppressed. Indeed, PCA and classical MDS are duals and yield effectively the same result. MDS MDS 1

7 Crabs Data Varieties of MDS Compare with previous PCA analysis. Classical MDS solution corresponds to the first 2 PCs Comp.3 Comp.4 Comp Generally, MDS is a class of dimensionality reduction techniques which represents data points x 1,..., x n R p in a lower-dimensional space z 1,..., z n R k which tries to preserve inter-point (dis)similarities. It requires only the matrix D of pairwise dissimilarities d ij = d(x i, x j ). For example, we can use Euclidean distance d ij = x i x j 2, but other dissimilarities are possible. MDS finds representations z 1,..., z n R k such that z i z j 2 d(x i, x j ) = d ij, and differences in dissimilarities are measured by the appropriate loss (d ij, z i z j 2 ). Goal: Find Z which minimizes the stress function S(Z) = i j (d ij, z i z j 2 ) Varieties of MDS Choices of (dis)similarities and (stress) functions lead to different algorithms. Classical/Torgerson: preserves inner products instead - strain function (cmdscale) S(Z) = (b ij z i z, z j z ) 2 i j Metric Shephard-Kruskal: preserves distances w.r.t. squared stress S(Z) = i j (d ij z i z j 2 ) 2 Sammon: preserves shorter distances more (sammon) S(Z) = i j (d ij z i z j 2 ) 2 Non-Metric Shephard-Kruskal: ignores actual distance values, only preserves ranks (isomds) i j S(Z) = min (g(d ij) z i z j 2 ) 2 g increasing i j z i z j 2 2 d ij