Computer exercise 3: PCA, CCA and factors. Principal component analysis. Eigenvalues and eigenvectors

Size: px

Start display at page:

Download "Computer exercise 3: PCA, CCA and factors. Principal component analysis. Eigenvalues and eigenvectors"

Reginald Young
5 years ago
Views:

1 UPPSALA UNIVERSITY Department of Mathematics Måns Thulin Multivariate Methods Spring 2011 Computer exercise 3: PCA, CCA and factors In this computer exercise the following topics are studied: Principal component analysis Factor analysis Canonical correlation analysis Principal component analysis Principal components are linear combinations of random variables, given by the eigenvalues and eigenvectors of the covariance or correlation matrix of the random variables. Eigenvalues and eigenvectors Sometimes a covariance or correlation matrix is given directly in the problem at hand. Consider for instance the covariance matrix in Example 8.1 on page 434 of J&W: Σ = To find the principal components, we must find the eigenvalues and eigenvectors of Σ. These are found by the command eigen; see the following script, where at the last line the first principal component is extracted: Sig <- cbind(c(1,-2,0),c(-2,5,0),c(0,0,2)); eigen(sig) pc1 <- eigen(sig)$vector[,1] Now let us verify that the vectors and scalars given by eigen really are the eigenvectors and corresponding eigenvalues of Σ. Make sure that you understand what the following commands do and why they yield the same result. Sig %*% eigen(sig)$vector[,1] eigen(sig)$values[1] * eigen(sig)$vector[,1] The vectors with the coefficients for the principal components should have lengh 1 and be orthogonal to eachother. We can verify that the eigenvectors of Σ satisfy this by R: t(eigen(sig)$vector[,1]) %*% eigen(sig)$vector[,1] t(eigen(sig)$vector[,1]) %*% eigen(sig)$vector[,2] 1

2 Principal components given a data structure If an entire multivariate dataset is given, the key command for PCA in R is princomp. We will study a dataset of air pollution in US cities, given in the file usair.r. Seven variables were recorded for 41 cities: SO2: Sulphur dioxide content of air in micrograms per cubic meter Temp: Average annual temperature in F Manuf: Number of manufacturing enterprises employing 20 or more workers Pop: Population size (1970 census) in thousands Wind: Average annual wind speed in miles per hour Precip: Average annual precipitation in inches Days: Average number of days with precipitation per year The example is taken from a book by Everitt. 1 We will for a moment disregard the SO2 variable and study the remaining six: two of these relate to human ecology (Pop, Manuf) and four to climate (Temp, Wind, Precip, Days). Everitt suggests looking at 1 times the temperatures, since then all six variables are such that high values represent a less attractive enviroment. We start by studying all combinations of scatter plots and computing correlations: source("usair.r") usair pairs(usair[,-1]) cor(usair[,-1]) Are there any outliers? For simplicity, we continue the analysis of the complete data set, but at this stage in a data analysis it is good practice to keep an eye out for possible outliers that may affect the analysis. Since the variables are measured on different scales, it seems reasonable to use the correlation matrix for the analysis, since the variables with big variance will dominate the principal components completely otherwise. The principal components are given by the eigenvectors of the correlation matrix: eigen(cor(usair[,-1])) Alternatively, the command princomp can be used. This gives us a nice summary of the principal components as well: usairpc<-princomp(usair[,-1],cor=true) summary(usairpc,loadings=true) How are the standard deviations listed for the principal components related to the eigenvalues of the correlation matrix? Everitt tries to interpret the first three principal components, and labels them quality of life, wet weather and climate type. Do you agree with his interpretation? To see how much of the variation in the data that is described by the principal components, we look at a screeplot: screeplot(usairpc,type="l") We can now study the data set by looking at a scatter plot of the first two principal components: 1 Everitt, B.S. (2005), An R and S-PLUS Companion to Multivariate Analysis, Springer 2

3 plot(usairpc$scores[,1],usairpc$scores[,2], ylim=range(usairpc$scores[,1]),xlab = "PC1",ylab="PC2",lwd=2) It can also be of interest to plot the first principal component versus the third. In the above plot, we can use the command identify to see which point belongs to which city. An alternative is to plot the (abbreviated) city names directly in the plot: par(pty="s") plot(usairpc$scores[,1],usairpc$scores[,2], ylim=range(usairpc$scores[,1]),xlab = "PC1",ylab="PC2",type="n") text(usairpc$scores[,1],usairpc$scores[,2], labels = abbreviate(row.names(usair)),cex=0.7,lwd=2) One purpose for collecting the air pollution data was to see how the SO2 variable is related to the other variables. We can study its relation the the principal components in different ways; for instance by a bubble plot: plot(usairpc$scores[,1],usairpc$scores[,2], ylim=range(usairpc$scores[,1]),xlab = "PC1",ylab="PC2",type="n") symbols(usairpc$scores[,1],usairpc$scores[,2],circles=sqrt(usair$so2), inches=0.2,add=true,bg="pink",fg="black") Or by plots versus the first three principal components: par(mfrow=c(1,3)) plot(usairpc$scores[,1],usair$so2,xlab="pc1") plot(usairpc$scores[,2],usair$so2,xlab="pc2") plot(usairpc$scores[,3],usair$so2,xlab="pc3") What are your conclusions? Is SO2 related to any of the principal components? Principal components and cluster analysis In cluster analysis, studied in the next block of the course, the goal is to find clusters sets of points that somehow belong together in the data set. Some methods for finding clusters require that the user states the number of clusters beforehand. If the data is high-dimensional, it may be hard to find the number of clusters by visual means. In this situation, principal components sometimes come in handy. Let s look at the four-dimensional data in data2.dat. data2<-read.table("data2.dat") pairs(data2) The pairwise scatter plots seem to indicate that there are indeed clusters within the data, but it is hard to say how many there are. Perhaps we can find out by looking at the principal components. data2p<-princomp(data2) screeplot(data2p,type="l") plot(data2p$scores[,1],data2p$scores[,2]) 3

4 Judging from the plot, how many clusters would you say there are in the data set? The screeplot tells us that the third and fourth principal components contain little information. However, in this particular case, including the third component in the analysis might change the conclusion regarding the number of clusters. Let s examine the clusters in three dimensions pay extra attention to the rightmost cluster in the two-dimensional scatter plot above. library(rgl) plot3d(data2p$scores[,1],data2p$scores[,3],data2p$scores[,2],col=2) Factor analysis The file life.dat contains data regarding life expectancy in years by country, age, and sex. The data comes from Keyfitz and Flieger (1971) and relate to life expectancies in the 1960s. life<-read.table("life.dat",header=true) life cor(life[,2:9]) m0 is the life expectancy of a 0 year old man, m25 is the life expectancy of a 25 year old man, and so on. It is assumed that the variation in the data can be described by a few underlying factors. A good start could be to think about what such factors could be. A factor analysis can be performed either by the principal component method or the maximum likelihood metod. Let s start with the principal component method. As before, we can find the principal components using princomp: lifepc<-princomp(life[,2:9]) summary(lifepc,loadings=true) Judging from the cumulative proportion of variance for the principal components, how many factors would you use in the factor model? The factor loadings are now obtained by scaling the principal components with their standard deviations (see page 490 of Johnson & Wichern). Here we choose m = 3 factors. L1pc<-lifepc$sdev[1]*lifepc$loadings[,1] L2pc<-lifepc$sdev[2]*lifepc$loadings[,2] L3pc<-lifepc$sdev[3]*lifepc$loadings[,3] Try to interpret the factor loadings. Sometimes the interpretation can be aided by rotating the factor using an orthogonal matrix (recall that the factor model with the rotated loadings is just as valid as the first model obtained). Many analysts like to use the varimax criterion for choosing which orthogonal matrix to use. In R, we can use the function varimax to perform the rotation: Lpc<-cbind(L1pc,L2pc,L3pc) varimax(lpc) 4

5 Interpret the rotated factor loadings. Does your interpretation differ from that from the unrotated loadings? The maximum likelihood approach to estimation of the the factor loadings is described on p. 495 in Johnson & Wichern. We do not go into details about the procedure, but we can try to use it as a black box method, just to see if it yields the same solution as the principal component method. The function factanal gives the ML estimates of the factor loadings. It also performs the test for the number of common factors detailed on p of Johnson & Wichern. The test is motivated by asymptotic results, so it might not be appropriate for our data, where we only have n = 31 observations. Nevertheless, we use it here to see what the results are. lifefa1<-factanal(life[,2:9],factors=1,method="mle") lifefa1 lifefa2<-factanal(life[,2:9],factors=2,method="mle") lifefa2 lifefa3<-factanal(life[,2:9],factors=3,method="mle") lifefa3 The loadings given by factanal are rotated using the varimax criterion by default. Interpret the loadings for m = 3 factors. Is your interpretation the same as those from the principal component analyses? Canonical correlation analysis Unlike principal component analysis and factor analysis, that deal with relationships within sets of variables, canonical correlation analysis deals with relationships between sets of variables. In this exercise, we will study the salespeople data from problem 9.19 in Johnson & Wichern. The data is given on the table on page 536 and is found in the file T9-12.DAT. We will study the canonical correlations between the first three variables (sales growth, sales profitability and new account sales) and the last four variables (creativity test, mechanical reasoning test, abstract reasoning test and mathematics test). T912<-read.table("T9-12.DAT") X1<-T912[,1:3] X2<-T912[,4:7] The coefficients for the canonical variates can be found by solving the eigenvector equations (10-11) on page 545 of Johnson & Wichern. First, let us partition the covariance matrix: covmatrix<-cov(t912) S11<-covmatrix[1:3,1:3] S22<-covmatrix[4:7,4:7] S12<-covmatrix[1:3,4:7] S21<-covmatrix[4:7,1:3] We can now solve the eigenvector equations: 5

6 bigmatrix1<-solve(s11)%*%s12%*%solve(s22)%*%s21 canon1<-eigen(bigmatrix1) bigmatrix2<-solve(s22)%*%s21%*%solve(s11)%*%s12 canon2<-eigen(bigmatrix2) Convince yourself that the non-standardized canonical variates and the (absolute values of the) canonical correlations now are given by canon1 canon2 sqrt(canon1$values) The variates obtained using eigen are standardized in the sense that the coefficient vectors have length 1, but that is not what we are looking for here. The canonical variates should be choosen so that V ar(u) = V ar(v ) = 1. Let s calculate the variance for U 1 = a 1 X(1) : varu1<-t(canon1$vectors[,1]) %*% S11 %*% canon1$vectors[,1] We can now use this to standardize a 1 so that V ar(u 1 ) = 1: aa<-1/sqrt(varu1) a<-aa*canon1$vectors[,1] t(a) %*% S11 %*% a # Variance after standardization The other coefficient vectors can be standardized in the same way. An alternative to the approach above is to use the cancor function to find the canonical correlations and variates. For the above data set, these quantities are found by writing can912<-cancor(x1,x2) can912 The coefficient vectors given by cancor are not standardized in the right way. Once again, we can standardize them to give linear combinations with variance 1: varu1c<-t(can912$xcoef[,1]) %*% S11 %*% can912$xcoef[,1] aa<-1/sqrt(varu1c) ac<-aa*can912$xcoef[,1] t(ac) %*% S11 %*% ac Finally, we can verify that the standardized vector given by the two methods is the same: a ac 6

FACTOR ANALYSIS AND MULTIDIMENSIONAL SCALING

FACTOR ANALYSIS AND MULTIDIMENSIONAL SCALING Vishwanath Mantha Department for Electrical and Computer Engineering Mississippi State University, Mississippi State, MS 39762 mantha@isip.msstate.edu ABSTRACT