Applied Multivariate Analysis

Transcription

1 Department of Mathematics and Statistics, University of Vaasa, Finland Spring 2017

2 Dimension reduction Principal Component Analysis (PCA)

3 The problem in exploratory multivariate data analysis usually is the large number of variables. Consentration of the number of variables to fewer new variables is one form of data reduction. Major tools in this process is principal component analysis (PCA) and exploratory factor analysis (FA). PCA is a technical transformation and FA is model based.

4 The aim in PCA is to replace the original variables, x 1, x 2,..., x p, by few new variables, y 1,..., y k, that are linear combinations of the x-variables, preserve essentially all the information in the x-variables, and are uncorrelated with each other.

5 More formally: The first principal component is y 1 = a 11 x 1 + a 12 x a 1p x p, (1) where the coefficients, a 1j (j = 1,..., p) are defined such that var[y 1 ] = under the restriction (scaling constraint) max var[a 11x a 1p x p ] (2) (a 11,...,a 1p ) a a 2 1p = 1. (3)

6 The second principal component is with a 2j defined such that var[y 2 ] = y 2 = a 21 x 1 + a 22 x a 2p x p (4) max var[a 21x a 2p x p ], (5) (a 21,...,a 2p ) a a 2 2p = 1, (6) and cov[y 1, y 2 ] = 0. (7)

7 Altogether there are p principal components, but not all of them are important. Thus, through the principal components a set of correlated variables are transformed a set of uncorrelated variables.

8 Mathematically the principal components are a solution of the eigenvalues of the covariance matrix of x-variables. The coefficients of the first PC are the elements of the eigenvector corresponding to the largest eigenvalue, the coefficients of the second PC are the elements of the eigenvector of the second largest eigenvalue, and so on. Remark 3.1: The principal component analysis is usually in practice obtained from the correlation matrix rather than the covariance matrix. Correlations are scale free, while covariances are not. Remark 3.2: PC solution from a correlation matrix is different from that of a covariance matrix.

9 Let l i denote the ith iegenvalue of or correlation matrix (or covariance matrix) of the the x-variables, such that l 1 l 2 l p, then p p var[x i ] = l i (8) and i=1 i=1 var[y i ] = l i. (9) Thus, the ith component explains l i 100 p j=1 var[x j] % (10) of the total variance of the x-variables. Remark 3.3: In the case of correlation matrix, the variables are standarized with unit variance, i.e., var[x j ] = 1 and p j=1 var[x j] = p. Thus the explanatory power of the ith component extracted from the correlation matix is lapplied i Multivariate Analysis

10 Assuming that the components are extracted form the correlatin matrix, correlation of the original variable x i with the component y j are given by corr[x i, y j ] = a ji lj, (12) and are called loadings. Thus, the loadings (correlations) are just scaled the eigenvector coefficients, but may be easier to interpret, because correlations are between 1 and 1. If varibales with high correlation have something common that can be used as the basis for the naming. Remark 3.4: If the components are extracted from the covariance matrix the loadings are where s i is the standard deviation of x i. corr[x i, y j ] = a ji lj s i, (13)

11 Example 1 Crime rates in the USA in 2005 per 100,000 people by states. Source: Violent crimes: murder and nonnegligent manslaughter, forcible rape, robbery, and aggarvated assault. Property crimes: burglary, larceny-theft, and motor vehicle theft. Using SAS PROC PRINCOMP, the results are: proc princomp data = uscrime2005 out = uscrime_components; title US crime rates per 100,000 population by state ; var murder rape robbery assault burglary larceny auto; run;

12 US crime rates per 100,000 population by state, year 2005 Simple Statistics murder rape robbery assault burglary larceny auto Mean StD Correlation Matrix murder rape robbery assault burglary larceny auto murder rape robbery assault burglary larceny auto Eigenvalues of the Correlation Matrix Eigenvalue Difference Proportion Cumulative

13 Eigenvectors Prin1 Prin2 Prin3 Prin4 Prin5 Prin6 Prin7 murder rape robbery assault burglary larceny auto The eigenvalues indicate that two (or three) components provide a good summary of the data. Of the total variance 76% is accounted by the first two components and 85% by the first three components.

14 The loadings matrix for the first three components: Principal component loadings Prin1 Prin2 Prin3 murder rape robbery assault burglary larceny auto All loadings for the first component are about the same and fairly high except for rape. Thus, the first component describes general criminality. The second component loads (positive) high on rape, larceny, and burglary and negative high on murder and assault. Thus this component seems to measure the preponderance of property and sexual crime over violent crimes (other than sexual) and vice versa (sign of an eigenvector can be changed). These kinds of components are called bipolar. Here it means that high

15 The third component is not that clear but high values of the component indicate those states where rape and assault crimes are high while property crimes tend to be below average. On the other hand, again high negative value indicate high level of property crime.

16 Number of components A rule of thumb to decide the number of meaningful components is select those for which the eigenvalue is equal or greater then 1 (e.g. SPSS uses this as an automatic rule). Another criterion is the so called Cattell s scree test. The rule is to retain all the eigenvalues (hence, the number of components) in the sharp descent (before the elbow point ) in the plot of eigenvalues against the their ordinal number. Usually there is a discernible drop (break point) before the eigenvalues start to level in the plot.

17 Cattell s Scree Plot for the Crime 2005 Data The eigenvalue criterion supports two components and the scree test two or three. We have selected three.

18 Significant coefficients The loadings (scaled component coefficients) are correlations. It can be shown that if the population correlation is zero, the sample correlation is asymptotically normally distributed with zero mean and variance 1/(n 1), where n is the sample size. Using this we can use the rule that those coefficients are statistically significant that are plus/minus two standard errors away from zero. stderr = 1/ n 1 (14)

19 In the crime data n = 52, thus those coefficients are statistically significant whose loadings are on absolute value larger than 2 n 1 = (15) Thus, for the first component all the coefficients are statistically significant, for the second all but assault and auto, and for the third rape and larceny, while assault and burglary are on the borderline.

20 Recap The main usage of principal components are for indexes and for new variables in subsequent studies. PCA is not a statistical model. It is merely a linear transformation of original variables to new variables for the purpose of reducing the dimensionality of the problem (concentrate information).