PRINCIPAL COMPONENTS ANALYSIS (PCA)

Transcription

1 PRINCIPAL COMPONENTS ANALYSIS (PCA) Introduction PCA is considered an exploratory technique that can be used to gain a better understanding of the interrelationships between variables. PCA is performed on a set of data with the hope of simplifying the description of a set of interrelated variables. Variables are treated equally and they are not separated into dependent and independent variables. In simplest terms, PCA transforms the original interrelated variables into a new set of uncorrelated variables call Principal Components. Each principal component is a linear combination of the original variables. The amount of information expressed by each principal component is its variance. Principal components often are displayed in rank order of decreasing variance. The principal component with the highest variance is termed the first principal component. An advantage of principal components to researchers is that the complexity in interpretation that can be caused by having a large number of interrelated variables can be reduced by utilizing only the first few principal components that explain a large proportion of the total variation. PCA can be used to test for normality. If the principal components are not normally distributed, then the original data weren t either. Basic Concepts Suppose we have a random sample of N observations for two variables, X 1 and X 2. o To simplify the description of these two variables, we will subtract the mean of each dataset from each observation; thus, x! = (X! X! ) and x! = (X! X! ) o The values of x 1 and x 2 would each have a mean of 0 and the sample variances S!! and S!! would be unaffected by using the deviations. o Our goal through PCA is to create two new variables C 1 and C 2, called principal components that are uncorrelated.

2 o The new variables are linear functions of x 1 and x 2 that can be written as: C! = a!! x! + a!" x! and C! = a!" x! + a!! x!, and Mean C 1 =Mean C 2 = 0 Variance C 1 = a!!! S!! + a!!" S!! + 2a!! a!" rs! S! Variance C 2 = a!!" S!! + a!!! S!! + 2a!" a!! rs! S! The variances for C 1 and C 2 are referred to the first and second eigenvalues of covariance matrix of X 1 and X 2 o The coefficients are chosen such that i. The Variance C 1 is maximized and greater than all other variances. The Var C 1 Var C 2... Var C P. ii. The N values of C 1 and C 2 are uncorrelated. iii. a!!! + a!!" + a!!" + a!!! = 1 (i.e. the sum of the squares of the coefficients is one). o Hotelling originally derived the mathematical solution for the coefficients. o PCA can be thought of as a rotation of the original x 1 and x 2 axes to new axes of C 1 and C 2. o The three items above that are related to how the coefficients are chosen determine the amount of the rotation of the new C 1 and C 2 axes. o The values for C 1 and C 2 are found by drawing perpendicular lines to the new axes from a given point, x 1, x 2.

3 Figure 1. Diagram showing the original x 1 and x 2 axes and the new C 1 and C 2 axes. Figure 2. Plot showing principal components for two variable.

4 The Number of Components to Retain An important concept of PCA is to reduce the number of variables or reduce dimensionality. An important decision that the researcher must make when using PCA is to determine the number of principal components to use. This decision has no hard-set rules, and the decision may seem subjective at times. Common methods to reduce the number of principal components include: o Determine the minimum amount of variation that you want defined by the principal components. Some individuals use a cutoff of 80%, or may even go lower to 50%. o Another option is to eliminate the principal components that explain insufficient variation. A common cutoff is <5%. o Another method is to eliminate all principal components that explain less than 70/P percent of the variation, where P = the total number variables. o Scree plots from the SAS analysis also can be used. The place where the plot has an elbow can be used as the cutoff. Example of using the scree plots will be discussed in the next section. Examples of SAS Analyses Using Proc Princomp Example 1: Using PCA to reduce the number of variables. This example starts with 20 variables X1 through X20. SAS commands ods graphics on; ods rtf file='pca.rtf'; proc princomp; var x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 x20; run; ods rtf close;

5 PCA of the Depression Data Set Observations 294 Variables 20 Simple Statistics x1 x2 x3 x4 x5 x6 x7 Mean StD Simple Statistixs x8 x9 x10 x11 x12 x13 x14 Mean StD Simple Statistics x15 x16 x17 x18 x19 x20 Mean StD Correlation Matrix x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x x x x x x x x x x x x x x x

6 PCA of the Depression Data Set Correlation Matrix x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x x x x x Correlation Matrix x14 x15 x16 x17 x18 x19 x20 x x x x x x x x x x x x x x x x x x x x

7 PCA of the Depression Data Set Eigenvalues of the Correlation Matrix Eigenvalue Difference Proportion Cumulative

8 PCA of the Depression Data Set Eigenvectors Prin1 Prin2 Prin3 Prin4 Prin5 Prin6 Prin7 Prin8 Prin9 Prin10 Prin11 x x x x x x x x x x x x x x x x x x x x Eigenvectors Prin12 Prin13 Prin14 Prin15 Prin16 Prin17 Prin18 Prin19 Prin20 x x x x x x x x x x x

9 PCA of the Depression Data Set Eigenvectors Prin12 Prin13 Prin14 Prin15 Prin16 Prin17 Prin18 Prin19 Prin20 x x x x x x x x x

10 Initial PCA Analysis of Malt Data to Determine the Number of Principal Components to Retain How many principal components should be retained? 1. Eleven principal components should be retained based on the rule of maintaining the total variation >80%. 2. Five principal components should be retained based on the rule of eliminating all principal components that explain less than 5% of the total variation. 3. Nine principal components should be retained based on the rule of eliminating all principal components that explain <70/P% of the variation (70/20 = 3.5%). 4. Around three principal components should be retained based on the scree plots. So what is the correct answer? o Decision should be based on your knowledge of the subject area. o I would select a number between 5-7. Example 2: Using PCA to determine the interrelationships between variables related to malt quality, particularly malt extract. Malt quality of barley lines is determined using a large number of correlated traits. PCA will be used to: o Reduce dimensionality between the 10 variables that define malt quality. o Determine which of the 10 variables contribute to explaining the most variability in each principal component based on the load.

11 Initial PCA Analysis of Malt Data to Determine the Number of Principal Components to Retain (Abbreviated Output) Eigenvalues of the Correlation Matrix Eigenvalue Difference Proportion Cumulative Eigenvectors Prin1 Prin2 Prin3 Prin4 Prin5 Prin6 Prin7 Prin8 Prin9 Prin10 kwt plump barcolor wrtcolor protein wrtprt kolbach dp alpha bglucan

12 Initial PCA Analysis of Malt Data to Determine the Number of Principal Components to Retain (Abbreviated Output) Based on the different methods of determining how many principal components to retain, I would keep five. The next step is to redo the analysis keeping in only five principal components. The different plots to interpret the results should be requested. SAS Commands ods graphics on; ods rtf file='maltpca.rtf'; proc princomp n=5 plots (ncomp=3)=pattern; id variety; var kwt plump barcolor wrtcolor protein wrtprt kolbach dp alpha bglucan; *title 'PCA of Malt Quality Using All Variables'; title 'PCA of Malt Quality Analyses Using 5 Principal Components'; run; ods rtf close; In the Proc Princomp statement, I use the option n=5 to have the PCA only calculate only the first five principal components. Additionally, the statement plots=pattern will provide the graphical plots of the PCA. The option (ncomp=3) requests for the graphical output comparing only the first three principal components.

13 PCA of Malt Quality Analyses Using 5 Principal Components Observations 20 Variables 10 Simple Statistics kwt plump barcolor wrtcolor protein wrtprt kolbach Mean StD Simple Statistics dp alpha bglucan Mean StD Correlation Matrix kwt plump barcolor wrtcolor protein wrtprt kolbach dp alpha bglucan kwt plump barcolor wrtcolor protein wrtprt kolbach dp alpha bglucan Eigenvalues of the Correlation Matrix Eigenvalue Difference Proportion Cumulative

14 PCA of Malt Quality Analyses Using 5 Principal Components Eigenvectors Prin1 Prin2 Prin3 Prin4 Prin5 kwt plump barcolor wrtcolor protein wrtprt kolbach dp alpha bglucan

15 PCA of Malt Quality Analyses Using 5 Principal Components

16 PCA of Malt Quality Analyses Using 5 Principal Components

17 PCA of Malt Quality Analyses Using 5 Principal Components