Principal Components Analysis

Transcription

1 Principal Components Analysis Lecture 9 August 2, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #9-8/2/2011 Slide 1 of 54

2 Today s Lecture Principal Components Analysis (PCA) Today s Lecture What it is How it works How to do such an analysis Examples of uses of PCA Lecture #9-8/2/2011 Slide 2 of 54

3 Principal components analysis is concerned with explaining the variance-covariance structure of a set of variables Variability PCA Features This explanation comes from a few linear combinations of the original variables Generally speaking, PCA has two objectives: Data reduction - moving from many original variables down to a few composite variables (linear combinations of the original variables) - which variables play a larger role in the explanation of total variance Think of the new variables, called the principal components, as composite variables consisting of a mixture of the original variables Com pos ite: A structure or an entity made up of distinct components (dictionary.com) Lecture #9-8/2/2011 Slide 3 of 54

4 Variability When talking about the joint variability of a set of variables, the concept of the total sample variance is important Variability PCA Features Essentially, the total sample variance provides a way to describe the sample covariance matrix, S, with a single number Recall from previous lectures the concept of total sample variance: Another way to characterize the sample variance is with the total variance Total variance equals tr(s) = s 11 + s s pp Like generalized variance, the total sample variance reflects the overall spread of the data Many multivariate techniques refer use total sample variance in computation of variance accounted for Lecture #9-8/2/2011 Slide 4 of 54

5 PCA and Variability In PCA, the goal is to find a set of k principal components (composite variables) that: Variability PCA Features 1. Is much smaller than the original set of p variables and 2. Accounts for nearly all of the total sample variance If these two goals can be accomplished, then the set of k principal components contains almost as much information as the original p variables This means that the k principal components can then replace the p original variables The original data set is thereby reduced, from n measurements on p variables to n measurements on k variables Lecture #9-8/2/2011 Slide 5 of 54

6 PCA Features PCA often reveals relationships between variables that were not previously suspected Variability PCA Features Because of such relationships, new interpretations of the data and variables often stem from PCA PCA usually serves more of a means to an end rather than an end in themselves in that the composite variables (the principal components) are often used in larger investigations or in other statistical techniques such as: Multiple regression Cluster analysis Scaled principal components are one factoring of the covariance matrix for the exploratory factor analytic approaches that will be discussed next class Lecture #9-8/2/2011 Slide 6 of 54

7 Linear Combinations Prinicpal Components Example #1 Principal components, the composite variables created by combinations of the original variables, are formed by a set of linear combinations For p variables, a total of p components can be formed using PCA (although much fewer are usually used) Letting Y stand for the new composite variables, or principal components, the linear combinations look like: Y 1 = a 1X = a 11 X 1 + a 12 X a 1p X p Y 2 = a 2X = a 21 X 1 + a 22 X a 2p X p. Y p = a px = a p1 X 1 + a p2 X a pp X p Lecture #9-8/2/2011 Slide 7 of 54

8 What are the Linear Combinations? Linear Combinations Prinicpal Components Example #1 In PCA, the linear combinations, each formed by weighting each original variable by a pp are formed so that the following conditions are met: The variance of each successive component is smaller than the previous component: Var(Y 1 ) > Var(Y 2 ) >... > Var(Y p ) The covariance (or correlation) between any two different principal components (i and j) is zero: Cov(Y i, Y j ) = 0 The sum of the variances of the principal components is equal to the total sample variance: p Var(Y p ) = Tr(S) = s 11 + s s pp i=1 Lecture #9-8/2/2011 Slide 8 of 54

9 What are the Linear Combinations? Principal components depend solely on the covariance matrix S or the correlation matrix R Linear Combinations Prinicpal Components Example #1 Linear combination weights in PCA aren t typically either ones or zeros (although there is one special case where that is the case) Rather, all the linear combination weights for each principal component come directly from the eigenvectors of S or R Recall that for p variables, the p p covariance/correlation matrix has a set of: p eigenvalues - {λ 1, λ 2,...,λ p } p eigenvectors - {e 1, e 2,...,e p } Lecture #9-8/2/2011 Slide 9 of 54

10 Forming the Principal Components Each principal component is formed by taking the values of the elements of the eigenvalues as the weights of the linear combination Linear Combinations Prinicpal Components Example #1 If each eigenvector has elements e ik : e 1 = e 11 e 21.., e 2 = e 12 e 22..,...,e p = e 1p e 2p.. e p1 e p2 e pp Then the principal components are formed by: Y 1 = e 11 X 1 + e 21 X e p1 X p Y 2 = e 12 X 1 + e 22 X e p2 X p. Y p = e 1p X 1 + e 2p X e pp X p Lecture #9-8/2/2011 Slide 10 of 54

11 SAS Example #1 Imagine that somehow (via the wonders of simulation) I collected data SAT test scores for both the Math (SATM) and Verbal (SATV) sections of 1,000 students Linear Combinations Prinicpal Components Example #1 The descriptive statistics of this data set are given below: Statistic SATV SATM Mean SD Correlation SATV SATM Note that this example is given for purposes of demonstration with a small number of variables In practice, this type of application of PCA with two variables is unrealistic Lecture #9-8/2/2011 Slide 11 of 54

12 SAS Example #1 The scatterplot of the data is shown below: Linear Combinations Prinicpal Components Example #1 SATM SATV Lecture #9-8/2/2011 Slide 12 of 54

13 SAS Code proc princomp cov data=sat out=pcsat; var satv satm; run; For more SAS code, see the proc princomp part of the SAS Manual at: Lecture #9-8/2/2011 Slide 13 of 54

14 13-1

15 SAS New Data Set The new SAS data set contains the two principal components: Lecture #9-8/2/2011 Slide 14 of 54

16 Principal Component Computation The way the principal components are often constructed (and are constructed in SAS) is to use mean-centered variables Linear Combinations Prinicpal Components Example #1 In the context of our example, the two principal components are formed by: Y 1 = 0.475(X 1 X 1 ) (X 2 X 2 ) Y 2 = 0.880(X 1 X 1 ) (X 2 X 2 ) For example, let s take the first individual s principle components, this person had a SATM of 580 and a SATV of 520: Y 1 = 0.475( ) ( ) = Y 2 = 0.880( ) ( ) = Lecture #9-8/2/2011 Slide 15 of 54

17 Principal Component of principal components can be with respect to two possible types of information: PC Variance Correlation 1. The importance of each component Measured by the proportion of total sample variance accounted for by the component 2. The importance of each original variable within each component The weight of the component for each variable (for interpretation of the relative importance of the original variables) The correlation between each of the original variables and the new principal components Lecture #9-8/2/2011 Slide 16 of 54

18 Principal Component Variance The variance of each principal component is equal to the corresponding eigenvalue of that component Var(Y p ) = s 2 Y p = λ p PC Variance Correlation The sum of the eigenvalues is equal to the total sample variance: λ 1 + λ λ p = p λ i = Tr(S) i=1 The corresponding percentage of total sample variance accounted for by each principal component, p, is then: λ p λ 1 + λ λ p Lecture #9-8/2/2011 Slide 17 of 54

19 Principal Component Variance From our SAT Example, the variance of each principal component (and the subsequent relationship to the total sample variance) is given by: PC Variance Correlation Lecture #9-8/2/2011 Slide 18 of 54

20 Component Weight Magnitude To interpret the relative importance of the original variables, the magnitude of the weights are examined PC Variance Correlation The magnitude of e ik measures the importance of the k th variable to the i th principal component, irrespective of the other variables For example, take the first weights of the first principal component from the SAT example: [ ] [ ] SATV e 1 = = SATM This shows that the SAT Math score plays a larger role in the development of the first principal component This also shows that the values of the weights are sensitive to the scale of the measures (SAT Math had a larger variance) Lecture #9-8/2/2011 Slide 19 of 54

21 Correlation of X and Y Occasionally, you will find people judging the relative importance of the original variables by use of their correlation with each of the principal components PC Variance Correlation The correlation between variable X k and principal component Y i can be computed by: ρ Yi,X k = e ik λi s 2 X k Many statisticians do not like this method for evaluation of the importance of a variable because it measures only the univariate contribution of a variable X to a component Y Some recommend only using the coefficients of the principal component to evaluate the relative importance of the original variables Lecture #9-8/2/2011 Slide 20 of 54

22 PCA provides a geometric shift and rescaling of the axes that the original variables were measured The two-variable example of SAT shows how this rotation occurs Original Variables Principal Components Lecture #9-8/2/2011 Slide 21 of 54

23 Original Variables Original Variables Principal Components Lecture #9-8/2/2011 Slide 22 of 54

24 Principal Components Original Variables Principal Components Lecture #9-8/2/2011 Slide 23 of 54

25 Principal Components Original Variables Principal Components Lecture #9-8/2/2011 Slide 24 of 54

26 Example #2 Rather than using the raw data, and hence, the covariance matrix to attain the eigenvalues and eigenvectors used in PCA, another alternative is to use the correlation matrix Using the correlation matrix is equivalent to using the covariance matrix of standardized observations The resulting principal component values (the Y i ), are calculated based on the standardized observations The results of such an analysis can produce different interpretations than a PCA with the original variables It is advised, however, that PCA be used with correlations when variables with widely different scales of measurement are included in the analysis To demonstrate, here is SAS Example #2... Lecture #9-8/2/2011 Slide 25 of 54

27 Example #2 Note the lack of cov in the proc line: proc princomp data=sat out=pcsat_cor; var satv satm; run; Example #2 Lecture #9-8/2/2011 Slide 26 of 54

28 26-1

29 Differing Component Values Also note that the values of the principal components are now based on the standardized variables: Y 1 = X 1 X 1 s X X 2 X 2 s X2 Y 2 = X 1 X 1 s X X 2 X 2 s X2 Lecture #9-8/2/2011 Slide 27 of 54

30 Differing Eigenvalues/Eigenvectors Note the comparison of the eigenvalues and eigenvectors (along with the comparison of the variances): Example #2 Covariance Matrix Variable e 1 e 2 Variance SATV SATM Correlation Variable e 1 e 2 Variance SATV SATM Look at the ratio of eigenvector weights between SATV and SATM and how it corresponds to the ratio of the variance between the two variables. Lecture #9-8/2/2011 Slide 28 of 54

31 PCA Assumptions PCA does not specifically presume any type of data for the analysis PCA Assumptions How Many PC? Uncorrelated Variables Many people prefer to think of using PCA for only continuous variables (although there are numerous examples of this not being the case) If the variables happen to be MVN, then the principal components will also be MVN, with a zero mean vector, and a covariance matrix that has zero off-diagonal elements and diagonal elements equal to the eigenvalues of the principal components Lecture #9-8/2/2011 Slide 29 of 54

32 How Many Principal Components? Although our example had only two variables, more often PCA is used to reduce the number of dimensions PCA Assumptions How Many PC? Uncorrelated Variables Used in that context, the main question of PCA regards how many principal components are to be used to describe the original data set Although no rules exist, most often people try to have a set of components that account for at least 70% to 80% of the total sample variance As a guide to where to decide upon the number of components, a scree plot is sometime used (I will show one in the next example) From dictionary.com: Scree (n): a sloping mass of loose rocks at the base of a cliff Lecture #9-8/2/2011 Slide 30 of 54

33 Uncorrelated Variables in PCA To finalize the theoretical topics of this lecture, consider what would happen to the PCA we had an uncorrelated variable in the analysis PCA Assumptions How Many PC? Uncorrelated Variables For instance, think about how the analysis would be if we included, for each person, the time it took to tie their shoes? The uncorrelated variable essentially cannot be summarized by the new principal components, and therefore becomes its own principal component Code for SAS Example #3: proc princomp data=sat3var out=pcsat3v_cor; var satv satm time; run; Lecture #9-8/2/2011 Slide 31 of 54

34 31-1

35 Gambling Explosion GRI Sample SAS Code SAS Output Accessability of gambling has increased exponentially in the past several years: State lotteries Native American Tribal Casinos Riverboat gambling Internet gambling As accessability has increased, incidences of pathological gambling has increased There is a need to be able to measure pathological gambling and to be able to understand its etiology Lecture #9-8/2/2011 Slide 32 of 54

36 Pathological Gambling The DSM-IV-TR defines pathological gambling as an impulse-control disorder (not elsewhere classified) Persistent and recurrent maladaptive gambling behavior as indicated by five (or more) of the following: A1 Is preoccupied with gambling A2 Needs to gamble with increasing amounts of money in order to achieve the desired excitement A3 Has repeated unsuccessful efforts to control, cut back, or stop gambling A4 Is restless or irritable when attempting to cut down or stop gambling A5 Gambles as a way of escaping from problems or of relieving a dysphoric mood A6 After losing money gambling, often returns another day to get even A7 Lies to family members, therapist, or others to conceal the extend of involvement with gambling A8 Has committed illegal acts such as forgery, fraud, theft, or embezzlement to finance gambling A9 Has jeopardized or lost a significant relationship, job, or educational or career opportunity because of gambling A10 Relies on others to provide money to relieve a desperate financial situation caused by gambling GRI Sample SAS Code SAS Output Behind the 10 criteria are a set of three personality traits (higher order factors): Dependence, Disruption, Loss of Control (Henson et al., 2000) Lecture #9-8/2/2011 Slide 33 of 54

37 Gambling Research Instrument GRI Sample SAS Code SAS Output Henson, Feasel, & Jones (2000) Forty-one Likert scale items Six response levels (Strongly Agree - Strongly Disagree) Twenty-five items used to create sum-scores for each of the 10 DSM criteria Increased variability in responses enabled structure of pathological gambling to be studied GRI Example Items 1. I would like to cut back on my gambling 2. There are few things I would rather do than gamble 3. If I lost a lot of money gambling one day, I would be more likely to want to play again the following day 4. I enjoy talking with my family and friends about my past gambling experiences (R) 5. I find it necessary to gamble with larger amounts of money (than when I first gambled) for gambling to be exciting 6. I have gone to great lengths to obtain money for gambling 7. I feel "high" when I gamble 8. I worry that I am spending too much money gambling 9. I feel restless when I try to cut down or stop gambling 10. It bothers me when I have no money to gamble. 11. I gamble to take my mind off my worries 12. When I lose money gambling, it is a long time before I gamble again (R) 13. I find it difficult to stop gambling 14. I am drawn more by the thrill of gambling than by the money I could win Lecture #9-8/2/2011 Slide 34 of 54

38 Sample Characteristics In a study of the gambling tendencies of college students, an online version of the GRI was created Study included 287 college students recruited from a large midwestern university Average age 19 GRI Sample SAS Code SAS Output Subjects provided responses to the GRI and also responded to an item asking about their overall gambling experience Lecture #9-8/2/2011 Slide 35 of 54

39 SAS Code *SAS ; PROC IMPORT OUT= WORK.gri DATAFILE= "C:\gri.xls" DBMS=EXCEL2000 REPLACE; RANGE="gri$"; GETNAMES=YES; RUN; GRI Sample SAS Code SAS Output proc princomp data=gri out=pcgri_cor outstat=pc_gri; var gri1-gri41; run; Lecture #9-8/2/2011 Slide 36 of 54

40 SAS Output GRI Sample SAS Code SAS Output Lecture #9-8/2/2011 Slide 37 of 54

41 SAS Output To account for 70% of the total sample variance, some 10 principal components must be used: Eigenvalues of the Correlation Matrix Eigenvalue Difference Proportion Cumulative Lecture #9-8/2/2011 Slide 38 of 54

42 SAS Output Highest magnitude component weights: GRI Sample SAS Code SAS Output Component 1 Variable Weight GRI GRI GRI Component 2 Variable Weight GRI GRI GRI GRI Lecture #9-8/2/2011 Slide 39 of 54

43 To demonstrate the use of the principal components in another analysis, consider trying to predict a person s gambling experience rating by the answers to the survey questions In the first part of this example, we will use the original 41 variables in a multiple regression In the second part of this example, we will use the first 10 principle components in a multiple regression Finally, to demonstrate the properties of the principal components (recovery of the R 2 ), all 41 principal components will be used in the regression Lecture #9-8/2/2011 Slide 40 of 54

44 Part 1 Code/Output proc glm data=pcgri_cor; model gambling=gri1-gri41/ss3; run; Lecture #9-8/2/2011 Slide 41 of 54

45 Part 2 Code/Output proc glm data=pcgri_cor; model gambling=prin1-prin10/ss3; run; Lecture #9-8/2/2011 Slide 42 of 54

46 Part 3 Code/Output proc glm data=pcgri_cor; model gambling=prin1-prin41/ss3; run; Lecture #9-8/2/2011 Slide 43 of 54

47 As a final example, consider the following data set (from the SAS user s guide): Police officers were rated by their supervisors in 14 categories as part of standard police departmental administrative procedure. Each variable contains the job ratings, using a scale measurement from 1 to 10 (1=fail to comply, 10=exceptional). Lecture #9-8/2/2011 Slide 44 of 54

48 The variables collected are: Communication Skills Problem Solving Learning Ability Judgment Under Pressure Observational Skills Willingness to Confront Problems Interest in People Interpersonal Sensitivity Desire for Self-Improvement Appearance Dependability Physical Ability Integrity Lecture #9-8/2/2011 Slide 45 of 54

49 Lecture #9-8/2/2011 Slide 46 of 54

50 Lecture #9-8/2/2011 Slide 47 of 54

51 Lecture #9-8/2/2011 Slide 48 of 54

52 Lecture #9-8/2/2011 Slide 49 of 54

53 Lecture #9-8/2/2011 Slide 50 of 54

54 Lecture #9-8/2/2011 Slide 51 of 54

55 Lecture #9-8/2/2011 Slide 52 of 54

56 Lecture #9-8/2/2011 Slide 53 of 54

57 Final Thought Final Thought Principal components analysis is a multivariate statistical technique for reducing the number of dimensions of a data set Often, PCA is an intermediate step between, providing variables that will be used in subsequent statistical analyses The method of gaining meaning of the components through use of their weights is a method that will carry over throughout the rest of this week Many multivariate techniques create new variables that can be interpreted by the nature of the values of the original variables Up next: Exploratory Factor Analysis Lecture #9-8/2/2011 Slide 54 of 54