The European Commission s science and knowledge service. Joint Research Centre

Transcription

1 The European Commission s science and knowledge service Joint Research Centre

2 Step 7: Statistical coherence (II) Principal Component Analysis and Reliability Analysis Hedvig Norlén COIN th JRC Annual Training on Composite Indicators & Scoreboards 05-09/11/2018 Ispra (IT)

3 Step 7: Statistical coherence (II) introduction New property will be given by a linear combination w 1 x + w 2 y y PCA will find this best line: 1) Maximum variance 2) Minimum error First Principal Component! x 3 JRC-COIN Step 7: Statistical Coherence (II)

4 Step 7: Statistical coherence (II) introduction Input Output Black Box PCA = Principal Component Analysis RA= Reliability Analysis 4 JRC-COIN Step 7: Statistical Coherence (II)

5 Outline talk Statistical coherence (II) How multivariate methods can help to understand the statistical coherence of our composite indicator Part 1 Principal Component Analysis Part 2 Reliability Analysis (Cronbach s alfa) Is the structure of our composite indicator statistically well-defined? Is the set of available indicators sufficient to describe the pillars and the sub-pillars of the composite indicator? 5 JRC-COIN Step 7: Statistical Coherence (II)

6 Ex 1. European Skills Index The European Skills Index (ESI) measures the performance of a country s skills system taking into account its manifold facets from continually developing the skills of the population to activating and effectively matching these skills to the needs of employers in the labour market. ESI launched in September JRC-COIN Step 7: Statistical Coherence (II)

7 European Skills Index Framework of the European Skills Index 2018 A Composite Indicator measures multifaceted phenomenon - combination of different aspects (Sub-pillars/Pillars). Each aspect can be measured by a set of observable variables (Indicators). Composite Indicator Pillars Sub-pillars Indicators Type equation here. 7 JRC-COIN Step 7: Statistical Coherence (II)

8 Statistical coherence (II) - unidimensionality PCA is used to verify the internal consistency, verify unidimensionality within: 1) each Sub-pillar (across indicators) Framework of the European Skills Index 2018 Type equation here. 8 JRC-COIN Step 7: Statistical Coherence (II)

9 Statistical coherence (II) - unidimensionality PCA is used to verify the internal consistency, verify unidimensionality within: 1) each Sub-pillar (across Indicators) 2) each Pillar (across Sub-pillars) Framework of the European Skills Index 2018 Type equation here. 9 JRC-COIN Step 7: Statistical Coherence (II)

10 Statistical coherence (II) - unidimensionality PCA is used to verify the internal consistency, verify unidimensionality within: 1) each Sub-pillar (across Indicators) 2) each Pillar (across Sub-pillars) 3) the Composite Indicator (across Pillars) Framework of the European Skills Index 2018 Type equation here. 10 JRC-COIN Step 7: Statistical Coherence (II)

11 PCA PCA Identify a small number of averages (PCs) that explain most of the variance observed. Indicator 1 X 1 Indicator 2 X 2... Indicator p X p w 2 w 1 w p Component PCA summarizes information of all indicators and reduces it into a fewer number of components Each principal component PC i is a new variable computed as a linear combination of the original (standardized) variables Observed indicators are reduced into components PC 1 = w 1 x 1 + w 2 x 2 + w p x p 11 JRC-COIN Step 7: Statistical Coherence (II)

12 European Skills Index 2 dimensions y PC2 x PC1 x RO BG IT BE SE FI PC1 RO BG IT BE FI SE Variance 81% y IT BGRO BE FI SE PC2 ITFI BG ROBESE Variance 19% 12 JRC-COIN Step 7: Statistical Coherence (II)

13 European Skills Index third piller dimension added PCA example to be completed in a little while! x RO FI Variance 58% y IT SE Variance 29% z EL CZ Variance 12% 13 JRC-COIN Step 7: Statistical Coherence (II)

14 First steps in PCA Check the correlation structure of the data and perform 2 pre-tests 1) Bartlett s sphericity test The test checks if the observed correlation matrix R diverges significantly from the identity matrix. H 0 : R = 1, H 1 : R 1 Want to reject H 0 to be able to do perform a valid PCA (Bartlett (1937)) 2) Kaiser-Meyer-Olkin (KMO) Measure for Sampling Adequacy Measure of the strength of relationship among variables based on correlations and partial correlations. KMO between [0;1]. Want KMO close to 1 to be able to perform a valid PCA. KMO>0.6 OK! (Kaiser (1970), Kaiser-Meyer (1974)) 14 JRC-COIN Step 7: Statistical Coherence (II)

15 First steps in PCA 2) Kaiser-Meyer-Olkin (KMO) Measure for Sampling Adequacy Kaiser s own interpretations of the KMO values KMO values: in the.90s, marvelous in the.80s, meritorious in the.70s, middling in the.60s, mediocre in the.50s, miserable below.50, unacceptable 1) Bartlett s sphericity test and 2) KMO test provide info whether it is possible to do PCA, but do not give info of the magic number - how many components are needed KMO>0.6 OK 15 JRC-COIN Step 7: Statistical Coherence (II)

16 How to get the PCs 1) Eigenvalue decomposition of a data correlation matrix (case of composite indicator), 2) Singular Value Decomposition (SVD) of a data matrix after mean centering (normalizing) the data matrix Assumptions: I) Linearity II)Normal distribution III) Principal components are orthogonal 16 JRC-COIN Step 7: Statistical Coherence (II)

17 Finding the magic number - determining how many components in PCA Several methods exist. The 3 most common are: 1) Kaiser Guttman Eigenvalues greater than one criterion (Guttman (1954), Kaiser (1960)). Select all components with eigenvalues over 1 (or 0.9) 2) Cattell s scree test (Cattell (1966)) Above the elbow approach Scree 3) Certain percentage of explained variance e.g., >2/3, 75%, 80%, 17 JRC-COIN Step 7: Statistical Coherence (II)

18 European Skills Index finalizing the PCA example Use PCA to explore the dimensionality of ESI across the three pillars. Framework of the European Skills Index 2018 The 3 pillars should be correlated. Supports the assumption that they are all measuring the a similar concept! Type equation here. 18 JRC-COIN Step 7: Statistical Coherence (II)

19 European Skills Index - PCA example Check the correlation structure Pearson correlation coefficients (Correlations not significant at significance level α = 0.01 are grey, n=28, critical value = 0.48) 19 JRC-COIN Step 7: Statistical Coherence (II)

20 European Skills Index - PCA example Check the correlation structure 1) Bartlett s sphericity test p-value (1.5e-17) < 0.01 Reject H 0 2) KMO Measure for Sampling Adequacy Overall MSA = 0.54 KMO>0.6! Pearson correlation coefficients 20 JRC-COIN Step 7: Statistical Coherence (II)

21 European Skills Index - PCA results 1.75= Total variance explained 58.41=1.75/3*100 Pearson correlation coefficients between pillar and principal component Component loadings Component Eigenvalue Variance Cumulative variance PC1 PC2 PC Skills Development Skills Activation ,00 3. Skills Matching Sum Sum of Squares Stopping criterion Eigenvalue > 1 PC 0. 51X 0.88X 0.84X 1 SD SA SM One dimension verified! 21 JRC-COIN Step 7: Statistical Coherence (II)

22 European Skills Index - Audit results ESI theoretical framework 22 JRC-COIN Step 7: Statistical Coherence (II)

23 Ex 2. Social Progress Index The Social Progress Index (SPI) is an international monitoring framework for measuring social progress without resorting to the use of economic indicators. SPI measures country performance on aspects of social and environmental performance, which are relevant for countries at all levels of economic development. 146 fully and 90 partially ranked countries. 23 JRC-COIN Step 7: Statistical Coherence (II)

24 Social Progress Index - PCA at Index level Social progress is measured taking into account the following three broad aspects: 1. Meeting everyone s basic needs for food, clean water, shelter and security. 2. Living long, healthy lives with basic knowledge and communication and a clean environment. 3. Practicing equal rights and freedoms and pursuing higher education. 5 th version of SPI was launched September PCA to explore the dimensionality of SPI across the three dimensions. 24 JRC-COIN Step 7: Statistical Coherence (II)

25 Social Progress Index - PCA example Check the correlation structure Correlation matrix Basic Human Needs Foundations of Wellbeing Opportunity Basic Human Needs 1 Foundations of Wellbeing Opportunity Pearson correlation coefficients (Significance level α = 0.01, n=146, critical value = 0.21) 25 JRC-COIN Step 7: Statistical Coherence (II)

26 Social Progress Index - PCA example Check the correlation structure Correlation matrix Basic Human Needs Basic Human Needs 1 Foundations of Wellbeing Foundations of Wellbeing Opportunity Opportunity ) Bartlett s sphericity test p-value (2.0e-116) < 0.01 Reject H 0 2) KMO Measure for Sampling Adequacy Overall MSA = 0.69 KMO>0.6 Pearson correlation coefficients 26 JRC-COIN Step 7: Statistical Coherence (II)

27 Social Progress Index - PCA results Total variance explained Pearson correlation coefficients between pillar and principal component Component loadings Component Eigenvalue Variance Cumulative variance PC1 PC2 PC Basic Human Needs 2. Foundations of Wellbeing ,00 3. Opportunity Sum Sum of Squares Stopping criterion Eigenvalue > 1 PC 0. 93X 0.96X 0.98X 1 BHN FoW Opp One dimension verified! 27 JRC-COIN Step 7: Statistical Coherence (II)

28 Social Progress Index - PCA results PCA results often summarized in a Factor map PC 0.96X 0.98X 0. 93X 1 BHN FoW PC 0.25X 0.09X 0. 35X PC 2 2 BHN FoW Second principal component is useful to evaluate the differences between the first two and third dimensions. Opp Opp 28 JRC-COIN Step 7: Statistical Coherence (II)

29 Social Progress Index - Audit results 29 JRC-COIN Step 7: Statistical Coherence (II)

30 Social Progress Index - Audit results Redundancies in SPI framework - very strong correlations between the SPI aggregates (dimensions and components). PCA was also performed on the whole set of 51 indicators Some components measure similar concepts: C1:Water and Sanitation C2:Shelter C5: Access to Basic Knowledge C12: Access to Advanced Education Index 3 Dim 12 Comp JRC-COIN Step 7: Statistical Coherence (II)

31 Social Progress Index - Audit results Very strong correlations between the SPI aggregates (dimensions and components). PCA was also performed on the whole set of 51 indicators. 7 latent dimensions (principal components) are retrieved, which capture 78% of the total variance in the underlying indicators. 31 JRC-COIN Step 7: Statistical Coherence (II) Total variance explained Component Eigenvalue Cumulative variance 1 27,55 54,02 2 5,30 64,42 3 1,98 68,31 4 1,63 71,51 5 1,24 73,94 6 1,22 76,34 7 1,07 78,44 8 0,97 80,33 9 0,84 81, ,76 83, ,67 84, ,67 86, ,01 100,00 Sum 51 Stopping criterion Eigenvalue > 1 Difference less than 8%

32 Social Progress Index - Audit results From JRC Audit report: Less number of components is also in line with the results from the beyond GDP approach by the Commission on the Measurement of Economic Performance and Social Progress*. The SPI conceptual framework has been influenced by this work. So apart from the statistical reasoning, the result from the beyond GDP approach, may also give conceptual justification for reducing the number of components in the SPI framework. * Stiglitz, J., Sen,A., Fitoussi,J.-P., Report by the Commission on the Measurement of Economic Performance and Social Progress. In: Commission on the Measurement of Economic Performance and Social Progress, Paris, France. 32 JRC-COIN Step 7: Statistical Coherence (II)

33 Historical notes PCA PCA is probably the oldest of the multivariate analysis methods. Pearson (1901) said that his methods can be easily applied to numerical problems and although he says that the calculations become cumbersome for four or more variables he suggests that they are still quite feasible. Hotelling (1933) chose his components so as to maximize their successive contributions to the total of the variances of the original variables, principal components. Karl Pearson ( ) Harold Hotelling ( ) 33 JRC-COIN Step 7: Statistical Coherence (II)

34 Part 2: Reliability Analysis Cronbach s α is measure of internal consistency and reliability Based upon classical test theory. Most common estimate of reliability, Cronbach (1951). Cronbach s α is defined as Ratio of the Total Covariance in the test to the Total Variance in the test. By test we mean the set of indicators constituting a Sub-pillar/Pillar/Composite Indicator. 34 JRC-COIN Step 7: Statistical Coherence (II)

35 Cronbach s α Cronbach s α is measure of internal consistency and reliability α ranges from 0 to 1. If all indicators are entirely independent from one another (i.e. are not correlated or share no covariance) α = 0. If all of the items have high covariances, then α will approach 1. Caution! α increases when the number of indicators increases. 35 JRC-COIN Step 7: Statistical Coherence (II)

36 Cronbach s α cut off values Cronbach s α is measure of internal consistency and reliability Lee Cronbach ( ) Cronbach's alpha Internal consistency 0.9 α Excellent 0.8 α < 0.9 Good 0.7 α < 0.8 Acceptable 0.6 α < 0.7 Questionable 0.5 α < 0.6 Poor α < 0.5 Unacceptable 36 JRC-COIN Step 7: Statistical Coherence (II)

37 Cronbach s α - European Skills Index Cronbach s α is measure of internal consistency and reliability European Skills Index Cronbach's Alpha Index level (3 pillars) Item level Corrected Item-Total Correlation 0.59 Alpha if Item is dropped Pillar 1: Skills Development Pillar 2: Skills Activation Pillar 3: Skills Matching Cronbach's alpha Internal consistency 0.9 α Excellent 0.8 α < 0.9 Good 0.7 α < 0.8 Acceptable 0.6 α < 0.7 Questionable 0.5 α < 0.6 Poor α < 0.5 Unacceptable 37 JRC-COIN Step 7: Statistical Coherence (II)

38 Cronbach s α - Social Progress Index Cronbach s α is measure of internal consistency and reliability Social Progress Index Index level (3 dimensions) Item level Cronbach's Alpha Corrected Item-Total Correlation 0.96 Alpha if Item is dropped 1: Basic Human Needs : Foundations of Wellbeing : Opportunity Cronbach's alpha Internal consistency 0.9 α Excellent 0.8 α < 0.9 Good 0.7 α < 0.8 Acceptable 0.6 α < 0.7 Questionable 0.5 α < 0.6 Poor α < 0.5 Unacceptable If α is too high it may suggest that some items are redundant as that are measuring very similar concepts. Maximum α 0.9 has been recommended. 38 JRC-COIN Step 7: Statistical Coherence (II)

39 Concluding words PCA and RA Input Output Open transparent Black Box box 39 JRC-COIN Step 7: Statistical Coherence (II)

40 References Books Johnson, Applied Multivariate Statistical Analysis (6th Revised edition), Pearson Education Limited (2013) Jolliffe, Principal Component Analysis, Second Edition, Springer (2002) Leandre, Fabrigar, Duane, Exploratory Factor Analysis, Oxford (2011) Rencher, Methods of multivariate analysis, Wiley (1995) Tinsley and Brown, Handbook of Applied Multivariate Statistics and Mathematical Modeling, Academic Press (2000) Tucker, MacCallum, Exploratory Factor Analysis, (1997) 40 JRC-COIN Step 7: Statistical Coherence (II)

41 THANK YOU Any questions? Welcome to us at: You may contact us & The European Commission s Competence Centre on Composite Indicators and Scoreboards COIN in the EU Science Hub COIN tools are available at:

42 Data matrix of indicators x x... x x x... x X.. x x... x p p n1 n2 np n = number of objects (countries, ) p = number of variables (indicators) The previous steps in building our CI - Step 3 Outlier detection and missing data estimation - Step 4 Normalisation/Standardisation X correlated data set (without outliers and (without or a few) missing values) PCA is based on correlations 42 JRC-COIN Step 7: Statistical Coherence (II)

43 First steps in PCA Check the correlation structure of the data and do 1) and 2) 1) Bartlett s sphericity test The Bartlett s test checks if the observed correlation matrix R diverges significantly from the identity matrix. H 0 : R = 1, H 1 : R 1 Test statistic 2 n 1 2 p 5 log 6 Under H 0, it follows a χ² distribution with a p(p-1)/2 degrees of freedom. R n = sample size p = number of indicators Want to reject H 0 to be able to do PCA,FA! (Bartlett (1937)) 43 JRC-COIN Step 7: Statistical Coherence (II)

44 First steps in PCA Check the correlation structure of the data and do 1) and 2) 2) Kaiser-Meyer-Olkin (KMO) Measure for Sampling Adequacy Indicators are more or less correlated, but the correlation between two indicators can be influenced by the others. Use partial correlation in order to measure the relation between two indicators by removing the effect of the remaining indicators (controlling for the confounding variable) KMO 2 rij 2 r ij 2 aij r ij Pearson correlation coefficient a ij partial correlation coefficient Want KMO close to 1 to be able to do PCA! KMO>0.6 ok (Kaiser (1970), Kaiser-Meyer (1974)) 44 JRC-COIN Step 7: Statistical Coherence (II)

45 PCA and sample size Sufficient number of cases (countries, regions, cities ) necessary to perform PCA. NO scientific answer opinions differ! Two different approaches : (A) Minimum total sample size (B) Examining the ratio of subjects to variables Rule of 10: These should be at least 10 cases for each variable 3:1 ratio: The cases-to variables ratio should be no lower than 3 5:1 ratio: The cases-to variables ratio should be no lower than 5 45 JRC-COIN Step 7: Statistical Coherence (II)

46 PCA and sample size (cont.1) (A) Minimum total sample size Rule of 100: Gorsuch (1983) and Kline (1979, p. 40) recommended at least 100 (MacCallum, Widaman, Zhang & Hong, 1999). No sample should be less than 100 even though the number of variables is less than 20 (Gorsuch, 1974, p. 333; in Arrindell & van der Ende, 1985, p. 166); Hatcher (1994) recommended that the number of subjects should be the larger of 5 times the number of variables, or 100. Even more subjects are needed when communalities are low and/or few variables load on each factor (in David Garson, 2008). Rule of 150: Hutcheson and Sofroniou (1999) recommends at least cases, more toward the 150 end when there are a few highly correlated variables, as would be the case when collapsing highly multicollinear variables (in David Garson, 2008). Rule of 200: Guilford (1954, p. 533) suggested that N should be at least 200 cases (in MacCallum, Widaman, Zhang & Hong, 1999, p84; in Arrindell & van der Ende, 1985; p. 166). Rule of 250: Cattell (1978) claimed the minimum desirable N to be 250 (in MacCallum, Widaman, Zhang & Hong, 1999, p84). Rule of 300: There should be at least 300 cases (Noruis, 2005: 400, in David Garson, 2008). Significance rule. Lawley and Maxwell (1971) suggested 51 more cases than the number of variables, to support chi-square testing (in David Garson, 2008). Rule of 500. Comrey and Lee (1992) thought that 100 = poor, 200 = fair, 300 = good, 500 = very good, 1,000 or more = excellent They urged researchers to obtain samples of 500 or more observations whenever possible (in MacCallum, Widaman, Zhang & Hong, 1999, p84). 46 JRC-COIN Step 7: Statistical Coherence (II)

47 PCA and sample size (cont.2) (B) Examining the ratio of subjects to variables A ratio of 20:1: Hair, Anderson, Tatham, and Black (1995, in Hogarty, Hines, Kromrey, Ferron, & Mumford, 2005) Rule of 10: There should be at least 10 cases for each item in the instrument being used. (David Garson, 2008; Everitt, 1975; Everitt, 1975, Nunnally, 1978, p. 276, in Arrindell & van der Ende,1985, p. 166; Kunce, Cook, & Miller, 1975, Marascuilor & Levin, 1983, in Velicer & Fava, 1998, p. 232) Rule of 5: The subjects-to-variables ratio should be no lower than 5 (Bryant and Yarnold, 1995, in David Garson, 2008; Gorsuch, 1983, in MacCallum, Widaman, Zhang & Hong, 1999; Everitt, 1975, in Arrindell & van der Ende, 1985; Gorsuch, 1974, in Arrindell & van der Ende, 1985, p. 166) A ratio of 3(:1) to 6(:1) of subjects-to-variables is acceptable if the lower limit of variables-to-factors ratio is 3 to 6. But, the absolute minimum sample size should not be less than 250.(Cattell, 1978, p. 508, in Arrindell & van der Ende, 1985, p. 166) Ratio of 2. There should be at least twice as many subjects as variables in factor-analytic investigations. This means that in any large study on this account alone, one should have to use more than the minimum 100 subjects" (Kline, 1979, p. 40). 47 JRC-COIN Step 7: Statistical Coherence (II)

48 Multivariate data The type of analysis to be performed depends on the data of the indicators Data Indicators Quantitative Qualitative Mixed/ structured PCA/FA PCA Principal Component Analysis FA Factor Analysis dispca/fa,(m)ca FAMD/MFA PCA for discrete data (based on polychoric correlations) (M)CA (Multiple) Correspondence Analysis. FAMD Factor Analysis of Mixed Data MFA Multiple Factor Analysis. 48 JRC-COIN Step 7: Statistical Coherence (II)

49 Exploratory FA Two main types of FA: Exploratory (EFA) and Confirmatory (CFA) EFA examining the relationships among the variables/indicators and do not have an à priori fixed number of factors CFA a clear idea about the number of factors you will encounter, and about which variables will most likely load onto each factor 49 JRC-COIN Step 7: Statistical Coherence (II)

50 Is PCA = Exploratory FA? PCA FA Similar in many ways but an important difference that has effects on how to use them 1) Both are data reduction techniques - they allow you to capture the variance in variables in a smaller set 2) Same (or similar) procedures used to run in many software (same steps extraction, interpretation, rotation, choosing the number of components/factors) 50 JRC-COIN Step 7: Statistical Coherence (II)

51 Is PCA = Exploratory FA? NO! PCA FA Similar in many ways but an important difference that has effects on how to use them 1) Both are data reduction techniques - they allow you to capture the variance in variables in a smaller set 2) Same (or similar) procedures used to run in many software (same steps extraction, interpretation, choosing the number of components/factors) Difference: PCA is a linear combination of variables FA is a measurement model of a latent variable (statistical model) 51 JRC-COIN Step 7: Statistical Coherence (II)

52 Is PCA = Exploratory FA? NO! PCA Observed indicators are reduced into components FA Latent factors drive the observed variables Item 1 X 1 w 1 λ 1 Item 1 X 1 ε 1 Item 2 X 2... w 2 Component Factor λ 2 Item 2 X 2... ε 2 ε i Error terms Item p X p w p λ p Item p X p ε p PCA summarizes information of all indicators and reduces it into a fewer number of components Each PC i is a new variable computed as a linear combination of the original variables PC 1 w x w x w p x p Model of the measurement of a latent variable. This latent variable cannot be directly measured with a single variable. Seen through the relationships it causes in a set of X variables. X X X 1 i p F 1 F i F p i p 52 JRC-COIN Step 7: Statistical Coherence (II)

53 PCA for discrete data A practically important violation of the normality assumption underlying the PCA occurs when the data are discrete! Several kinds of discrete data (binary, ordinal, count, nominal ) Two ways on how to proceed: 1) Simple approach, use the discrete x s as if they were continuous in the PCA but instead of using the Pearson moment correlation coefficients use Spearman or Kendall rank correlation coefficients. 2) PCA using polychoric correlations. Pearson (1901) founder of this approach (as well!), developed further by Olsson (1979) and Jöreskog (2004) 53 JRC-COIN Step 7: Statistical Coherence (II)

54 PCA using polychoric correlations The polychoric correlation (PCC) coefficient is a measure of association for ordinal variables which rests upon an assumption of an underlying joint continuous normal distribution. (Later generalizations of different continuous distributions). PCC coefficient is MLE of the product-moment correlation between the underlying normal variables 54 JRC-COIN Step 7: Statistical Coherence (II)

55 55 JRC-COIN Step 7: Statistical Coherence (II) Cronbach s α is measure of internal consistency and reliability Based upon classical test theory. Most common estimate of reliability, Cronbach (1951) Cronbach s α is defined as Ratio of the Total Covariance in the test to the Total Variance in the test. By test we mean the set of indicators constituting a sub-pillar/ pillar/ci. Let be the variance of indicator i and total variance in sub-pillar/pillar/ci Average covariance of an indicator with any other indicator is Part 2: Reliability Analysis ) ( i X Var p i i X Var 1 1) ( ) ( 1 1 p p X Var X Var p i i p i i p i i p i i p i i p i i p i i p i i X Var X Var X Var p p p X Var p p X Var X Var ) ( 1 / 1) ( / ) ( p = number of indicators p i i p i i X Var X Var p p 1 1 ) ( 1 1