Factor Analysis. Qian-Li Xue

Transcription

1 Factor Analysis Qian-Li Xue Biostatistics Program Harvard Catalyst The Harvard Clinical & Translational Science Center Short course, October 7, 06

2 Well-used latent variable models Latent variable scale Observed variable scale Continuous Discrete Continuous Factor analysis Discrete LISREL Latent profile Growth miture Discrete FA IRT (item response) Latent class analysis, regression General software: MPlus, Latent Gold, WinBugs (Bayesian), NLMIXED (SAS)

3 Objectives What is factor analysis? What do we need factor analysis for? What are the modeling assumptions? How to specify, fit, and interpret factor models? What is the difference between eploratory and confirmatory factor analysis? What is and how to assess model identifiability? 3

4 What is factor analysis Factor analysis is a theory driven statistical data reduction technique used to eplain covariance among observed random variables in terms of fewer unobserved random variables named factors 4

5 An Eample: General Intelligence (Charles Spearman, 904) Y δ Y δ General Intelligence Y 3 δ 3 F Y 4 δ 4 Y 5 δ 5 5

6 Why Factor Analysis?. Testing of theory Eplain covariation among multiple observed variables by Mapping variables to latent constructs (called factors ). Understanding the structure underlying a set of measures Gain insight to dimensions Construct validation (e.g., convergent validity) 3. Scale development Eploit redundancy to improve scale s validity and reliability 6

7 Part I. Eploratory Factor Analysis (EFA) 7

8 One Common Factor Model: Model Specification Y δ Y = F + δ F 3 Y δ Y = F + δ Y 3 δ 3 Y 3 = 3 F + δ 3 The factor F is not observed; only Y, Y, Y 3 are observed δ i represent variability in the Y i NOT eplained by F Y i is a linear function of F and δ i 8

9 One Common Factor Model: Model Assumptions Y δ Y = F + δ F Y δ Y = F + δ 3 Y 3 δ 3 Y 3 = 3 F + δ 3 Factorial causation F is independent of δ j, i.e. cov(f,δ j )=0 δ i and δ j are independent for i j, i.e. cov(δ i,δ j )=0 Conditional independence: Given the factor, observed variables are independent of one another, i.e. cov( Y i,y j F ) = 0 9

10 One Common Factor Model: Model Interpretation F Y Y Y 3 = = = 3 Y Y δ δ Y 3 δ 3 3 F F F δ δ δ 3 Given all variables in standardized form, i.e. var(y i )=var(f)= Factor loadings: i i = corr(y i,f) Communality of Y i : h i h i = i = [corr(y i,f)] =% variance of Y i eplained by F Uniqueness of Y i : -h i = residual variance of Y i Degree of factorial determination: =Σ i /n, where n=# observed variables Y 0

11 Two-Common Factor Model (Orthogonal): Model Specification F F Y Y Y 3 Y 4 Y 5 δ δ δ 3 δ 4 δ 5 Y Y Y Y Y Y = F = = F = F F = F = F + F F F F F F + δ + δ + δ + δ + δ + δ Y 6 δ 6 F and F are common factors because they are shared by variables!

12 Matri Notation with n variables and m factors Y n = Λ nm F m + δ n + = n n m m m n nm n m n F F Y Y δ δ

13 Factor Pattern Matri Columns represent derived factors Rows represent input variables Loadings represent degree to which each of the variables correlates with each of the factors Loadings range from - to Inspection of factor loadings reveals etent to which each of the variables contributes to the meaning of each of the factors. High loadings provide meaning and interpretation of factors (~ regression coefficients) n m nm n m 3

14 Two-Common Factor Model (Orthogonal): Model Assumptions F F Y Y Y 3 Y 4 Y 5 δ δ δ 3 δ 4 δ 5 Factorial causation F and F are independent of δ j, i.e. cov(f,δ j )= cov(f,δ j )= 0 δ i and δ j are independent for i j, i.e. cov(δ i,δ j )=0 Conditional independence: Given factors F and F, observed variables are independent of one another, i.e. cov( Y i,y j F, F ) = 0 for i j Orthogonal (=independent): cov(f,f )=0 Y 6 δ 6 4

15 Two-Common Factor Model (Orthogonal): Model Interpretation F Y Y Y 3 Y 4 δ δ δ 3 δ 4 Given all variables in standardized form, i.e. var(y i )=var(f i )=; AND orthogonal factors, i.e. cov(f,f )=0 Factor loadings: ij ij = corr(y i,f j ) Communality of Y i : h i h i = i + i =% variance of Y i eplained by F AND F F 6 5 Y 5 δ 5 Uniqueness of Y i : -h i Y 6 δ 6 Degree of factorial determination: =Σ ij /n, n=# observed variables Y 5

16 Two-Common Factor Model : The Oblique Case F Y Y Y 3 δ δ δ 3 Given all variables in standardized form, i.e. var(y i )=var(f i )=; AND oblique factors (i.e. cov(f,f ) 0) The interpretation of factor loadings: ij is no longer correlation between Y and F; it is direct effect of F on Y 3 4 Y 4 δ 4 The calculation of communality of Yi (h i ) is more comple F 5 Y 5 δ 5 6 Y 6 δ 6 6

17 Etracting initial factors Least-squares method (e.g. principal ais factoring with iterated communalities) Maimum likelihood method 7

18 d) Repeat b) and c) until no improvement can be made 8 Model Fitting: Etracting initial factors Least-squares method (LS) (e.g. principal ais factoring with iterated communalities) v Goal: minimize the sum of squared differences between observed and estimated corr. matrices v Fitting steps: a) Obtain initial estimates of communalities (h ) e.g. squared correlation between a variable and the remaining variables b) Solve objective function: det(r LS -ηi)=0, where R LS is the corr matri with h in the main diag. (also termed adjusted corr matri), η is an eigenvalue c) Re-estimate h

19 Model Fitting: Etracting initial factors Maimum likelihood method (MLE) v v v v v v Goal: maimize the likelihood of producing the observed corr matri Assumption: distribution of variables (Y and F) is multivariate normal Objective function: det(r MLE - ηi)=0, where R MLE =U - (R-U )U - =U - R LS U -, and U is diag(-h ) Iterative fitting algorithm similar to LS approach Eception: adjust R by giving greater weights to correlations with smaller unique variance, i.e. - h Advantage: availability of a large sample χ significant test for goodness-of-fit (but tends to select more factors for large n!) 9

20 Choosing among Different Methods Between MLE and LS LS is preferred with v few indicators per factor v Equeal loadings within factors v No large cross-loadings v No factor correlations v Recovering factors with low loadings (overetraction) MLE if preferred with v Multivariate normality v unequal loadings within factors Both MLE and LS may have convergence problems 0

21 Factor Rotation Goal is simple structure Make factors more easily interpretable While keeping the number of factors and communalities of Ys fied!!! Rotation does NOT improve fit!

22 Factor Rotation To do this we rotate factors: redefine factors such that loadings (or pattern matri coefficients) on various factors tend to be very high (- or ) or very low (0) intuitively, it makes sharper distinctions in the meanings of the factors

23 Factor Rotation (Intuitively) F,, F 5 4 F 4 F Factor Factor Factor Factor

24 Factor Rotation Uses ambiguity or non-uniqueness of solution to make interpretation more simple Where does ambiguity come in? Unrotated solution is based on the idea that each factor tries to maimize variance eplained, conditional on previous factors What if we take that away? Then, there is not one best solution 4

25 Factor Rotation: Orthogonal vs. Oblique Rotation Orthogonal: Factors are independent varima: maimize variance of squared loadings across variables (sum over factors) v Goal: the simplicity of interpretation of factors quartima: maimize variance of squared loadings across factors (sum over variables) v Goal: the simplicity of interpretation of variables Intuition: from previous picture, there is a right angle between aes Note: Uniquenesses remain the same! 5

26 Factor Rotation: Orthogonal vs. Oblique Rotation Oblique: Factors are NOT independent. Change in angle. oblimin: minimize covariance of squared loadings between factors. proma: simplify orthogonal rotation by making small loadings even closer to zero. Target matri: choose simple structure a priori. Intuition: from previous picture, angle between aes is not necessarily a right angle. Note: Uniquenesses remain the same! 6

27 Pattern versus Structure Matri In oblique rotation, one typically presents both a pattern matri and a structure matri Also need to report correlation between the factors The pattern matri presents the usual factor loadings The structure matri presents correlations between the variables and the factors For orthogonal factors, pattern matri=structure matri The pattern matri is used to interpret the factors 7

28 Factor Rotation: Which to use? Choice is generally not critical Interpretation with orthogonal (varima) is simple because factors are independent: Loadings are correlations. Configuration may appear more simple in oblique (proma), but correlation of factors can be difficult to reconcile. Theory? Are the conceptual meanings of the factors associated? 8

29 Factor Rotation: Unique Solution? The factor analysis solution is NOT unique! More than one solution will yield the same result. 9

30 Derivation of Factor Scores Each object (e.g. each person) gets a factor score for each factor: The factors themselves are variables Object s score is weighted combination of scores on input variables Fˆ = WY ˆ, where Wˆ is the weight matri. These weights are NOT the factor loadings! Different approaches eist for estimating Wˆ (e.g. regression method) Factor scores are not unique Using factors scores instead of factor indicators can reduce measurement error, but does NOT remove it. Therefore, using factor scores as predictors in conventional regressions leads to inconsistent coefficient estimators! 30

31 Factor Analysis with Categorical Observed Variables Factor analysis hinges on the correlation matri As long as you can get an interpretable correlation matri, you can perform factor analysis Binary/ordinal items? Pearson corrlation: Epect attenuation! Tetrachoric correlation (binary) Polychoric correlation (ordinal) To obtain polychoric correlation in STATA: polychoric var var var3 var4 var5 To run princial component analysis: pcamat r(r), n(38) To run factor analysis: factormat r(r), fa() ipf n(38) 3

32 Criticisms of Factor Analysis Labels of factors can be arbitrary or lack scientific basis Derived factors often very obvious defense: but we get a quantification Garbage in, garbage out really a criticism of input variables factor analysis reorganizes input matri Correlation matri is often poor measure of association of input variables. 3

33 Major steps in EFA. Data collection and preparation. Choose number of factors to etract 3. Etracting initial factors 4. Rotation to a final solution 5. Model diagnosis/refinement 6. Derivation of factor scales to be used in further analysis 33

34 Part II. Confirmatory Factor Analysis (CFA) 34

35 Eploratory vs. Confirmatory Eploratory: summarize data Factor Analysis describe correlation structure between variables generate hypotheses Confirmatory Testing correlated measurement errors Redundancy test of one-factor vs. multi-factor models Measurement invariance test comparing a model across groups Orthogonality tests 35

36 Confirmatory Factor Analysis (CFA) Takes factor analysis a step further. We can test or confirm or implement a highly constrained a priori structure that meets conditions of model identification But be careful, a model can never be confirmed!! CFA model is constructed in advance number of latent variables ( factors ) is pre-set by analyst (not part of the modeling usually) Whether latent variable influences observed is specified Measurement errors may correlate Difference between CFA and the usual SEM: SEM assumes causally interrelated latent variables CFA assumes interrelated latent variables (i.e. eogenous) 36

37 Eploratory Factor Analysis Two factor model: = Λξ + δ = ξ ξ + δ δ δ 3 δ 4 δ 5 δ 6 ξ ξ δ δ δ 3 δ 4 δ 5 δ 6 37

38 CFA Notation + = δ δ δ δ δ δ ξ ξ Two factor model: = + Λξ δ δ δ δ 3 δ 4 δ 5 δ ξ ξ 38

39 Difference between CFA and EFA CFA = ξ + δ = ξ + δ = ξ + δ = ξ + δ = ξ + δ = ξ + δ cov( ξ, ξ ) = ϕ = ξ + ξ + δ = ξ + ξ + δ = ξ + ξ + δ = ξ + ξ + δ = ξ + ξ + δ = ξ + ξ + δ ξ ξ = 0 cov(, ) EFA 39

40 Model Constraints Hallmark of CFA Purposes for setting constraints: Test a priori theory Ensure identifiability Test reliability of measures 40

41 Identifiability Let θ be a t vector containing all unknown and unconstrained parameters in a model. The parameters θ are identified if Σ(θ )= Σ(θ ) θ =θ Estimability Identifiability!! Identifiability attribute of the model Estimability attribute of the data 4

42 Model Constraints: Identifiability Latent variables (LVs) need some constraints Because factors are unmeasured, their variances can take different values Recall EFA where we constrained factors: F ~ N(0,) Otherwise, model is not identifiable. Here we have two options: Fi variance of latent variables (LV) to be (or another constant) Fi one path between LV and indicator 4

43 Necessary Constraints Fi variances: Fi path: δ δ ξ δ ξ δ ξ 3 4 δ 3 δ 4 ξ 3 4 δ 3 δ 4 5 δ 5 5 δ 5 6 δ 6 6 δ 6 43

44 Model Parametrization Fi variances: = ξ + δ = ξ + δ = ξ + δ = ξ + δ = ξ + δ = ξ + δ cov( ξ, ξ ) var( ξ ) = var( ξ ) = = ϕ Fi path: = ξ + δ = ξ + δ = ξ + δ = ξ + δ 4 4 = ξ + δ = ξ + δ cov( ξ, ξ ) var( ξ ) var( ξ ) = = = ϕ ϕ ϕ 44

45 Identifiability Rules for CFA () Two-indicator rule (sufficient, not necessary) ) At least two factors ) At least two indicators per factor 3) Eactly one non-zero element per row of Λ (translation: each only is pointed at by one LV) 4) Non-correlated errors (Θ is diagonal) (translation: no double-header arrows between the δ s) 5) Factors are correlated (Φ has no zero elements)* (translation: there are double-header arrows between all of the LVs) * Alternative less strict criteria: each factor is correlated with at least one other factor. (see page 47 on Bollen) 45

46 = ξ ξ + δ δ δ 3 δ 4 δ 5 δ 6 ξ ξ δ δ δ 3 δ 4 δ 5 δ 6 Θ = var( δ) = θ θ θ θ θ θ66 Φ = = ϕ var( ξ) ϕ 46

47 Eample: Two-Indicator Rule δ ξ δ ξ 3 δ 3 4 δ 4 ξ 3 5 δ 5 6 δ 6 47

48 Eample: Two-Indicator Rule δ ξ δ ξ 3 δ 3 4 δ 4 ξ 3 5 δ 5 6 δ 6 48

49 Eample: Two-Indicator Rule δ ξ δ ξ 3 δ 3 4 δ 4 ξ 3 5 δ 5 6 δ 6 ξ 4 7 δ 7 8 δ 8 49

50 Identifiability Rules for CFA () Three-indicator rule (sufficient, not necessary) ) at least one factor ) at least three indicators per factor 3) one non-zero element per row of Λ (translation: each only is pointed at by one LV) 4) non-correlated errors (Θ is diagonal) (translation: no double-headed arrows between the δ s) [Note: no condition about correlation of factors (no restrictions on Φ).] δ ξ δ 3 δ 3 50