EVALUATION OF STRUCTURAL EQUATION MODELS

Transcription

1 1 EVALUATION OF STRUCTURAL EQUATION MODELS I. Issues related to the initial specification of theoretical models of interest 1. Model specification: a. Measurement model: (i) EFA vs. CFA (ii) reflective vs. formative indicators [see Appendix A] (iii) number of indicators per construct [see Appendix B] total aggregation model partial aggregation model total disaggregation model b. Latent variable model: (i) recursive vs. nonrecursive models (ii) alternatives to the target model [see Appendix C for an example] 2. Model misspecification a. omission/inclusion of (ir)relevant variables b. omission/inclusion of (ir)relevant relationships c. misspecification of the functional form of relationships 3. Model identification 4. Sample size 5. Statistical assumptions

2 2 II. Data screening 1. Inspection of the raw data a. detection of coding errors b. recoding of variables c. treatment of missing values (FIML) In Lisrel: Missing Value Code -99 In Mplus: VARIABLE: NAMES = id aa1t1 aa2t1 aa3t1 aa4t1 aa1t2 aa2t2 aa3t2 aa4t2; MISSING = ALL (-99); In SAS: PROC CALIS DATA=couponmiss METHOD=fiml; 2. Outlier detection 3. Assessment of normality 4. Measures of association a. regular vs. specialized measures b. covariances vs. correlations c. non-positive definite input matrices III. Model estimation and testing 1. Model estimation 2. Estimation problems a. nonconvergence or convergence to a local optimum b. improper solutions c. problems with standard errors d. empirical underidentification

3 3 3. Overall fit assessment [see Appendix D] 4. Model modification and model comparison [see Appendix E] a. Measurement model b. Latent variable model 5. Local fit measures [see Appendix F on how to obtain robust standard errors] a. Measurement model (i) factor loadings, factor (co)variances, and error variances (ii) reliability/convergent validity and discriminant validity b. Latent variable model (i) structural coefficients and equation disturbances (ii) direct, indirect, and total effects [see Appendix G] (iii) explained variation in endogenous constructs 6. Power [see Appendix H] 7. Model-based residual analysis 8. Cross-validation 9. Model equivalence and near equivalence [see Appendix I] 10. Latent variable scores [see Appendix J]

4 4 Appendix A: Reflective vs. formative indicators Reflective indicators Formative indicators focal construct focal construct Criteria for distinguishing between reflective and formative indicator models Are the indicators manifestations of the underlying construct (rather than defining characteristics of it)? Are the indicators conceptually interchangeable? Are the indicators expected to covary? Are all of the indicators expected to have the same antecedents and/or consequences? Based on MacKenzie, Podsakoff and Jarvis, JAP 2005, pp

5 5 Appendix B: Number of indicators per factor Total aggregation model: (a) If measurement error in indicator i is ignored, then 1 and 0 ij ii (b) If indicator i is treated as fallible but reliability is unknown, a small amount of measurement error might be assumed: 2 ij 1 and ii.1s xi (c) If indicator i is treated as fallible and an estimate of reliability ( ii ) is available: 2 ij 1 and ii (1 ii) sx i Partial aggregation model: Recommendations for item parceling (based on Holt 2004): if a factor is unidimensional, the items may be parceled randomly; if a factor is multidimensional, item parcels should be based on similar facets of the construct; item parceling may be used strategically to correct for non-normality problems; if the factor structure is not well-understood, do not parcel; Total disaggregation model: in principle, more indicators are better than fewer, but if the number of indicators is too large, it will be difficult to achieve a good model fit;

6 6 Appendix C: Specification of a priori models to be compared using SEM Consumer Behavior Attitudes A ad as a mediator of advertising effectiveness: Four structural specifications (MacKenzie et al. 1986) Affect transfer hypothesis Dual mediation hypothesis C ad A ad C ad A ad C b A b BI C b A b BI Reciprocal mediation hypothesis Independent influences hypothesis C ad A ad C ad A ad C b A b BI C b A b BI

7 7 Appendix D: Classification of overall fit indices Overall fit indices Stand-alone fit indices Incremental fit indices 2 test and variations Noncentralitybased measures Information theory-based measures Others Type I indices Type II indices minimum fit function 2 normal theory WLS 2 S-B scaled 2 2 corrected for nonnormality 2 /df NCP Rescaled NCP (t) RMSEA MC AIC SBC CIC ECVI (S)RMR GFI PGFI AGFI Gamma hat CN NFI RFI [ 2 or f] [ 2 /df] CFI [ 2 -df] TLI [( 2 -df)/df] IFI TLI minimum fit function f Scaled LR

8 8 Summary of overall fit (or lack-of-fit) indices (a) stand-alone fit indices: Index Definition of the index 1 Characteristics 2 Interpretation and use of the index Minimum fit function chi-square (N-1)f BF, SA, NN, NP tests the hypothesis that the specified model provides a perfect fit (within the limits of sampling error); the obtained 2 value should be smaller than 2 crit; note that different discrepancy functions will yield different 2 values; Normal theory weighted least squares (WLS) chi-square Satorra-Bentler scaled (robust) chisquare Chi-square corrected for nonnormality (Browne 1984) Elliptical corrected 2 (Browne 1982; 2 ELL) Chi-square over degrees of freedom ratio ( 2 /df) Minimum fit function value or fit criterion (f) Scaled likelihood ratio (Marsh et al. 1988; LHR) 2 BF, SA, NN, NP same interpretation as C1 and asymptotically equivalent to it; 2 robust BF, SA, NN, NP scaled 2 statistic corrected for violations of multivariate normality (such that the adjusted value has the correct asymptotic mean or the correct asymptotic mean and variance); same interpretation as C1; 2 nn BF, SA, NN, NP 2 statistic corrected for violations of multivariate normality; same interpretation as C1; 2 /(1+ ) BF, SA, NN, NP 2 when the variables have a multivariate elliptical distribution; the multivariate normal distribution is a special case with =0; same interpretation as C1; 2 /df BF, SA, NN, P estimate of how many times larger the obtained 2 value is than the expected value (under H 0) of a 2 variate with a given number of df; ratios smaller than 2 or 3 are sometimes considered desirable; f BF, SA, NN, NP minimum of the discrepancy function comparing S with ; if the fit of the model is perfect (i.e., S= ), f will be zero; can be used to compare competing models; exp [-(1/2)f ] GF, SA, N, NP f scaled to fall within the [0, 1] interval; the closer to one the better the fit;

9 9 Noncentrality parameter (Browne and Cudeck 1993; NCP) Rescaled NCP (McDonald and Marsh 1990) or population discrepancy function value (t, F 0) Root mean squared error of approximation (Browne and Cudeck 1993; Steiger 1990; RMSEA) McDonald s (1989) measure of centrality (MC) ( 2 -df) BF, SA, NN, NP estimate of how well the fitted model approximates the population covariance matrix; can be interpreted as a weighted sum of squares of discrepancies between the parameters of the approximating model and the parameters of the model yielding ; a confidence interval around NCP is available; ( 2 -df)/(n-1) BF, SA, NN, NP estimate of error due to approximation; a confidence interval around t is available; (t/df) ½ BF, SA, NN, P estimate of how well the fitted model approximates the population covariance matrix per df; Browne and Cudeck (1993) suggest that a value of.05 indicates a close fit and that values up to.08 are reasonable; Hu and Bentler (1999) recommend a cutoff value of.06; a p-value for testing the hypothesis that the discrepancy is smaller than.05 may be calculated (test of close fit); exp [-(1/2)t] GF, SA, AN, NP t scaled to fall within the [0, 1] interval; the closer to one the better the fit; Hu and Bentler (1999) recommend a cutoff value of close to.9; Akaike s (1987) information criterion (AIC) [ 2 +2r] or [ 2 2df] BF, SA, NN, P based on statistical information theory and used for testing competing models; the model with the smallest AIC is selected; Schwarz s (1978) Bayesian criterion (SBC) [ 2 +r lnn] or [ 2 df lnn] BF, SA, NN, P modification of AIC; same interpretation and use, although it may lead to a different rank order of models; Bozdogan s (1987) consistent information criterion (CIC) [ 2 +r(lnn+1)] or [ 2 df(lnn+1)] BF, SA, NN, P modification of AIC; same interpretation and use, although it may lead to a different rank order of models; Expected cross-validation index (ECVI; Browne and Cudeck 1989) [f+(2r)/(n 1)] or [f+(2r)/(n (p+q) 2)] BF, SA, NN, P single-sample approximation to the cross-validation coefficient obtained from a validation sample; similar to AIC and same rank order; a confidence interval can be calculated to take into account the precision of the estimate; not recommended by Hu and Bentler (1999); Root mean residual (Jöreskog and Sörbom 1993; (S)RMR) {2 (s ij - ij ) 2 / [(p+q)(p+q+1)]} 1/2 BF, SA, N or NN, NP measure of the average size of residuals between the fitted and sample covariance matrices; if a correlation matrix is analyzed, RMR is standardized to fall within the [0, 1] interval (SRMR), otherwise it is only bounded from below; a cutoff of.05 is often used for SRMR; Hu and Bentler (1999) recommend a cutoff value close to.08;

10 10 Goodness-of-fit index (Jöreskog and Sörbom 1993; GFI) 1-{f[S, ]/f[s, (0)]} GF, SA, AN, NP proportion of the variances and covariances in S accounted for by the fitted model; should fall between 0 and 1, although it can be smaller than 0; values greater than.9 are sometimes deemed desirable; not recommended by Hu and Bentler (1999); Parsimonious goodness-of-fit index (PGFI; Mulaik et al. 1989) 2df/[(p+q)(p+q+1)] GFI or [df t/df n] GFI GF, SA, AN, P GFI adjusted for (loss of) df (by multiplying by the parsimony index); can be used to compare competing models; Adjusted goodness-of-fit index (Jöreskog and Sörbom 1993; AGFI) 1-[(p+q)(p+q+1)/2df] (1-GFI) GF, SA, AN, P GFI adjusted for (loss of) df (adjustment based on the same logic as in the case of an adjusted R 2 ); values greater than.9 are sometimes deemed desirable; not recommended by Hu and Bentler (1999); Gamma hat (Steiger 1989) (p+q)/(p+q+2t) GF, SA, AN, NP population equivalent of GFI expressed in terms of the non-centrality parameter; Hu and Bentler (1999) recommend a cutoff value of.95; Hoelter s (1983) critical N (CN) [ 2 crit/f+1] or {[Z crit+(2df-1) 1/2 ] 2 / [2 2 /(N-1)]}+1 GF, SA, NN, NP estimate of the sample size at which the null hypothesis that the model fits is rejected; Hoelter suggests that CN should be larger than 200; not recommended by Hu and Bentler (1999);

11 11 (a) incremental fit indices: Bentler and Bonett s (1980) normed fit index (NFI) or 1 1-[ 2 t / 2 n ] GF, IM, N, NP measure of the proportionate improvement in fit (defined in terms of f or 2 ) as one moves from the baseline to the target model; values greater than.9 are usually deemed desirable; problem that it is biased downward for small N; not recommended by Hu and Bentler (1999); Parsimonious normed fit index (James et al. 1982; PNFI) df t/df n [1-2 t / 2 n ] GF, IM, N, P NFI adjusted for (loss of) df (by multiplying by the parsimony index); can be used to compare competing models; Bollen s (1989) nonnormed fit index 2 or incremental fit index (IFI) [ 2 n - 2 t ]/[ 2 n -df t ] GF, IM, AN, P modified version of NFI designed to lessen its dependence on sample size; however, it may be biased upward for small N when the model is misspecified, and the parsimony correction may be inappropriate; Hu and Bentler (1999) recommend a cutoff value of.95; Bollen s (1986) normed index 1 or relative fit index (RFI) 1-[( 2 t /df t )/( 2 n /df n )] GF, IM, AN, P similar to NFI, except that the fit of the target model is compared to the fit of the baseline model per df; problem that it is biased downward for small N; not recommended by Hu and Bentler (1999); Bentler s (1990) normed Comparative Fit Index (CFI) McDonald and Marsh s (1990) nonnormed relative noncentrality index (RNI) 1-[max( 2 t -df t,0)/ GF, IM, N, NP measure of the proportionate improvement in fit (defined in terms of max( 2 n -df n, 2 t -df t,0)] noncentrality) as one moves from the baseline to the target model; Hu and Bentler (1999) recommend a cutoff value of.95; 1-[( 2 t -df t )/( 2 n -df n )] GF, IM, AN, NP similar to CFI in most cases, but not restricted to fall strictly within the [0, 1] interval; Hu and Bentler (1999) recommend a cutoff value of.95; Tucker and Lewis (1973) nonnormed fit index (TLI, NNFI) or 2 [ 2 n /df n - 2 t /df t ]/[ 2 n /df n -1] = 1-[(t t /df t )/(t n /df n )] GF, IM, AN, P measure of the proportionate improvement in fit (defined in terms of noncentrality) as one moves from the baseline to the target model, per df; Hu and Bentler (1999) recommend a cutoff value of.95; (footnotes on next page)

12 12 1 N = sample size; df = degrees of freedom; r = number of parameters estimated; (p+q) = number of observed variables; = Mardia-based kappa, an estimate of multivariate kurtosis (Browne 1982); 2 crit = critical value of the 2 distribution with the appropriate number of degrees of freedom and for a given significance level; Z crit = critical value of the normal distribution for a given level of statistical significance. The subscripts n and t refer to the null (or baseline) and target models, respectively. In the definition of GFI, f[s, ] refers to the minimum of the fitting function after the target model has been fitted and f[s, (0)] refers to the minimum of the fitting function before any model has been fitted. 2 GF = goodness-of-fit index; BF = badness-of-fit index; SA = stand-alone fit index; IM = incremental fit index; N = normed (in the sample) fit index; AN = normed (in the population) fit index, but only approximately normed in the sample (i.e., can fall outside the [0, 1] interval); NN = nonnormed fit index; NP = no correction for parsimony; P = correction for parsimony.

13 13 Types of error in covariance structure modeling best fit of the model to S for a given discrepancy function ˆ known - random ~ 0 unknown - fixed best fit of the model to 0 for a given discrepancy function error of approximation (an unknown constant) 0 unknown - fixed population covariance matrix Incremental fit indices type I indices: GF t GF GF n t or BF n BF t BF n type II indices: GF GF t n E( GF ) GF t n or BF n BF n BF t E( BF ) t where: GF t, BF t = value of some stand-alone index for the target model; GF n, BF n = value of the stand-alone index for the null model; E(GF t ), E(BF t ) = expected value of GF t or BF t assuming that the target model is true;

14 14 Appendix E: Model comparisons Model comparisons saturated structural model (M s ) lowest 2 lowest df next most likely unconstrained model (M u ) target model (M t ) next most likely constrained model (M c ) null structural model (M n ) highest 2 highest df

15 15 APPENDIX F: HOW TO OBTAIN ROBUST MODEL STATISTICS IN LISREL: Title: Confirmatory factor model (attitude toward using coupons measured at two points in time) Raw Data from File factor.lsf Latent Variables AAT1 AAT2 Relationships aa1t1 aa2t1 aa3t1 aa4t1 = AAT1 aa1t2 aa2t2 aa3t2 aa4t2 = AAT2 Set the Variance of AAT1 to 1 Set the Variance of AAT2 to 1 Set the Covariance of AAT1 and AAT2 free Set the Error Covariance of aa1t1 and aa1t2 free Set the Error Covariance of aa2t1 and aa2t2 free Set the Error Covariance of aa3t1 and aa3t2 free Set the Error Covariance of aa4t1 and aa4t2 free Robust Estimation Options sc rs mi wp nd=3 End of Problem IN MPLUS: Use Estimator=MLM, i.e., ANALYSIS: ESTIMATOR=MLM; IN SAS: Use METHOD=MLSB as an option following the PROC CALIS statement (this is only available in the most recent versions of SAS).

16 16 Overall goodness-of-fit statistics: Degrees of Freedom for (C1)-(C3) 15 Maximum Likelihood Ratio Chi-Square (C1) (P = ) Browne's (1984) ADF Chi-Square (C2_NT) (P = ) Browne's (1984) ADF Chi-Square (C2_NNT) (P = ) Satorra-Bentler (1988) Scaled Chi-Square (C3) (P = ) Satorra-Bentler (1988) Adjusted Chi-Square (C4) (P = ) Degrees of Freedom for C Local fit indices: construct parameter parameter estimated estimated estimate standard robust standard (ML) error (ML) error Aact(t1) Aact(t2) x x x x x x x x Note: Error covariances not shown.

17 direct indirect total 17 Appendix G: Direct, indirect and total effects Direct, indirect, and total effects inconveniences rewards Aact BI B encumbrances -.05 inconveniences rewards encumbrances BI B inconveniences rewards encumbrances Aact BI.24 B Note: To get indirect and total effects in LISREL, use the EF keyword on the OU line.

18 18 Appendix H: Power True state of nature H 0 true H 0 false Decision Accept H 0 Reject H 0 Correct decision Type I error ( Type II error ( Correct decision low power high test statistic nonsignificant significant

19 19 Appendix I: Four equivalent models (Kline 2005) η 3 η 3 η 1 η 1 η 4 η 4 η 2 η 2 η 5 η 5 η 3 η 3 η 1 η 1 η 4 η 4 η 2 η 2 η 5 η 5

20 20 Appendix J: Latent variable scores Sometimes it is of interest to get estimated latent variable scores for η and ξ (e.g., if one wants to estimate nonlinear relationships among the latent variables) and estimated observational residuals. IN LISREL: Title: Confirmatory factor model (attitude toward using coupons measured at two points in time) Raw Data from File factor.lsf Latent Variables AAT1 AAT2 Relationships aa1t1 aa2t1 aa3t1 aa4t1 = AAT1 aa1t2 aa2t2 aa3t2 aa4t2 = AAT2 Set the Variance of AAT1 to 1 Set the Variance of AAT2 to 1 Set the Covariance of AAT1 and AAT2 free Set the Error Covariance of aa1t1 and aa1t2 free Set the Error Covariance of aa2t1 and aa2t2 free Set the Error Covariance of aa3t1 and aa3t2 free Set the Error Covariance of aa4t1 and aa4t2 free LSFfile factor.lsf Estimate Residuals Options sc rs mi wp nd=3 PT End of Problem The latent variable scores and residuals will be contained in a new lsf file called factornew.lsf. IN MPLUS: SAVEDATA: FILE IS scores.txt; SAVE = FSCORES; IN SAS: PROC CALIS DATA=raw OUTSTAT=ostat; [your model goes here] RUN; PROC SCORE DATA=raw SCORE=ostat OUT=scores; RUN;