FIT CRITERIA PERFORMANCE AND PARAMETER ESTIMATE BIAS IN LATENT GROWTH MODELS WITH SMALL SAMPLES

Transcription

1 FIT CRITERIA PERFORMANCE AND PARAMETER ESTIMATE BIAS IN LATENT GROWTH MODELS WITH SMALL SAMPLES Daniel M. McNeish Measurement, Statistics, and Evaluation University of Maryland, College Park

2 Background Latent Growth Models (LGMs) are conceptually similar to multilevel models (MLMs) for repeated measures Can often address similar research questions to reach similar conclusions with LGMs being favored in behavioral science research There are many exceptions, particularly for models with non-linear change

3 Background II With MLMs and the related GEEs, there is a growing literature on small sample size bias of parameter and standard error estimates* Simulation studies have found that, depending on the estimate of interest (e.g., fixed effect, variance components in MLMs), as many as 100 people are needed to yield unbiased estimates *e.g., Bell et al, 2014; Maas & Hox, 2005; Morel et al, 2003

4 Background III Within both MLMs and GEEs, small sample procedures and corrections such as Kenward-Roger and REML estimation in MLMs and the MBN correction in GEEs have been developed to handle small sample scenarios Despite the recent focus on small samples in MLMs and GEEs, methodological research on small sample properties of LGMs will small samples is nearly non-existent. No LGM specific studies with samples below 100

5 Are Small Samples a Problem? Percentage of repeated measures studies with fewer than 100 people (based on 4 meta-analytic studies)* Personality change over time 35% (92/265) Preschool prevention programs 41% (14/34) Schizophrenic brain volume 93% (13/14) The methodological properties of a fair amount of studies are not being investigated *Roberts & DelVecchio, 2000; Roberts et al.,2006; Nelson et al., 2003; Steen et al., 2006

6 Why Does This Matter? If there is a growing literature on MLMs with small samples and MLMs are often similar to LGMs, why does this matter? Different estimation methods (REML vs. FIML) LGMs facilitate non-linear change much more easily by simply allowing the slope loadings to be freely estimated Also more interpretable Assessment of global model fit is done with LGMs but not with MLMs

7 Goals of Study Investigate the extent of bias in LGMs and small samples as has been done recently with MLMs Model fit criteria (e.g., T ML, CFI) have been noted to perform poorly with small sample sizes. Three corrections to T ML are investigated Approximate fit indices that are a function of T ML (RMSEA, TLI, CFI) can also use the corrected T ML

8 Bartlett Correction Outlined in Bartlett (1950) T B = 1 2p + 4k n 1 T ML T ML is the minimum fit function χ 2 p is the number of observed variables k is the number of latent factors n is the sample size

9 Yuan Correction Outlined in Yuan (2005), argued that Bartlett correction is based on too many parameters and tends to overcorrect T B = 1 2p + 2k n 1 T ML T ML is the minimum fit function χ 2 p is the number of observed variables k is the number of latent factors n is the sample size

10 Swain Correction Outlined in Swain (1975) T S = 1 p 2p2 + 3p 1 q(2q 2 + 3q 1) 12n df T ML T ML is the minimum fit function χ 2 p is the number of observed variables k is the number of latent factors n is the sample size q = 1+4p p+1 8df 1 2

11 Why T ML Corrections vs. Finite Sample Methods? Yuan and Bentler have proposed additional methods for finite sample sizes (often based on ADF). Sample size must be larger than the non-duplicated entries in the observed variable covariance matrix For LGMs used in practice, this minimum can be fairly sizeable 6 repeated measures and 3 time-invariant predictors, the minimum sample would be (9 10) / 2 = 45 5 repeated measures with 1 time-varying covariate, (10 11) / 2 = 55

12 Simulation Design 2 Generation models: 1) Simple Linear Growth 2) non-linear growth -Latent Basis Model Sample Size conditions : 20, 30, 50, 100 Both had 2 binary exogenous predictors (X1 and X2) and a heterogeneous diagonal error structure Repeated Measure conditions: 4, replications per cell of the design

13 Non-linear Growth For the 8 repeated measures condition, the population loadings from the slope to the observed variables were Similar to what might be seen in educational data were the largest gains occur at earlier time points and the growth dampens over time Modeling this type of change is very difficult in a MLM framework The first loading was constrained to 0 and the second was constrained to 1.

14 Simulation Outcome Measures Parameter Relative Bias θ 2500 θ θ Values between -5% and 5% are negligibly biased Between 5% and 10% are moderately biased, 10% < is considerably biased 95% CI Coverage Proportion of replications where the population value was contained in the estimated confidence interval Values between and are considered acceptable Fit Criteria Approximate indices Mean value across replications, acceptable values based on Hu and Bentler (1999) cutoffs Chi-square tests Proportion of replications with significant test Between 2.5% and 7.5% within sampling error of 5%

15 Parameter Outcomes Linear Growth Parameter Relative Bias 95% Coverage Rate 4RM Number of Individuals (N:t) Number of Individuals (N:t) Parameter 20 (1.5) 30 (2.3) 50 (3.8) 100 (7.7) 20 (1.5) 30 (2.3) 50 (3.8) 100 (7.7) γ γ Var(α) Var(β) Cov(α,β) RM Number of Individuals (N:t) Number of Individuals (N:t) Parameter 20 (1.2) 30 (1.8) 50 (2.9) 100 (5.9) 20 (1.2) 30 (1.8) 50 (2.9) 100 (5.9) γ γ Var(α) Var(β) Cov(α,β)

16 Parameter Outcomes Non-Linear Growth Parameter Relative Bias 95% Coverage Rate 4RM Number of Individuals (N:t) Number of Individuals (N:t) Parameter 20 (1.3) 30 (2.0) 50 (3.3) 100 (6.7) 20 (1.3) 30 (2.0) 50 (3.3) 100 (6.7) γ γ Var(α) Var(β) Cov(α,β) RM Number of Individuals (N:t) Number of Individuals (N:t) Parameter 20 (0.9) 30 (1.4) 50 (2.3) 100 (4.5) 20 (0.9) 30 (1.4) 50 (2.3) 100 (4.5) γ γ Var(α) Var(β) Cov(α,β)

17 Fit Criteria Results Linear Growth T ML Based Criteria Bartlett Corrected Criteria Swain Corrected Criteria Condition N:t SRMR RMSEA CFI TLI T ML RMSEA CFI TLI T B RMSEA CFI TLI T S 8RM repeated measure condition was fairly similar but omitted to avoid an abundance of tables -CFI and TLI are calculated manually since LGMs are not nested within the traditional null model Yuan Corrected Criteria Cond. RMSEA CFI TLI T Y

18 The T ML criteria perform very poorly when N:t < 4. T Bartlett criteria no longer overcorrected when the Fit Criteria Results Linear Growth Bartlett Corrected T ML Based Criteria Criteria slope loadings were estimated and performed well Swain Corrected Criteria Condition N:t SRMR RMSEA CFI TLI T ML RMSEA CFI TLI T B RMSEA CFI TLI T S even with less than 1 observations per estimated parameter 8RM T Swain criteria performed fairly well so long as N:t > T Yuan criteria performed well across all conditions repeated measure condition was fairly similar but omitted to avoid an abundance of tables -CFI and TLI are calculated manually since LGMs are not nested within the traditional null model Yuan Corrected Criteria Condition RMSEA CFI TLI T Y

19 T ML Based Criteria Bartlett Corrected Criteria Fit Criteria Results Non-Linear Growth Swain Corrected Criteria Condition N:t SRMR RMSEA CFI TLI T ML RMSEA CFI TLI T B RMSEA CFI TLI T S 8RM repeated measure condition was fairly similar but omitted to avoid an abundance of tables -CFI and TLI are calculated manually since LGMs are not nested within the traditional null model Yuan Corrected Criteria Cond. RMSEA CFI TLI T Y

20 The T ML criteria perform very poorly when N:t < 4. T Bartlett criteria no longer overcorrected when the T ML Based Criteria slope loadings were estimated and performed well even with less than 1 observations per estimated Bartlett Corrected Criteria Fit Criteria Results Non-Linear Growth Swain Corrected Criteria Condition N:t SRMR RMSEA CFI TLI T ML RMSEA CFI TLI T B RMSEA CFI TLI T S parameter 8RM T Swain criteria performed fairly well so long as N:t > T Yuan performed well across all conditions repeated measure condition was fairly similar but omitted to avoid an abundance of tables -CFI and TLI are calculated manually since LGMs are not nested within the traditional null model Yuan Corrected Criteria Condition RMSEA CFI TLI T Y

21 Freely Estimated Slope Loadings for Non-Linear Growth Parameter Relative Bias Parameter Relative Bias 4RM 4RM 8RM Parameter 20 (1.3) 30 (2.0) 50 (3.3) 100 (6.7) 20 (0.9) 30 (1.3) 50 (2.2) 100 (4.3) Y1 Constrained Constrained Y2 Constrained Constrained Y3 Not-Present in 4RM Condition Y Y5 Not-Present in 4RM Condition Y6 Not-Present in 4RM Condition Y7 Not-Present in 4RM Condition Y

22 Discussion LGMs with small samples exhibit some similar patterns of bias to MLMs; both factor mean standard errors and factor variances are downwardly biased Model fit was not highly problematic if using the Swain correction with as little as 1.5 observations per parameter in the model. The Yuan correction did not encounter issues for any conditions in this study Still must be aware of parameter bias

23 Discussion II The MLM literature on small sample size is still far more advanced Includes REML estimation for unbiased variance components and Kenward-Roger for unbiased standard errors with as few as 5 people These procedures or their analog in LGMs are potential topics for future research, especially given the performance of the Yuan and Swain corrections for model fit MLMs do not have a widely used method for global model fit At the present time, for scenarios where either framework can be used, small samples may be better addressed in the MLM framework with REML and Kenward-Roger

24 Thank You! Questions or requests for the working paper: