Terrence D. Jorgensen*, Alexander M. Schoemann, Brent McPherson, Mijke Rhemtulla, Wei Wu, & Todd D. Little

Transcription

1 Terrence D. Jorgensen*, Alexander M. Schoemann, Brent McPherson, Mijke Rhemtulla, Wei Wu, & Todd D. Little KU Center for Research Methods and Data Analysis (CRMDA) Presented 21 May 2013 at Modern Modeling Methods, Storrs, CT

2

3 Incorporating planned missing data into research designs has advantages: Cost-effective Reduces participant burden/fatigue Limits unplanned missingness Ensures that data are MCAR Three-form design Background Useful when measuring many items (e.g., several surveys) e.g., testing (in)direct effects among several constructs

4

5 Research Question At Wave 1, randomly assign participants to complete 1 of the 3 forms (XAB, XAC, or XBC) At subsequent waves, does it matter how you assign forms? Use the same form each wave? Use a different form each wave? Use random assignment each wave? Random assignment each time might take less effort Are practice effects attenuated by seeing different forms? Is systematic assignment more efficient than random?

6 Different patterns, but same proportions of missing data

7 Study 1 Baseline differences, given the best-case scenario No practice effects Cole & Maxwell s (2003) longitudinal mediation model Study 2 Outline Practice effects: mean-increase of Cohen s d = 0.10 for consecutively measured items Only for indicator intercepts; latent means remain 0 No practice effects in the covariance structure

8 Study 1: Method 3 assignment methods same, random, or different 436 combinations of population values 4 factor loadings: λ = 0.70, 0.75, 0.80, or indicators per factor 2 levels of proportion missing (11.1% or 22.2%) 4 within-time covariances: ψ = 0.2, 0.3, 0.4, or cross-lagged regressions: β = 0.0, 0.1, 0.2, 0.3, or autoregressive: β = 0.4, 0.5, 0.6, 0.7, 0.8, or 0.9 N = 540, standard multivariate normal data

9

10 Study 1: Results Given the following: items within a construct are distributed evenly across forms NO practice effects (unreasonable assumption) No practical differences were observed between different methods of assignment, with respect to: Point and SE estimates, absolute and relative bias MSE, 95% coverage rates Convergence rates

11

12

13

14 Study 1: Conclusion So in the absence of any practice effects, the assignment methods are functionally equivalent. However, the probability that practice effects are precisely zero (nonexistent) is ZERO!

15 3-factor CFA Study 2: Method Same construct measured at 3 times 24 combinations of population values: 3 assignment methods (same, random, or different) 4 factor loadings: λ = 0.70, 0.75, 0.80, or indicators per factor 2 levels of proportion missing (14.3% or 28.6%) 6 factor correlations: ψ = 0.4, 0.5, 0.6, 0.7, 0.8, or 0.9 N = 270, standard multivariate normal data

16

17 Study 2: Results Still no practical differences were observed between different methods of assignment, with respect to: Point and SE estimates, absolute and relative bias MSE, 95% coverage rates COVARIANCE STRUCTURE ONLY However, bias due to practice effects manifested in the mean structure Indicator intercepts increased when items were seen on consecutive occasions, but latent means were 0 When strong invariance model was fit, the increase in intercepts manifested as an increase in the latent mean

18

19 Conclusion Bias can manifest in many ways Mean increases on ability measures due to practice No reason to assume seeing different items won t lead practice effects greater correlations/regressions when items are measured in closer proximity or with greater frequency The best way to prevent such bias is to use a multiform design and assign different forms over time to prevent practice effects

20 Conclusion More forms means greater proportion MCAR, thus less potential for bias Perhaps in combination with a wave-missing design (yet to be studied) No reason for any scale items to be in X block Opens up possibilities to test for practice effects e.g., researchers could examine differences between X- set and A-set items from each time to the next

21 Acknowledgements CRMDA Missing Data Workgroup Paul Johnson for much help with parallel computing Support for this project provided by NSF grant (Wei Wu & Todd D. Little, co-pis) CRMDA at the University of Kansas (Todd D. Little, director) Look for the manuscript in a special issue of International Journal of Behavioral Development

22 References Cole, D. A., & Maxwell, S. E. (2003). Testing mediational models with longitudinal data: Questions and tips in the use of structural equation modeling. Journal of Abnormal Psychology, 112(4), doi: / x All data were simulated using the R package simsem, developed at KU, which fits models using lavaan (can also use OpenMx). Pornprasertmanit, S., Miller, P., & Schoemann, A. M. (2012). simsem: SIMulated structural equation modeling (version 0.4-1) [R package]. Available from the Comprehensive R Archive Network: Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, Available from

23 Contact Information I m happy to share these slides, or a poster presentation of this material (SRCD 2013) tdj@ku.edu