Statistical Power and Autocorrelation for Short, Comparative Interrupted Time Series Designs with Aggregate Data

Transcription

1 Statistical Power and Autocorrelation for Short, Comparative Interrupted Time Series Designs with Aggregate Data Andrew Swanlund, American Institutes for Research Kelly Hallberg, University of Chicago Ryan Williams, American Institutes for Research

2 Background In designing studies, applied researchers must be able to assess whether their study will be sensitive enough to detect the effects they are purporting to study. In education research, a robust literature has developed to explore the design sensitivity of randomized controlled trials. Methodologists have detailed the factors that influence the power of a given study to detect effects (e.g. Raudenbush, 1997; Schochet, 2005; Bloom, Richburg-Hayes, & Black, 2007; Raudenbush, Spybrook, Martinez, & Blooom, 2011), and have provided empirical parameters to guide power calculations (e.g. Hedges & Hedberg, 2007; Hill, Bloom, Black, & Lipsey, 2008). Less work has been done on design sensitivity for CITS. Bloom (1999; 2003) and Dong and Maynard, (2012) have done the most extensive work on power in CITS to date. They derive power equations for CITS that incorporate student level data for a baseline linear trend model. In recent years, longitudinal, aggregate student achievement scores have become increasingly available on public education department websites. Jacob, Goddard, and Kim (2013) explored the usefulness of such aggregate data and found them to be valid for studying educational policy. Recently, Jacob, Somers, Zhu, and Bloom (2016) examined the validity of CITS designs employing aggregated data. However, using school level data rather than student level data has implications for the distribution of variance and thus statistical power. Furthermore, little work has been done on the impact of autocorrelation in short CITS designs (as opposed to long time series in general). Purpose Our paper will build on this work in several ways. First, we derive power calculations for two CITS model specifications, and discuss general influences on the power of such designs. Second, we explore the impact of autocorrelation on standard errors, bias, and coverage probabilities. Third, we use real world data to study the extent to which autocorrelation is apparent in schoollevel aggregate achievement data. Research Design Our research includes both mathematical derivations of the variance of the treatment effect estimate in two CITS designs as well as Monte Carlo simulations involving both real world data and data simulated to have autocorrelation (autoregressive first order). Due to space limits, the derivations are not included. A simple baseline mean CITS model takes the following form: Y ij = β 0 + β 1 TX j + β 2 Post ij + β 3 Post ij TX j + u j + ε ij This model includes random effects for schools, and the treatment effect is represented by β3 with variance: (n pre + n post )σ 2 var(β 3) = (S T + S C )(1 P T )P T n pre n post

3 Where n (pre/post) is the number of pre and post measurements, St and Sc are the number of treatment and comparison schools, Pt is the proportion of treatment schools, and σ 2 is the withinschool variance. The baseline linear trend can be written as follows: Y ij = β 0 + β 1 TX j + β 2 time ij + β 3 Post ij + β 4 TX j time ij + β 5 Post ij TX j + u j + ε ij Where the terms are as above, with time representing the year for a particular measurement. Here the treatment effect is estimated by β5, with variance var(β ) 5 = σ 2 (n pre + n post ) 2 2 σ t ( (S T + S C )(1 P T )P 2 2 T n pre n post (n pre σ tpre + n post σ tpost ) ) Where terms are defined as above, and σ t 2 is the variance of the time variable. In addition to the above equations, we also examined how the variance term would change with the introduction of autocorrelation. To date, we have been unable to simplify this equation to a reasonable degree, and therefore, we look at the impact of autocorrelation of inference through a series of simulations. Data Sources and Analysis To test the extent to which autocorrelation affected inference for short CITS, we simulated school aggregate data with an AR(1) correlation structure. Data were generated with the following specifications: 20 schools vs 50 schools (half treatment schools) 3 pre (with one or two follow-ups); 5 pre (with one or two follow-ups) Autocorrelation sampled uniformly between.1 and.9 These specifications yielded 8 data frameworks, each of which was simulated 1000 times. For each simulation, we fit a baseline mean baseline linear trend model using OLS (with school fixed effects), a mixed effects model (with school random effects) with uncorrelated error structure (which we posit is the typical way these models are analyzed in practice), and a mixed effects model with AR(1) error structure. For each simulation, we examined the bias from the true treatment effect, the root mean squared error, and coverage probability of the confidence interval. To test the impact of autocorrelation (and to estimate its prevalence), we ran simulations using publically available school/grade-aggregate mathematics data from Minnesota in grades 3 through 8 (we will be expanding on this prior to the conference). For each grade, we sampled either 20 or 50 schools (1000 times), with 4 pre and 1 post time point, and calculated the same terms as above (also capturing the estimate of the autocorrelation coefficient).

4 Findings The data analysis yielded the following major findings: The number of pre and post data points, along with the balance between pre and post data points, affects power. Between school variance does not factor into statistical power (although it would for determining effect sizes in terms of student standard deviations) Autocorrelation affects statistical inference, even for short CITS designs. Failing to account for autocorrelation yields greater bias (see tables 2 and 4), and confidence intervals with low coverage probabilities (see tables 1 and 3). Fitting a baseline linear trend can help with the coverage probability as autocorrelation manifests as a trend in the short-term, but results are still more biased. Real world data suggests that some autocorrelation is present in aggregate school-level data (see table 6). Our simulations showed a wide range of autocorrelation, with the average being around.25 in our data. The difference in the modeling approaches in terms of bias and coverage probability was less apparent with real world data (see table 5), which may be due to the data not strictly following an autoregressive first order pattern. Conclusions Our derivation of statistical formulae for use in power analysis can help in the design and implementation of CITS studies. In addition, our simulation work suggests that autocorrelation can degrade statistical inference for short CITS designs; however, the amount of autocorrelation varies significantly in practice. This would suggest implementing models with error structures beyond uncorrelated as a robustness check on CITS model results.

5 References Bloom, H. S. (1999). Estimating program impacts on student achievement using short interrupted time series. MDRC. New York, NY. Retrieved from Bloom, H. S. (2003). Using short interrupted time-series analysis to measure the impacts of whole-school reforms: With applications to a study of accelerated schools. Evaluation Review, 27, Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), doi: / Dong, N., & Maynard, R. (2013). PowerUP!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning grouprandomized trials in education. Educational Evaluation and Policy Analysis, 29, 1, Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), doi: /j x Jacob, R. T., Goddard, R. D., & Kim, E. S. (2013). Assessing the use of aggregate data in the evaluation of school-based interventions: Implications for evaluation research and state policy regarding public-use data. Educational Evaluation and Policy Analysis, 36(1), Jacob, R. T., Somers, M. A., Zhu, P. & Bloom, H. S. (2016). The validity of the comparative interrupted time series design for evaluating the effect of school-level interventions. Evaluation Review, 40(4), Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173. Raudenbush, S. W., Spybrook, J., Congdon, R., Liu, X., Martinez, A., Bloom, H., et al. (2011). Optimal design software for multi-level and longitudinal research (Version 3.01) [Software]. Schochet, P. Z. (2005). Statistical power for random assignment evaluations of education programs. Princeton, NJ: Mathematica Policy Research, Inc.

6 Tables Table 1. by Error Structure and Autocorrelation Coefficient Baseline Mean Model Pre- Post Schools Error Structure (.1 < rho <.3) (.3 < rho <.5) (.5 < rho <.7) (.7 < rho <.9) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1)

7 Table 2. Root Mean Squared Error (RMSE) by Error Structure and Autocorrelation Coefficient Baseline Mean Model Pre- Post Schools Error Structure RMSE (.1 < rho <.3) RMSE (.3 < rho <.5) RMSE (.5 < rho <.7) RMSE (.7 < rho <.9) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1)

8 Table 3. by Error Structure and Autocorrelation Coefficient Baseline Linear Trend Model Pre- Post Schools Error Structure (.1 < rho <.3) (.3 < rho <.5) (.5 < rho <.7) (.7 < rho <.9) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1)

9 Table 4. Root Mean Squared Error (RMSE) by Error Structure and Autocorrelation Coefficient Baseline Linear Trend Model Pre- Post Schools Error Structure RMSE (.1 < rho <.3) RMSE (.3 < rho <.5) RMSE (.5 < rho <.7) RMSE (.7 < rho <.9) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Table 5. RMSE and : 4 pre and 1 post, Minnesota Mathematics Aggregate Data by Grade, Model, and Error Structure 20 Schools 50 Schools RMSE RMSE Grade Model Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) Uncorrelated AR(1) 3 Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean BLT

10 Table 6. Descriptive Statistics for Autocorrelation Coefficient: 4 pre and 1 post, Minnesota Mathematics Aggregate Data by Grade and Model Schools Grade Model Mean SD Min Max 20 3 Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend Baseline Mean Baseline Linear Trend