Small-Sample Methods for Cluster-Robust Inference in School-Based Experiments

Size: px

Start display at page:

Download "Small-Sample Methods for Cluster-Robust Inference in School-Based Experiments"

Abel Patrick
5 years ago
Views:

1 Small-Sample Methods for Cluster-Robust Inference in School-Based Experiments James E. Pustejovsky UT Austin Educational Psychology Department Quantitative Methods Program Elizabeth Tipton Teachers College, Columbia University Dept. of Human Development March, 017 Society for Research on Educational Effectiveness Washington, DC

2 In brief Analysis of social experiments often requires handling dependencies among outcomes using: Multi-level modeling Regression with cluster-robust variance estimation (CRVE) Conventional CRVE behave poorly when the number of clusters is small, and small depends on the model. McCaffrey, Bell, & Botts (001; Bell & McCaffrey, 00) proposed biasreduced linearization variance estimator (BRL), Satterthwaite t-test Our work (Pustejovsky & Tipton, 017) extends BRL so that it works in panel models with fixed effects F-test for multi-parameter hypothesis tests software implementation in R and Stata (clubsandwich package)

3 Model Main impacts model: Y T β x e t ij 0 1 ij ij ij More generally, Models with multiple treatment indicators Treatment-by-covariate interactions In matrix form: Y Xβ e, Var e Σ, 1,..., i i i i i i n 3

4 Estimation Estimate β by weighted least squares: ˆ β M X W Y, X W X Standard CRVE: n n t t i i i M i i i i1 i1 n V M X W W X M n CR t t ˆ ˆ i i i i i i n 1 ee i1 Conventional to use n 1 degrees of freedom for t-tests. 1 4

5 Bias-reduced linearization Corrects V CR based on a working model for the error covariance structure: n BRL t t t V M X ˆˆ iwi Aiee i iaiwi Xi M i1 with adjustment matrices A 1,,A n chosen to satisfy BRL V β ˆ E Var Degrees of freedom corrections for hypothesis tests Satterthwaite d.f. for t-tests (Bell & McCaffrey, 00) Approximate Hotelling s T d.f. for F-test (Tipton & Pustejovsky, 015; Pustejovsky & Tipton, 017) 5

6 Approximate Hotelling Test We propose a generalization of the Satterthwaite approximation to the multi-dimensional case, with H : Cβ 0 0 Approximate the distribution of V BRL using a Wishart distribution with degrees of freedom η. Estimate η by matching mean and total variance of V BRL. F F AHT AHT q 1 ˆ q ~ F q, t BRL 1 Cβ CV C Cβ ˆ q 1 6

7 Effects of Tribes Learning Communities (Hanson et al., 011) Social-Emotional Learning curriculum. Classroom-level randomization to TRIBES or BAU control. 10 participating schools in Grades 1-. Original analysis used HLM with classroom level random effects, school fixed effects. 7

8 Effects of Tribes Learning Communities (Hanson et al., 011) OLS estimation (seemingly unrelated regressions) Cluster SEs by school Conventional CRVE Bias-Reduced Impact Linearization Est. Outcome (ES units) SE df p SE df p Aggressive behavior (T) Rule-breaking (T) Interpersonal strength (P) Intrapersonal strength (P) Joint test of outcomes Conventional: F(4, 9) = 6.8, p =.008 Bias-reduced linearization: F(4, 4.3) = 3.70, p =.109 8

9 Angrist & Lavy (009) Cluster-randomized trial in 40 high schools in Israel. Tested effects of monetary incentives on post-secondary matriculation exam (Bagrut) completion rates. Longitudinal data, difference-in-differences specification. Focus on effects for higher-achieving girls Hypothesis Test F df p-value treatment effect (q = 1) Moderation by school sector (q = ) Standard Satterthwaite Standard AHT

10 Further considerations Magnitude of SE adjustment and degrees of freedom depend on: Weighting Cluster sizes Balance Covariate distribution Given these complexities, we recommend applying small-sample adjustment by default when using CRVE. 10

11 Software R package clubsandwich Available on Comprehensive R Archive Network (v0..1) Development version at Works with a wide variety of models (lm, lme, plm) Stata package clubsandwich Available on Github: Wraps reg and areg 11

12 Future directions Performance comparisons versus other small-sample corrections Cluster-wild bootstrap (Cameron, Gelbach, & Miller, 008; MacKinnon & Webb, 016). Randomization tests (Canay, Romano, & Shaikh, 014). Other degrees-of-freedom corrections from GEE literature (e.g., Fay & Graubard, 001; Wang & Long, 011). Robust score (LM) tests. Extensions Instrumental variables (-stage least squares) GEE models Multi-way clustering (Cameron, Gelbach, & Miller, 011) 1

13 References Angrist, J. D., & Lavy, V. (009). The effects of high stakes high school achievement awards : Evidence from a randomized trial. American Economic Review, 99(4), Bell, R. M., & McCaffrey, D. F. (00). Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 8(), Cameron, A. C., Gelbach, J. B., and Miller, D. (008). Bootstrap-based improvements for inference with clustered errors. The Review of Economics and Statistics, 90(3): Cameron, A. C., Gelbach, J. B., & Miller, D. L. (011). Robust inference with multiway clustering. Journal of Business & Economic Statistics, 9(), Canay, I. A., Romano, J. P., & Shaikh, A. M. (014). Randomization tests under an approximate symmetry assumption. Working paper. Fay MP and Graubard BI. Small-sample adjustments for Wald-type tests using sandwich estimators. Biometrics 001;57: Hanson, Thomas L., Jo Ann Izu, Anthony Petrosino, Bo Delong-Cotty, and Hong Zheng. Outcome Evaluation of Tribes Learning Communities in California, ICPSR381-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], Imbens, G. W. and Kolesar, M. (016). Robust Standard Errors in Small Samples: Some Practical Advice. Review of Economics and Statistics, forthcoming. James-Burdurmy, Susanne. Randomized Experiment of Playworks Analytic Files for and Cohorts in Six United States Cities. ICPSR35638-v1. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], Lee, D. S., & Card, D. (008). Regression discontinuity inference with specification error. Journal of Econometrics, 14(), MacKinnon, J. G. and Webb, M. D. (016). Wild bootstrap inference for wildly different cluster sizes. Journal of Applied Econometrics, forthcoming. McCaffrey, D. F., Bell, R. M., & Botts, C. H. (001). Generalizations of biased reduced linearization. In Proceedings of the Annual Meeting of the American Statistical Association. Pustejovsky, James E. & Elizabeth Tipton (017). Small sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. Journal of Business and Economic Statistics. In Press. Tipton, E., & Pustejovsky, J. E. (015). Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. Journal of Educational and Behavioral Statistics, 40(6), Wang M and Long Q. Modified robust variance estimator for generalized estimating equations with improved small-sample performance. Statistics in Medicine 011;30(11):

14 Simulation results: Block-randomized trials Note: q is the dimension of the hypothesis test. Source: Pustejovsky & Tipton (017). 14

15 Simulation results: Cluster-randomized trials Note: q is the dimension of the hypothesis test. Source: Pustejovsky & Tipton (017). 15

16 Block-randomized/multi-site trials Model with block fixed effects: Y Overall impact estimate: T e ij i ij ij n n ˆ 1 w ˆ, W w W i i i1 i1 where መδ 1,, መδ n are treatment effect estimates from each block. i Conventional CRVE (clustering by block): V CR n 1 w ˆ i i1 W ˆ i 16

17 Block-randomized/multi-site trials (cont.) BRL correction: V BRL 1 W Satterthwaite df: n i1 i w ˆ ˆ i 1 w / W i df n n 3 n wi w 1 i w i i1 W wi W i1 W wi W i1 W w i 1 Satterthwaite df = n 1 if w j are equal (otherwise df < n 1). 17

18 Cluster-randomized trials Model (without covariates): Y T e ij 0 i ij Overall impact estimate: ˆ 1 1 ˆ ˆ W T n T n C T C wi i wi i i1 WC i1 where μƹ T 1,, Ƹ estimates. T μ nt and μƹ C 1,, Ƹ C μ nc are cluster-specific mean 18

19 Cluster-randomized trials (cont.) Conventional CRVE: V BRL correction: V 1 1 C n T n CR T T C C w ˆ ˆ ˆ ˆ i i w i i WT i1 WC j1 BRL W n T T C C T wi i i i ˆ ˆ ˆ ˆ n w 1 C 1 1 w / W W 1w / W T i1 i T C j1 i C If w i are approximately equal (cf. Imbens & Kolesaar, 016): df nt nc nt 1nC 1 n n 1 n n 1 T T C C 19

20 Effects of Playworks on school climate, student social skills and behavior (James-Burdurmy et al., 013) Structured physical activity and recess coaching program. 9 participating schools, grouped in 9 blocks School-level block randomization to Playworks or BAU control. 17 treatment schools 1 control schools OLS estimation, including block fixed effects Cluster SEs by school 0

21 Effects of Playworks on school climate, student social skills and behavior (James-Burdurmy et al., 013) Conventional CRVE Bias-Reduced Impact Linearization Est. Outcome (ES units) SE df p SE Df p Teacher support for organized play Staff support for organized play Student bullying/exclusion Difficult transitioning to learning after recess < < < <.001 Joint test of outcomes Conventional: F(4, 8) = 3.5, p <.001 Bias-reduced linearization: F(4, 9.0) = 10.6, p =.00 1

Small-sample adjustments for multiple-contrast hypothesis tests of meta-regressions using robust variance estimation

Small-sample adjustments for multiple-contrast hypothesis tests of meta-regressions using robust variance estimation James E. Pustejovsky The University of Texas at Austin Elizabeth Tipton Teachers College,