Addressing Analysis Issues REGRESSION-DISCONTINUITY (RD) DESIGN

Transcription

1 Addressing Analysis Issues REGRESSION-DISCONTINUITY (RD) DESIGN

2 Overview Assumptions of RD Causal estimand of interest Discuss common analysis issues In the afternoon, you will have the opportunity to work through the analysis issues I talk about today

3 The Regression Discontinuity Design Comparison Treatment Discontinuity, or treatment effect Counterfactual regression line

4 Treatment Assignment Process Units are assigned to treatment on the basis of a cutoff score (z c ) on a continuous assignment variable (Z) Unit i receives treatment condition if the unit scores above the cutoff ( =1 if Z z c ) Unit i receives control condition if the unit scores below the cutoff ( =0 if Z < z c )

5 Causal Estimand of Interest The treatment effect is the difference in the potential outcomes at the cutoff: t ATE(C) = E[Y i (1) -Y i (0) = z c ] = E[Y i (1) = z z ]- E[Y i (0) = z c ] This is the average treatment effect at the cutoff.

6 RD Estimates the Average Treatment Effect at the Cutoff Comparison Treatment Discontinuity, or treatment effect Counterfactual regression line

7 RD Design Assumptions 1. Probability of treatment is discontinuous at the cutoff 2. No alternative explanations for the treatment effect except through the treatment 3. Non-interference between units

8 Why People Like RDD Theoretical warrants 1. Selection process is known 2. It is like an experiment around the cutoff Empirical warrants Validated by 7 within-study comparisons (see Cook and Wong (2010 ) for review) Deeper point is that assumptions for a causally valid RD are transparent, and can often be probed empirically be bed empirically.

9 Example: Five State Pre-K Study Wong, Cook, Barnett, & Jung (2008) Assignment to pre-k based on children s date of birth and state cutoff Data gathered in by National Institute for Early Education Research (NIEER) Comparison group consisted of kids who were entering pre-k in fall of 2004 Treatment group consisted of kids who were entering kindergarten in 2004 (and were in pre-k last year)

10 Sample of States Program started Spent per student % of 4 y.o. enrolled Duration of program Teacher Ed Michigan 1985 $3,367 19% Half-day BA New Jersey 1998 $10,361 79% in Abbotts Full day BA Oklahoma 1990 universal in 1998 $2,517 65% Varied BA South Carolina 1984 $1,575 32% Half-day BA West Virginia 1983 universal by 2010 $6,829 33% Varied BA or AA

11 school year Sampling design: Five State Pre-K Study Sampling design Purposive sampling of 5 states with high quality standards for pre-k Simple random sampling of classrooms in 4 states, stratified random sampling of classrooms in New Jersey Four children were randomly chosen from each classroom

12 Outcome Measures Peabody Picture Vocabulary Test (PPVT) Early Math (Woodcock-Johnson Subtest of Applied Problems) Print Awareness

13 Visual Depiction of Pre-K Study Control children Pre-Kindergarten children

14 RD Implementation Issues 1. Correct specification of response function 2. Treatment non-compliance 3. Limited generalization of treatment effects 4. Discontinuity in potential outcomes at the cutoff 5. Reduced statistical power

15 Issue 1: Correct specification of the functional form Recall that anything that affects the size of the discontinuity at the cutoff besides treatment can lead to a spurious effect Suppose we assume that the relationship between the assignment variable and outcome is linear But it is actually quadratic! This could result in a biased estimate of the treatment effect

16

17

18

19 Estimation strategies 1. Parametric approaches: Y i = a +t + f ( ) + C i b + e Use regression to model the outcome as a function of treatment status, the assignment variable, and the interaction of the two Requires correct modeling of response function 2. Non-parametric approaches t = lim E[Y i = z c ]- lim E[Y i = z z ] z z Use local linear c z z kernel regression c to estimate the difference in the limits for the treatment and control groups at the cutoff No assumptions about functional form, but difficult to include baseline covariates

20 Estimation strategies 3. Semi-parametric approaches Y i = a +t + f ( ) + C i b + e Implemented as a semi-parametric partially linear model Allows for flexibility in the response function while controlling for baseline covariates

21 Analytic strategies Model Linear regression Quadratic regression Cubic regression Local linear kernel regression (at =0) Partial linear regression (at =0) Equation Y i = a +t + b 1 + b 2 + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 + b 4 Z 2 i + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 Z 3 i + b 4 + b 5 Z 2 i + b 6 Z 3 i + C ' ig + e i Y i = a +t + b 1 + b 2 + e i Y i = a +t + g( ) + C ' ig + e i with triangular kernel weights K( /h) with triangular kernel weights K( /h)

22 Analytic strategies Model Linear regression Quadratic regression Cubic regression Local linear kernel regression (at =0) Partial linear regression (at =0) Equation Y i = a +t + b 1 + b 2 + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 + b 4 Z 2 i + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 Z 3 i + b 4 + b 5 Z 2 i + b 6 Z 3 i + C ' ig + e i Y i = a +t + b 1 + b 2 + e i Y i = a +t + g( ) + C ' ig + e i with triangular kernel weights K( /h) with triangular kernel weights K( /h)

23 Analytic strategies Model Linear regression Quadratic regression Cubic regression Local linear kernel regression (at =0) Partial linear regression (at =0) Equation Y i = a +t + b 1 + b 2 + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 + b 4 Z 2 i + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 Z 3 i + b 4 + b 5 Z 2 i + b 6 Z 3 i + C ' ig + e i Y i = a +t + b 1 + b 2 + e i Y i = a +t + g( ) + C ' ig + e i with triangular kernel weights K( /h) with triangular kernel weights K( /h)

24 Analytic strategies Model Linear regression Quadratic regression Cubic regression Local linear kernel regression (at =0) Partial linear regression (at =0) Equation Y i = a +t + b 1 + b 2 + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 + b 4 Z 2 i + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 Z 3 i + b 4 + b 5 Z 2 i + b 6 Z 3 i + C ' ig + e i Y i = a +t + b 1 + b 2 + e i Y i = a +t + g( ) + C ' ig + e i with triangular kernel weights K( /h) with triangular kernel weights K( /h)

25 Analytic strategies Model Linear regression Quadratic regression Cubic regression Local linear kernel regression (at =0) Partial linear regression (at =0) Equation Y i = a +t + b 1 + b 2 + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 + b 4 Z 2 i + C ' ig + e i Y i = a +t + b 1 + b 2 Z 2 i + b 3 Z 3 i + b 4 + b 5 Z 2 i + b 6 Z 3 i + C ' ig + e i Y i = a +t + b 1 + b 2 + e i Y i = a +t + g( ) + C ' ig + e i with triangular kernel weights K( /h) with triangular kernel weights K( /h)

26 Check Functional Form Using Multiple Specifications Results from New Jersey PPVT Treatment Effect (1) Linear regression 5.71* (1.44) (2) Quadratic regression 5.36* (2.02) (3) Cubic regression 5.26* (2.71) (4) Local linear regression 6.98 (3.92) (5) Partial linear regression 5.47 (4.14) * p <.05

27 Add Pretest to Check Functional Form Comparison Treatment

28 Add Pretest to Check Functional Form Use pretest to check shape of the response function across distribution of assignment variable Allow for a simple time period fixed effect, but assume that the functional form did not change between two time periods Key assumption is that the pretest function is time invariant except for an intercept shift

29 Issue #2: Treatment noncompliance Occurs when units that are assigned to the control condition take up the treatment, and units assigned to the treatment condition fail to show up Could bias treatment effects downwards In pre-k, red shirting and green shirting are versions of treatment non-compliance (Bassok & Reardon, 2011) In a sharp RD, the probability of treatment receipt changes discontinuously from 1 to 0 In fuzzy RD, the discontinuity in the probability of treatment receipt is less than 1

30 No-Show and Cross-Over Rates (bandwidth = 14 days) RD Treatment RD Control Received Treatment Did Not Receive Treatment

31 Panel 1: Full compliance Comparison students Cutoff Pre-K students Panel 2: 10% non-compliance Comparison students Cutoff Pre-K students

32 Noncompliance in Pre-K Study

33 Addressing Noncompliance in RD Hahn, Todd, and van der Klaauw (2001) show that when there is fuzziness, we can use instrumental variable analysis to estimate the local average treatment effect: lim z z c E[Y i z i = z c ]- lim z z c E[Y i z i - z c ] lim z z c E[ z i = z c ]- lim z z c E[ z i - z c ]

34 Addressing Noncompliance in RD lim z z c E[Y i z i = z c ]- lim z z c E[Y i z i - z c ] lim z z c E[ z i = z c ]- lim z z c E[ z i - z c ] RD ITT Effect

35 Addressing Noncompliance in RD lim z z c E[Y i z i = z c ]- lim z z c E[Y i z i - z c ] lim z z c E[ z i = z c ]- lim z z c E[ z i - z c ] ITT-D = =.84 RD Treatment RD Control Received Treatment Did Not Receive Treatment

36 Requirements for Estimating Treatment Effects in Fuzzy RDDs Similar to how non-compliance is handled in randomized experiments, except ITT and ITT-D are estimated at the RD cutoff Requires the following: 1. That the instrument (treatment assignment) is correlated with treatment receipt; and 2. That the instrument is not correlated with errors in the outcome model 3. There are no defiers

37 PPVT Summary of Analytic Strategy New Jersey Treatment Effect (1) Linear regression 5.71* (1.44) (2) Quadratic regression 5.36* (2.02) (3) Cubic regression 5.26* (2.71) (4) Local linear regression 6.98 (3.92) (5) Partial linear regression 5.47 (4.14) (6) Wald estimate 6.99 (4.02) (7) Two stage least squares 6.10* (1.44) * p <.05

38 Issue 3: Limited Generalization of Treatment Effects at the Cutoff Comparison Treatment

39 Oklahoma Pre-K Example Treatment Effect Estimated at School Cutoff 9/1/2009 Kid i Birthday Y i (0) School Readiness if Kid Does Not Attend Pre-K Y i (1) School Readiness if Kid Attends Pre-K Kid 1 1/1/ Kid 2 3/3/ Kid 3 8/31/09? 30 Kid 4 8/31/09? 15 Kid 5 9/1/09 10? Kid 6 9/1/09 15? Kid 7 11/1/ Average

40 Outcome Approach 1: Use Multiple Cutoffs All sites assign treatment based on an assignment score and cutoff Assignment variable is the same across sites But cutoffs vary across sites Identify average treatment effects across an interval on the assignment variable (the discontinuity frontier) Average treatment effect across a frontier Site 1 Site 2 Site 3 Site 4 Site Assignment variable

41 School Cutoffs in Pre-K Evaluation Study State School Cutoff Dates Michigan 12/1/2003 Oklahoma 9/1/2003 New Jersey 9/30/03 10/1/03 10/15/03 10/31/03 11/1/03 11/30/99 12/31/99 West Virginia 9/1/2003 South Carolina 9/1/2003

42 Outcome Analyzing RDs with Multiple Cutoffs 1. For each kid, center the assignment score (birthdays) at the district cutoff 2. Pool observations of all kids into a single dataset 3. Analyze as a two dimensional RD with a single assignment variable and a cutoff at zero 4. Estimate treatment effect across a range of the assignment variable using pooled data 0 Assignment variable

43 Approach 2: Add Pretest to Improve Generalization Average treatment effect at cutoff Average treatment effect of treated

44 Add Pretest to Improve Generalization Use pretest to check shape of the response function across range of assignment variable Allow for a simple time period fixed effect, but assume that the functional form did not change between two time periods Using pretest data, you can extend the untreated outcome line beyond cutoff to calculate average treatment effect for the treated Key assumption is that the pretest function is time invariant except for an intercept shift

45 Issue 3: Potential Confounds at the Cutoff Any jump in the regression line at the cutoff should be due to the treatment and nothing else There are lots of ways this could fail to be true Many of these problems are detectable, some are solvable and others are not Important to be specific about threats

46 Potential Confounders in Pre-K Study Recall that measurement place took place in the fall of 2004 for both treatment and comparison kids Three plausible threats

47 Diagnostic Test #1 Compare Compositional Differences at the Cutoff Threat 1: Attrition children as movers We may be concerned that movers left the state or entered private school, and were different from children who stayed What to do? Check for compositional differences between the two groups at the cutoff Run an RD of baseline covariates and the assignment variable

48 Panel 1: Proportion of high SES kids without attrition Comparison students Cutoff Pre-K students Panel 2: Proportion of high SES kids with attrition 1.5 Attrition kids 0 Comparison students Cutoff Pre-K students

49

50 Diagnostic Test #1 Compare Compositional Differences at the Cutoff Is the mover story a problem for the RD? No significant differences in any of our control covariates at the cutoff (within states and across states) Treatment effects for parametric and semi-parametric estimates were robust to whether control covariates were included in the model or not But only a partial test

51 Diagnostic Test #2 Examine Density Functions at the Cutoff Threat 2: Younger children may have disproportionally attrited from the sample, and age is correlated with children s achievement The hypotheses here is that young children in the treatment group are now missing What to do? We can examine this empirically by looking for dips in the number (density) of cases at or above the cutoff Density test can also provide important clues as to whether selection occurred on other unobserved variables

52 Number of births Number of births Panel 1: Distribution of births with no attrition Comparison students Cutoff Pre-K students Panel 2: Distribution of births with young kids as attriters Attrition kids Comparison students Cutoff Pre-K students

53 *

54 Density Test Assess discontinuity at threshold by estimating log difference ˆq in the height of the intercept ˆ ln fˆ ln fˆ ˆf + ˆf - Where and are the estimated density values using non-parametric regression immediately above and below the cutoff. Formal t-test that H 0 : ˆq = 0 using the estimated log difference and bootstrapped standard errors with 500 replications

55 Interpreting Density Tests Is the young kids story a problem for the RD? Unclear Discontinuities in the density functions may represent actual discontinuities in the population We can check this by using birth records from our sample states to check the distribution of births in the overall population around the cutoff period If such discontinuities exist, the question then is why?

56 Diagnostic #3 Examine Density Functions at the Cutoff Threat 3: Sorting around the school cutoff Assume there is no gaming of school cutoffs (Dickert-Conlin & Elder, 2010) But, there is evidence that health workers and parents may avoid births on weekends and holidays What to do? Check to see whether school cutoffs coincide with weekends or holidays that may introduce a discontinuity in the density of observations at the cutoff

57 Weekday versus Friday Cutoff Dates

58 New Jersey Friday Birth Cutoff 10/1/99 Weekend

59 Diagnostic #3 Density Test Weekend effects are a problem if: 1. The distribution of children s birthdates is not uniform 2. Kids born on weekend have different attributes from those who are born on weekdays 3. These attributes are correlated with children s later achievement

60 Issue 4: Reduced statistical power RD has less statistical power than an RE by a factor of 3 5 times (Goldberger, 1972; Cappelleri et al. 1994; Bloom et al., 2005; Schochet, 2009); This means that for the RD to detect the same effects, the RD sample has to be 3-5 times the size of the RE. Similar to randomized experiments, pretest and baseline covariates can drastically improve precision by explaining variation in the outcome

61 Variance of Impact Estimator in RA ^ Var ra (α 0 ) = σ 2 (1-R ra2 ) np(1-p) σ 2 is the variance of mean student outcomes across schools within the treatment or comparison group R ra2 is the square of the correlation between school outcomes and the assignment variable np(1-p) is the total variation in treatment status across schools

62 Variance of Impact Estimator in RDD Var rd (α^ 1 ) = σ 2 (1-R rd2 ) np(1-p)(1-r ts2 ) All other terms are the same, except: R ts2 is the square of the correlation between school treatment status and assignment variable

63 Power Considerations Common to Randomized Experiments and RDD 1. Sample size 2. Distribution of outcome variable 3. Misallocation rates 4. R-square of school- and student-level covariates 5. Clustered designs and intra-class correlations

64 Power Considerations Unique to RDD 1. Bandwidth around cutoff 2. Non-linear response function 3. Shape of the distribution around cutoff 4. Location of cutoff in distribution 5. Balance of treatment and comparison

65 Power Considerations Unique to RDD 1. Bandwidth around cutoff 2. Non-linear response function 3. Shape of the distribution around cutoff 4. Location of cutoff in distribution 5. Balance of treatment and comparison

66 Special Role of the Pretest In RD Note that the pretest can be used to address most major concerns in the RD design: 1. Help check functional form assumption 2. Improve generalization of treatment effects 3. Identify sorting problems at the cutoff 4. Improve statistical precision of treatment effect

67 Summary: Empirical Tests Specification of the functional form Apply parametric, non-parametric, semi-parametric approaches for analyzing data Misallocation of treatment (Fuzzy RD) Inspect data to assess discontinuities in the probability of treatment receipt at the cutoff Use treatment assignment as an instrument for treatment receipt to estimate the local average treatment effect at the cutoff Differential attrition Examine discontinuities in third variables at cutoff Creative use of McCrary density tests Sorting at the cutoff McCrary density tests