Regression Discontinuity Design

Transcription

1 Chapter 11 Regression Discontinuity Design 11.1 Introduction The idea in Regression Discontinuity Design (RDD) is to estimate a treatment effect where the treatment is determined by whether as observed assignment (or forcing or running variable) exceeds a known cutoff point. 1 Thistlethwaite and Campbell (1960) introduced RDD by analyzing the impact of merit awards of future economic outcomes using the fact that the allocation of these awards was based on an observed test score. The key idea behind RDD is that individuals with scores just below the cutoff (who did not receive the award) were good comparisons to those just above the cutoff (who did receive the award). Even though this evaluation strategy has been around for almost fifty years, it did not attract attention until relatively recently (Lee and Lemieux, 2010). RDD has been used exploit threshold rules to study education outcomes, exploit the presence of discontinuities at the geographical area to estimate willingness to pay for good schools, as well as other examples in labor supply, unemployment insurance, the effects of Medicaid and unionization. One important reason behind this surge is that RDD requires seemingly mild assumptions compared to those needed for other nonexperimental approaches. Another reason is the belief that RDD is not just another evaluation strategy, and that causal inferences from RDD are potentially more credible than those from typical natural experiment strategies (e.g., difference-in-differences or instrumental variables), which have been widely applied for research. Lee (2008) justifies this notion and formally shows that one need not assume that RDD isolates treatment variation that is as good as randomized, instead such randomization is a consequence of agents inability to precisely control the assignment near the known cutoff. Following most of the RDD literature in this chapter we refer to X as the assignment variable. Then the treatment is assignment to individuals with a value of X 1 This chapters follows the structure of Lee and Lemieux (2010) very closely. Refer to this article for a more extensive discussion. 93

2 94 11 Regression Discontinuity Design greater than or equal to a cutoff value c. These are key elements to keep in mind when using RDD: 1. RDD can be invalid if individuals can precisely manipulate the assignment variable. The existence of a treatment being a discontinuous function of the assignment variable is not sufficient to justify the validity of an RDD. 2. If individuals (even while having some influence) are unable to precisely manipulate the assignment variable, a consequence of this is that variation in the treatment near the threshold is randomized as though from a randomized experiment. Even if some individuals are specially likely to have values of X near the cutoff, every individual will have approximately the same probability of having an X that is just above or just below the cutoff. Note that when using an IV approach under exact identification one must assume that the instrument is exogenously generated (as if by a coin-flip). The variation in RDD isolates the randomized as a consequence of the assumption that individuals have imprecise control over the assignment variable. 3. RDD can be analyzed (and tested) like randomized experiments. If variation in the treatment near the threshold is approximately randomized, then it follows that baseline characteristics (all those variables determined prior to the realization of the assignment variable) should have the same distributions just above and just below the cutoff. Thus, the baseline covariates are used to test the validity of the RDD. Notice that when using IV or a matching/regression-control strategy, assumptions typically need to be made about the relationship of these other covariates to the treatment and outcome variables. 4. Graphical representation of an RDD is helpful and informative, but the visual presentation should not be tilted toward either finding an effect or finding no effect. A graph can also give the reader a sense of whether the jump in the outcome variable at the cutoff is unusually large compared to the bumps in the regression curve away from the cutoff. 5. Nonperametric estimation does not represent a solution to functional form issues raised by RDD. It is therefore helpful to view it as a complement to (rather than a substitute for) parametric estimation. There might be some functions where a low-order polynomial is a very good approximation and produces little or no bias, and therefore it is efficient to use all data points (both close to and far away from the threshold). For example, the procedure of regressing the outcome Y on X and a treatment dummy D can be viewed as a parametric regression or as a local linear regression with a very large bandwidth. 6. Goodness-of-fit and other statistical tests can help rule out overly restrictive specification Intuition of Sharp RDD One of the main virtues of the RDD approach is that it can be naturally presented using simple graphs, which greatly enhances its credibility and transparency. Consider

3 11.1 Introduction 95 Fig Simple Linear RD Setup (Lee and Lemieux, 2010). the RDD introduced by Thistlethwaite and Campbell (1960) in their study of impact merit awards on the future academic outcomes. Their study exploited the fact that these awards were allocated on the bases of an observed test score. Students with test scores X, greater than or equal to a cutoff value c, received the award, while those with scores below the cutoff were denied the award. This generated a sharp discontinuity in the treatment (receiving the award) as a function of the test score. Let the receipt of treatment be denoted by the dummy variable D {0,1}, so that we have D=1 if X c and D=0 if X < c. At the same time, there appears to be no reason, other than the merit award, for for future academic outcomes, Y, to be discontinuous function of the test score. This simple reasoning suggests attributing the discontinuous jump in Y at c to the causal effect of the merit award. Assuming that the relationship between Y and X is otherwise linear, a simple way of estimating the treatment effect τ is by fitting the linear regression: Y = α+ Dτ+ Xβ + ε (11.1) where ε is the usual error term that can be viewed as a purely random error generating variation in the value of Y around the regression line α + Dτ + Xβ. This is viewed in Figure In this example it is reasonable to assume that all factors (other than the award) are evolving smoothly with respect to X, then B would be a reasonable guess for the value of Y of an individual scoring c (and receiving the treatment) while A would be the reasonable guess for the same individual in

4 96 11 Regression Discontinuity Design the counterfactual state of not having received the treatment. It follows that B A would be the causal estimate. This illustrates the intuition that the RD estimates should use observations close to the cutoff. In practice one cannot only use data close to the cutoff because the narrower the area that is examined, the less data there are. This illustrates two important features of the RDD. First, in order for this approach to work all factors determining Y must evolve smoothly with respect to X. Second, since an RD estimate requires data requires data away from the cutoff, the estimate will be dependent on the chosen functional form Potential Outcomes Framework Hahn, Todd, and van der Klaauw (2001) noted that the key assumption of a valid RDD was that all other factors were continuous with respect to X, and suggested an nonparametric procedure for estimating τ that did not assume underlying linearity as in the simple example of Figure The necessity of the continuity assumption is seen more formally using the potential outcomes framework of the treatment effects literature. It is typically imagined that, for each individual i, there exists a pair of potential outcomes: Y i (1) for what would occur if the unit were exposed to the treatment and Y i (0) if not exposed. The causal effect of the treatment is represented by the difference Y i (1) Y i (0). The fundamental problem with causal inference is that we cannot observe the pair Y i (0) and Y i (1) at the same time. We therefore typically focus on average effects of the treatment, that is, averages of Y i (1) Y i (0) over (sub-)populations, rather than on unit-level effects. In the RD setting, we can imagine there are two underlying relationships between average outcomes and X, represented by E[Y i (1) X] and E[Y i (0) X], as in Figure It is easy to see that with what is observed, we could try to estimate the quantity which would equal B A=lim ε 0 E[Y i X = c+ε] lim ε 0 E[Y i X = c+ε], (11.2) E[Y i (1)= Y i (0) X = c] (11.3) This is the average treatment effect at the cutoff c. This is possible because of the continuity of the underlying functions E[Y i (1) X] and E[Y i (0) X]. Note that in standard regression analysis we assume that all relevant regressors are controlled for, and that no omitted variables are correlated with the treatment dummy. In an RD design, however, this crucial assumption is trivially satisfied. When X c, the treatment dummy is always equal to 1. When X < c, D is always equal to 0. Conditional on X, there is no variation left in D, so it cannot, therefore, be correlated with any other factor.

5 11.1 Introduction 97 Fig Assignment Variable X (Lee and Lemieux, 2010) A Comparison of RDD and Other Identification Strategies In this section we compare the RDD with other evaluation approaches. Consider the randomized experiment where the subjects where the subjects are assigned to a random number X and are given the treatment if X c. By construction, X is independent and not systematically related to any observable or unobservable characteristic determined prior to the randomization. This is illustrated in panel A of Figure The first column shows the relationship between the treatment variable D and X, a step function, going from 0 to 1 at the X = c threshold. The second column shows the relationship between the observables W and X. This is flat because X is completely randomized. The same is true for the unobservable variable U, depicted in the third column. These three graphs capture the appeal of the randomized experiment: treatment varies while all other factors are kept constant (on average). Panel B of Figure 11.3 considers the RDD where individuals have imprecise control over X. Both W and U may be systematically related to X, perhaps to the actions taken by units to increase their probability of receiving treatment. Whatever the shape of the relation, as long as individuals have imprecise control over X, the relationship will be continuous. And therefore, as we examine Y near the X = c cutoff, we can be assured that like an experiment, treatment varies (the first column) while other factors are kept constant. Consider the following model: Y = Dτ+Wδ 1 +U (11.4)

6 98 11 Regression Discontinuity Design Fig Treatment, Observables, and Unobservables in Randomized Experiment and RDD (Lee and Lemieux, 2010). D=1 [X c] (11.5) X = Wδ 2 +V (11.6) The basic idea of the selection on unobservables approach is to adjust for differences in the W s between treated and control individuals. It is usually motivated by the fact that it seems implausible that the unconditional mean of Y for the control group represents a valid counterfactual for the treatment group. So it is argued that, conditional on W, treatment-control contrasts may identify the (W-specific) treatment effect. The underlying assumption is that conditional on W, U and V are independent. The selection on observables approach is illustrated in panel C of Figure Observables W can help predict the probability of treatment (first column), but ultimately one must assume that unobservable factors U must be the same for treated and control units for every value of W. That is, the crucial assumption is that the two lines in the third column be on top of each other. Importantly, there is no comparable graph in the second column because there is no way to test the design since all the W s are use for estimation. A less restrictive assumption is to allow U and V to be correlated, conditional on W. But because of the more realistic /flexible data generating process, another assumption is needed to identify τ. One such assumption is that some elements of W (call them Z) enter the selection equation, but not the outcome equation and are uncorrelated with U. This situation is illustrated in panel D of Figure It is

7 11.2 Regression-Discontinuity Design Approach to Price Discrimination 99 Fig Treatment, Observables, and Unobservables in Matching on Observables and IV (Lee and Lemieux, 2010). necessary that the instrument Z is related to the treatment (as in the first column). The crucial assumption is regarding the relation between Z and the unobservables U (the third column). In order for an IV approach to work, the function in the third column needs to be flat (something that we cannot observe if it is true) Regression-Discontinuity Design Approach to Price Discrimination Why do we observe different passengers paying a different price for the same product? There exists well documented price dispersion in airlines. Prices can can be different because of ticket characteristics, refundability, blackouts, one-way vs. round trip (open jaws), competition, demand expectations, capacity costs, seat availability. Borestein and Rose (1994) explain that identifying price discrimination is difficult because a price discrimination component is usually correlated with other ticket restrictions that affect costs. This section summarizes Escobari, Rupp, and Meskey (2015). The intuition of the problem can be seen in the LHS panel of Figure Price changes frequently as the flight date approaches, but we cannot claim that those changes are price discriminatory. We use the known thresholds at 7, 14, and 21 days in advance. The RHS panel of Figure 11.4 illustrates the

8 Regression Discontinuity Design Data Posted prices consistent with the Econometrica paper by Deneckere and Peck (2012). We have propietary data set with over 4 million observations on prices for 1,754 flights, 158 city pairs and for 9 carriers Empirical Strategy For each observation i in the data, the random variable FARE i denotes our outcome of interest. The scalar regressor TIME i (time to departure day) is the running variable that determines the treatment assignment based on a known cutoff. Following the framework in Heckman and Vytlacil (2007) and Imbens and Wooldridge (2009), let{(fare i (0), FARE i (1), TIME i ) : i=1,2,...,n} be a random sample from (FARE(0), FARE(1), TIME), with FARE(0) and FARE(1) being the outcomes with and without the price discrimination treatment. FARE i is assigned to the price discrimination treatment condition if TIME i T and is assigned to the control (no price discrimination) condition if TIME i < T for a specific and known fixed value T. Here we use the two known cutoffs, the seven and the fourteen day in advance purchase restriction, i.e., T = 7, 14. The observed outcome is { FARE FARE i = i (0) if TIME i < T (11.7) FARE i (1) if TIME i T. We identify price discrimination (PD) as the sharp average treatment effect at the threshold T and it is given by PD=E[FARE i (1) FARE i (0) TIME i = T]. (11.8) We can estimate PD nonparametrically following the regression-discontinuity design literature under mild continuity conditions. In particular where µ + = lim E[FARE i TIME i = T] TIME T PD=µ + µ, (11.9) µ = lim E[FARE i TIME i = T]. TIME T (11.10) Using kernel-based local polynomials on either side of the threshold we can estimate PD following Hahn, Todd and van der Klaauw (2001) and Porter (2003). The local polynomial regression discontinuity estimator of order p is PD(h n )= µ +,p (h n ) µ,p (h n ) (11.11)

9 11.2 Regression-Discontinuity Design Approach to Price Discrimination 101 Fig Treatment, Observables, and Unobservables in Matching on Observables and IV (Lee and Lemieux, 2010). where µ +,p (h n )=e 0 β +,p (h n ) and µ,p (h n )=e 0 β,p (h n ) (11.12) denote the intercept at the threshold of a weighted p-th order polynomial regression for only treated and control groups respectively. e 0 = (1,0,0,...,0) R p+1 is the first unit vector. Moreover, β +,p (h n ) and β +,p (h n ) are β +,p (h n )=arg min n β R p+1 i=1 β,p (h n )=arg min n β R p+1 i=1 I TIMEi T (FARE i r p (TIME i T) β) 2 K hn (TIME i T) (11.13) I TIMEi <T (FARE i r p (TIME i T) β) 2 K hn (TIME i T) (11.14) whereiis an indicator function, r p (T)=(1, T, T 2,..., T p ), K h (u)=k(u/h)/h with K( ) is a kernel function, and h n is a positive band sequence RDD Estimates The RDD estimates are reported in 11.1.

10 Regression Discontinuity Design Table 11.1 RD Estimates: Monopolies VARIABLES (1) (2) (3) (4) (5) At 7 Days to Departure (T=7): PD Observations Robust 95% CI [.11 ;.16] [.11 ;.16] [.13 ;.16] [.12 ;.17] [.11 ;.16] Robust p-value BW Loc. Poly. (h) BW Bias (b) At 14 Days to Departure (T=14): PD Observations Robust 95% CI [.03 ;.11] [.05 ;.11] [.01 ;.1] [.03 ;.11] [.04 ;.11] Robust p-value e BW Loc. Poly. (h) BW Bias (b) At 21 Days to Departure (T=21): PD Observations Robust 95% CI [-.03 ;.04] [-.02 ;.04] [-.03 ;.01] [-.03 ;.04] [-.04 ;.05] Robust p-value BW Loc. Poly. (h) BW Bias (b) Kernel Type Triangular Triangular Triangular Uniform Triangular BW Type CCT IK CV CCT CCT Order Loc. Poly. (p) Order Bias (q) Notes: From Escobari, Rupp, and Meskey (2015) Estimation in Stata The estimation follows article Robust Data-Driven Inference in the Regression- Discontinuity Design published in The Stata Journal in 2014 by Calonico, Cattaneo and Titiunik. We use the following data: use airlines.dta which is a selection of flights from the original data set. For the RD plot we need: rdplot fare timeid if timeid > 1086 & timeid < 1422, /// c(1255) h(76.026) rho(0.479) graph_options(title(rd Plot 7 Days) /// ytitle(fares) xtitle(time to Departure)) This should give us an RD plot similar to the RHS of Figure For the RD estimates using the methods in Calonico et al.(2014) we have: rdrobust fare timeid if timeid > 1086 & timeid < 1422, c(1255) The regression output is:

11 11.3 Estimation in Stata 103 Sharp RD estimates using local polynomial regression. Cutoff c = 1255 Left of c Right of c Number of obs = NN matches = 3 Number of obs BW type = CCT Order loc. poly. (p) 1 1 Kernel type = Triangular Order bias (q) 2 2 BW loc. poly. (h) BW bias (b) rho (h/b) Outcome: fare. Running variable: timeid Method Coef. Std. Err. z P> z [95% Conf. Interval] Conventional Robust