A Test for Rank Similarity and Partial Identification of the Distribution of Treatment Effects Preliminary and incomplete

Transcription

1 A Test for Rank Similarity and Partial Identification of the Distribution of Treatment Effects Preliminary and incomplete Brigham R. Frandsen Lars J. Lefgren August 1, 2015 Abstract We introduce a test of the rank invariance or rank similarity assumption common in treatment effects and instrumental variables models. The test probes the implication that the conditional distribution of ranks should be identical across treatment states using a regression-based test statistic. When justified by the test, natural extensions of rank similarity partially identify the distribution of treatment effects. The resulting bounds are substantially tighter than those based on the classical Frechet-Hoeffding bounds. We apply the test to school classroom interventions, and show that systematic slippages in rank can be important statistically and economically, and that when rank similarity is plausible, the implied bounds are sharp enough to rule out economically and policy-relevant hypotheses such as harm to students at the bottom of the distribution. 1 Introduction Many recent developments in econometric theory concern the importance of treatment effect heterogeneity. Heterogeneity matters for the policy relevance and interpretation of treatment effects: a treatment which is beneficial on average may nevertheless inflict substantial harm, depending on how subjects differ in their response to the treatment. Treatment effect heterogeneity also matters for econometric identification Department of Economics, Brigham Young University 1

2 of structural (causal) parameters. When relative ranks across treatment states are preserved, population as opposed to local average treatment effects are identified and models restricting to scalar outcome disturbances are justified. Finally, characterizing treatment effect heterogeneity provides insights regarding the economics surrounding individuals selection into treatment. Generally researchers observe a subject in only one treatment state. Consequently, in order to characterize the heterogeneity of treatment effects, researchers must make some assumption about a subject s rank in counterfactual distributions. A common benchmark is to assume rank invariance, which in the case of a binary treatment implies that the subject s rank is the same in both the treated and control distributions. Under this assumption, the distribution of treatment effects is identified and quantile treatment effects can be interpreted as effects on individuals at given points of the distribution. Furthermore, researchers can also identify an intervention s ATE even in the absence of perfect compliance to the instrument, where otherwise only a LATE may be identified. There is generally little reason to believe, however, that rank invariance holds. A generalization, rank similarity, requires only the conditional distribution of ranks in all treatment states to be identical, conditional on factors influencing treatment status (Chernozhukov and Hansen, 2005). This weaker assumption loses the ability to point identify the distribution of treatment effects, but it preserves the ability to identify ATE. Even the weaker rank similarity condition may be implausible in many settings, however, and deserves scrutiny. For example, it rules out the possibility that individuals use private information on comparative advantage when selecting into (or out of) treatment. We propose a method for examining rank similarity and, by extension, its stronger version, rank invariance. We develop the test in a setting with a binary treatment 2

3 that is either exogenously assigned, or partially determined by an instrument satisfying standard conditions, but note that it generalizes in a straightforward way to multivalued (including continuous) treatments. The test consists of comparing the estimated distribution of treated and control ranks, conditional on some observed variable S, using a regression-based statistic. Under rank invariance or similarity, the treated and control ranks will have identical distributions conditional on S and the test statistic has a well-known limiting distribution, while under violations the distributions will diverge, provided S is related to outcomes. For example, consider the case of a randomized job training program with perfect compliance. One might suspect that prior earnings a candidate S variable is correlated with outcomes in the absence of treatment. Consequently, the distribution of ranks for control subjects with high prior earnings should have higher mass in the top of the distribution. If the treated and control distributions exhibit rank similarity with respect to prior earnings, the distribution of ranks for treated subjects with high prior earnings should be shifted in precisely the same way. In settings with imperfect compliance but monotonicity, one can perform a similar test by examining the rank distributions of treated and untreated compliers. A failure to reject may justify rank similarity or invariance assumptions and the additional identification power they garner. For example, Chernozhukov and Hansen s (2005) estimator leverages rank similarity to identify ATE; imposing the special case of rank invariance further identifies the treatment effect on individuals at each quantile of distribution, as well as the distribution of treatment effects. Rank invariance also justifies classes of models with scalar outcome disturbances (Newey and Powell, 2003; Chesher, 2005; Chernozhukov, Imbens, and Newey, 2007). Even without further imposing rank invariance, we show how rank similarity can be used to identify bounds on the distribution of treatment effects. 3

4 Rejecting rank similarity is also potentially interesting. In this case the testing procedure shows which groups have comparative advantage under the treatment. For example, finding that conditional on low levels of education treated ranks are shifted up relative to untreated ranks implies that the intervention benefits less-educated individuals relative to the more-educated. When rank similarity holds, we show the distribution of individual-level effects is partially identified. Bounds on the distribution of treatment effects at each point of the control outcome distribution are implied by a natural extension of rank similarity, namely that individual ranks in the treated and control distribution are mutually stochastically increasing. Bounds on the overall distribution of treatment effects can be further tightened by assuming additionally that ranks are conditionally exchangeable, which implies rank similarity. These bounds are sharper when the conditioning covariate S is more strongly predictive of outcomes. Exchangeability, while strictly stronger than rank similarity, is still testable, since it implies the same restrictions on observed data as rank similarity. Our procedure can be viewed, then, as not only a test of rank similarity, but as a motivation for the exchangeability condition and its accompanying bounds on the distribution of treatment effects. Our paper complements the literature on treatment effects, nonseparable models, and quantile instrumental variables models. The notion of rank invariance in treatment effects was introduced by Doksum (1974), who used the idea of an underlying proneness to interpret effects on different quantiles of the outcome distribution (see also Lehmann, 1975; Imbens and Wooldridge, 2009). Heckman et al. (1997) consider rank invariance (perfect positive dependence) as an extreme case to obtain bounds on the distribution of treatment effects. In the nonparametric IV literature, the assumption that the outcome is monotonic in a scalar disturbance separable 4

5 or nonseparable imposes rank invariance. For example, Newey and Powell s (2003) nonparametric instrumental variables model relies on a rank invariance assumption in the form of a scalar separable outcome disturbance for identification, as do Chesher (2005), Athey and Imbens (2006) and Chernozhukov, Imbens, and Newey (2007), but with a nonseparable outcome disturbance. The IV quantile model of Chernozhukov and Hansen (2005, 2008) imposes rank similarity for identification of average treatment effects and rank invariance for interpretation of quantile treatment effects. Finally, Imbens and Newey s (2009) nonseparable triangular model achieve identification for a general non-scalar outcome disturbance, but introduces rank invariance in the form of a scalar outcome disturbance for interpretation of the estimates. Our paper builds on this literature by introducing a test of the assumptions needed for identification or interpretation in these models. The intuition for our testing procedure is similar to Bitler et al. s (2005) idea for testing for rank reversals using pre-determined covariates in the context of an exogenous treatment. Their method examines whether individuals at a given quartile in the control distribution exhibit similar pre-treatment characteristics to individuals at the corresponding quartile of the treatment distribution. Our method builds on their work by generalizing to endogenous or multi-valued treatments. Dong and Shen (2015) develop a test for rank similarity based on a similar framework. Our paper complements theirs by developing a regression-based test and by constructing bounds on the distribution of treatment effects when rank preservation is consistent with the data. Our paper is also related to the literature on bounding the distribution of treatment effects. Heckman et al. (1997) show how the classical Frechet-Hoeffding limits bound features of the joint distribution of potential outcomes of interest in program evaluation, and Williamson and Downs (1990) calculate bounds on the distribution 5

6 of treatment effects, specifically, and Fan and Park (2010) show how to perform inference on those bounds. Our paper builds on this work, showing how these bounds can be sharpened using the additional information imposing rank similarity provides. In the remainder of this paper we describe our econometric framework and define rank similarity and specify our test in a setting with an exogenous treatment. We then generalize our test to consider cases where treatment is endogenous but where an instrument is available. We continue by describing how to estimate bounds on the distribution of treatment effects. We conclude with two empirical applications in which we examine the effect of class size and ability-tracking on student achievement. 2 Econometric Framework and Test Procedures Consider the standard treatment effects framework with a binary treatment. The test we develop can be generalized to multi-valued (including continuous) treatments. A binary treatment, D, potentially affects a continuously distributed outcome Y. Let Y (1) and Y (0) be potential outcomes with and without treatment, with cdfs F 1 and F 0. The observed outcome is Y = Y (D). In addition to outcomes and treatment, we observe a q-vector of pre-treatment variables S and, in the case of an endogenous treatment, an instrumental variable Z. Define an individual s rank in the untreated and treated distributions to be, respectively: U (0) : = F 0 (Y (0)) U (1) : = F 1 (Y (1)). The marginal distributions of U (0) and U (1) are, as a consequence of the Skorokhod 6

7 representation of Y (0) and Y (1), uniform: U (0) U (0, 1) U (1) U (0, 1). Rank invariance and rank similarity restrict the relationship between U (0) and U (1). The stricter notion, rank invariance, is defined as follows: Definition 1 (Rank Invariance) A treatment exhibits rank invariance if and only if U (0) = U (1) almost surely. Rank invariance means an individual of a given rank in the control distribution would have the same rank in the treated distribution. It implies that individuals with the same control outcome would have responded identically to treatment, as noted by Doksum (1974). While this is restrictive, it is imposed by models with scalar outcome disturbances (Newey and Powell, 2003; Chesher, 2005; Athey and Imbens, 2006; Chernozhukov, Imbens, and Newey, 2007). A generalization of rank invariance that captures a similar notion is rank similarity, introduced by Chernozhukov and Hansen (2005): Definition 2 (Rank Similarity) A treatment exhibits rank similarity with respect to S if and only if conditional on S = s, U (0) and U (1) are identically distributed for each s in the support of S. Rank similarity means a subpopulation (defined by S = s) will have the same distribution of ranks across treatment states. Rank similarity only has meaning with 7

8 respect to some rank-shifting variable S, since unconditionally U (0) and U (1) have identical distributions by definition. Note that rank invariance implies rank similarity, and can be seen as the special case of rank similarity where S is itself U (0) or Y (0). The rank shifting variable S plays a crucial role in the testability of this restriction; it obviates the need for knowledge of or assumptions about the joint distribution of potential outcomes (Y (0), Y (1)) required in prior work on treatment effect heterogeneity (Heckman et al., 1997). Rank similarity, and by extension rank invariance, imposes testable restrictions on observed data, provided the distributions of potential outcomes are identified. Intuitively, it means that conditional on S, treatment status has no effect on the distribution of ranks. The proposed test directly examines this implication via the following procedure: 1. Estimate potential outcome cdfs ˆF 0 and ˆF 1 ; 2. Construct sample ranks Û i = (1 D i ) ˆF 0 (Y i ) + D i ˆF1 (Y i ) ; (1) 3. Estimate the following specification: Û i = α 0 + α 1 D i + S iα 2 + D i S iδ + ε i ; (2) 4. Test the hypothesis ( H 0 : δ = 0 0 ) using the test statistic ( ) 1 ˆ = nˆθ h h ˆV h h ˆθ, (3) 8

9 where n is the sample size, ˆθ is an appropriate estimator (OLS or instrumental variables) for specification (2), h is the selection matrix ( h = 0 q q+2 I q ), and ˆV is the estimated asymptotic variance-covariance matrix of ˆθ. As shown below, under the null hypothesis of rank similarity the test statistic converges asymptotically to a chi-squared random variable with q degrees of freedom. Since the Ûis are constructed conditional on D i, they are normalized within each treatment category. Therefore, α 1 is not a free parameter in regression (2), and so is not included in the test statistic. Note that in the special case of a scalar rank shifter (q = 1), this test is asymptotically equivalent to a standard t-test. The regression-based procedure can be adapted to maintain power against different alternatives. For example, if the violations which one would like the test to detect affect features of the conditional distribution of ranks other than the mean, one can ) replace the left hand side of (2) with other functions of the rank, such as 1 (Ûi u for one or several values of u. A uniform test over u (0, 1) would be akin to a two-sample Kolmogorov-Smirnov test of rank distribution equivalence. The choice of estimators for ˆF 0, ˆF1, and specification (2) depends on the specific empirical setting. For exogenously assigned treatments, empirical cdfs and ordinary least squares estimators suffice; settings with endogenous treatments require instrumental variables methods. The following subsections specify the details of the test for these two cases in turn. 2.1 Unconfounded treatment First, consider the case where treatment is as good as randomly assigned: 9

10 Condition 3 (Treatment Unconfoundedness) (Y (0), Y (1)) are jointly independent of D. Under this condition, the marginal distributions of potential outcomes are identified by the conditional empirical distribution functions: ˆF 0 (y) = ˆF 1 (y) = n i=1 1 (Y i y) (1 D i ) n i=1 (1 D, i) n i=1 1 (Y i y) D i n i=1 D. i Sample ranks Ûi are constructed from these estimators via (1), and specification (2) can be estimated via ordinary least squares: ˆθ = (W W ) 1 W Û, where W is a matrix of observations on W i := ). Given independent sampling, ˆV is given by the usual White formula: ( 1 D i S i D i S i ( W ˆV W = n ) 1 1 n n i=1 ( ) W W i W i ˆε 2 1 W i. n 2.2 Endogenous treatment Now suppose treatment status D i is possibly confounded, but there exists an exogenous instrument Z i ; for example, randomized treatment assignment. Let potential treatment status as a function of the instrument be D (0) and D (1). We assume the standard conditions for a valid instrument: Condition 4 (Instrument Validity) (Y (0), Y (1), D (0), D (1)) are jointly independent of Z; and E [D Z = 1] > E [D Z = 0]. 10

11 Condition 4 and the assumption implicit in the potential outcomes notation corresponds to the usual exclusion and relevance conditions. Under this condition and rank similarity the marginal distributions of potential outcomes are identified (Chernozhukov and Hansen, 2005). If rank similarity is replaced by a monotonicity condition on the response of treatment status to the instrument, the marginal distributions of potential outcomes among the subpopulation that responds to the instrument compliers are identified (Imbens and Angrist, 1994). Implementing the test in a setting with an endogenous treatment requires instrumental variables estimation of ˆF 0, ˆF1, and regression specification (2). This can be done in at least two ways, depending on which assumptions are appropriate for the specific empirical setting. The first is Chernozhukov and Hansen s (2006) quantile instrumental variables estimator. Given Condition 4 this approach is valid under the null hypothesis of rank similarity, and leads to a test of asymptotically correct size. The cdfs F 1 and F 0 can be estimated by inverting estimates of the quantiles of Y i (1) and Y i (0). The quantiles of Y i (1) can be obtained by quantile instrumental variables estimation of the effect of D i on Y i D i, using Z i as an instrument. The quantiles of Y i (0) can likewise be obtained by quantile instrumental variables estimation of the effect of (1 D i ) on Y i (1 D i ), using Z i as an instrument. The resulting estimates ˆF 1 and ˆF 0 then can be used to construct the sample ranks Ûi via (1). The test consists ( ) of estimating the effect of the vector W i := 1 D i S i D i S i on Ûi, which can be done for a particular quantile (say, the median), or integrated over all quantiles for the mean effect. The second method is Abadie s (2003) κ-weighted least squares estimator. Given Condition 4, this approach is valid when treatment status can be assumed to respond monotonically to the instrument almost surely. A procedure based on this approach then tests rank similarity among the subpopulation of compliers. Rank invariance 11

12 (and rank similarity as imposed in Chernozhukov and Hansen (2005)) implies rank similarity among compliers, so a test based on this approach has asymptotically correct size. Using this approach, the cdfs F 1 and F 0 can be estimated directly by weighted least squares estimation of the effect of D i on 1 (Y i y) D i (for F 1 (y)) and the effect of (1 D i ) on 1 (Y i y) (1 D i ) (for F 0 (y)), using Abadie s κ as weights: κ i = 1 D i (1 Z i ) Pr (Z i = 0) (1 D i) Z i Pr (Z i = 1). (4) The resulting estimates ˆF 1 and ˆF 0 then can be used to construct the sample ranks Ûi via (1). The test is then carried out by weighted regression of Ûi on the vector W i. 3 Partial identification of the distribution of treatment effects If rank similarity holds, the data are informative about individual-level treatment effects, i := Y i (1) Y i (0), and bounds can be constructed for their distribution by assuming ranks are weakly stochastically increasing, a natural extension of rank similarity. The distribution of treatment effects is something of a holy grail in the program evaluation and treatment effects literature, but is typically not identified in experimental data without strong assumptions (Heckman et al., 1997). The fundamental difficulty in identifying the distribution of treatment effects is that experimental data provide information on the marginal distributions of potential outcomes, but not the joint distribution. The marginal distributions themselves impose some restrictions on the joint distribution via the Frechet-Hoeffding bounds, but these are typically quite wide. The bounds we construct here sharpen the Frechet-Hoeffiding bounds and the related bounds on the distribution of treatment effects discussed by Fan and Park 12

13 (2010) and Fan et al. (2014) by imposing natural restrictions on the joint distribution of potential outcomes suggested by rank similarity. One important source of treatment heterogeneity is how the average effect of treatment varies with observable characteristics. This heterogeneity can be explored either by examining subgroups separately or by interacting treatment with the covariates of interest. However, it is of significant and independent interest to understand heterogeneity of treatment effects across the distribution of control group outcomes. In particular, agents may have private information regarding their likely outcome in the absence of treatment. In this case, providing separate estimates of treatment effects for individuals likely to do well and do poorly in the absence of treatment can improve the efficiency of treatment assignment and minimize the probability that individuals are harmed by inappropriate treatment. Also, it is only by exploring how treatment effects vary across the distribution of control outcomes that we can estimate not only the average effect of treatment but also the fraction of individuals that was benefited by the treatment. This is of great importance in that the average benefit of treatment may not be robust to the scaling of the outcome variable. Furthermore, subjects might be rightly wary of treatment if it exposes them to a significant risk of harm, even if the average outcome is improved. We propose the following natural extension to rank similarity in order to identify the distribution of treatment effects conditional on Y (0): Definition 5 U (1) is weakly stochastically increasing in U (0) conditional on S if Pr (U (1) t U (0) = u, S) is nonincreasing in u almost surely. Lehmann (1966) described this property, referring to it as positive regression dependence. It means that individuals with higher U (0) have a conditional distribution of U (1) that weakly stochastically dominates the distribution of U (1) among those 13

14 with lower U (0). It includes as limiting cases rank invariance and conditional rank independence, the special cases of rank similarity in which the ranks in one treatment state are respectively perfectly informative or perfectly uninformative about ranks in the other state. This condition rules out negative dependence between ranks conditional on S, a plausible restriction, since conditional rank similarity implies a perfectly positive relationship between ranks on the basis of observables. This restriction is how we sharpen the classical Frechet-Hoeffding bounds on joint distributions (Heckman et al., 1997). Under the weak stochastically increasing property, the conditional distribution of the individual level treatment effect can be bounded, as the following theorem establishes. Theorem 6 Suppose U (0) and U (1) are rank similar and U (1) is weakly stochastically increasing in U (0) conditional on S. Then F Y (0),S (t Y (0), S) := Pr ( t Y (0), S) is bounded from below by F Y L (0),S (t Y (0), S) := 0, F 1 (Y (0) + t) < U (0) F U S (F 1 (Y (0)+t) S) F U S (U(0) S) 1 F U S (U(0) S), F 1 (Y (0) + t) U (0) (5) and from above by F Y U (0),S (t Y (0), S) := F U S (F 1 (Y (0)+t) S) F U S (U(0) S), F 1 (Y (0) + t) U (0) 1, F 1 (Y (0) + t) U (0). (6) Proof. See the appendix. The cdf bounds (5) and (6) can be consistently estimated via the following procedure for any value t given consistent estimators ˆF 1 and ˆF 0 for the cdfs of potential 14

15 outcomes. The procedure consists of the following steps for each untreated observation j: on S i and con- 1. Nonparametrically regress an indicator function 1 (Ûi Ûj struct predicted value ˆF ) U S (Ûj S i ) 2. Construct Ũj (t) = ˆF 1 (Y j + t) and nonparametrically regress an indicator function 1 ) (Ûi Ũi (t) on S i and construct predicted value ˆF ) U S (Ũj (t) S j 3. Form estimates of the bounds ˆF Y L (0),S (t Y j (0), S j ) ) ˆF U S (Ũj (t) S j ˆF ) U S (Ûj S j = max 1 ˆF ), 0 U S (Ûj S (7) j ˆF Y U (0),S (t Y j (0), S j ) ) ˆF U S (Ũj (t) S j = min ), 1 ˆF U S (Ûj S. (8) j Bounds on the overall distribution of treatment effects can be obtained from sample averages of these conditional bounds. The bounds on the treatment effect cdf given by (5) and (6) also imply bounds on the average treatment effect conditional on Y (0), a quantity that is frequently of great interest in applications, but not point identified. Let the average treatment effect conditional on Y (0) and S be denoted (Y (0), S) := E [Y (1) Y (0) Y (0), S]. By definition, bounds on the expectation are given by integrating the derivative of the cdf bounds: L (Y (0), S) = U (Y (0), S) = tdf U Y (0),S (t Y (0), S), tdf L Y (0),S (t Y (0), S). 15

16 These bounds can be computed by integrating the bounds (7) and (8) on a discrete grid, followed by integration over S via nonparametric regression on Y (0). The bounds (5) and (6) on the conditional distribution of treatment effects are by construction sharp pointwise in Y (0) and S, but not uniformly. Thus integrating (5) and (6) over Y (0) and S yields conservative bounds on the unconditional distribution of treatment effects. The unconditional bounds can be further tightened by assuming that U (1) and U (0) are exchangeable conditional on S: Definition 7 U (0) and U (1) are exchangeable conditional on S if F 01 S (u, v s) = F 01 S (v, u s) for all u, v in the support of U (0) and U (1) conditional on S, where F 01 S is the joint distribution function of U (0) and U (1) conditional on S. Exchangeability implies rank similarity with respect to S, and thus the testing procedure developed in Section 2 can also be considered a test of exchangeability. Exchangeability potentially tightens the bounds on the distribution of treatme effects dramatically, as the following result shows: Theorem 8 Suppose U (0) and U (1) are exchangeable conditional on S. Then (1) F 1 S (y s) F 0 S (y t s) for all y implies Pr ( t S = s) 1/2; and (2) F 1 S (y s) F 0 S (y t s) for all y implies Pr ( t S = s) 1/2. Proof. See the appendix. This result allows knowledge of the separate distributions of Y (0) and Y (1) conditional on S which are identified in the data to bound the distribution of the treatment effect. These bounds are typically much tighter than the classical bounds in Williamson and Downs (1990) derived from the Frechet-Hoeffding bounds, as the simulations and empirical application below shows. These bounds based on exchangeability are most useful when the treatment has a modest, stochastically dominant shifting effect on outcomes. 16

17 As mentioned above, the testing procedure for rank similarity also motivates the exchangeability assumption, since a rejection implies exchangeability, too, is false. The intuition underlying our test for rank similarity is equally compelling for exchangeability as it is for rank similarity. In particular, any rank-shifting covariate S must have the same effect on ranks in both the treated and untreated state in order for unconditional exchangeability to hold. Consequently, our test of rank similarity can be reinterpreted as a test of rank exchangeability. However, conditional rank similarity may hold when exchangeability fails to hold. Such situations tend to include either negative dependence between ranks in the treated and untreated states or non-monotonicities in this relationship. One plausible case of rank similarity that implies exchangeability is the case of rank slippage as described by Chernozhukov and Hansen (2005) in which an individual s rank in either state is given by an expected rank plus a rank slippage term that has the same distribution in both states. As long as the rank slippage term is independently distributed in each of the two states, this simple model implies both rank similarity and rank exchangeability. In cases in which a researcher is uncomfortable imposing exchangeability, the pointwise bounds (5) and (6) can be integrated to construct bounds on the marginal distribution of treatment effects. While wider than the bounds that incorporate exchangeability, they can still be informative in situations with large treatment effects or highly predictive rank shifters. They also involve only the assumption that the rank in the treated state is stochastically increasing with the rank in the control state. 17

18 4 Asymptotic Theory The test statistics are based on consistent, asymptotically normal estimators, and thus have straightforward limiting distributions under standard regularity conditions. Critical values and p-values can then be calculated by consistent estimation of the limiting distribution, or by simulation methods. This section establishes the limiting distribution of the test statistic in the case of an endogenous regressor, using the Abadie (2003) IV estimator. The setting with an endogenous treatment includes unconfounded treatment as a special case. The test statistic (3), as a quadratic form in an asymptotically normal estimator, converges to a χ 2 random variable, as the following theorem establishes. Theorem 9 Suppose Condition 4 holds and (1) D i (1) D i (0) almost surely; (2) the ( ) data are i.i.d; and (3) E [κ i W i W i ] is nonsingular for W i := 1 D i S i D i S i. Let ˆθ be the Abadie-κ-weighted least squares estimator of specification (2), with asymptotic variance matrix V, defined in Abadie (2003). Then the test statistic ˆ has the following limiting distribution: ( ) 1 ˆ = nˆθ h h ˆV h h ˆθ d χ 2 (q), where ˆV is a consistent estimator of V, given in Abadie (2003) and ( h = 0 q q+2 I q ). The result guarantees that the test will have asymptotically correct size. The test s power derives from differences in the conditional distribution of ranks across 18

19 treatment states under alternatives to rank similarity. If these differences lead to shifts in the mean or other features of the distribution captured in specification (2), then h θ will have nonzero elements, and the test s power will depend on the corresponding noncentral χ 2 distribution. 5 Monte Carlo Simulations 5.1 Finite-sample size and power of the rank similarity test The asymptotic theory implies that for sufficiently large samples the proposed testing procedures will have approximately correct size. This section shows that the tests also have good size and power properties in finite samples. The simulations generate iid samples of size n from the following data generating process. An observed rank-shifting variable is generated as S i N (0, σs 2 ). The untreated potential outcome is generated as Y i (0) = βs i + ε i, where ε i N (0, σ 2 ε), independently of S i. For fixed σ 2 S and σ2 ε, β parameterizes the informativeness of the rank-shifting variable S i. The treated potential outcome is constructed as Y i (1) = (1 ω) Y i (0) + ωη i, where η i N ( 0, ση) 2 independently of Si and ε i and ω [0, 1]. The parameter ω controls the degree of departure from rank invariance (or similarity). The special case ω = 0 corresponds to rank invariance, while ω = 1 corresponds to a setting where one s treated rank is orthogonal to one s untreated rank. The instrument Z i is assigned as in a totally randomized experiment where half of the subjects are assigned Z i = 0 and the other half are assigned Z i = 1, independently of (S i, ε i, η i ). Treatment status exhibits negative selection: potential treatment status for Z i = z is D i (z) = 1 (ρy i (0) + ζ i γ (z 1/2)), where ζ i N ( 0, σζ) 2 independently of (S i, ε i, η i, Z i ), γ is a constant that governs the strength of the instrument, 19

20 and ρ [0, 1]. The special case of ρ = 0 corresponds to an exogenous treatment. In this setup, the local average treatment effect (LATE) is equal to zero, as is the average treatment effect (ATE). Unless stated otherwise, the simulations set n = 1, 000, σ 2 S =.5, β =.75, σ2 ε =.75, σ 2 η = 1, ρ = 0, σ 2 ζ = 1, and γ = 3. The first set of simulations examines the finite-sample size of the test for sample sizes ranging from n = 100 to n = 1, 000 for an exogenous and endogenous treatment. The degree of departure from rank invariance or rank similarity is naturally set at ω = 0. The degree of endogeneity is set at ρ = 0 for the exogenous treatment and ρ =.3 for the endogenous treatment. The simulations show that even for relatively small sample sizes the test maintains accurate size under both exogeneity and endogeneity. Figure 1 plots the simulated rejection rate for tests of nominal size α =.05 by sample size. The rejection rate hovers around five percent over the entire range. The next set of simulations explores the test s power to detect departures from rank invariance or similarity. The simulations vary the degree of departure between ω = 0 and ω = 1. The simulations show that the rejection probability increases rapidly as a function of the degree of departure from rank invariance or rank similarity. Figure 2 plots the simulated rejection rates by ω for tests of nominal size α =.05. At the far left of the plot, where ω = 0, the rejection rate is approximately.05, indicating correct size. As violations of rank invariance are introduced, however, the rejection rate increases steeply. The test rejects 75 percent of the time when ω =.5 and essentially 100 percent of the time when ω = 1. The next set of simulations examines the test s power when the sample size ranges from n = 100 to n = 1, 000. The degree of departure from rank invariance is set at ω =.75. The simulations show the test is lower-powered for small sample sizes, but has excellent power for sample sizes common in empirical analysis. Figure 3 plots the simulated rejection rates by n for tests of nominal size α =.05. The rejection rate 20

21 is about 40 percent for the smallest sample size of n = 100, but increases to over 99 percent for n = 1, 000. The next set of simulations examines how the test s power varies with the informativeness of the rank-shifting variable S i. The simulations vary the β parameter from zero to one, corresponding to an R 2 between the control outcome and S i of 0 to.3. The degree of departure from rank invariance is set at ω =.75. The simulations show the test s power depends heavily on the informativeness of the rank-shifting variable. Figure 4 plots the simulated rejection rates by R 2 for tests of nominal size α =.05. The rejection rate is.05 when S i is completely uninformative, but reaches essentially 100 percent when the R 2 reaches.3. The next set of simulations shows how endogeneity affects the test s power. The simulations vary the degree of endogeneity from ρ = 0 to ρ = 1. For purposes of comparison, the tests use instrumental variables estimation in all cases, even ρ = 0, where OLS would be valid. The degree of departure from rank similarity is set at ω =.75. The simulations show the test maintains excellent power across the range of endogeneity, although it is more powerful when there is less endogeneity. Figure 5 plots simulated rejection rates by ρ for tests of nominal size α =.05. Power exceeds 97 percent for all levels of ρ shown, but is decreasing in the degree of endogeneity. The final set of simulations shows how the strength of the instrument affects the test s power. The simulations vary the instrument strength parameter from γ = 0 to γ = 6, corresponding to compliance rates from zero to 100 percent. The simulations show the test s power increases substantially with the strength of the instrument. Figure 6 plots simulated rejection rates by instrument strength as measured by compliance rate for tests of nominal size α =.05. The figure shows the power is near the size for a completely weak instrument, and thus the test is essentially uninformative, as expected. Power increases significantly, however, as the instrument becomes 21

22 stronger, and reaches essentially 100 percent for compliance rates near 100 percent. 5.2 Bounds on the distribution of treatment effects The final set of simulations illustrates the bounds on the distribution of individuallevel treatment effects that are implied when rank similarity holds. These simulations adopt the same setup as in the previous section, but with a positive treatment effect and in the special case of rank invariance (ω = 0) and an exogenous treatment (D i = Z i ). Thus potential outcomes are generated as Y i (0) = βs i + ε i, Y i (1) = Y i (0) + δ, and D i is assigned independently of S i and ε i. The simulations set n = 500, and δ = σs 2 = 1. The simulations vary σ2 ε from.01 to 1, corresponding to an R 2 between Y i (0) and S from.99 to zero, and β is set accordingly to R 2 /σs 2. The simulations show that the bounds sharpen dramatically as the predictive power of S increases. Figure 7 plots the bounds on the individual-level treatment effect by rank in the control distribution for three levels of R 2 between Y i (0) and S: 0, 0.5, and.99. The top panel, where S is completely unrelated to outcomes, shows wide bounds. The rank invariance bound, which corresponds to the quantile treatment effects, is flat, while the rank similarity bound corresponds to complete mean reversion. The lower panels show that as S becomes more informative, the bounds sharpen, and come close to point identifying the individual-level treatment effect as R 2 approaches one. Figure 8 plots the bounds for the probability of a negative treatment effect by rank in the control distribution. The lower bound in this case is zero, since all of the quantile treatment effects are positive. As with the bounds of the individuallevel expected treatment effect, the top panel, corresponding to R 2 = 0 shows wide bounds with a substantial probability of negative treatment effects at the top of 22

23 the distribution because of mean reversion. The lower two panels show that as the informativeness of the rank shifter increases the bound tightens toward zero. 6 Empirical Examples 6.1 Class Size and Student Achievement In our first example, we test for rank similarity in the context of the Project STAR class size reduction experiment. Starting in 1985, the state of Tennessee randomized kindergarten students either into small classes of 13 to 17 students or into larger classes of 22 to 25 students. Some larger classrooms were randomly assigned a teacher s aide. Given the importance of the question and strength of the research design, data from this study have been analyzed many times. These studies include works by Folger and Breda (1989), Finn and Achilles (1990), Word et al. (1990) and Krueger (1999). For this analysis, we consider only the effect of assignment to a small class. Prior research (e.g. Krueger, 1999) suggests the impact of an aide was minimal. We restrict our sample to students whose assignment occurred in kindergarten and were still in the sample at the end of first grade. This includes 1,368 treated students and 3,040 control students. We examine math performance at the end of first grade. On average, students assigned to the smaller classes performed 0.20 standard deviations better than students assigned to the larger classes. This positive effect is statistically significant at the 1 percent level. To test for rank similarity, we compute math performance rank in both the treatment and control groups. Because we do not observe math performance prior to classroom assignment, we choose eligibility for free lunch in kindergarten as our rank shifting variable, S. Figure 9 shows how the distribution of ranks in the control distri- 23

24 bution varies according to whether subjects were eligible (lower income) or ineligible (higher income) for free lunch. The density of ranks is shifted clearly to the right for control subjects with higher family income relative to subjects with lower family income. We now examine the distributions of ranks in the treated and control distributions for different values of our rank shifter. In Figure 10, we compare the distribution of ranks for treated and untreated subjects who had lower family incomes. Figure 11 represents the corresponding figure for subjects who had higher family incomes. Under the null hypothesis of rank similarity, we expect the rank distributions between treated and control observations to be similar in both figures. Examining the rank distributions of students who had lower family incomes, we see that the bottom of the distribution has less mass in the treatment group relative to the control group. Similarly, if we examine the rank distribution of students with higher family incomes, the bottom of the distribution has more mass in the treatment group than the control group. This suggests that treatment may have been more efficacious for struggling students with low incomes than for struggling students with higher incomes. This visual evidence suggests that the assumption of rank similarity may be unfounded. To test more formally for rank similarity and establish the power of the rank shifter, we estimate regression specification (2) via ordinary least squares: Û i = α 0 + α 1 D i + α 2 S i + δs i D i + ε i Table 1 shows the results of this estimation. We see that the estimate for α 2 is 0.208, which implies that students who are eligible for free lunch are located nearly 21 percentage points lower in the control distribution than students who are ineligible for free lunch. This is highly significant and together with Figure 9 suggests that our 24

25 rank shifter has strong power. The estimate for δ is and statistically significant at the 1 percent level suggesting that the rank disadvantage of being from a low income family is smaller in the treatment group than in the control group. This test suggests that we should reject the null hypothesis of rank similarity. 6.2 Tracking and Student Achievement For a third example, we examine the impact of tracking on student achievement for primary school children in Kenya. We use data from Duflo et al. (2011), who randomized schools into two treatments in In the control treatment, students were randomly assigned to classrooms within their school. In the tracking treatment, students were assigned to classrooms within their school on the basis of their prior performance. Because policy makers worry that tracking may have differential effects for high and low achieving students, tracking is a particularly interesting example to explore rank similarity and heterogeneity in treatment effects. For our analysis, we examine the sample of 5,304 students for whom we observe academic achievement both post and prior to treatment. Post-treatment performance is a composite performance measure normalized so that it has mean zero and unit standard deviation. Pre-treatment performance is normalized to represent the student s percentile rank within their school. Using this sample, we find that tracking on average is associated with a 0.15 standard deviation increase in student performance, which is statistically significant and quite close to the effect found by Duflo et al. (2011). To test rank similarity, we examine how prior achievement percentile shifts the rank distribution in the treated and untreated states. In Figure 12, we show how the distribution of ranks in the control distribution varies according to whether subjects 25

26 had above-median or below-median prior achievement. We see that the density of ranks is shifted strongly to the right for control students with high initial performance relative to students with low initial performance. We now examine the distributions of ranks in the treated and control distributions for different values of our rank shifter. In Figure 13, we compare the distribution of ranks for treated and untreated subjects who had low initial achievement. Figure 14 represents the corresponding figure for subjects who had high initial achievement. Under the null hypothesis of rank similarity, we expect the rank distributions between treated and control observations to be similar in both figures. We see that the distributions do indeed look quite similar. We test this explicitly by estimating equation (2). The estimates of this regression are found in Table 3. In the first column, our rank shifter is a binary variable that takes on a value of one if the subject had above-median prior achievement and zero otherwise. In this specification, our estimate for δ 1 is and is highly significant. This suggests that students with above-median prior achievement scored 24 percentage points higher in the control distribution than students who scored below the median. This provides strong numeric evidence regarding the power of our rank shifter. δ 3 is and statistically insignificant. Given the small size of this coefficient and precise standard errors, it appears that high prior achievement had essentially the same effect on students rank in both the treatment and control rank distributions. In column 2, we use the continous rank shifter of rank in the prior achievement distribution. δ 1 is 0.514, suggesting that each percentage point increase in the prior earnings distribution is associated with a 0.51 percentage point increase in the control distribution. δ 3 is and again statistically insignficant. Collectively, the evidence suggests we should fail to reject the null hypothesis of rank similarity. 26

27 Having failed to reject the null hypothesis of rank similarity, we now employ our method for the partial identification of treatment effects across ranks of control outcomes. We first consider the expected performance increase associated with tracking at each point in the control distribution. The bounds are on the individual-level treatment effects are estimated via (7) and (8). Figure 15 shows these bounds. The bounds are quite wide because prior achievement has such little predictive power, but they are informative enough to rule out negative expected effects among individuals at the bottom of the control distribution. The probability of a negative treatment effect at each point in the control distribution tells a similar story. Figure 16 shows these bounds. At the very bottom of the distribution the upper and lower bounds coincide at zero, again ruling out that individuals at the bottom were harmed by the treatment. The bounds become quite wide at the top of the distribution, however. Nevertheless, these bounds are strongly suggestive that concerns regarding the adverse effects of tracking at the bottom of the achievement distribution are unwarranted, at least when considering measured achievement. The ability to make this conclusion illustrates the value of these bounds; conventional estimators like quantile treatment effects even if positive at lower quantiles do not allow such conclusions without imposing rank invariance. The pointwise bounds in Figure 16 can be aggregated to construct bounds on the overall fraction of individuals hurt by treatment. Table prhurttable presents bounds on the fraction of students hurt by academic tracking, calculated in three ways to illustrate the identifying power of each assumption. The top row shows the bounds obtained when imposing conditional rank exchangeability and weak stochastic increasingness, as described in Section 3. This assumption bounds the fraction hurt between zero and 50 percent. The next row relaxes exchangeability, but maintains stochastic increasingness. The bounds somewhat widen to between zero and 27

28 61 percent of individuals are harmed by tracking. Finally, the bottom row reports the Williamson and Downs (1990) bounds based on the classical Frechet-Hoeffding bounds. These are wider still, pinning the fraction hurt only within the range zero to 82 percent. The bounds we propose therefore substantially tighten the classical abounds, albeit at the expense of additional assumptions. The null results of the test in Table 3 provide some support for the exchangeability assumption, however. 7 Conclusion This paper proposed a test for rank invariance or rank similarity in a treatment effects setting, and showed how to construct bounds on the distribution of individual-level treatment effects when rank similarity holds. The test is based on standard least squares or instrumental variables estimation methods, and applies to exogenous and endogenous treatments. The test is consistent and simulations show it has good size and power properties in finite samples. The bounds can be constructed from standard estimates of the conditional distributions of potential outcomes. Empirical examples from job training programs and classroom interventions show rank preservation deserves scrutiny in real-life data. The test failed to reject rank preservation in earnings from the JTPA experiment and student achievement in a Kenyan tracking experiment. Imposing rank similarity in the student tracking example produced bounds on the distribution of treatment effects that rules out substantial harm to students at the bottom of the distribution. Rank similarity was rejected in test scores from the Tennessee STAR class-size experiment. The proposed test and bounds should prove useful in examining the identification and interpretation of program evaluations, clinical trials, and other treatment effect estimates. The procedure applies to binary treatments, but the framework can be 28

29 applied more generally. Specifying the test and constructing bounds for multi-valued or continuous treatments is left to future research. References Alberto Abadie. Semiparametric instrumental variable estimation of treatment response models. Journal of Econometrics, 113: , Susan Athey and Guido Imbens. Identification and inference in nonlinear differencein-difference models. Econometrica, 74: , March Marianne P. Bitler, Jonah B. Gelbach, and Hilary W. Hoynes. Distributional impacts of the self-sufficiency project. Working Paper 11626, National Bureau of Economic Research, September Victor Chernozhukov and Christian Hansen. An IV model of quantile treatment effects. Econometrica, 73(1): , January Victor Chernozhukov and Christian Hansen. Instrumental quantile regression inference for structural and treatment effect models. Journal of Econometrics, 132(2): , Victor Chernozhukov and Christian Hansen. Instrumental variable quantile regression: A robust inference approach. Journal of Econometrics, 142(1): , January Victor Chernozhukov, Guido W. Imbens, and Whitney K. Newey. Instrumental variable estimation of nonseparable models. Journal of Econometrics, 139(1):4 14, Endogeneity, instruments and identification. 29