Testing for Rank Invariance or Similarity in Program Evaluation: The Effect of Training on Earnings Revisited

Transcription

1 Testing for Rank Invariance or Similarity in Program Evaluation: The Effect of Training on Earnings Revisited Yingying Dong University of California, Irvine Shu Shen University of California, Davis First version, February 2015; this version, September 2015 PRELIMINARY; Please do not cite without permission Abstract This paper discusses the testable implications of rank invariance or similarity, assumptions that are common in program evaluation and quantile treatment effect (QTE) models, and proposes nonparametric tests applicable to both. The tests examine whether the distribution of potential ranks (or features of the distribution) for observationally equivalent individuals remains the same across treatment states. The tests allow treatment to be endogenous, while essentially not requiring any additional assumptions other than those used to identify and estimate QTEs. We focus on testing for invariance or similarity of ranks in the unconditional distribution of potential outcomes, but briefly discuss extensions to testing for conditional rank invariance or similarity. The proposed tests are used to examine whether training causes individuals to systematically change their ranks in the earnings distribution using JTPA data. JEL Codes: C12, C14, J31 Keywords: Treatment effect heterogeneity, Distributional treatment effect, Rank invariance, Rank similarity, Quantile treatment effect (QTE), IVQR, LQTE 1 Introduction The recent program evaluation literature has increasingly sought to identify and estimate distributional effects. This reflects growing awareness that program impacts can be heterogeneous, and economic policies should take account of these heterogeneous impacts. Recent examples include Heckman, Smith, and Clements (1997); Black, Smith, Berger, and Noel The authors would like to thank Alberto Abadie for providing the data. yyd@uci.edu shushen@ucdavis.edu

2 (2003); Bitler, Gelbach, and Hoynes (2006a,b); Dammert (2008); Djebbari and Smith (2008), among others. These studies show that there can be substantial heterogeneity in the program effects that a focus on mean effects may miss. The distributional effects of a program may be of interest in their own right. However, it is well known that without rank preservation, distributional effects are not equivalent to the distribution of program impacts. Rank preservation requires that an individual s potential rank remains the same regardless of being treated or not. It is also known as rank invariance in the quantile treatment effect (QTE) literature. Without such an assumption, it is possible that the quantile differences of potential outcomes are zero at any quantiles, but due to individuals moving up and down in the distribution, the true treatment effects are not zero. The distribution of program impacts can be important from the policy perspective. Heckman, Smith, and Clements (1997) note that undesirable distributional aspects of programs cannot always be offset by transfers governed by a social welfare function. Some outputs of programs (such as test scores, other forms of human capital or non-transferable payments in kind) simply cannot be valued, summed, and then redistributed. One may wish to know whether a welfare program helps some individuals as well as that it does not hurt others. To gain insight into the distribution of program impacts for a training program, Heckman, Smith, and Clements (1997) explore the rank preservation assumption and find that it is not plausible. In various QTE models, rank invariance or rank similarity is required either for the identification or the interpretation of the identified treatment effects. There is a growing literature on identification and estimation of QTEs since the pioneering work of Koenker and Bassett (1978) 1. Unlike rank invariance, which requires an individual s rank in the potential out- 1 See, for recent examples, Chesher (2003, 2005) for identification of quantile effects in nonseparable models with endogeneity; Firpo (2007) for estimation of unconditional QTEs under the unconfoundedness assumption; Frolich and Melly (2013), for estimation of unconditional LQTEs, Firpo, Fortin, and Lemieux (2007) for identification of the effect of a marginal change in an exogenous explanatory variable on the unconditional quantiles of an outcome; Rothe (2010) and Imbens and Newey (2009) for identification and 2

3 come distribution to be the same across treatment states (Doksum, 1974; Lehmann, 1974), rank similarity only requires that an individual s potential rank has the same probability distribution (Chernozhukov and Hansen, 2005). For example, Chernozhukov and Hansen s (2005, 2006, 2008) instrumental variable quantile regression (IVQR) model restricts the evolution of individual ranks across treatment states and thereby identifies QTEs for the whole population. Rank invariance or less restrictively rank similarity is one of the key identifying assumptions. Similarly the nonparametric IV quantile regression models of Chernozhukov, Imbens, and Newey (2007) and Horowitz and Lee (2007) implicitly impose rank invariance by imposing a scalar disturbance. In contrast, the LQTE framework of Abadie, Angrist, and Imbens (2002) permits essential heterogeneity (cf. Heckman and Vytlacil, 2006) in treatment effects by not restricting how individuals ranks change across treatment states and, therefore, identifies QTEs only among compliers. Frolich and Melly (2013) adopt the LQTE framework to identify unconditional QTEs while allowing for covariates. Rank invariance is not required for identification in the LQTE framework but is required for the interpretation of the identified QTEs as individual causal effects. The same holds true for other studies that identify QTEs as the horizontal differences between the marginal distributions of potential outcomes across treatment states (see, e.g., Firpo, 2007 or Imbens and Newey, 2009). Comparison between the LQTE model and the IVQR model can be found in Wüthrich (2014). In light of their empirical and theoretical importance, this paper proposes nonparametric tests for rank invariance or rank similarity. We discuss their testable implications and provide identification of the implied (conditional) moment restrictions on the distribution of potential ranks. The discussion focuses on ranks in the unconditional distribution of the potential outcome, which are arguably more policy relevant (Frolich and Melly, 2013). For example, the welfare of the (unconditionally) poor or the test performance of the (unestimation of unconditional QTEs of a continuous endogenous treatment; and Powell (2014) for estimation of unconditional QTEs in a general framework. 3

4 conditionally) lowest-ranking students attracts more public attention than, say, the welfare of highly educated people with relatively low wages or the test performance of relatively low-ranking students from high-income families. Testing for conditional rank invariance or similarity is briefly discussed at the end of the paper, extending results from the benchmark unconditional tests. At first, one may think that testing for rank invariance or similarity is not feasible since the same individual cannot be observed under treatment and also under no treatment. Let Y 1 and Y 0 be the potential outcome of an individual under treatment and no treatment, respectively. It is well known that without functional form restrictions, one cannot identify the joint distribution of Y 0 and Y 1. This makes a direct test for rank invariance or similarity seemingly impossible. Several empirical papers evaluating distributional effects of welfare or training programs tentatively investigate the plausibility of rank preservation in their contexts (see, e.g., Bitler, Gelbach, and Hoynes, 2006a,b; Dammert, 2008). Treatment is exogenous, so they can directly compare the treatment and control groups to see whether the means of the covariates are similar at the same quantiles of the outcome variable. Less well known among empirical researchers is that a weaker condition, rank similarity, can also justify a causal interpretation of the horizontal differences between the potential outcome distributions as individual causal impacts. Similar to Bitler, Gelbach, and Hoynes (2006a,b), our tests draw on the implications of the joint distribution of Y 1 and observable covariates as well as that of Y 0 and observable covariates. Additionally, we address endogeneity and construct test statistics with wellbehaved asymptotic distributions. Our tests directly explore whether the distribution of potential ranks for observationally equivalent individuals remains the same across treatment states. Intuitively, rank invariance or similarity implies that individuals with the same characteristics should have the same probability distribution of potential ranks with or without treatment. In an independent and contemporaneous work, Frandsen and Lefgren (2015) also 4

5 leverage observable covariates and propose a rank similarity test. They focus on a parametric test for the equality of mean ranks under treatment and no treatment. In contrast, this paper focuses on nonparametric identification and testing of the distribution of potential ranks. Except for very mild regularity conditions, the proposed tests do not require any additional assumptions other than those used to identify and estimate QTEs (or LQTEs). The tests allow treatment to be endogenous. Covariates are permitted in estimating the unconditional QTEs, so instrumental variables are allowed to be valid conditional on covariates. Since the proposed tests rely on the predictive power of observable covariates for potential ranks, they may not work for conditional rank invariance or rank similarity if all the relevant observables are in the conditioning set of the conditional QTEs. Therefore, we do not view them as tests for the identifying assumption of the Chernozhukov and Hansen (2005, 2006, 2008) IVQR model or the nonparametric IV quantile regression model of Chernozhukov, Imbens, and Newey (2007) and Horowitz and Lee (2007). Monte Carlo simulations show that the proposed tests have good size and power properties in finite samples. The tests are applied to examine whether job training causes individuals to systemically change their ranks in the distribution of earnings using the Job Training Partnership Act (JTPA) data. We show that rank similarity can be strongly rejected for both male and female trainees. In contrast, falsification tests using age as the outcome variable fail to reject rank similarity for either group. Further, while rank similarity can be rejected at almost all quantiles for males, deviations from rank similarity occur mainly at the lower tail for females. Overall, the evidence suggests that the impacts of the JTPA training program are more complicated than may be suggested by standard QTE estimates. One should, therefore, be cautious in equating the distributional impacts of training with the effects of training on individual trainees. The rest of the paper is organized as follows. Section 2 discusses testable implications 5

6 of rank invariance and rank similarity. Section 3 provides identification of the conditional moment equality on which our test is based. Section 4 discusses the primary test statistic along with its asymptotic distribution. Section 5 presents Monte Carlo simulations. Section 6 presents the empirical application. Section 7 discusses various extensions. Concluding remarks are provided in Section 8. 2 Testable Implications of Rank Invariance or Similarity This section defines rank invariance and rank similarity, and discusses their testable implications. Here we focus on the unconditional rank, or an individual s ranking in the unconditional distribution of the potential outcome. We defer the discussion of conditional rank invariance or similarity to Section 7. Let T be a binary treatment indicator that is equal to one if an individual is treated and zero if not. Let Y be the observed outcome. Y = Y 0 (1 T )+Y 1 T, where Y 1 and Y 0, as defined above, are the potential outcomes with and without treatment, respectively. We use F t : R [0, 1] and q t : [0, 1] R to denote the unconditional cumulative distribution function and unconditional quantile function of Y t for t = 0, 1. Following Doksum (1974) and Lehmann (1974), unconditional QTEs are defined in this paper as horizontal differences between the marginal distributions of potential outcomes. Specifically, QT E(τ) = q 1 (τ) q 0 (τ), for all τ (0, 1). Let U t = F t (Y t ) be potential ranks; U t U (0, 1) for both t = 0, 1. In practice, we never observe both U 0 and U 1 for the same individual, and so do not actually know whether the same individual remains at the same rank or not across treatment states. Unless stated otherwise, rank invariance as well as rank similarity are used in this paper to refer to conditions imposed on the unconditional ranks. Definition 1. Rank invariance is the condition where U 0 = U 1. Rank invariance states that U 0 and U 1 are the same random variable, so an individual s 6

7 potential rank with or without the treatment remains exactly the same. Let X be a vector of observables and V be a vector of unobservables. Suppose that for t = 0, 1, Y t = g t (X, V ) : W R, where W is the support of (X, V ). U t = F t (g t (X, V )) is then a function that maps from W to [0, 1] and is deterministic given X and V. Rank invariance holds if and only if U 0 (X = x, V = v) = U 1 (X = x, V = v) for all (x, v) W. Immediately rank invariance implies U 0 (X = x) U 1 (X = x) for all x X, where X is the support of X. Rank invariance may be restrictive in practice, however. Consider the following thought experiment. A test is given to a random sample of students and a cloned sample that consists of the same students. The outcome of interest is the test score. The treatment is simply the cloning indicator, so the treatment effect is (supposed to be) zero for everyone. However, due to random chance or luck, a student and her clone may not have the same test score or same rank in the test score distribution. Nevertheless, if we repeat this experiment infinitely many times, the student and her clone will have the same rank distribution and any features (e.g. mean, median) of their rank distributions will be the same. Rank similarity relaxes rank invariance by allowing random deviations, or slippages in one s rank away from some common level, so that the exact rank for an individual may not be the same in different treatment states (Chernozhukov and Hansen, 2005). Assume that for both t = 0, 1, Y t X = x, V = v and hence U t X = x, V = v is not deterministic as in the case of rank invariance. In particular, let Y t = g t (X, V, S t ), where X and V are observables and unobservables that determine the common rank level of an individual, and S t is an idiosyncratic shock, such as luck as in the above thought experiment. S t is responsible for random slippages from the common rank level in the treatment state t. Note that Y t = g t (X, V, S t ) is a representation rather than a real restriction. The difference between S t and V is that unlike V, S t is realized only after the treatment is chosen. If one were to specify a treatment model, then V, if observed, would enter the treatment model while S t would not. Similar to Chernozhukov and Hansen (2005), we also...implicitly make the 7

8 assumption that one selects the treatment without knowing the exact potential outcomes... Rank similarity can then be analogously defined as follows. Definition 2. Rank similarity is the condition where U 0 (X = x, V = v) U 1 (X = x, V = v) for any (x, v) W. The above definition states that given (X, V ), the unconditional ranks of potential outcomes, U 1 and U 0, are identically distributed. Chernozhukov and Hansen (2005) consider a weaker condition, conditional rank similarity, which assumes that conditional ranks conditional on observables X are identically distributed across treatment status conditioning on X and unobservables (V here). 2 Rank invariance is a special case of rank similarity with S t being a null set of random variables and the distribution of U t X, V degenerating. The following discussion therefore focuses on rank similarity. All the conclusions hold strictly for rank invariance as well, as it is a special case of rank similarity. Lemma Given rank similarity, F X,V U0 (x, v τ) = F X,V U1 (x, v τ), for all τ (0, 1) and (x, v) W. 2. Given rank similarity, E [Y 1 Y 0 X, V ] = 1 0 (q 1(u) q 0 (u)) df U X,V (u x, v) for all (x, v) W, where F U X,V (. x, v) F Ut X,V (. x, v) for both t = 0, 1 and all (x, v) W. 3. (Main Testable Implication) Given rank similarity, F U0 X(τ x) = F U1 X(τ x) for all τ (0, 1) and x X. Lemma 1 summarizes several implications of rank similarity. The proof is given in the appendix. The first part of the lemma says that given rank similarity, the distribution of 2 To see the difference clearly, by the Skorohod representation of potential outcomes, one can write Y t = q(t, U t ), where q(t, U t ) is the quantile function of Y t, e.g., q(t, τ) is τ unconditional quantile of Y t. U t is then the unconditional potential rank when T = t. In contrast, Chernozhukov and Hansen (2005, 2006, 2008) define the conditional rank using Y t q(t, x, Ũt), so Ũt represents the conditional rank conditional on X = x. Ũ t is responsible for the heterogeneity of outcomes among individuals with the same observed characteristics x and treatment state t. Their rank invariance then requires Ũ0 = Ũ1 conditional on X, while their rank similarity requires Ũ0 Ũ1 conditional on X and the treatment model unobservables, our V here. 8

9 observable and unobservable covariates is the same at the same quantile of the potential outcome distribution with or without treatment. The statement follows from the definition of rank similarity and the Bayes rule. The second part of the lemma states that for any individual defined on observables X and unobservables V, the average treatment effect is a weighted average of the unconditional QTEs weighted by the individual s probability of being at each quantile. This result implies that although one loses the ability to point identify individual treatment effects under rank similarity (see, e.g., Imbens and Newey, 2009), the QTE still carry causal implications on individuals expected treatment effects. In fact, under the assumption of rank similarity, expected individual treatment effects may be of greater policy interest rather than exact individual treatment effect, since the random slippages S t, luck or any counterparts of it, are not manipulable by nature. The last part of Lemma 1 follows immediately from the definition of rank similarity. It states that the distribution of potential ranks among those with X = x are the same across treatment states. In another word, under rank similarity, treatment should not affect the distribution of ranks for observationally equivalent individuals. This result is the main testable implication that we employ to construct our tests for rank similarity. The next section provides identification of the potential rank distribution among observationally equivalent individuals, under essentially the same conditions as those used for identifying QTEs (or LQTEs). Given an additional assumption, we also establish consistency of our test. This additional assumption may also be used to justify the identification of the potential rank distribution for individuals that are equivalent both in observables and unobservables, and hence provides point identification of the individual expected treatment effects. 9

10 3 Identification For any τ (0, 1) and x X, let M(τ x) = F U1 X(τ x) F U0 X(τ x) be a measure of the difference between the distributions of potential ranks conditional on observables. Intuitively, this shows how the probability of staying at the same rank changes with treatment among those with X = x. This section discusses the identification of M (τ x), which serves as the basis for our proposed tests. Exogenous Treatment If T is exogenous, as in randomized experiments with perfect compliance, identification of F Ut X(τ x) for t = 0, 1 and hence M (τ x) is trivial. In this case for any τ [0, 1] and x X, we have M (τ x) = E [1(U 1 τ) X = x] E [1(U 0 τ) X = x] = E [1(Y 1 q 1 (τ) X = x] E [1(Y 0 q 0 (τ)) X = x] = E [1(Y q 1 (τ)) X = x, T = 1] E [1(Y q 0 (τ)) X = x, T = 0], where q 1 (τ) and q 0 (τ) are directly identified from the subsamples with T = 1 and T = 0, respectively. Endogenous Treatment When T is endogenous, a valid IV is required to identify QTEs and further the impact of treatment T on the distribution of ranks given X. We adopt the LQTE framework, whose identifying assumption is also suitable for our empirical application. The LQTE framework identifies distributional effects among compliers only and hence the test for rank similarity is only relevant to compliers. In models with essential heterogeneity, compliers are the largest sub-population for which QTEs can be point identified. However, if assumptions are made to identify unconditional QTEs for the whole population, one could analogously test for 10

11 rank similarity for the whole population. For example, one could adopt the IVQR model to estimate unconditional QTEs for the whole population, assuming that the IVQR identifying assumptions hold without conditioning on covariates. Under the null hypothesis of rank similarity, the QTEs identified by the IVQR model are valid, and hence the test statistic under the null would have the correct asymptotic distribution. As is clear from Theorem 1 and its proof, essentially what is required is a valid IV that allows the identification (and estimation) of unconditional QTEs in the first stage and then the effect of treatment on the distribution of ranks in the second stage. Let Z be an IV for the endogenous treatment T. For simplicity, assume that the instrument Z takes on two values, 0 and 1, although the identification results can be extended straightforwardly to a continuous IV. Further, let T z be the potential treatment status if Z = z. The observed treatment status can then be written as T = T 0 (1 Z) + T 1 Z. Define compliers as individuals with T 1 > T 0 (Angrist, Imbens, and Rubin, 1996). Further define the distribution function of the potential outcome among compliers as F t C (y) = Pr[Y t y T 1 > T 0 ], t = 0, 1. When the treatment is endogenous, only rank distributions among compliers are identified and we are interested in testing for rank similarity among compliers. Let U t C = F t C (Y t ) be the potential rank among compliers only. Similar to the last section, we define rank similarity among compliers as the condition where U 0 C (T 1 > T 0, X = x, V = v) U 1 C (T 1 > T 0, X = x, V = v), for all (x, v) W c = {(x, v) W : Pr [T 1 > T 0 X = x, V = v] > 0}. W c is the support of observables X and unobservable V among compliers. For all (x, v) W c, Pr[T 1 > T 0, X = x, V = v] > 0, so the conditional set is nontrivial. Let X c = {x X : Pr [T 1 > T 0 X = x] > 11

12 0} and F Ut C C,X(τ x) = Pr[U t C τ T 1 > T 0, X = x] : [0, 1] X c [0, 1] for both t = 0, 1. Rank similarity among compliers imply that F U1 C C,X(. x) = F U0 C C,X(. x) for all x X c. To save on notation, unless stated otherwise we will use U t to refer to potential ranks among compliers only. Note that this relabelling does not change the definition of U t in the exogenous treatment case since Z = T under exogenous treatment and everyone is a complier. Consequently, F Ut C,X(. x) is used to refer to F Ut C C,X(. x) for both t = 0, 1. The following conditions gaurantee that the testable implication stated in Lemma 1 is identified. Assumption 1. Let (Y t, T z, X, Z), t, z = 0, 1 be random variables mapped from the common probability space (Ω, F, P). The following conditions hold jointly with probability one. 1. Independence: (Y 0, Y 1, T 0, T 1 ) Z X. 2. First stage: E(T 1 ) E(T 0 ). 3. Monotonicity: Pr(T 1 T 0 ) =1. 4. Nontrivial assignment: 0 < Pr (Z = 1 X = x) < 1 for all x X. Assumption 1 is the standard LQTE identifying assumption used in Abadie, Angrist, and Imbens (2002) and Abadie (2003), except that we allow for a weaker first stage. In particular we require E(T 1 ) E(T 0 ) to hold without conditioning on X, so compliers do not have to exist at every value of X. This is because we test whether rank similarity is violated at any value X = x and because we identify and estimate the unconditional QTE instead of conditional QTE (for the latter point, see the discussion in Frolich and Melly 2013). Assumption 1.1 subsumes the exclusion restriction and the IV independence of the first stage error (Angrist, Imbens, and Rubin, 1996). Assumption 1.3 rules out defiers, which can be weakened by the assumption that there are conditionally more compliers than defiers 12

13 (see, e.g., de Chaisemartin, 2014). Assumption 1.4 is also known as a common support assumption requiring Supp (X Z = 0) = Supp(X Z = 1). When Z is a random assignment of the treatment, as in our empirical application, the independence restriction is valid without conditioning on covariates X; however, including covariates can remove any chance association between T and X or improve efficiency (Frolich and Melly, 2013). Let q t C (τ) for t = 0, 1 be the τ quantile of Y t distribution among compliers. Theorem 1. Given Assumption 1, q t C (.) is identified for both t = 0, 1, F Ut C,X(. x) is identified for both t = 0, 1 and all x X c. Let I(τ) 1 ( Y ( T q 1 C (τ) + (1 T ) q 0 C (τ) )). For all τ (0, 1) and x X c, F U1 C,X(τ x) F U0 C,X(τ x) = E [I(τ) Z = 1, X = x] = E [I(τ) Z = 0, X = x]. (1) E[T Z = 1, X = x] E[T Z = 0, X = x] Therefore, rank similarity among compliers implies that for all τ (0, 1) and x X E [I(τ) Z = 1, X = x] = E [I(τ) Z = 0, X = x.] (2) I(τ) can be seen as a rank indicator. Equation (2) in Theorem 1 states that given rank similarity, the instrument Z has no impact on the distribution of ranks conditional on X = x. Note that 1) although the conditional distribution F Ut C,X(. x) is only defined for x X c, Equation (2) holds for all values of x X since it holds trivially for any x X /X c, and 2) Theorem 1 nests the exogenous treatment as a special case: when T is exogenous, Z = T, and everyone is a complier. Theorem 1 suggests that one can test for rank similarity by a two-step procedure: first estimate the unconditional quantiles q 0 C (τ) and q 1 C (τ), and then test whether Equation (2) in Theorem 1 holds for all τ (0, 1) and x X, replacing q 0 C (τ) and q 1 C (τ) with their estimates from the first step. If desired, one can also test a particular quantile or some other features of the potential rank distribution, such as the median rank or the mean rank. 13

14 The idea of constructing our tests for rank similarity based on Equation (2) rather than the whole fraction term identified in Equation (1) is similar to the strategy used by Abadie (2002) to test conditional distributional treatment effects in IV models. Upon rejection of rank similarity, researcher can futher employ the identification result of Equation (1) to quantify the degree of violation in rank similarity for different values of τ and x. In practice, one needs the covariates X to be non-trivial, or F t C,X F t C for either t = 0 or 1, in order for a test based on Equation (2) to have any power. If F t C,X = F t C for both t = 0 or 1, then Equation (2) holds by construction regardless of whether rank similarity holds or not. 3 Testing Mean Rank Above, we have discussed testing whether the distribution of potential ranks for those with X = x remains the same across treatment states. One may also test any functionals of potential ranks, as is discussed in below. Remember that rank similarity among compliers implies that for all all x X c, F U1 C,X(. x) = F U0 C,X(. x). This further implies that for all x X c, E[U 1 C, X = x] = E[U 0 C, X = x]. Let U T U 1 +(1 T )U 0 = 1 1 ( T q 0 1 C (τ) + (1 T )q 0 C (τ) < Y ) dτ = 1 1 I(τ)dτ. U is 0 identified because I(τ) is. Following the same identification strategy that leads to Theorem 1, one can test whether the instrument Z has an impact on the mean rank conditional on X by testing whether E [U Z = 1, X = x] = E [U Z = 0, X = x] holds for all x X. In addition, the average change in outcome rank for individuals with 3 That is because if F t C,X (q t C (τ) x) = F t C (q t C (τ)), F t C,X (q t C (τ) x) = τ. Then the testable implication of rank similarity holds trivially, and the identified equation in (2) holds trivially. 14

15 observed characteristic X = x can be identified by E[U 1 C, X = x] E[U 0 C, X = x] = E [U Z = 1, X = x] E [U Z = 0, X = x] E [T Z = 1, X = x] E [T Z = 0, X = x] for all x X c. Consistency of the Rank Similarity Test So far we have only considered observables, so having F U0 X(τ x) = F U1 X(τ x) hold for all τ (0, 1) and x X is a necessary but not sufficient condition for rank similarity. Below we provide an assumption under which the testable implication in Lemma 1 is also a sufficient condition for rank similarity. Assumption 2. F V X,U0 (v x,τ) = F V X,U1 (v x,τ) for all τ (0, 1) and x X Assumption 2 says that conditional on observables X, the distribution of unobservables is the same at the same rank of the potential outcome distribution. Alternatively, once the distribution of the observables is the same at the same potential rank, the distribution of unobservables will also be the same. Assumption 2 resembles in spirit the unconfoundedness assumption that is popular in program evaluation, e.g., in various matching estimators (Rubin, 1990). Assumption 2 essentially assumes away unobservables and is untestable, just as unconfoundedness. Corollary 1. Given Assumption 2, rank similarity holds if and only if F U0 X(τ x) = F U1 X(τ x). Given Assumption 2, F X,V U0 (x,v τ) = F X,V U1 (x,v τ) if and only if F X U0 (x τ) = F X U1 (τ x). Then, by Bayes rule F U0 X,V (τ x, v) = F U1 X,V (τ x,v) if and only if F U0 X(τ x) = F U1 X(τ x). It follows that F U0 X(τ x) = F U1 X(τ x) is not only a necessary but also a sufficient condition for rank similarity. Further, under Assumption 2, rank similarity implies that for all (x, v) W c, E [Y 1 Y 0 X, V ] = 1 0 (q 1(u) q 0 (u)) df U X (u x), where F U X (u x) F Ut X(u x)du for 15

16 both t = 0, 1. This means that expected individual treatment effect is point identified and is a weighted average of the QTEs. Note that without Assumption 2, our test can only detect whether observationally equivalent individuals have the same potential rank distributions under treatment and no treatment. And if one is only interested in compliers, Assumption 2 can be made conditional on compliers and hence we have that Corollary 1 holds among compliers. 4 Testing Null Hypothesis and Test Statistic This section discusses the test statistic along with its asymptotic properties, given the identification results in the previous section. Treatment is endogenous here, with exogenous treatment following as a special case. We focus on the case where X is discrete with finite support. An extension to the continuous covariate case is provided later in Section 7. Discrete X with finite support is a reasonable assumption, since typically one has a limited number of covariates and one can always discretize covariates. These assumptions are also appropriate for our empirical application where all explanatory covariates are discrete. Our goal is to test whether equation (2) in Theorem 1 holds or not. Let X = {x 1, x 2,..., x J } be the support of X and Ω = {τ 1, τ 2,..., τ K } be a set of unconditional quantiles of interest. 4 Recall that I(τ) = 1 ( Y ( T q 1 C (τ) + (1 T ) q 0 C (τ) )). For any j = 1,..., J and k = 1,..., K, define m z j(τ k ) =E [I (τ k ) Z = z, X = x j ]. 4 The proposed test can be extended to test for rank similarity over a continuous range of quantiles. In that case, the limiting distribution in Theorem 2 becomes a centered Gaussian process. Then, a Kolmogorov- Smirnov-type L -norm test could be constructed. We do not pursue this route here because the asymptotics become quite complicated and obtaining numerical critical values in such a case is computationally demanding. Also, standard empirical practice is just to consider a finite set of quantiles. 16

17 We are interested in the null hypothesis H 0 : m 0 j(τ k ) = m 1 j(τ k ), for all j = 1,..., J 1 and k = 1,..., K. Only J 1 values of X are included in the null hypothesis, since J j=1 mz j(τ) = 1 for any τ (0, 1) and z = 0, 1. Let {Y i, T i, Z i, X i } n i=1 be a sample of i.i.d. draws of size from (Y, T, Z, X), and I i (τ) = 1 ( Y i ( T i q 1 C (τ) + (1 T i ) q 0 C (τ) )) be the rank indicator for individual i. If the unconditional quantiles q 0 C (τ k ) and q 1 C (τ k ) were known, the conditional expectation m z j(τ k ) can be estimated by the proportion of individuals with I i (τ k ) = 1 in the subsample with Z i = z and X i = x j. However, q 0 C (τ k ) and q 1 C (τ k ) are unknown, so we have to estimate them first. Here we adopt the n-consistent estimator proposed in Frolich and Melly (2013), which has the advantage of estimating unconditional quantiles but still allowing for covariates. Denote these estimates as ˆq 0 C (τ k ) and ˆq 1 C (τ k ). They can be obtained by minimizing a weighted check function: (ˆq0 C (τ k ), ˆq 1 C (τ k ) ) 1 = arg min q 0,q 1 n n ρ τ k (Y i q 0 (1 T i ) q 1 T i )ˆω i, ( where ρ τ k (u) = u (τ k 1(u < 0)) is the standard check function, ˆω i = i=1 Z i π(x i ) ) 1 Z i 1 π(x i (2T ) i 1) and π(x j ) is a consistent estimator of the instrument probability π(x) = Pr (Z i = 1 X i = x). 5 Given ˆq 0 C (τ k ) and ˆq 1 C (τ k ), the conditional expectation m z j(τ k ) for all z = 0, 1, j = 1,..., J 1 and k = 1,..., K can be estimated by ˆm z j(τ k ) = 1 n z j Z i =z,x i =x j 1 ( Y i ( T iˆq 1 C (τ k ) + (1 T i )ˆq 0 C (τ k ) )), where n z j = n i=1 1(Z i = z, X i = x j ). 5 The Stata command ivqte can be conveniently used to estimate ˆq 0 C (τ k ) and ˆq 1 C (τ k ). ˆω i in practice is replaced by the projected weights projected onto Y and T to make sure that the weight is nonnegative. 17

18 For j = 1,..., J, let ˆm z j = ( ˆm z j(τ 1 ),..., ˆm z j(τ K )) and m z j = (m z j(τ 1 ),..., m z j(τ K )) be K 1 vectors. Let ˆm z = ( ˆm z 1,..., ˆm z J 1 ) and m z = (m z 1,..., m z J 1 ) be K(J 1) 1 vectors. Let ˆV be a consistent estimator of the variance-covariance matrix of ˆm 1 ˆm 0. We propose to test the null hypothesis H 0 using a Wald-type test statistic W n ( ˆm 1 ˆm 0) ˆV 1 ( ˆm 1 ˆm 0). Asymptotics To derive asymptotic properties of the nonparametric estimator ˆm z j(τ k ) for all z = 0, 1, j = 1,..., J 1 and k = 1,..., K and of the test statistic W, we make the following assumptions regarding the underlying distribution of the data. Assumption i.i.d. data: the data (Y i, T i, Z i, X i ) for i = 1,..., n is a random sample of size n from (Y, T, Z, X). Let X and Y be the supports of X and Y, respectively. X = {x 1, x 2,..., x J } R L for some finite integer J, Y R, while T and Z are both binary. 2. For all τ Ω = {τ 1, τ 2,..., τ K }, the random variable Y 1 and Y 0 are continuously distributed with positive density in a neighborhood of q 0 C (τ) and q 1 C (τ) in the subpopulation of compliers. 3. Let π(x) = P (Z = 1 X = x). For all j = 1,..., J, its estimator ˆπ(x j ) p π(x j ) (0, 1). 4. Let f Y T,Z,X be the conditional density of Y given T, Z and X. For all t, z = 0, 1 and j = 1,..., J, f Y T,Z,X (y t, z, x j ) has bounded first derivative with respect to y. Let f Y X (y t, z, x) be the conditional density of Y given X. For all τ Ω and j = 1,..., J, f Y X (. x j ) is positive and bounded in a neighborhood of q t C (τ). Assumptions guarantee the consistency of ˆq 0 C (τ k ) and ˆq 1 C (τ k ) for k = 1,..., K. Assumption 3.4 ensures that the asymptotic variance-covariance matrix of ˆm 1 ˆm 0 is 18

19 bounded and has full rank. Let f t C be the density of potential outcome Y t among compliers, p Z,X be the joint probability of Z and X, p T Z,X (z, x j ) = p Tz X(x j ) be the probability of receiving treatment given instrument status 6, and P c = E[T = 1 Z = 1] E[T = 1 Z = 0] ( be the proportion of compliers. For all k = 1,..., K, f t C qt C (τ k ) ) > 0 by Assumption 3.2. Meanwhile, p Z,X > 0 by Assumptions 1.4 and 3.3, and P c > 0 by Assumptions 1.2 and 1.3. The following theorem discusses the asymptotic distribution of ( ˆm 1 ˆm 0 ). Theorem 2. Given Assumptions 1 and 3, n ( ˆm 1 ˆm 0 ( m 1 m 0)) N(0, V) where V is the K(J 1) K(J 1) asymptotic variance-covariance matrix. The ( J 1 j=1 K(j 1) + k, J 1 j =1 K(j 1) + k )-th element of V is equal to E [( φ 1 j(τ k ) φ 0 j(τ k ) ) ( φ 1 j (τ k ) φ0 j (τ k ))] with φ z j(τ k ) φ z j(τ k ; Y, T, Z, X) = I(τ k) m z j(τ k ) 1(Z = z, X = x j ) p Z,X (z, x j ) f Y T,Z,X(q 0 C (τ k ) 0, z, x j )(1 p T Z,X (z, x j )) ψ P c f 0 C (q 0 C (τ k )) 0 (Y, T, Z, X) f Y,T Z,X(q 1 C (τ k ) 1, z, x j )p T Z,X (z, x j ) ψ P c f 1 C (q 1 C (τ k )) 1 (Y, T, Z, X), where ψ 0 (Y, T, Z, X) and ψ 1 (Y, T, Z, X) are defined in the proof of Theorem 7 in Frolich and Melly (2007), and restated in the proof of this theorem in the Appendix. The last two terms of the φ z j(τ k ) function come from the estimation error of ˆq 0 C (τ k ) and ˆq 1 C (τ k ). If q 0 C (τ k ) and q 1 C (τ k ) were known, φ z j(τ k ) would reduce to I(τ k) m z j (τ k) p Z,X (z,x j ) 1(Z = z, X = x j ) and E [( φ 1 j(τ k ) φ 0 j(τ k ) ) ( φ 1 j (τ k ) φ0 j (τ k ))] would be simplified to z=0,1 mz j(τ k τ k ) m z j(τ k )m z j(τ k ) if j = j, and 0 j j. 6 Notice that by the independence assumption in Assumption 1, p Tz X(x j ) = E[T z X = x j ] = E[T z Z = z, X = x j ] = E[T Z = z, X = x j ] = p T Z,X (z, x j ). 19

20 Given the above theorem, it follows immediately that with a consistent estimator ˆV for the variance-covariance matrix V, the test statistic W converges to a Chi-squared distribution with K(J 1) degrees of freedom under the null hypothesis, where K(J 1) is the number of moment restrictions in H 0 as well as the rank of V given our assumption. Under the alternative hypothesis, the test statistic W explodes. Given the complicated nature of the variance-covariance matrix resulting from the first stage estimation of the unconditional quantile functions, we recommend estimating V by bootstrapping. Notice that the set X is finite here. In Section 7, we discuss an extension that allows J to increase with sample size. There, the estimation of the unconditional quantile functions does not play a role in the asymptotics and the variance-covariance matrix of the moment function estimators can be estimated analytically. Let the critical value c α of the test be the (1 α) 100-th percentile of the χ 2 (K(J 1)) distribution. Define the decision rule of the test as reject the null hypothesis H 0 if W > c α. The following Corollary summarizes the asymptotic properties of the proposed test. Corollary 2. Given Assumptions 1 and 3, the constant c α which is the (1 α) 100% quantile of the χ 2 (K(J 1)) distribution, the test satisfies: 1. if H 0 is true, lim n P (W > c α ) = α, and 2. if H 0 is false, lim n P (W > c α ) = 1. Note that Assumption 3.5 guarantees that for all j = 1,..., J 1 and k = 1,..., K, φ 1 j(τ k ) φ 0 j(τ k ) is not zero and hence the variance-covariance matrix V has full rank. In practice with a finite sample, it is possible that for some small cells defined by values of X and Z, both ˆm 1 j(τ k ) and ˆm 0 j(τ k ) degenerate. In that case, ˆV would not have full rank. The 20

21 effective number of moment restrictions in H 0 is then the rank of ˆV, which should be used as the degrees of freedom for the the test statistic. The Mean Rank Similarity Test Let m z j = E[U Z = z, X = x j ] for both z = 0, 1. As discussed in Section 2, one can also construct a mean test for rank similarity by testing the null hypothesis H 0,mean : m 0 j = m 1 j, for all j = 1,..., J 1. Let R(y, t) = ( tq 1 C (τ) + (1 t)q 0 C (τ) y ) dτ be the rank function such that U = R(Y, T ). Let ( τ 1,...τ S) be S random numbers drawn from a uniform distribution that is independent of the data. The rank of any interior point y can be estimated by ˆR(y, t) = 1 S S 1 (( tˆq 1 C (τ s ) + (1 t)ˆq 0 C (τ s ) ) y ), s=1 which converges to R(y, t) in probability as S, n. Let Ûi = ˆR(Y i, T i ) for i = 1,..., n. For z = 0, 1 and j = 1,..., J, define m z j = 1 n z j Z i =z,x i =x j Û i as the estimator of m z j. 21

22 Let m z = ( m z 1,..., m z J 1 ) and m z = ( m z 1,..., m z J 1 ) be J 1 dimensional vectors. Since [ 1 m z j = E = 1 0 m z j = 1 n z j = 1 S 0 1 (( T q 1 C (τ) + (1 T )q 0 C (τ) ) < Y ) ] Z = z, X = x ( 1 m z j (τ) ) dτ, and 1 S 1 (( T iˆq 1 C (τ s ) + (1 T i )ˆq 0 C (τ s ) ) ) Y i S Z i =z,x i =x j s=1 S ( 1 ˆm z j (τ s ) ), s=1 the asymptotic property of m 1 m 0 can be derived straightforwardly following results of Theorem 2. It is summarized in the following Corollary. Corollary 3. Given Assumptions 1 and 3, when S, n n ( m 1 m 0 ( m 1 m 0)) N(0, V mean ), where V mean is the (J 1) (J 1) asymptotic variance-covariance matrix. The (j, j )-th element of V mean is E [( 1 ( 0 φ 1 j(τ) ) dτ 1 ( 0 φ 0 j(τ) ) ) ( 1 ( dτ 0 φ 1 j (τ)) dτ 1 ( )] 0 φ 0 j (τ)) dτ, where 1 0 ( φ z j(τ) ) dτ = U mz j p Z,X (z, x j ) 1(Z = z, X = x j) f Y T,Z,X (q 0 C (τ) 0, z, x j ) dτ f 0 C (q 0 C (τ)) ( 1 PT Z,X (z, x j ) ) ψ 0 (Y, T, Z, X) f Y T,Z,X (q 1 C (τ) 1, z, x j ) dτ P T Z,X(z, x j )ψ 1 (Y, T, Z, X). f 1 C (q 1 C (τ)) P c P c Again, the last two terms of 1 ( 0 φ z j(τ) ) dτ come from the estimation error in the first step estimation of q 0 C (.) and q 1 C (.). 7 This theorem leads to the following Wald-type test 7 If q 0 C (.) and q 1 C (.) were known, 1 ( 0 φ z j (τ) ) U m dτ would reduce to z j p Z,X (z,x 1(Z = z, X = x j) j) and the off-diagonal elements in matrix V mean would reduce to zero. 22

23 statistic W mean n ( m 1 m 0) V 1 ( m 1 m 0) with bootstrapped variance-covariance matrix V. The test statistic W mean converges to a Chi-squared distribution with J 1 degrees of freedom under the null hypothesis, and explodes under the alternative. Therefore, a level α mean test for rank similarity can be constructed by the rejection rule that rejects H 0,mean if W mean exceeds the (1 α) 100% quantile of the χ 2 (J 1) distribution. Note that 1) in practice, the mean rank similarity test is not computationally simpler than the rank similarity test using the distribution of ranks, due to the estimation of the whole quantile curve and then individual ranks, and 2) since rank similarity is a distributional concept, the mean rank similarity test, which only tests one feature of the distribution, may have less power against different alternatives. We demonstrate this point in both the Monte Carlo simulations and the empirical application. 5 Simulation In this section we conduct Monte Carlo studies to illustrate the finite sample size and power properties of the proposed tests. We study both the distributional test and the mean test for rank similarity. We first consider the exogenous treatment case and then the endogenous treatment case. For all the Monte Carlo simulations, the observed covariate X is generated to take values 0.4, 0.8, 1.2, 1.6, and 2 with equal probability. The unobserved covariate V, as well as the idiosyncratic shocks S 0 and S 1 are generated following independent N(0, 1) distributions. The potential outcomes under treatment and control are Y 0 = X + V + S 0 and Y 1 = X + 1 βxv + V + S 1. For the exogenous treatment case (DGP 1), the treatment variable T is randomly generated. For the endogenous treatment case (DGP 2), we generate a random instrument Z and let the treatment variable T determined by the difference in potential 23

24 outcomes and the realization of Z. To summarize, DGP 1: P (X = 0.4j) = 0.2, for j = 1,..., 5; V, S 0, S 1 N(0, 1); P (T = 0) = P (T = 1) = 0.5; Y = X + (1 βxv )T + V + S 0 + (S 1 S 0 )T. DGP 2: P (X = 0.4j) = 0.2, for j = 1,..., 5; V, S 0, S 1 N(0, 1); P (Z = 0) = P (Z = 1) = 0.5; T = 1(0.15(Y 1 Y 0 ) + Z 0.5 > 0); Y = X + (1 βxv )T + V + S 0 + (S 1 S 0 )T. For both GDP s, rank similarity holds when β = 0 and is violated when β 0. Figure 1 illustrate the design of DGP 1 for different β values. In each graph, the solid lines represent the conditional expectation E[1(Y q 1 (τ)) X = x j, T = 1] for x j = 0.4, 0.8,..., 2, while the dotted lines represent the conditional expectation E[1(Y q 0 (τ)) X = x j, T = 0]. When b = 0, the treatment effect is a constant and hence rank is invariant to treatment. Consequently, the two conditional expectations are equal for all values of x j. When b 0, the treatment effect depends on both X and V and rank similarity is violated. Figure 1 shows that the degree of violation increases with the scale of β. The violation is quite small with β = 1 and substantially larger with β = 2 and 3. Also, with β = 2 and 3 rank similarity is violated more strongly at the lower quantiles. Table 1 reports the small sample performance of the proposed rank similarity tests under DGP 1. For each β = 0, 1, 2, 3, we draw samples of size 500, 1000, 1500, 2000 and For each sample size, we conduct the mean test of rank similarity as well as the distributional tests of rank similarity with four different sets of quantiles. All test statistics are constructed using bootstrapped variance-covariance matrices calculated with 1,000 bootstrap repetitions and compared to the 5% critical value. For each test, 1,000 simulations are conducted and we report in the table the rejection rates among all simulations. Results for β = 0 in the table shows that the proposed distributional test and mean test for rank similarity both control size well. Results for β = 1, 2 and 3 show that the proposed tests have power increasing with the sample size. Further, when β = 2 and 3 and the violation in rank similarity is nontrivial, the rejection rate goes to one rapidly with the 24

25 Figure 1: Conditional Distributions of Potential Ranks b=0 b=1 Conditional CDF Conditional CDF Percentile Rank X=0.4, T=1 X=0.4, T=0 X=0.8, T=1 X=0.8, T=0 X=1.2, T=1) X=1.2, T=0 X=1.6, T=1 X=1.6, T=0 X=2.0, T=1 X=2.0, T= Percentile Rank X=0.4, T=1 X=0.4, T=0 X=0.8, T=1 X=0.8, T=0 X=1.2, T=1) X=1.2, T=0 X=1.6, T=1 X=1.6, T=0 X=2.0, T=1 X=2.0, T=0 b=2 b=3 Conditional CDF Conditional CDF Percentile Rank X=0.4, T=1 X=0.4, T=0 X=0.8, T=1 X=0.8, T=0 X=1.2, T=1) X=1.2, T=0 X=1.6, T=1 X=1.6, T=0 X=2.0, T=1 X=2.0, T= Percentile Rank X=0.4, T=1 X=0.4, T=0 X=0.8, T=1 X=0.8, T=0 X=1.2, T=1) X=1.2, T=0 X=1.6, T=1 X=1.6, T=0 X=2.0, T=1 X=2.0, T=0 Note: The plotted conditional expectations are simulated with sample size equal to 1,000,000. Table 1: size and power of the proposed test for the exogenous treatment case N β = 0 β = 1 Test 1: Ω = {0.5} Test 2: Ω = {0.2, 0.3, 0.4} Test 3: Ω = {0.5, 0.6, 0.7, 0.8} Test 4: Ω = {0.2, 0.3,..., 0.8} Test 5: Mean Test β = 2 β = 3 Test 1: Ω = {0.5} Test 2: Ω = {0.2, 0.3, 0.4} Test 3: Ω = {0.5, 0.6, 0.7, 0.8} Test 4: Ω = {0.2, 0.3,..., 0.8} Test 5: Mean Test

26 Figure 2: Small Sample Performance of the Proposed Tests Sample Size = 1000 beta = 2 Rejection Rate Rejection Rate beta Distributional Test 1 Distributional Test 2 Distributional Test 3 Distributional Test 4 Mean Test Sample Size Distributional Test 1 Distributional Test 2 Distributional Test 3 Distributional Test 4 Mean Test Note: The plotted conditional expectations are simulated with sample size equal to 1,000,000. increase in sample size. In addition, since rank similarity is violated more strongly at the lower quantiles when β = 2 and 3, the distributional tests of rank similarity are seen to have higher power when the set of testing quantiles includes the lower ranks. In contrast, the mean and median tests for rank similarity have lower power. Figure 2 demonstrates the power performance of the proposed tests visually. Simulations reported in the left graph fix the sample size at 1000 and vary the value of β, while those reported in the right graph fix β at 2 and vary the sample size. It is clear that the distributional tests with a range of different quantiles values have the best small sample performance given the data generating process under study. Next we study the performance of the proposed tests under DGP 2 with an edogenous treatment. Figure 3 illustrates the design of the DGP. The solid lines now represent the m 1 j(τ) = E[I(τ) X = x j, Z = 1] functions, while the dotted lines m 0 j(τ) = E[I(τ) X = x j, Z = 0]. When b = 0, rank similarity holds and m 1 j(τ) = m 0 j(τ) for all j = 1,..., 5. When b = 1, the violation in m 1 j(τ) = m 0 j(τ) is very small and hard to detect with small samples. When b = 2 and 3, the violation has larger magnitude, especially at the lower quantiles. When compared with the corresponding graphs in Figure 1, the violation in the moment equalities are found to be weaker in DGP 2, when the treatment is endogenous. Therefore, we will expect the tests to have smaller power under DGP 2 for given a sample size, and this 26

27 Figure 3: Conditional Distributions of Potential Ranks b=0 b=1 Conditional CDF Conditional CDF Percentile Rank X=0.4, Z=1 X=0.4, Z=0 X=0.8, Z=1 X=0.8, Z=0 X=1.2, Z=1 X=1.2, Z=0 X=1.6, Z=1 X=1.6, Z=0 X=2.0, Z=1 X=2.0, Z= Percentile Rank X=0.4, Z=1 X=0.4, Z=0 X=0.8, Z=1 X=0.8, Z=0 X=1.2, Z=1 X=1.2, Z=0 X=1.6, Z=1 X=1.6, Z=0 X=2.0, Z=1 X=2.0, Z=0 b=2 b=3 Conditional CDF Conditional CDF Percentile Rank X=0.4, Z=1 X=0.4, Z=0 X=0.8, Z=1 X=0.8, Z=0 X=1.2, Z=1 X=1.2, Z=0 X=1.6, Z=1 X=1.6, Z=0 X=2.0, Z=1 X=2.0, Z= Percentile Rank X=0.4, Z=1 X=0.4, Z=0 X=0.8, Z=1 X=0.8, Z=0 X=1.2, Z=1 X=1.2, Z=0 X=1.6, Z=1 X=1.6, Z=0 X=2.0, Z=1 X=2.0, Z=0 Note: The plotted conditional distributions of the percentile ranks of potential outcomes are simulated with sample size equal to 1,000,000. is what we find in rejection rates reported in Table 2. In general, however, the proposed tests again control size well and have power increasing with the sample size. The distributional tests with a range of different quantiles values also outperform the mean and median tests in terms of power. 6 Empirical Application The impact of job training programs on the earnings of trainees, especially those with low income, is of great interest to both policy makers and economists. Abadie, Angrist, and Imbens (2002) and Chernozhukov and Hansen (2008) utilize data from a randomized experiment conducted under the JTPA to estimate the impact of the JTPA training program on the distribution of trainee earnings. They both focus on conditional QTEs. An interesting feature of the JTPA training experiment is that there are almost no always takers, so the 27