Comparing Distributional Policy Parameters between Populations with Different Outcome Structures

Transcription

1 Comparing Distributional Policy Parameters between Populations with Different Outcome Structures Anthony Strittmatter September 2014 Discussion Paper no School of Economics and Political Science, Department of Economics University of St.Gallen

2 Editor: Publisher: Electronic Publication: Martina Flockerzi University of St.Gallen School of Economics and Political Science Department of Economics Bodanstrasse 8 CH-9000 St. Gallen Phone Fax seps@unisg.ch School of Economics and Political Science Department of Economics University of St.Gallen Bodanstrasse 8 CH-9000 St. Gallen Phone Fax

3 Comparing Distributional Policy Parameters between Populations with Different Outcome Structures 1 Anthony Strittmatter Author s address: Anthony Strittmatter, Ph.D. SEW-HSG Varnbüelstrasse 14 CH-9000 St. Gallen Phone Fax Anthony.strittmatter@unisg.ch 1 I am also affiliated with the Albert-Ludwigs-University Freiburg. I am grateful for helpful comments and suggestions from Jaap Abbring, Manuel Arellano, Stéphane Bonhomme, Annabelle Doerr, Bernd Fitzenberger, Michael Lechner, Jan Nimczik, Stefan Sperlich and Andreas Steinmayr. Further, I would like to thank participants at the NOeG in Vienna, SSES in Zürich, EALE in Torino, SOLE in Washington D.C., and IAAE in London for useful comments. The usual disclaimer applies.

4 Abstract In this study, we promote the estimation of alternative distributional policy parameters that identify horizontal changes in conditional distributions at quantiles from a reference distribution. The motivation underlying the introduction of these new parameters is the comparison of distributional policy parameters between populations with different outcome structures. If the outcome structures of different populations are not taken into consideration, then such comparisons could potentially yield to results that cannot be associated with the policy variation of interest. The suggested alternative distributional policy parameters account for the outcome structure and enable straightforward comparisons between different populations. The relevance of these parameters is demonstrated in an application of the Job Training Partnership Act. Keywords Quantile Regression, Program Evaluation, Potential Outcome Distribution. JEL Classification C21, J38, C46.

5 1 Introduction Most evaluation studies focus on average treatment eects for a specic population. Nevertheless, the idea of eect heterogeneity is widely acknowledged. Distributional policy parameters emphasize heterogeneity with respect to the outcome variable. Recently, many studies have considered the estimation of quantile treatment eects (QTE). As an example, consider a government introduces a training program with the aim to support economically disadvantaged individuals. The treatment is participation in the training program and the outcome variable is earnings. QTE enable the comparison of quantiles between the counterfactual earnings distributions when everyone or nobody in the population of interest would participate in the program. Two programs might be evaluated dierently, when one shifts the lower and the other the upper part of the earnings distribution, even when the average eects are similar. Many studies do not only consider eect heterogeneity with respect to the outcome variable, but also with respect to dierent populations (e.g. Angrist, Chernozhukov, and Fernandez-Val, 2006, Angrist, Lang, and Oreopoulos, 2009, Bitler, Gelbach, and Hoynes, 2006, Eren and Ozbeklik, 2014). To make an example, Abadie, Angrist, and Imbens (2002) evaluate the inuence of participation in the Job Training Partnership Act (JTPA) on the earnings distributions of males and females separately. The JTPA was a program in the 1980s and 1990s, providing training for economically disadvantaged individuals. Earnings distributions of males and females are typically unequal even in the counterfactual distribution when nobody in the population of interest is treated. This makes comparisons between QTE by gender complicated to interpret, because not everything else than the gender is held constant between these two populations. In particular it could be, that QTE dierences between males and females can be partly associated with the dierent values of the outcomes at which they are measured, which might be unrelated to the treatment. Since the aim of QTE is to estimate heterogeneity with respect to the relative position in the outcome distribution, one might want to take the dierent earnings structures between males and females into consideration. 3

6 Up to date, only the study of Bitler, Hoynes, and Domina (2014) explicitly takes account of the outcome structure when comparing QTE between dierent populations. They suggest putting QTE of each population on the same absolute scale. For this purpose, they use quantiles from the observed distribution of the control group as anchor points. They establish the term translated quantile treatment eects (TQTE) for this policy parameter. However, they do not give any guidance how to implement this parameter in a specic application a researcher might has in mind. In this study, we take up this discussion and formally dene TQTE. Our contribution is the suggestion of dierent ways to implement this parameter. For this we use the framework of relative distributions (see Handcock and Morris, 1998, Morris, Bernhardt, and Handcock, 1994). With this concept, it is possible to make a mapping between ranks in the conditional and the reference distribution. In contrast to Bitler, Hoynes, and Domina (2014), we suggest using quantiles from the counterfactual outcome distributions as anchor points. We will argue that these reference distributions make more meaningful interpretations of the policy parameter of interest possible, than using the observed distribution. Therefore, we suggest a new set of distributional policy parameters based on the concept of TQTE. Comparisons of these parameters between dierent populations are straightforward to interpret, because they are evaluated at the same reference distribution. The relevance of TQTE is demonstrated with an application of the JTPA. We estimate the impact of the randomized oer to participate in the JTPA on the earnings distribution of males and females separately. Our ndings suggest that QTE for females are larger than QTE for males in the lower part of the respective potential outcome distributions. Contrarily, QTE for males exceed that of females in the upper part of the respective potential outcome distributions. Applying the concept of TQTE, we show that these ambiguous results can be explained by the dierent earnings structures of males and females. We show that TQTE for females are always larger or equal than TQTE for males. This suggests that the impact of the JTPA on the females earnings distribution dominates that on the males earnings distribution at any given value of the outcome. 4

7 TQTE propose one possible way to make non-separable transformations between QTE for dierent populations. The transformations that we apply are well established in the literature on distributional policy parameters and their decomposition. Altonji and Blank (1999), Blau and Kahn (1992, 2003), and Juhn, Murphy, and Pierce (1991, 1993) propose similar transformations to decompose wage structures into dierent components. Machado and Mata (2005) drastically improve the estimation of transformed conditional quantiles, by suggesting to make the proposed transformation only at specic quantiles instead of for each actual value of the outcome separately. Further contributions about asymptotic features of the proposed transformation are made by Albrecht, Björklund, and Vroman (2003) and Melly (2006). Athey and Imbens (2006) and Bonhomme and Sauder (2011) apply these principal transformation concepts to an extended version of the dierence-in-dierence estimator. However, none of these studies has the focus on comparing QTE between dierent populations. The remainder of this study is structured as follows. In the next section we formally dene QTE. In Section 3 we illustrate the problem of comparing QTE between dierent populations in a very simplistic example. In Section 4 we discuss intensively the denition and implementation of TQTE. In Section 5 we discuss briey estimation and provide some nite sample properties of TQTE. The application is presented in Section 6 and we conclude in Section 7. 2 Quantile treatment eects (QTE) Let F Y D (y d) denote the cumulative distribution function (cdf) of Y given D, where D is a binary treatment indicator. The uth quantile of Y given D is, Q Y D (u d) = inf{y : F Y D (y d) u}, for the real u (0, 1). Q Y D (u d) is the minimum value of Y such that the amount u of the probability distribution given D lies below this value. 5

8 Following the framework of Rubin (1974), the potential outcomes are denoted by Y (d), where d = 1 under treatment and zero otherwise. For each individual only the realized outcome, Y = Y (1) D + Y (0) (1 D), is observed, implying that either Y (0) or Y(1) is counterfactual. Under the assumption of random treatment assignment (Y (1), Y (0)) D, the potential outcome distributions are identied F Y (d) (y) = F Y D (y d). The corresponding potential quantiles are Q Y (d) (u) = Q Y D (u d). QTE are dened as, QT E = Q Y (1) (u) Q Y (0) (u), the dierence between the potential quantiles. Accordingly, QTE identify horizontal differences in the potential outcome distributions at specic quantiles. It is not possible to identify individual specic QTE in this setting, unless very strong assumptions like rank preservation are made. Rank preservation implies that individuals do not change their rank as a result of their treatment status (see discussion in Firpo, 2007). 1 Now we discuss conditional quantile treatment eects (CQTE). This might be QTE for two mutually exclusive populations, for example gender. Let F Y (d) X (y x) = F Y D,X (y d, x) denote the conditional potential outcome distribution of Y (d) given X, where X denotes some exogenous pre-treatment control variables. The uth quantile of Y (d) given X is, Q Y (d) X (u x) = inf{y : F Y D,X (y d, x) u}. Q Y (d) X (u x) is the minimum value of Y (d) such that the amount u of the conditional probability distribution lies below this value. CQTE are dened by, CQT E = Q Y (1) X (u x) Q Y (0) X (u x), 1 Chernozhukov and Hansen (2005) use the terminology rank invariance for the same assumption. Rank similarity is a weaker assumption than rank preservation since it allows for random deviations in the ranks (see Chernozhukov and Hansen, 2005, for details). 6

9 the dierence between the conditional potential quantiles. In the following we are interested to compare CQTE between dierent populations, i.e. for dierent values of X. If X is a binary indicator for gender, then we would compare CQTE between the male and female population. 3 Example To explain the problem when comparing CQTE between dierent populations, we start with a simplistic model. The general arguments of this section extend to more complicated settings in a natural way. Consider a classical location-scale model. The model for Y (0) is, Y (0) = X β + X γ ɛ, X γ > 0, where the impact of X on Y (0) is linearly increasing with the continuously distributed error term ɛ, with ɛ X. In this special case, Y (0) = Q Y (0) X (U X) = X β + X γ Q ɛ (U), so that Q Y (0) X (u x) = x β(u) with β(u) = β + γ Q ɛ (u) for all u (0, 1). Accordingly, the slope functions β(u) are monotone and linearly increasing in u. The model for Y (1) is, Y (1) = X δ + (1 + α) Y (0), α > 0, such that Q Y (1) X (u x) = δ CQT E (u, x) + x β(u), with δ CQT E (u, x) = Q Y (1) X (u x) Q Y (0) X (u x) = x δ + α Q Y (0) X (u x), being the CQTE. CQTE have in this example a part that depends on the characteristics, x δ, and a part that depends on the level of the outcome under non-treatment, 7

10 α Q Y (0) X (u x). The level or location eect is linearly increasing in Q Y (0) X (u x). Accordingly, the size of the treatment eect depends on the level of the outcome under non-treatment, which is a reasonable assumption in many economic settings. The slope α is similar for all values of X. At any given xed value of the outcome Y (0), the treatment eects of populations with dierent values of X dier only with respect to x δ and not with α Q Y (0) X (u x). We are interested in CQTE for two dierent values of X, lets say x 1 and x 2. The values x 1 and x 2 might represent two mutually exclusive populations, e.g. males and females, young and old, black and white. CQTE for these populations can be indicated by, δ CQT E (u, x 1 ) = Q Y (1) X (u x 1 ) Q Y (0) X (u x 1 ) = x 1δ + α Q Y (0) X (u x 1 ), and δ CQT E (u, x 2 ) = Q Y (1) X (u x 2 ) Q Y (0) X (u x 2 ) = x 2δ + α Q Y (0) X (u x 2 ). A comparison between these two distributional parameters leads to, δ CQT E (u, x 1 ) δ CQT E (u, x 2 ) = (x 1 x 2)δ + α (Q Y (0) X (u x 1 ) Q Y (0) X (u x 2 )). The rst term on the right side is the characteristics eect, (x 1 x 2)δ. This is the part of the treatment eect which can be associated with the characteristics X. In this simple example, the characteristics eect shifts the entire outcome distribution to the left or right. Accordingly, the characteristics eect does not depend on the value of the outcome in this example. The last term on the right side is the structural eect, α (Q Y (0) X (u x 1 ) Q Y (0) X (u x 2 )). This term cannot be associated with an interaction between X and the treatment eect, because α is similar under dierent values of X (in this simple example). The variation in this term comes from the dierent values of Y (0) at which the CQTE are measured under x 1 and x 2. When the outcome structures would be similar under x 1 and x 2, then Q Y (0) X (u x 1 ) = Q Y (0) X (u x 2 ) and the last term disappears. Otherwise, this last term can be associated with dierences in the outcome structures, which are not necessarily related to the treatment. In this study, we propose 8

11 a number of alternative distributional policy parameters that take the outcome structure into consideration. 4 Alternative policy parameters A way to account for the outcome structure is to x the value of the outcome at which CQTE are measured. This requires a reference distribution. Quantiles of this reference distribution are used as xed anchor points. CQTE are translated to these reference quantiles to put each CQTE on the same absolute scale. Therefore, the concept of relative distributions is applied (Morris, Bernhardt, and Handcock, 1994). Let Q Yr (u) denote the quantiles of the reference distribution. Relative distributions are dened as, R d (u) = F Y (d) X (Q Yr (u) x), which are assumed to be continuously distributed and strictly increasing in the interval (0, 1). The relative distribution can be interpreted as the proportion of the conditional population whose outcome lies below the u th quantile of the reference distribution. Accordingly, the relative distribution is the rank that an individual with the outcome value Q Yr (u) would have, if the reference distribution would have a similar structure as the conditional potential outcome distribution F Y (d) X (y x). Let Y(d) X denote the support of Y (d) X. Let Y r denote the support of Y r. Assume that Y(d) X Y r and dene, u dxr = min F Y r (y), u dxr = max F Y r (y). y Y(d) X y Y(d) X Bitler, Hoynes, and Domina (2014) established the term translated conditional quantile treatment eects (TQTE) for CQTE that are estimated at the relative distribution. For any u [u dxr, u dxr ], TQTE are dened by, δ d T QT E(u, x) = Q Y (1) X (R d (u) x) Q Y (0) X (R d (u) x), 9

12 implying that TQTE measures the horizontal distance between the two conditional potential outcome distributions. Only the location has shifted from u to R d (u) in comparison between CQTE and TQTE. TQTE make a mapping between the conditional and the reference distributions. Notice that δ CQT E (R d (u), x) = δt d QT E (u, x). Further, CQTE are nested in TQTE in the special case when all required distributions are continuous, strictly increasing, and F Y (d) X (y x) = F Yr (y). Then the outcome structures of the conditional and reference distributions are equal and R d (u) = F Y (d) X (Q Yr (u) x) = u (see Lemma A.1 in Athey and Imbens, 2006). Several decisions have to be made to give TQTE a meaningful interpretation. First, we have to choose the appropriate reference distribution. A natural choice would be to use the observed outcome distribution. For example, Bitler, Hoynes, and Domina (2014) use the observed outcome distribution of the control units as reference distribution. Another possibility is to use the conditional potential outcome distribution as reference distribution. This corresponds to the specications of Machado and Mata (2005) in their decomposition approach. Such procedures have the drawback that the results depend on the choice of the conditional distribution. Therefore, one might prefer to use the unconditional potential outcome distribution as reference distribution. Second, we have to decide whether to use the relative distribution under treatment or non-treatment. If there is a strong prior how non-treatment is dened, then one can possibly argue from an economic perspective, that policy eects at relative distributions, which are not already aected by the treatment, are more interesting. Nevertheless, the results critically depend on the choice of the treatment status of the relative distribution. This might be a problem, because TQTE could be dierent, when treatment and nontreatment are switched. Before we go into the details of TQTE, we illustrate the basic idea in the stylized Figure 1. The abscissa displays the continuous outcome variable Y and the ordinate shows the cumulative distribution function. CQTE are indicated by the horizontal distance between D and C at rank u. TQTE identify the relative rank R 0 (u), which is dened under non- 10

13 Figure 1: Illustration of CQTE and TQTE. F 1 F Yr (y) u R 0 (u) F Y (0) X (y x) F Y (1) X (y x) A B C D Note: CQTE are the horizontal distance between D and C at rank u. TQTE identify the relative rank R 0 (u) that refers to the reference distribution and estimates the horizontal distance between A and B at the relative distribution. Y treatment in this gure, in the rst place. The relative distribution refers to rank u in the reference distribution. Afterwards, the horizontal distance in the conditional potential outcome distributions between B and A is estimated at the relative distribution. 4.1 Observed reference distribution First we consider the observed outcome distribution as reference distribution. Let Q Y (u) be the u th quantile of the observed outcome distribution. The relative distribution is R d O (u) = F Y (d) X(Q Y (u) x), which indicates the proportion of F Y (d) X (y x) lying below the u th quantile of the observed outcome distribution. The corresponding TQTE are, δ d O(u, x) = Q Y (1) X (R d O(u) x) Q Y (0) X (R d O(u) x). (1) 11

14 We have to specify the treatment status of the relative distribution. For now we choose to specify d = 0. The rst right hand term in (1) can be extended to, Q Y (1) X (R 0 O(u) x) = Q Y (1) X (F Y (0) X (Q Y (u) x) x), using the relative distribution. If the conditional potential outcome distributions are continuous and strictly increasing, then the second right hand term in (1) equals, Q Y (0) X (R 0 O(u) x) = Q Y (0) X (F Y (0) X (Q Y (u) x) x) = Q Y (u), using Q Y (F Y (y)) = y. Accordingly, δo 0 (u, x) can be characterized by, δ 0 O(u, x) = Q Y (1) X (F Y (0) X (Q Y (u) x) x) Q Y (u). Analogue transformations for d = 1 result in, δ 1 O(u, x) = Q Y (u) Q Y (0) X (F Y (1) X (Q Y (u) x) x). The parameters δo 0 (u, x) and δ1 O (u, x) are not necessarily similar. A comparison between δ d O (u, x 1) and δ d O (u, x 2) leads to, δ d O (u, x 1 ) δ d O (u, x 2) = QY (1) X (F Y (d) X (Q Y (u) x 1 ) x 1 ) Q Y (1) X (F Y (d) X (Q Y (u) x 2 ) x 2 ), which accounts for the outcome structure. The conditional treatment eects impute the observed outcome structure of F Y (y). This means they are measured at the observed quantile Q Y (u). 12

15 4.2 Conditional reference distribution Next we consider the conditional potential outcome distribution as reference distribution. Dene the relative distribution RC d (u) = F Y (d) X(Q Y (d) X (u x ) x), which indicates the proportion of the population with X = x whose outcome lies below the u th quantile of the population with X = x. TQTE at conditional quantiles are dened by, δ d C(u, x) = Q Y (1) X (R d C(u) x) Q Y (0) X (R d C(u) x). In the following discussion we choose R 0 C2 (u) = F Y (d) X(Q Y (0) X (u x 2 ) x) to be the relative distribution. Obviously one could dene analogue policy parameters by replacing x 2 with x 1 or non-treatment with treatment. Making similar transformations as before, δc2 d (u, x) could be represented by, δ d C2(u, x) = Q Y (1) X (F Y (0) X (Q Y (0) X (u x 2 ) x) x) Q Y (0) X (F Y (0) X (Q Y (0) X (u x 2 ) x) x)). This allows us to dene, δ d C2(u, x 1 ) =Q Y (1) X (F Y (0) X (Q Y (0) X (u x 2 ) x 1 ) x 1 ) Q Y (0) X (F Y (0) X (Q Y (0) X (u x 2 ) x 1 ) x 1 ), =Q Y (1) X (F Y (0) X (Q Y (0) X (u x 2 ) x 1 ) x 1 ) Q Y (0) X (u x 2 ), and δ d C2(u, x 2 ) =Q Y (1) X (F Y (0) X (Q Y (0) X (u x 2 ) x 2 ) x 2 ) Q Y (0) X (F Y (0) X (Q Y (0) X (u x 2 ) x 2 ) x 2 )), =Q Y (1) X (u x 2 ) Q Y (0) X (u x 2 ) = δ CQT E (u, x 2 ), using Q Y (F Y (y)) = y and F Y (Q Y (u)) = u. Both parameters δ d C2 (u, x 1) and δ d C2 (u, x 2) use the outcome structure of F Y (0) X (y x 2 ). A comparison between δ d C2 (u, x 1) and δ d C2 (u, x 2) 13

16 leads to, δ d C2(u, x 1 ) δ d C2(u, x 2 ) = Q Y (1) X (F Y (0) X (Q Y (0) X (u x 2 ) x 1 ) x 1 ) Q Y (1) X (u x 2 ), which takes the outcome structure into consideration. In this case, conditional treatment eects are dened at Q Y (0) X (u x 2 ). A nice property of δ 0 C2 (u, x) is that if F Y (0) X(y x 2 ) = F Y (0) X (y x 1 ), then Q Y (1) X (F Y (0) X (Q Y (0) X (u x 2 ) x 1 ) x 1 ) = Q Y (1) X (F Y (0) X (Q Y (0) X (u x 1 ) x 1 ) x 1 ) = Q Y (1) X (u x 1 ), using F Y (Q Y (u)) = u. Accordingly, δc2 0 (u, x 1) coincides with δ CQT E (u, x 1 ) if the structures of the conditional potential outcome distributions under non-treatment are similar for x 1 and x 2. A drawback of using the conditional reference distribution is that the results depend on the choice of x 1 or x 2 as reference category. If x 1 and x 2 are switched, then the results are not generally the same. This is known as the sequential decomposition problem (see discussion in Machado and Mata, 2005). Similarly, as for the observed reference distribution, the results depend also on the treatment status of the relative distribution. 4.3 Unconditional reference distribution Now we consider unconditional potential outcome distributions as reference distribution. The relative distribution is dened by RU d (u) = F Y (d) X(Q Y (d) (u) x), which indicates the proportion of the subgroup with X = x whose outcome lies below the u th quantile of the larger population. TQTE are in this case, δ d U(u, x) = Q Y (1) X (R d U(u) x) Q Y (0) X (R d U(u) x). 14

17 Making similar transformations as before, TQTE at the unconditional distribution can be represented by, δ 0 U(u, x) = Q Y (1) X (F Y (0) X (Q Y (0) (u) x) x) Q Y (0) (u), and δ 1 U(u, x) = Q Y (1) (u) Q Y (0) X (F Y (1) X (Q Y (1) (u) x) x), using Q Y (F Y (y)) = y. The parameters δu 0 (u, x) and δ1 U (u, x) are not necessarily similar. In the following we consider δu 0 (u, x) and dene, δ 0 U(u, x 1 ) = Q Y (1) X (F Y (0) X (Q Y (0) (u) x 1 ) x 1 ) Q Y (0) (u), and δ 0 U(u, x 2 ) = Q Y (1) X (F Y (0) X (Q Y (0) (u) x 2 ) x 2 ) Q Y (0) (u). A comparison between δ 0 U (u, x 1) and δ 0 U (u, x 2) leads to, δ 0 U(u, x 1 ) δ 0 U(u, x 2 ) = Q Y (1) X (F Y (0) X (Q Y (0) (u) x 1 ) x 1 ) Q Y (1) X (F Y (0) X (Q Y (0) (u) x 2 ) x 2 ), which uses the outcome structure of F Y (0) (y). If F Y (0) X (y x 2 ) = F Y (0) X (y x 1 ), and x 1 and x 2 are two mutually exclusive subpopulations (e.g. males and females) with P r(x = x 1 )+P r(x = x 2 ) = 1 (where P r(x = x) indicates the probability that X takes the value x) such that, F Y (0) (y) = F Y (0) X (y x 1 )P r(x = x 1 ) + F Y (0) X (y x 2 )P r(x = x 2 ) using the law of iterative expectations, then Q Y (1) X (F Y (0) X (Q Y (0) (u) x) x) = Q Y (1) X (u x), using F Y (Q Y (u)) = u and, Q Y (0) (u) = Q Y (0) X (u x), 15

18 using F Y (0) (y) = F Y (0) X (y x 1 ). Accordingly, δ U0 (u, x) coincides with δ CQT E (u, x) under these conditions if the outcome structures under non-treatment are similar under x 1 and x 2. Equivalent results can be derived for δu 1 (u, x). The advantages of using the unconditional distribution as reference category are threefold. First, δ d U (u, x) does not depend on the choice of x 1 or x 2 as reference category. Second, the support conditions are satised, because Y(d) X Y(d) holds by denition, with Y(d) being the support of Y (d). Third, the unconditional distribution could be of particular interest for the researcher. In the optimal case, the unconditional distribution is identied for the entire population. This is the largest distribution for which policy eects can be identied. Many economic variables, like poverty measures, are dened relative to specic ranks in the population distribution. Therefore, locations in the population distribution could make meaningful interpretations of the estimated parameter possible. 4.4 Potential extensions A potential drawback of TQTE is that they depend on the choice of the treatment status of the relative distribution. In most applications, from an economic perspective, it makes sense to use the relative distribution under non-treatment. Usually, researchers think about distributions that are not already aected by the treatment, and want to know how they change if something is added. However, for some economic questions, it might be unclear how the treatment status is dened. Let TQTE c be the policy parameter corresponding to TQTE when treatment and non-treatment are switched. The absolute values of TQTE and TQTE c are potentially unequal, which might, from a statistical view, be an unfavorable property. In contrast, the absolute values of the corresponding CQTE and CQTE c are numerically equivalent. To overcome this potential drawback, we dene the augmented translated quantile 16

19 treatment eects (ATQTE), δ AT QT E (u, x) = 1 2 [ QY (1) X (R 0 (u) x) Q Y (0) X (R 0 (u) x) +Q Y (1) X (R 1 (u) x) Q Y (0) X (R 1 (u) x) ], which are the average between TQTE and TQTE c. ATQTE do not depend upon the choice of the treatment status of the relative distribution. However, the reference distribution is dicult to interpret. R d (u) could be replaced with R d O (u), Rd C (u), or Rd U (u). Analogous transformations as above can be made. For example, the ATQTE at the unconditional distribution are, δ U (u, x) = 1 2 [ QY (1) X (F Y (0) X (Q Y (0) (u) x) x) Q Y (0) X (F Y (1) X (Q Y (1) (u) x) x) +Q Y (1) (u) Q Y (0) (u) ]. This parameter is of particular interest, because it is not subject to the sequential decomposition problem and not subject to the choice of the treatment status of the relative distribution. 5 Parameter estimation and simulations In this section we briey discuss the estimation of the proposed parameters. In this study we consider only the simple case where all required counterfactual distributions are identied from a randomized experiment. Extensions to more complicated identication strategies are straightforward. An extensive literature on how to estimate counterfactual distributions under dierent identication strategies and model specications is readily available. Recent examples include Chernozhukov, Fernández-Val, and Melly (2013), Firpo, Fortin, and Lemieux (2009), Machado and Mata (2005), and Rothe (2012). Fortin, Lemieux, and Firpo (2010) give an up-to-date review about dierent estimation strategies. We use the quantile regression estimator of Koenker and Bassett (1978), to estimate the 17

20 inverse distribution functions, ˆQ Y (u) = arg min q ˆQ Y (d) (u) = arg min q ˆQ Y (d) X (u x) = arg min q N ρ u (Y i q), i=1 N 1{D i = d}ρ u (Y i q), and i=1 N 1{D i = d, X i = x}ρ u (Y i q), i=1 where 1{ } is the indicator function and ρ u (a) = a(u 1{a 0}) is the check function. This estimator minimizes the weighted absolute deviations. CQTE are estimated by ˆδ CQT E (u, x) = ˆQ Y (1) X (u x) ˆQ Y (0) X (u x). The cumulative distribution functions are estimated using the empirical distributions, ˆF Y (d) X (y x) = 1 N i=1 1{D i = d, X i = x} N 1{Y i y, D i = d, X i = x}. i=1 The relative distributions and TQTE are estimated using a plug-in approach, i.e. ˆRd (u) = ˆF Y (d) X ( ˆQ Yr (u) x) and ˆδ T d QT E (u, x) = ˆQ Y (1) X ( ˆR d (u) x) ˆQ Y (0) X ( ˆR d (u) x). Athey and Imbens (2006) show that such procedures are n consistent. The standard deviations of all parameters are estimated using non-parametric bootstrap procedure (sampling individual observations with replacement). Firpo (2010) shows that bootstrapping works for the type of estimator that we apply. To give an intuition about the nite sample behavior of the proposed parameters, we provide some simulation evidence. For this purpose, we go back to the example from Section 3. We consider a data generating process that is similar to this example. The matrix X = (1, Z) includes a constant term and the control variable Z. Two mutually exclusive populations are dened by x 1 = (1, 1) and x 2 = (1, 0). We specify α = 1 and β, δ, γ = (1, 1). The observed outcome equals, Y = DX δ + (1 + D) Y (0), 18

21 Table 1: Simulation results for the median. Mean Bias Std. Dev. RMSE # Observations 1,000 4,000 1,000 4,000 1,000 4,000 1,000 4,000 δ CQT E (0.5, x 1 ) δ CQT E (0.5, x 2 ) Dierence δo 0 (0.5, x 1) δo 0 (0.5, x 2) Dierence δc1 0 (0.5, x 1) δc1 0 (0.5, x 2) Dierence δc2 0 (0.5, x 1) δc2 0 (0.5, x 2) Dierence δu 0 (0.5, x 1) δu 0 (0.5, x 2) Dierence δu 1 (0.5, x 1) δu 1 (0.5, x 2) Dierence δ U (0.5, x 1 ) δ U (0.5, x 2 ) Dierence Note: All parameters are evaluated at the median. Simulations are based on 10,000 replications. with Y (0) = X β + X γ ɛ and ɛ N(0, 1). The treatment dummy D and the observed control variable Z are two binary indicators, which are specied such that D = 1{u 1 > 0} and Z = 1{u 2 > 0} with u 1, u 2 N(0, 1). In Table 1 we show the results from the simulation exercise for the median. In the rst two columns we show the mean of the simulated parameters for a sample size of 1,000 and 4,000, respectively. Depending on how the parameter of interest is dened, the mean varies between 1 and 4. This is also demonstrated in Table 2, where we derive the expected values of all distributional parameters. This demonstrates that the choice of the relative and reference distribution is an important factor for the parameter of interest. We also show the dierence between the parameter estimate for the populations x 1 and x 2 in Table 1. The dierence between CQTE for x 1 and x 2 is 2. This dierence can be decomposed 19

22 Table 2: Expected distributional eects in the local-scale model. Parameter Expected Eect δ CQT E (u, x 1 ) x 1δ + α Q Y (0) X (u x 1 ) δ CQT E (u, x 2 ) Dierence x 2δ + α Q Y (0) X (u x 2 ) (x 1 x 2 ) δ + α (Q Y (0) X (u x 1 ) Q Y (0) X (u x 2 ) ) δo 0 (u, x 1) x 1δ + α Q Y (u) δo 0 (u, x 2) x 2δ + α Q Y (u) Dierence (x 1 x 2 ) δ δc1 0 (u, x 1) x 1δ + α Q Y (0) X (u x 1 ) δc1 0 (u, x 2) x 2δ + α Q Y (0) X (u x 1 ) Dierence (x 1 x 2 ) δ δc2 0 (u, x 1) x 1δ + α Q Y (0) X (u x 2 ) δc2 0 (u, x 2) x 2δ + α Q Y (0) X (u x 2 ) Dierence (x 1 x 2 ) δ δu 0 (u, x 1) x 1δ + α Q Y (0) (u) δu 0 (u, x 2) x 2δ + α Q Y (0) (u) Dierence (x 1 x 2 δ δu 1 (u, x ( ) 1) 1 α 1+α x 1 δ + α 1+α Y (1)(u) δu 1 (u, x ( ) 2) 1 α x ( 2 δ + α 1+α Y (1)(u) Dierence 1 α ( 1+α) (x1 ) x 2 ) δ ( δ U (u, x 1 ) 1 α x 2(1+α) 1δ + α QY 2 (0) (u) α Y (1)(u) ) ( ) ( δ U (u, x 2 ) 1 α x 2(1+α) 2δ + α QY 2 (0) (u) α Y (1)(u) ) ( ) Dierence 1 α (x 1 x 2 ) δ 2(1+α) Note: See description of the local-scale model in Sections 3 and 5. into a characteristic and a structural component. Dening the relative distribution under non-treatment, the characteristics component is equal to 1. Accordingly, the dierence for the parameters δo 0 (u, x), δ0 C1 (u, x), δ0 C1 (u, x), and δ0 U (u, x) between x 1 and x 2 leads to the same results. 2 However, when we dene the relative distribution under treatment, then the dierence between the parameters estimated under x 1 and x 2 change, as it is demonstrated in Table 2. An intuitive explanation for this is that the outcome structure is dierent in the relative distributions under treatment and non-treatment. Therefore, a dierent share of the total process variation is associated with the structural component. In our simulation, the outcome structures have a larger variation under treatment than under non-treatment. Accordingly, the structural component is increasing and the characteristic component is decreasing to 0.5. ATQTE at the unconditional distribution are the average between 2 However, this is only the case because we use a very simple model in these exemplary simulations. 20

23 δu 0 (u, x) and δ1 U (u, x). Accordingly, the dierence between the ATQTE under x 1 and x 2 is the average between the characteristics components when the relative distributions under treatment and non-treatment are used (= 0.75). We show in Table 1, that the bias, standard deviation, and RMSE of the parameters are expected to decrease when the sample size is increasing. 6 Application In this section we apply TQTE to data from the Job Training Partnership Act (JTPA). The JTPA's Title II program includes a randomized trial for training evaluation (see Bloom et al., 1997). Within 16 Service Delivery Areas, training was oered randomly to applicants who applied between November 1987 and September In Table 3 we show descriptive statistics for data from this trial. 64% of all individuals who received the oer for training participated in the program. Due to the random instrument assignment, it is found that the pre-treatment control variables gender, school degree, marital status, race and past work experience are almost balanced between the subpopulations with and without an oer for training. Accordingly, we do not need to control for these variables. The outcome variable is future 30-months earnings. Individuals with an oer for training earned in the 30-months following the treatment on average $1,150 more than individuals without an oer. Abadie, Angrist, and Imbens (2002) identify changes in the earnings distributions of males and females. They consider the randomized oer to participate in the JTPA as instrument for actual participation. In contrast to this study, we estimate intention-totreat (ITT) eects. This means we consider the oer to participate in the JTPA as the treatment (and not the participation). Accordingly, ITT eects ignore potential noncompliance. ITT eects receive increasingly more attention in the literature, because this is the treatment that is actually under the control of the policy maker and thus of particular interest. Further, alone the oer to participate in training could aect the outcome of interest already, which raises identication issues for treatments that take 21

24 Table 3: Means and standard deviations for males and females. Total Sample Oer for Training No Oer Mean Std. Dev. Mean Std. Dev. Mean Std. Dev. Males Treatment Dummy 0.42 (0.49) 0.62 (0.48) 0.01 (0.11) Highschool or GED 0.69 (0.45) 0.69 (0.45) 0.69 (0.45) Black 0.25 (0.44) 0.25 (0.44) 0.25 (0.44) Hispanic 0.10 (0.30) 0.10 (0.30) 0.09 (0.29) Married 0.35 (0.47) 0.36 (0.47) 0.34 (0.46) Worked less than 13 weeks in past year 0.40 (0.47) 0.40 (0.47) 0.40 (0.47) 30-months earnings 19,147 (19,540) 19,520 (19,912) 18,404 (18,760) (U.S. $) # Observations 5,102 3,399 1,703 Females Treatment Dummy 0.45 (0.50) 0.66 (0.47) 0.02 (0.13) Highschool or GED 0.72 (0.43) 0.73 (0.43) 0.70 (0.44) Black 0.26 (0.44) 0.27 (0.44) 0.26 (0.44) Hispanic 0.12 (0.32) 0.12 (0.32) 0.12 (0.33) Married 0.22 (0.40) 0.22 (0.40) 0.21 (0.39) Worked less than (0.47) 0.52 (0.47) 0.52 (0.47) weeks in past year AFDC 0.31 (0.46) 0.30 (0.46) 0.31 (0.46) 30-months earnings 13,029 (13,415) 13,439 (13,614) 12,197 (12,964) (U.S. $) # Observations 6,102 4,088 2,014 place later in time (see mediation literature, e.g. Albert and Nelson, 2011). In Figure 2 the potential outcome distributions under treatment and non-treatment are reported for males and females separately. It can be found that the male, as well as the female potential outcome distributions, are shifted to the right when they receive an oer to participate in the JTPA. However, comparing the two horizontal changes could be complicated, because the potential outcome distributions without an oer are unequal. As an example, the median earnings of males is $5,500 higher than for females. Accordingly, CQTE for males and females at the median are measured at dierent values of the outcome. The dierence between CQTE for males and females can be potentially explained by dierent genders, but potentially also with the dierent earnings of males and females at a specic quantile in the conditional distributions. This dierence might 22

25 Figure 2: 30-month earnings potential outcome distributions of males and females. Figure 3: CQTE of receiving an oer to participate in the JTPA on 30-months future earnings (in U.S. $) for males and females. Note: Triangles, diamonds, and circles indicate signicance at the 5%-level. Standard errors are based on 999 bootstrap replications. be explained with discrimination, which is possibly unrelated to receiving an oer to participate in the JTPA. Accordingly, it makes sense to estimate TQTE in this application to compare CQTE between males and females. In Figure 3 and Table 4 the estimated CQTE for males and females are reported. The results are strongly signicant for females. For males we only nd signicant impacts of 23

26 Table 4: Estimated CQTE and TQTE of receiving an oer to participate in the JTPA on earnings for males and females. Outcome Variable: 30-months Earnings (in U.S. $) Quantiles δ CQT E (u, Females) 295** 930*** 1292*** 1557*** 1755*** 2037*** 1364* (129) (288) (423 (591) (569) (718) (745) δ CQT E (u, Males) ,863** 1, (219) (524) (671) (748) (919) (966) (1,204) Dierence (257) (606) (818) (977) (1,102) (1,225) (1,445) δo 0 (u, Females) 550** 1,185*** 1,743*** 1,515** 2,205*** 1,425** 1,329 (254) (430) (538) (608) (604) (653) (834) δo 0 (u, Males) ,236 1,467 1,467 (224) (471) (607) (760) (855) (1,086) (1,126) Dierence 680** 906 1,487* (345) (647) (836) (983) (1,060) (1,286) (1,409) δc1 0 (u, Females) 294** 936*** 1,290*** 1,565*** 1,781*** 2,046*** 1,484** (131) (290) (424) (586) (561) (718) (745) δc1 0 (u, Males) ,248 1,780* (154) (298) (568) (716) (754) (947) (917) Dierence 389** 851** 974 1,047 1, (193) (410) (722) (1,002) (967) (1,257) (1,196) δc2 0 (u, Females) 682** 1,137** 1,568*** 2,039*** 1,777*** 1,282* 1,279 (328) (490) (536) (670) (678) (714) (892) δc2 0 (u, Males) ,842** 1,324 1,481 (221) (527) (672) (747) (905) (970) (1,206) Dierence , (438) (760) (870) (1,072) (1,060) (1,132) (1,518) δu 0 (u, Females) 456** 1,040*** 1,462*** 1,481** 2,064*** 1,381** 1,258* (208) (366) (531) (603) (626) (664) (741) δu 0 (u, Males) ,440 1,279 (181) (413) (635) (785) (830) (1,083) (1,045) Dierence 654** 851 1, , (304) (603) (894) (1,012) (1,062) (1,264) (1,295) (u, Females) 380** 1,029*** 1,419*** 1,610*** 2,025*** 1,357* 1,214* δ 1 U δ 1 U (150) (308) (482) (597) (617) (720) (730) (u, Males) ,023 1,461 1,218 (188) (426) (577) (768) (730) (1,048) (1,030) Dierence 396* 729 1, , (233) (522) (770) (999) (955) (1,316) (1,307) δ U (u, Females) 418** 1,035*** 1,441*** 1,546*** 2,045*** 1,369** 1,236* (176) (334) (501) (596) (617) (685) (725) δ U (u, Males) ,451 1,249 (180) (412) (599) (772) (772) (1,060) (1,030) Dierence 525** 790 1, , (263) (555) (823) (999) (1,000) (1,281) (1,289) Note: Standard errors (based on 999 bootstrap replications) are in parentheses. ***, **,* indicate signicance at 1-, 5-, and 10-percent level, respectively. C1 is the reference distribution of females and C2 the reference distribution of males. 24

27 receiving an oer to participate in the JTPA in the upper part of the earnings distribution. At the lower quantiles, larger impacts of a JTPA oer are found in the female, than in the male earnings potential outcome distributions. Above the eighth decile, CQTE for males exceed that for females. The dierence between the impacts for males and females is generally not signicant. However, the dierence is large, in particular above the eighth decile. The impact of a JTPA is almost $2,000 higher for males than for females at the ninth decile. The results in Figure 4 and Table 4 show changes in conditional potential outcome distributions for males and females with adjusted quantiles, which are obtained from applying the concept of TQTE. TQTE adjust the estimated parameters to the scale of the reference distribution. The estimated treatment eects are similar, only the location is changed. Approximately, one can say that this corresponds with a shift of CQTE for males to the right, and a shift of CQTE for females to the left. The size of these shifts depends on the choice of the reference and relative distribution. The estimated TQTE for males and females dier in several dimensions from the corresponding CQTE, irrespective of the choice of the relative and reference distributions. Firstly, the impact of receiving an oer to participate in the JTPA, is virtually always larger or equal for females than for males. In particular, the large negative dierences above the eighth decile disappear. Secondly, we nd signicant dierences in the lower part of the respective relative distributions (with the exception of the case where we use the male distribution under non-treatment as reference category, Figure 4c). In Table 4 we report that this dierence is between $400 and $700 at the 20% quantile. Whereas the same dierence is only $150 between CQTE for males and females and insignicant. The choice of the treatment status of the relative distribution appears to have little inuence on the TQTE in this application. In Figure 4d we show TQTE using the unconditional distribution under non-treatment and in Figure 4e we show TQTE using the unconditional distribution under treatment. Both parameters are very similar to each other. The same holds for AQTET at unconditional quantiles reported in Figure 4f. 25

28 Figure 4: TQTE of receiving an oer to participate in the JTPA on 30-months future earnings (in U.S. $) for males and females. (a) TQTE at observed distribution (relative distribution under non-treatment) (b) TQTE at conditional distribution of females under non-treatment (c) TQTE at conditional distribution of males under non-treatment (d) TQTE at unconditional distribution under non-treatment (e) TQTE at unconditional distribution under treatment (f) ATQTE at unconditional distribution Note: Triangles, diamonds, and circles indicate signicance at the 5%-level. Standard errors are based on 999 bootstrap replications. 26

29 Accordingly, estimating CQTE and TQTE makes a much broader picture, of the inuence of the JTPA, on the earnings potential outcome distributions of males and females, than just estimating CQTE. There are only few females in the upper part of the earnings potential outcome distributions. This results in converse conclusions at low and high quantiles estimating CQTE. TQTE show that CQTE are equal or larger for females than for males at any given value of the outcome. Accordingly, the earnings structure has an important inuence on CQTE in this application. We show that the positive shift in the females earnings distribution dominates the impact of an oer to participate in the JTPA on the earnings distribution of males, at any xed value of the outcome. It is left open whether our results might be biased from cream-skimming of the case workers, or alternative treatments which potentially invalidate the identifying assumptions (see Heckman, LaLonde, and Smith, 1999, and Heckman, Smith, and Taber, 1996, for a discussion). 7 Conclusions We suggest an alternative policy parameter, which identies horizontal changes in conditional potential outcome distributions under treatment and non-treatment at quantiles from reference distribution. It is not argued that CQTE should be replaced by TQTE. However, estimating TQTE as an additional parameter allows additional policy conclusions to be drawn. This can help to develop a deeper understanding of the inuence of policy interventions on potential outcome distributions. In an application of the JTPA, it is shown that TQTE have the potential to complement and improve future policy conclusions. This suggests that researchers should pay much more attention to the quantiles at which distributional policy parameters are measured, especially when they make a comparison between CQTE for dierent populations. 27

30 References Abadie, A., J. D. Angrist, and G. W. Imbens (2002): Instrumental Variables Estimates of the Eect of Subsidized Training on the Quantiles of Trainee Earnings, Econometrica, 70(1), Albert, J. M., and S. Nelson (2011): Generalized Causal Mediation Analysis, Biometrics, 67(3), Albrecht, J., A. Björklund, and S. Vroman (2003): Is there a Glass Ceiling in Sweden?, Journal of Labor Economics, 21(1), Altonji, J. G., and R. Blank (1999): Race and Gender in the Labor Market, in Handbook of Labor Economics Volume 3c, ed. by O. Ashenfelter, and D. Card, pp Elsevier Science B.V. Angrist, J., V. Chernozhukov, and I. Fernandez-Val (2006): Quantile Regression under Misspecication, with an Application to the U.S. Wage Structure, Econometrica, 74(2), Angrist, J., D. Lang, and P. Oreopoulos (2009): Incentives and Services for College Achievement: Evidence from a Randomized Trial, American Economic Journal: Applied Economics, 1(1), Athey, S., and G. W. Imbens (2006): Identication and Inference in Nonlinear Dierence-In-Dierence Models, Econometrica, 74(2), Bitler, M. P., J. B. Gelbach, and H. W. Hoynes (2006): What Mean Impacts Miss: Distributional Eects of Welfare Reform Experiments, American Economic Review, 96(4), Bitler, M. P., H. W. Hoynes, and T. Domina (2014): Experimental Evidence on Distributional Eects of Head Start, Working Paper. 28

31 Blau, F. D., and L. M. Kahn (1992): The Gender Earnings Gap: Learning from International Comparisons, American Economic Review, 82(2), (2003): Understanding International Dierences in the Gender Pay Gap, Journal of Labor Economics, 21(1), Bloom, H. S., L. L. Orr, S. H. Bell, G. Cave, F. Doolittle, W. Lin, and J. M. Bos (1997): The Benets and Costs of the JTPA Title II-A Programs, Journal of Human Resources, 32(3), Bonhomme, S., and U. Sauder (2011): Recovering Distributions in Dierence-in- Dierences: A Comparison of Selective and Comprehensive Schooling, Review of Economics and Statistics, 93(2), Chernozhukov, V., I. Fernández-Val, and B. Melly (2013): Inference on Counterfactual Distributions, Econometrica, forthcoming. Chernozhukov, V., and C. Hansen (2005): An IV Model of Quantile Treatment Eects, Econometrica, 73(1), Eren, O., and S. Ozbeklik (2014): Who Benets from Job Corps? A Distributional Analysis of an Active Labor Market Program, Journal of Applied Econometrics, 29(4), Firpo, S. (2007): Ecient Semiparametric Estimation of Quantile Treatment Eects, Econometrica, 75(1), (2010): Identication and Estimation of Distributional Impacts of Interventions Using Changes in Inequality Measures, IZA Discussion Paper, Firpo, S., N. M. Fortin, and T. Lemieux (2009): Unconditional Quantile Regressions, Econometrica, 77(3),

32 Fortin, N. M., T. Lemieux, and S. Firpo (2010): Decomposition Methods in Economics, in Handbook of Labor Economics, Vol. 4 Part A, ed. by O. Ashenfelter, and D. Card, pp North Holland. Handcock, M. S., and M. Morris (1998): Relative Distribution Methods, Sociological Methodology, 28(1), Heckman, J. J., R. J. LaLonde, and J. A. Smith (1999): The Economics and Econometrics of Active Labor Market Programs, in Handbook of Labor Economics, Vol. 3, ed. by O. C. Ashenfelter, and D. Card, pp North Holland. Heckman, J. J., J. A. Smith, and C. Taber (1996): What Do Bureaucrats Do? The Eects of Performance Standards and Bureaucratic Preferences on Acceptance into the JTPA Program, in Advances in the Study of Entrepreneurship, Innovation and Growth, Vol. 7, ed. by G. Libecap, pp JAI Press. Juhn, C., K. M. Murphy, and B. Pierce (1991): Accounting for the Slowdown in Black-White Wage Convergencee, in Workers and their Wages: Changing Patterns in the United States, ed. by M. H. Kosters, pp American Enterprise Insitute, Washigton. (1993): Wage Inequality and the Rise in Returns to Skill, Journal of Political Economy, 101(3), Koenker, R., and G. Bassett (1978): Regression Quantiles, Econometrica, 46(1), Machado, J. A. F., and J. Mata (2005): Counterfactual Decomposition of Changes in Wage Distributions using Quantile Regression, Journal of Applied Econometrics, 20(4), Melly, B. (2006): Estimation of Counterfactual Distributions using Quantile Regression, Working Paper. 30