Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies

Transcription

1 Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies Carlos A. Flores Oscar A. Mitnik May, 2012 Abstract We consider the problem of using data from several programs, each implemented at a different location, to compare what their effect would be if they were implemented at a specific location. In particular, we study the effectiveness of nonexperimental strategies in adjusting for differences across comparison groups arising from two sources. First, we adjust for differences in the distribution of individual characteristics simultaneously across all locations by using unconfoundedness-based and conditional difference-in-difference methods for multiple treatments. Second, we explicitly adjust for differences in local economic conditions. We stress the importance of analyzing the overlap of, and adjusting for, local economic conditions after program participation. Our results suggest that the strategies studied are valuable econometric tools for the problem we consider, as long as we adjust for a rich set of individual characteristics and have sufficient overlap across locations for both individual and local labor market characteristics. Our results show that the overlap analysis of these two sets of variables is critical for identifying non-comparable groups and they illustrate the difficulty of adjusting for local economic conditions that differ greatly across locations. Detailed comments from two anonymous referees and an editor greatly improved the paper and are gratefully acknowledged. We are also grateful for comments by Chris Bollinger, Matias Cattaneo, Alfonso Flores-Lagunes, Laura Giuliano, Guido Imbens, Kyoo il Kim, Carlos Lamarche, Phil Robins, Jeff Smith, and seminar participants at the 2009 AEA meetings, 2009 SOLE meetings, Second Joint IZA/IFAU Conference on Labor Market Policy Evaluation, 2009 Latin American Econometric Society Meeting, Third Meeting of the Impact Evaluation Network, Second UCLA Economics Alumni Conference, Abt Associates, Federal Deposit Insurance Corporation, Institute for Defense Analyses, Interamerican Development Bank, Queen Mary University of London, Tulane University, University of Freiburg, University of Kentucky, University of Miami, University of Michigan, and VU University Amsterdam. Bryan Mueller and Yongmin Zang provided excellent research assistance. This is a heavily revised version of IZA Discussion Paper No. 4451, Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data. The usual disclaimer applies. A supplemental Appendix is available online at [to be determined]. Department of Economics, University of Miami. caflores@miami.edu. Department of Economics, University of Miami and IZA. oscar@mitnik.net.

2 1 Introduction We consider the problem of using data from several programs, each implemented at a different location, to compare what their effect would be if they were implemented at a specific location. One example would be a local government considering the implementation of one of several possible job training programs. This is a problem many policy makers are likely to face given the heterogeneity of job training programs worldwide and the scope that many local governments have in selecting their programs. In this case the policy maker may want to perform an experiment in her locality and randomly assign individuals to all the possible programs to assess the effectiveness of the programs and choose the best one for her locality. This would require implementing all the candidate programs in her locality. Thus, the most likely scenario is that the policy maker would have to rely on nonexperimental methods and use data on implementations of the programs at various locations and times. In this paper, we assess the likely effectiveness of nonexperimental strategies for multiple treatments to inform a policy maker about the effects of programs implemented in different locations if they were implemented at her location. This paper is closely related to the literature analyzing whether observational comparison groups from a different location than the treated observations can be used to estimate average treatment effects by employing nonexperimental identification strategies (Friedlander and Robins, 1995; Heckman et al., 1997; Heckman et al., 1998; Michalopoulos et al., 2004). Our work is also closely related to Hotz et al. (2005), who analyze the problem of predicting the average effect of a new training program using data on previous program implementations at different locations. Our paper differs from the rest of the literature, which focuses on pairwise comparisons between groups in different locations. Instead, we focus on comparing all groups simultaneously, which requires the use of nonexperimental methods for multiple treatments. This is an important distinction in practice. For instance, suppose that the policy maker above was given the average effects of each of 2 programs estimated from the binary-treatment versions of the nonexperimental strategies we study in this paper. If so, each estimate would give the average effect for a particular subpopulation, namely, those individuals in the common support region of the corresponding pairwise comparison (Heckman et al., 1997; Dehejia and Wahba, 1999, 2002). Thus, the policy maker would be unable to compare the average effects across the different programs simultaneously because the effects are likely to apply to different subpopulations. We employ data from the National Evaluation of Welfare-to-Work Strategies (NEWWS) study, which combines high quality survey and administrative data. The NEWWS evaluated a set of social experiments conducted in the United States in the 1990s in which individuals in several locations were randomly assigned to a control group or to different training programs. These data allow us to restrict the analysis to a relatively well-defined target population, which is welfare recipients in the different locations. This is consistent with our example above, where we would expect the policy maker to be interested in comparing programs that applied to relatively similar target populations. Even in cases like this, differences across locations in the average outcomes of individuals who participated in the site-specific programs can arise because of (at least) three important factors (Hotz 1

3 et al., 2005). First, the effectiveness of the programs can differ (this is the part that the policy maker in the example above would like to know). Second, the distribution of individual characteristics across sites may be different (e.g., individuals may be more educated, on average, in some locations). Third, the labor market conditions may differ across locations (e.g., the unemployment rate may be higher at some sites). In this paper, we focus on the control groups in the different locations and use the insight that since the control individuals were excluded from all training services, the control groups should be comparable after adjusting for differences in individual characteristics and labor market conditions. If by using nonexperimental strategies to adjust for those differences we are able to simultaneously equalize average outcomes among the control groups in the different locations, we would expect the average effects resulting from the implementation of those strategies on the treated observations to reflect actual program heterogeneity across the sites (e.g., Heckman et al., 1997; Heckman et al., 1998; Hotz et al., 2005). 1 The first source of heterogeneity across sites that we consider is the distribution of individual characteristics. We consider unconfoundedness-based and conditional difference-in-difference strategies for multiple treatments to adjust simultaneously across all sites for this source of heterogeneity. This is related to the literature on program evaluation using nonexperimental methods, most of which focuses on the binary-treatment case (see Heckman et al., 1999; Imbens and Wooldridge, 2009). More recently, however, there has been a growing interest on methods to evaluate multiple treatments (Lechner, 2001, 2002a, 2002b; Frölich, 2004; Plesca and Smith, 2007) and treatments that are multivalued or continuous (Imbens, 2000; Imai and van Dyk, 2004; Hirano and Imbens, 2004; Abadie, 2005; Cattaneo, 2010); we employ some of these methods in this paper. In addition, we study the use of the generalized propensity score (GPS), defined as the probability of receiving a particular treatment conditional on covariates, in assessing the comparability of individuals among the different comparison groups, and we propose a strategy to determine the common support region that is less stringent than those previously used in the multiple treatment literature (Lechner, 2002a, 2002b; Frölich et al., 2004). We also consider several GPS specifications and GPS-based estimators, and we analyze the performance of the nonexperimental strategies considered when controlling for alternative sets of covariates. The second source of heterogeneity across the comparison groups that we consider is local economic conditions (LEC). The literature on program evaluation methods stresses the importance of comparing treatment and nonexperimental comparison groups from the same local labor market (Friedlander and Robins, 1995; Heckman et al., 1997; Heckman et al., 1998). In our setting, the analyst likely has information about specific programsonlywhentheywereimplementedin different local labor markets. Thus, we consider the challenging problem of explicitly adjusting for differences in LEC across our comparison groups. The literature on how to adjust for these differences is scarce compared to that on how to adjust for individual characteristics, as we discuss later. We stress the importance of adjusting for differences in LEC after program participation, as 1 Another potential source of heterogeneity across control groups is substitution by their members into alternative training programs (Heckman and Smith, 2000; Heckman et al., 2000; Greenberg and Robins, 2011). Our analysis leads us to believe that it is unlikely that this potential source of heterogeneity drives our results, as we discuss later. 2

4 they can be a significant source of heterogeneity in average outcomes across sites. We also highlight the importance of analyzing the overlap in LEC across locations, as well as the advantages of separating such analysis from that of the overlap in individual characteristics. Our results suggest that the strategies studied are valuable econometric tools when evaluating the likely effectiveness of programs implemented at different locations if they were implemented at another location. The keys are adjusting for a rich set of individual characteristics (including labor market histories) and having sufficient overlap across locations in both the individual and local labor market characteristics. More generally, our results also illustrate that in the multiple treatment setting, there may be treatment groups for which it is extremely difficult to find valid comparison groups, and that the strategies we consider perform poorly in such cases. Importantly, the overlap analyses (both in individual characteristics and LEC) are critical for identifying those noncomparable groups. In addition, our results show that when the differences in post-randomization LEC are large across sites, the methods we consider to adjust for them substantially reduce this source of heterogeneity, although they are not able to eliminate it. These results highlight the importance of adjusting for post-treatment LEC. The paper is organized as follows. Section 2 presents our general setup and the econometric methods we employ; Section 3 describes the data; Section 4 presents the results; and Section 5 provides our conclusions. An online Appendix provides further details on the econometric methods and data. It also presents supplemental estimation results, including those from several robustness checks briefly mentioned throughout the paper and those complementing the analysis in Section 4. 2 Methodology 2.1 General Framework We use data from a series of randomized experiments conducted in several locations. In each location, individuals were randomly assigned either to participate in a site-specific training program or to a control group that was excluded from all training services. We study whether the nonexperimental strategies considered in this paper can simultaneously equalize average outcomes for control individuals across all the sites by adjusting for differences in the distribution of individual characteristics and local economic conditions. Each unit in our sample, = 1 2, comes from one of possible sites. Let {1 2 } denote the location of individual. For our purposes, it is convenient to write the potential outcomes of interest as (0 ), where denotes the site and zero only emphasizes the fact that we focus exclusively on the control groups. Thus, (0 ) is the outcome (e.g., employment status) that individual would obtain if that person were a control in site. The data we observe for each unit are ( ),where is a set of pre-treatment covariates and = (0 ). Let be a subset of the support of, X. Our parameters of interest in this paper are ( ) = [ (0 ) ],for =1 2 (1) 3

5 The region can include all or part of the support of and is determined by the overlap region, which we discuss below. To simplify the notation, we omit the conditioning on unless necessary for clarity. The object in (1) gives the average potential outcome under the control treatment at location for someone with randomly selected from any of the sites. This is different from [ (0 ) = ], which gives the average control outcome at site for someone with randomly selected from site. While this last quantity is identified from the data because of random treatment assignment within site, is not. Rather than focusing on (1), we could have focused on [ (0 ) = ] for =1, which gives the average outcome for the individuals in a particular site {1 } within the overlap region for site. Byfocusingontheaverageoverall sites in (1), we avoid having our results depend on the selection of site as a reference point. However, the estimation problem becomes more challenging because we need to find comparable individuals in each of the sites for every individual in the sites (as opposed to finding comparable individuals in each site for only those in site ). We assess the performance of the strategies presented below in several ways. First, given estimates b 1 b of the corresponding parameters in (1), we test the joint hypothesis 1 = 2 = = (2) One drawback of this approach is that we could fail to reject the null hypothesis in (2) only because the variance of the estimators is high, rather than because all the estimated s b are sufficiently close to each other. Hence, we focus our analysis mostly on an overall measure of distance among the estimated means. Letting b = X 1 b,wedefine the the root mean square distance as =1 s = 1 1 b P =1 ³ 2 b b (3) wherewedivideby b to facilitate its interpretation and the comparison across outcomes. Due to pure sample variation, we would never expect the to equal zero, even if were randomly assigned. In Section 4, we present benchmark values of the as a reference point for reasonable values of the. These benchmarks give the value of the thatwouldbeachievedbyan experiment in a setting where 1 = 2 = = holds. 2 In the following two sections, we discuss the strategies we employ to adjust for differences across the sites in the distribution of individual characteristics and local economic conditions. 2.2 Adjusting for Individual Characteristics Nonexperimental Strategies. We consider two identification strategies to adjust for differences in individual characteristics across sites. Let 1( ) be the indicator function, which equals one if event is true and zero if otherwise. The first strategy is based on the following assumption: 2 In addition to the in (3), we considered two other measures: the mean absolute distance and the maximum pairwise distance among all the estimated means. All three measures give similar results. 4

6 Assumption 1 (Unconfounded site) 1( = ) (0 ), for all {1 2 }. This assumption is an extension to the multiple-site setting of the one in Hotz et al. (2005), and in the program evaluation context it is called weak unconfoundedness (Imbens, 2000). The problem when estimating is that we only observe (0 ) for individuals with 1( = ) =1, while it is missing for those with 1( = ) =0. Assumption 1 implies that conditional on,the two groups are comparable, which means that we can use the individuals with 1( = ) =1to learn about. A key feature of Assumption 1 is that all that matters is whether or not unit is in site ; hence, the actual site of an individual not in is not relevant for learning about. In addition to Assumption 1, we impose an overlap assumption that guarantees that, in infinite samples, we are able to find comparable individuals in terms of the covariates for all units with 6= in the subsample with = for every site. Assumption 2 (Simultaneous strict overlap) 0 Pr ( = = ), for all and X. This assumption is stronger than that of the binary treatment case, as it requires that for every individual in the sites, we find comparable individuals in terms of covariates in each of the sites. Intuitively, while we want to learn about the potential outcomes in all sites for every individual, we observe only one of those potential outcomes, so the fundamental problem of causal inference (Holland, 1986) is worsened. When the strict overlap assumption is violated, semiparametric estimators of can fail to be -consistent, which can lead to large standard errors and numerical instability (Busso et al., 2009a, 2009b; Khan and Tamer, 2010). In practice, Assumption 2 may hold only for some of the sites and/or regions in the support of, inwhich case one may have to restrict the analysis to those sites and/or regions with common support. The second identification strategy we consider to adjust for differences in individual characteristics across locations is a multiple-treatment version of the conditional difference-in-differences (DID) strategy (Heckman et al., 1997; Heckman et al., 1998; Smith and Todd, 2005). In our setting, the main identification assumption of this strategy is 1( = ) { (0 ) 0 (0 )}, for all {1 2 }, where and 0 are time periods after and before random assignment, respectively. This assumption is similar to Assumption 1 but employs the difference of the potential outcomes over time. This identification strategy allows for systematic differences in time-invariant factors between the individuals in site and those not in site for every site. Thus, DID methods can also help to adjust for time-invariant heterogeneity in local economic conditions across sites (Heckman et al., 1997; Heckman et al., 1998; Smith and Todd, 2005). Estimators. We consider several estimators to implement the strategies discussed above. For simplicity, we present the estimators in the context of the unconfoundedness strategy. The DID estimators simply replace the outcome in levels with the outcome in differences. Under Assumption 1 we can identify (omitting for brevity the conditioning on ) as = [ [ (0 ) = ]] = [ [ (0 ) 1( = ) =1 = ]] = [ [ = = ]] 5

7 where, following similar steps to those used in a binary treatment setting (e.g., Imbens, 2004), the first equality uses iterated expectations and the second uses Assumption 1. This result suggests estimating using a partial mean, which is an average of a regression function over some of its regressors while holding others fixed (Newey, ). To estimate, we first estimate the expectation of conditional on and, and then we average this regression function over the covariates holding fixed at. Thefirst estimator we consider uses a linear regression of on the site dummies and to model the conditional expectation [ = = ]. We refer to it as the partial mean linear estimator. We also consider a more flexible model of that expectation by including polynomials of the continuous covariates and various interactions, and refer to the corresponding estimator as the partial mean flexible estimator. The rest of the estimators we consider are based on the generalized propensity score (GPS). Its binary-treatment version, the propensity score, is widelyusedintheprogramevaluationliterature (Heckman et al., 1999; Imbens and Wooldridge, 2009). In our setting, the GPS is defined as the probability of belonging to a particular site conditional on the covariates (Imbens, 2000): ( ) =Pr( = = ) (4) We let = ( ) be the probability that individual belongs to the location she actually belongs to, and let = ( ) be the probability that belongs to a particular location conditional on her covariates. Clearly, = for those units with =. We consider GPS-based estimators of based on partial means and inverse probability weighting. Imbens (2000) shows that the GPS can be used to estimate within a partial mean framework by first estimating the conditional expectation of as a function of and = ( ), andthen by averaging this conditional expectation over = ( ) while holding fixed at. Weconsider two estimators based on this approach. The first follows Hirano and Imbens (2004) and estimates [ = = ] by employing a linear regression of the outcome on the site dummies and the interactions of the site dummies with and 2. The second estimator employs a local polynomial regression of order one to estimate that regression function. We refer to these two estimators of as the GPS-based parametric and nonparametric partial mean estimators, respectively. Finally, we also employ the GPS in a weighting scheme. Intuitively, reweighting observations by the inverse of creates a sample in which the covariates are balanced across all locations. The first inverse probability weighting (IPW) estimator of that we consider equals the coefficient for 1( = ) from a weighted linear regression of on all the site indicator variables, with weights equal to = p 1. We also consider an estimator that combines IPW with linear regression by adding covariates to this regression. We refer to it as the IPW with covariates. Determination of the Overlap Region. A key ingredient in the implementation and performance of the estimators previously discussed is the identification of regions of the data in which there is overlap in the covariate distributions across the different comparison groups. The usual approach in the binary-treatment literature is to focus on the overlap or common support 6

8 region by dropping those individuals whose propensity score does not overlap with the propensity score of those in the other treatment arm (Heckman et al., 1997; Heckman et al., 1998; Dehejia and Wahba, 1999, 2002). We propose a GPS-based trimming procedure for determining the overlap region in the multiple-treatment setting that is motivated by a procedure commonly used when the treatment is binary (Dehejia and Wahba, 1999, 2002). We determine the overlap region in two steps. First, we define the overlap region with respect to a particular site,, as the subsample of individuals whose conditional probability of being in site (i.e., ) is greater than a cutoff value, which is given by the highest -quantile of the distribution for two groups: those individuals who are in site and those who are not. This rule ensures that within the subsample, for every individual with 6=, wecan find comparable individuals with =, which implies that we can estimate the mean of (0 ) for the individuals in under Assumption 1. Second, we define the overlap region as the subsample given by those individuals who are simultaneously in the overlap regions for all the different sites. This step guarantees that we estimate the mean of (0 ) for =1 2 for the set of individuals who are simultaneously comparable in all the sites. More formally, let = { : max{ { : = } { : 6= } }}, where { } denotes the -th quantile of the distribution of over those individuals in subsample. Wedefine the overlap region as 3 = T =1 (5) Lechner (2002a, 2002b) and Frölich et al. (2004) use a rule akin to (5) but define = { : [max =1 {min { : = } } min =1 {max { : = } }]}. Our rule is less stringent in two important ways. First, as implied by Assumption 1, it does not require the comparison of among all locations but only between the groups of individuals with = and 6=. Second, it identifies individuals outside the overlap region based only on the lower tail of the distributions of. This is because we impose the overlap condition for all sites, so it is not necessary to look at the upper tail of the GPS distributions Adjusting for Local Economic Conditions Even after adjusting for differences in individual characteristics, there could still be differences in the average outcomes of the comparison groups across sites because of differences in LEC. The main difficulty in adjusting for differences in LEC is that these variables take on the same value within each location, or within each cohort in a location. When each comparison group belongs to adifferent location, as in our setting, the variation of the LEC variables can be limited and they are likely to fail the overlap condition. In such cases, the literature provides little guidance on how to adjust for differences in LEC across comparison groups. 3 Alternatively, the overlap region could be defined as = =1 { X : ( ) max{ { : = } { : 6= } }}. 4 For example, when estimating the average treatment effect in the binary-treatment setting both tails of the distribution of the propensity score are considered. This is equivalent to considering the two lower tails of the distributions of Pr ( =1 ) and Pr ( =0 ) because Pr ( =1 )+Pr( =0 ) =1. 7

9 In the context of evaluating training programs, a strategy followed in practice to control for differences in LEC across comparison groups is to include LEC measures and/or geographic indicators in the propensity score (Roselius, 1996; Heckman et al., 1998; Dyke et al., 2006; Mueser et al., 2007; Dolton and Smith, 2011). Using regression-adjustment methods Hotz et al. (2006) and Flores-Lagunes et al. (2010) include as regressors measures of post-treatment LEC. Most of these papers differ from ours because we do not have a large time series or cross sectional variation in LEC. In a setting more similar to ours, Hotz et al. (2005) include two measures of pre-randomization LEC in the regression step of the bias-corrected matching estimator in Abadie and Imbens (2011). 5 We pay particular attention to adjusting for differences across sites in LEC after randomization, which can be a significant source of heterogeneity in average outcomes. For example, Hotz et al. (2006) and Flores-Lagunes et al. (2010) find that it is very important to adjust for post-treatment LEC when evaluating training programs. The approach we use to adjust for post-randomization LEC consists of modeling the effect of the pre-randomization LEC on pre-randomization values of the outcomes, and then using the estimated parameters to adjust the outcomes of interest for the effect of post-randomization LEC. We implement our approach in two ways depending on whether we use the outcome in levels or in differences. Assume that the potential outcomes (0 ) are separable functions of the individual characteristics and the LEC, and for the sake of presentation, ignore the presence of the covariates. For the outcome in levels, we use the pre-randomization outcome and LEC to run the regression 0 = P =1 [1 ( = ) 0] ,where 0 denotes a time period before random assignment, 0 is a vector of LEC at time 0,and 0 is an error term. The availability of different cohorts within each site in our data provides us with sufficient variation in the LEC to identify the parameter vector. We use the estimated coefficient b to adjust the outcome of interest ( )as e = 0 b, and apply the estimators discussed in Section 2.2 on the LEC-adjusted outcome e. For the outcome in differences, we follow a similar approach but employ in the regression the differences of the outcomes and LEC in two periods prior to randomization. In both cases (for the outcome in levels and in differences), we normalize the adjustments to have mean zero in order to allow for easier comparisons between the unadjusted and adjusted outcomes. Further details are provided in the Appendix. Clearly, the flexibility of the specific model employed to adjust for differences across locations in LEC depends heavily on the available data and the amount of variation in the LEC. For instance, one may be able to estimate more flexible models if there is a large number of locations or cohorts in each location. In this paper, we are faced with the challenging (but not uncommon) problem that there is limited variation in the LEC measures (we have thirteen cohorts total); so it is difficult to use a more flexible model than the one we use. 5 Dehejia (2003) uses an approach based on hierarchical models to deal with site effects when evaluating multisite training programs, although he does not directly address LEC differences. Galdo (2008) considers a setting where there is lack of overlap due to site and time effects, and proposes a procedure that removes those two fixed effects before implementing a matching estimator. As pointed out by Dehejia (2003), in a framework that aims at predicting the effects of programs if implemented in different sites, it is problematic to model site effects as fixed effects. 8

10 An attractive feature of our approach is that it is not estimator-specific, meaning that once we adjust the outcomes for differences in post-randomization LEC, we can employ any of the estimators from Section 2.2 to adjust for differences in individual characteristics. Another potentially attractive feature is that the effects of the LEC on the outcome variables are estimated using the outcomes prior to random assignment, so part of the analysis (e.g., modeling the effect of the LEC on outcomes) can be performed without resorting to the outcomes after randomization. Finally, note that while we do not explicitly adjust for pre-randomization LEC in our preferred specification, we include a rich set of welfare and labor market histories as covariates, which can proxy for those variables (Hotz et al., 2005). In Section 4, we also present results from models that include measures of both pre- and post-randomization LEC in the GPS estimation. 3 Data The data we use comes from the National Evaluation of Welfare-to-Work Strategies (NEWWS), a multi-year study conducted in seven United States cities to compare the effects of different approaches for helping welfare recipients (mostly single mothers) to improve their labor market outcomes and leave public assistance. The individuals in each location were randomly assigned to different training programs or to a control group that was denied access to the training services offered by the program for a pre-set embargo period. The year of randomization differed across sites, with the earliest randomization taking place in and the latest in. We rely on the public-use version of the NEWWS data, which is a combination of high quality survey and administrative data. For further details on the NEWWS study, see Hamilton et al. (2001). We restrict the analysis to female control individuals in the five sites for which two years of pre-randomization individual labor market histories are available and in which individuals were randomized after welfare eligibility had been determined. The five sites we use are Atlanta, GA; Detroit, MI; Grand Rapids, MI; Portland, OR; and Riverside, CA (we exclude Columbus, OH and OklahomaCity,OK).Thefinal sample size in our analysis is 9,351 women. 6 The outcome we use is an indicator variable equal to one if the individual was ever employed during the two years (quarters two to nine) following randomization, and zero otherwise. 7 We focus on an outcome that is measured only up to two years after randomization because, in most sites, we cannot identify which control individuals were embargoed from receiving program services after year two. We define the outcome in differences as the original outcome minus an indicator equal to one if the individual was ever employed during the two years prior to randomization. Our data contain information on demographic and family characteristics, education, housing type and stability, pre-randomization welfare and food stamps use (seven quarters), and pre- 6 Michalopoulos et al. (2004) use the NEWWS data to perform binary comparisons between control groups in different locations. See the Appendix for a discussion of the differences between their sample and ours. 7 Using one-year employment indicators instead does not change our conclusions substantively. We focus on an employment indicator because the public-use data available to us contain only the year of random assignment and nominal earnings measures. Thus, we could create only rough measures of real earnings. By focusing on employment, we also avoid having to deal with issues related to cost-of-living differences across sites. 9

11 randomization earnings and employment histories (eight quarters). The first five columns of Panel A in Table 1 show the averages of the outcomes and selected covariates for each site. As expected, there are important and usually large differences in the means of all the variables across sites (e.g., percentage of blacks). We employ three measures of LEC for the different cohorts within each site at the metropolitan statistical area (MSA) level: employment-to-population ratio, average real earnings, and unemployment rate. The bottom of Panel A presents the average of these variables during the calendar year of random assignment, as well as their two-year growth rates in the two years before and after random assignment. The differences in LEC across all sites clearly suggest that we are working with five distinct local labor markets, especially with respect to Riverside. 4 Results 4.1 GPS Estimation and Covariate Balancing We estimate the GPS using a multinomial logit model (MNL) that includes 52 individual-level covariates (estimation results are provided in the Appendix). In Section 4.4, we consider alternative estimators of the GPS. An important property of the GPS is that it balances the covariates between the individuals in a particular site and those not in that site (Imbens, 1999). In the binary treatment setting, this property is commonly used to gauge the adequacy of the propensity score specification (Dehejia and Wahba, 1999; Smith and Todd, 2005; Lee, 2011). We follow two strategies to examine the balance of each covariate across the different sites after adjusting for the GPS. The first strategy tests whether there is joint equality of means across all five sites after each observation is weighted by 1 b. The second strategy consists of a series of pairwise comparisons of the mean in each site versus the mean in the (pooled) remaining sites, adjusting for the GPS within a blocking or stratification framework in the spirit of Dehejia and Wahba (1999, 2002) and Hirano and Imbens (2004). Both strategies are implemented after imposing the overlap rule (5). An important issue when implementing this rule is selecting the quantile that determines the amount of trimming. Even in the binary treatment literature, there is no consensus about how to select the amount of trimming (e.g., Imbens and Wooldridge, 2009). Based on attaining good balancing under both strategies, we set = For brevity, we focus on the first strategy. The last four columns of Panel A in Table 1 show the p-valuefromajointequalitytestandtherootmeansquaredistance( ) as given in (3) for the means of the variables in each site, before and after weighting by the GPS. The complete variableby-variable results are presented in the Appendix. Weighting by the GPS brings the means of all of the covariates much closer together (except for the percentage of individuals with a high school degree or GED, which becomes slightly less balanced). For example, the for the percentage of blacks is reduced from 0.64 to 0.06 after weighting by the GPS. Despite the significant improvement 8 We also examined covariate balancing properties for values of equal to and Our main results presented in Section 4.3 are, in general, robust to the choice of. 10

12 in balancing, the joint equality of means test is still rejected for some of the covariates. Panel B in Table 1 shows summary results for the covariate balancing analysis. Weighting by the GPS greatly improves the balancing among the five sites, as the number of unbalanced covariates at the 5 percent significance level drops from 52 to 6. The second strategy used to analyze covariate balancing, blocking on the GPS, yields similar results. Overall, we conclude that the covariates in the raw data are highly unbalanced across sites, and that the estimated GPS does a reasonably good job in improving their balance across all five sites. 4.2 Overlap Analysis We first analyze the overlap in individual characteristics across sites, and then we analyze the overlap in LEC. Panel C in Table 1 presents the percentage of individuals dropped from each site after imposing overlap, and the overall percentage of observations dropped (26 percent). In Riverside, almost half of the observations are dropped, implying that many of the individuals in Riverside are not comparable to individuals in at least one of the other sites. The sites with the fewest number of observations dropped are Atlanta and Detroit (about 5 percent in each). In Figure 1, we examine more closely the overlap quality in individual characteristics. Each panel in the Figure shows, for a given site, the number of observations with 6= (left bars) and = (right bars) whose GPS falls within given percentile intervals of the support of. The percentile intervals, for each site, are constructed using the distribution of only for individuals in site before imposing overlap. The outside wide (inside thin) bars display the number of individuals before (after) imposing overlap. As discussed in Section 2.2, we are interested only in the lower tail of these distributions because, for estimation of, we need to find for every individual with 6= comparable individuals in terms of in the = group, and not vice versa. However, note that the intersection in the overlap rule (5) results in individuals being dropped from all regions of the GPS distributions in Figure 1. This illustrates a key difference with the binary-treatment case, where usually individuals are only dropped from the tails. Figure 1 makes clear that the GPS distributions of the 6= and = groups differ greatly in some sites, which is consistent with the large differences in individual characteristics documented in Table 1. Figure 1 also shows that imposing overlap significantly decreases the number of individuals at the lower tail of the distribution of for the 6= group in most sites. The biggest changes, mostly arising from the large number of Riverside individuals dropped, are in Atlanta and Detroit. For example, the bottom one percent of the individuals in Atlanta (about 14 observations) are comparable to 3,245 individuals (about 41 percent of those not in Atlanta) before imposing overlap, and to 938 individuals after imposing overlap (about 17 percent of those not in Atlanta). Despite the improvement in overlap after imposing (5), the overlap quality remains relatively poor in some regions of the distributions, as many observations not in site are comparable to few observations in site in those regions. This poor overlap quality can lead to high variance of the GPS-based semiparametric estimators, just as in the binary-treatment setting (Black and Smith, 2004; Busso et al., 2009b; Imbens and Wooldridge, 2009; Khan and Tamer, 11

13 2010). In general, the overlap quality in individual characteristics between units with = and 6= in each site is similar to that from previous studies in a binary-treatment setting (e.g., Dehejia and Wahba, 1999; Black and Smith, 2004; Smith and Todd, 2005). We now examine the overlap in pre- and post-randomization LEC across the different cohorts in our sites. Figure 2 presents plots of the LEC faced by each of the cohorts in our analysis before (top figures) and after (bottom figures) random assignment. It is clear from the Figure that all the cohorts in Riverside experienced, especially after randomization, extremely different LEC than the other sites with respect to the employment-to-population ratio and the unemployment rate. Moreover, the comparison of the top and bottom panels shows that the behavior over time ofthelecinriversideisalsodifferent, with earlier cohorts experiencing a deterioration of the LEC. As for the rest of the sites, while their LEC are not the same, they are roughly similar. Figure2highlightsthatitisalmostimpossibletofind individuals in other sites that faced postrandomization LEC similar to those faced by the individuals in Riverside. Thus, the adjustment for LEC in Riverside heavily depends on how well the parametric model used to adjust for LEC is able to extrapolate to these regions of the data. 4.3 Main Results Table 2 presents the results for the assessment measures of the different estimators used to implement the unconfoundedness (first three columns) and conditional DID (last three columns) strategies. Panel A shows the results when the outcome is not adjusted for differences in postrandomization LEC, while Panel B shows the results when adjusting for such differences following the procedure discussed in Section 2.3. The first and second columns of the Table show, respectively, the p-value from the joint equality test in (2) and the in (3) with its corresponding 95 percent confidence interval based on 1,000 bootstrap replications. To have a reference level for the reduction in the for the different estimators, the third column displays the relative to that from the raw mean estimator using the outcome in levels before imposing overlap and not adjusting for LEC. In addition, to have a benchmark against which to compare our results, the last row in each of the panels gives the value of the that would be achieved by an experiment in a setting where 1 = = holds. In particular, we perform a placebo experiment in which we assign placebo sites randomly to the individuals in the data set (keeping the number of observations in each site the same as in our sample), and we let b be the average outcome in each placebo site. These placebo values are also calculated using the outcome in differences and with and without adjusting for the LEC. We implement all the GPS-based estimators only within the overlap region, and the non-gps-based estimators before and after imposing overlap. 9 9 For the partial mean linear X and IPW with covariates estimators, we include all 52 individual covariates used in the GPS estimation. For the partial mean flexible X estimator, we add 50 higher order terms and interactions. We implement the GPS-based nonparametric partial mean estimator using an Epanechnikov kernel and selecting the bandwidth using the plug-in procedure proposed by Fan and Gijbels (1996). For the GPS-based parametric partial mean estimator, we also explored using a cubic and a quartic polynomial in, which made the results very close to those from the nonparametric partial mean estimator. Finally, the results from the regressions in the first step of the procedure used to adjust for LEC are shown in the Appendix. 12

14 The of the raw mean in levels (0.129) is about six times what one would expect to see in an experiment (0.021). Both strategies used to adjust for individual characteristics do very little in reducing the, regardless of the estimator used. This illustrates that in some instances, DID estimators are unable to adjust for differences across sites. In our case, there seems to be important differences in time-variant characteristics across sites (LEC) that are not captured by the individual characteristics we control for. The only adjustment that helps to reduce the differences in outcomes across comparison groups is the one for post-randomization LEC. In general, this adjustment reduces the by about half whether we employ the outcome in levels or in differences; however, the still remains more than three times that of the placebo experiment. Table 3 presents the estimates and confidence intervals of for each site. This table shows that the methods are fairly successful in equalizing the average outcomes among all sites but Riverside. In fact, with the exception of the LEC adjustment, the strategies considered change very little the average outcome of Riverside. This is not surprising given the very different LEC in this site. Tables 2 and 3 highlight the difficulty of adjusting for LEC when they differ substantially across sites and the number of cohorts and/or locations is small. This is consistent with Dehejia (2003), who finds that it is difficult to predict site effects when the site being predicted differs greatly from the other sites. The lack of overlap in post-randomization LEC in Riverside prevents us from finding individuals in other sites who faced post-randomization LEC like those in Riverside. Moreover, the small number of cohorts and limited LEC variation in our sample makes it extremely difficult to model the effect of the LEC on the outcome and to extrapolate to regions with no overlap. Thus, we are unable to compare the outcomes in Riverside to those from the other sites. More generally, our results illustrate that when evaluating multiple treatments, there may be treatment groups for which it is not possible to draw inferences because they are not comparable to the other ones. Table 4 shows our assessment measures calculated using the estimated means from Table 3 (i.e., from the five-sitemodel)forallsites,exceptriverside. The for the raw mean of the outcome in levels starts at 0.077, which is about three times that of the placebo experiment. In this case, regardless of the estimator employed, the adjustment for individual characteristics significantly reduces the to levels that are much closer to those of the placebo experiment. 10 As in Table 2, the DID estimators do not seem to improve on the results from the outcomes in levels. The LEC adjustment does not affect the results nearly as much as it did in Table 2, and for most estimators it slightly increases the. Thissmalleffect of the LEC adjustment, as compared to the case when Riverside is included, is consistent with the other four sites having more similar LEC than does Riverside. It is also important to compare the confidence intervals from the different estimators to those from the placebo experiment. In general, the confidence intervals from the linear regression-based estimators without adjusting the outcome for LEC are similar to those from the placebo experiment. However, those from the GPS-based estimators are wider, especially those 10 For estimation of, it does not matter which site a unit comes from if the unit is not in site. Thus,performing the whole analysis using only the observations not in Riverside gives results similar to those in Table 4. In addition, carrying out the four-site analysis keeping Riverside, while dropping any of the other sites, gives estimates as high as those for the five-site analysis in Table 2. 13

15 from the IPW estimators, because of the poor overlap quality in some sites (e.g., Atlanta). 11 In sum, the results from this section highlight the importance of the LEC when employing comparison groups from different locations. Our results suggest that, given our set of conditioning variables, the unconfoundedness and DID strategies work reasonably well in simultaneously equalizing average outcomes across our comparison groups when they come from sites with similar LEC. This is consistent with some of the previous findings in the binary-treatment literature (Roselius, 1996; Heckman et al., 1997; Heckman et al., 1998; Michalopoulos et al., 2004; Smith and Todd, 2005; Hotz et al., 2005). Our results show that, in the presence of large differences in post-randomization LEC across comparison groups, our LEC adjustment can reduce but not eliminate these differences. Our findings regarding the importance of adjusting for post-randomization LEC complement, and are consistent with, those in the previous literature (Hotz et al., 2006; Flores-Lagunes et al., 2010). We briefly relate our results to our example of a policy maker who wants to implement in her site one of the programs from the other sites. If the policy maker were in any of the sites but Riverside, she would clearly be able to compare all the programs, except the one in Riverside, for the people in her site in the overlap region. For the program in Riverside, she could not separate program heterogeneity from LEC heterogeneity. A subtler case is that of a policy maker in Riverside, who would be able to compare the effects of the other sites programs for the individuals in Riverside in the overlap region, had they participated in these programs in any of the sites but Riverside. However, she would not be able to make the same comparison if the programs had been implemented in Riverside because she could not assess the effects of the programs in a labor market like Riverside, which differs greatly from the labor markets of the other sites. A potential threat to our analysis is differential substitution into alternative training programs by the control individuals (Heckman and Smith, 2000; Heckman et al., 2000; Greenberg and Robins, 2011). Analyzing a subsample of the individuals in the NEWWS study, Freedman et al. (2000, Table A.1) find differential participation rates in self-initiated activities among the control individuals. For instance, the control individuals in Atlanta (19%) and Riverside (around 26%) show much lower participation rates than do those in the other three sites (around 40%), with most of the differences driven by participation in post-secondary education and vocational training activities. Freedman et al. (2000) partially attribute these differences to individual characteristics heterogeneity, which should be adjusted by the individual-level covariates we control for. Although we lack the data to directly deal with differential control group substitution, our results do not seem to support the notion that this source of heterogeneity drives our results. For example, while we do not have trouble adjusting the average outcomes in Atlanta (the site with the lowest participation among the control individuals), we have the most trouble in Riverside (the site with the worst LEC). Unfortunately, we cannot completely rule out this additional source of heterogeneity, nor can we precisely determine the extent to which it may affect our results When overlap is poor, the large standard errors of semiparametric propensity-score based estimators of treatment effects may more accurately reflect uncertainty than regression based estimators, which extrapolate to regions of poor overlap using parametric assumptions (Black and Smith, 2004; Imbens and Wooldridge, 2009). 12 Another potential source of heterogeneity across control groups is welfare rules. For instance, welfare waivers 14