Recent Developments in the Econometrics of Program Evaluation

Transcription

1 Recent Developments in the Econometrics of Program Evaluation The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Imbens, Guido W. and Jeffrey M. Wooldridge Recent developments in the econometrics of program evaluation. Journal of Economic Literature 47, no. 1: Published Version Citable link Terms of Use This article was downloaded from Harvard University s DASH repository, and is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at nrs.harvard.edu/urn-3:hul.instrepos:dash.current.terms-ofuse#oap

2 NBER WORKING PAPER SERIES RECENT DEVELOPMENTS IN THE ECONOMETRICS OF PROGRAM EVALUATION Guido M. Imbens Jeffrey M. Wooldridge Working Paper NATIONAL BUREAU OF ECONOMIC RESEARCH 1050 Massachusetts Avenue Cambridge, MA August 2008 Financial support for this research was generously provided through NSF grants SES and We are grateful for comments by Caroline Hoxby, Roger Gordon, Larry Katz, The views expressed herein are those of the author(s) and do not necessarily reflect the views of the National Bureau of Economic Research by Guido M. Imbens and Jeffrey M. Wooldridge. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including notice, is given to the source.

3 Recent Developments in the Econometrics of Program Evaluation Guido M. Imbens and Jeffrey M. Wooldridge NBER Working Paper No August 2008 JEL No. C01 ABSTRACT Many empirical questions in economics and other social sciences depend on causal effects of programs or policiy interventions. In the last two decades much research has been done on the econometric and statistical analysis of the effects of such programs or treatments. This recent theoretical literature has built on, and combined features of, earlier work in both the statistics and econometrics literatures. It has by now reached a level of maturity that makes it an important tool in many areas of empirical research in economics, including labor economics, public finance, development economics, industrial organization and other areas of empirical micro-economics. In this review we discuss some of the recent developments. We focus primarily on practical issues for empirical researchers, as well as provide a historical overview of the area and give references to more technical research. Guido M. Imbens Department of Economics Littauer Center Harvard University 1805 Cambridge Street Cambridge, MA and NBER imbens@fas.harvard.edu Jeffrey M. Wooldridge Department of Economics, Michigan State University. wooldri1@msu.edu

4 1 Introduction Many empirical questions in economics and other social sciences depend on causal effects of programs or policies. In the last two decades, much research has been done on the econometric and statistical analysis of the effects of such programs or treatments. This recent theoretical literature has built on, and combined features of, earlier work in both the statistics and econometrics literatures. It has by now reached a level of maturity that makes it an important tool in many areas of empirical research in economics and suitable for a review. In this article we attempt to present such a review. We will focus on practical issues for empirical researchers, as well as provide an historical overview of the area and give references to more technical research. This review complements and extends other reviews and discussions by Blundell and Costa-Dias (2002), Imbens (2004), Angrist and Krueger (2000), and the books by Rosenbaum (1995), Pearl (2000), Lee (2005a), Rubin (2006), Caliendo (2006), Angrist and Pischke (2008), and Morgan and Winship (2007). In addition the reviews in Heckman, Lalonde and Smith (2000), Heckman and Vytlacil (2007a, 2007b), and Abbring and Heckman (2007) provide an excellent overview of the important theoretical work by Heckman and his coauthors in this area. The central problem studied in this literature is that of evaluating the effect of the exposure of a set of units to a program or treatment on some outcome. In economic studies, the units are typically economic agents such as individuals, households, markets, firms, counties, states or countries, but in other disciplines where evaluation methods are used the units can be animals, plots of land, or pieces of material. The treatments can be job search assistance programs, educational programs, vouchers, laws or regulations, medical drugs, environmental exposure, or technologies. A critical feature is that, in principle, each unit can be exposed to one or more different levels of the treatment. An individual may enroll or not in a training program, or he or she may receive or not receive a voucher, or be subject to a particular regulation or not. The object of interest is a comparison of the two outcomes for the same unit when exposed, and when not exposed, to the treatment. The problem is that we can at most observe one of these outcomes because the unit can be exposed to only one level of the treatment. Holland (1986) refers to this as the fundamental problem of causal inference. In order to evaluate the effect of the treatment we therefore always need to compare distinct units receiving the different levels of the treatment. Such a comparison can involve different physical units, or the same physical unit at different times. The problem of evaluating the effect of a binary treatment or program is a well studied problem with a long history in both econometrics and statistics. This is true both in the theoretical literature as well as in the more applied literature. The econometric literature goes back to early work by Ashenfelter (1978) and subsequent work by Ashenfelter and Card (1985), Heckman and Robb (1985), Lalonde (1986), Fraker and Maynard (1987), Card and Sullivan (1988), and Manski (1990). Motivated primarily by applications to the evaluation of labor market programs in observational settings, the focus in the econometric literature is traditionally on endogeneity, or self-selection, issues. Individuals who choose to enroll in a training program are by definition different from those who choose not to enroll. These differences, if they influence the response, may invalidate causal comparisons of outcomes by treatment status, possibly even after ad- [1]

5 justing for observed covariates. Consequently, many of the initial theoretical studies focused on the use of traditional methods for dealing with endogeneity, such as fixed effect methods from panel data analyses and instrumental variables methods. Subsequently, the econometrics literature has combined insights from the semiparametric literature to develop new estimators for a variety of settings, requiring fewer functional form and homogeneity assumptions. The statistics literature starts from a different perspective. This literature originates in the analysis of randomized experiments by Fisher (1925) and Neyman (1923). From the early seventies, Rubin (1973a,b, 1974, 1977, 1978), in a series of papers, formulated the now dominant approach to the analysis of causal effects in observational studies. Rubin proposed the interpretation of causal statements as comparisons of so-called potential outcomes: pairs of outcomes defined for the same unit given different levels of exposure to the treatment. Models are developed for the pair of potential outcomes rather than solely for the observed outcome. Rubin s formulation of the evaluation problem, or the problem of causal inference, labeled the Rubin Causal Model (RCM) by Holland (1986), is by now standard in both the statistics and econometrics literature. One of the attractions of the potential outcomes setup is that from the outset it allows for general heterogeneity in the effects of the treatment. Such heterogeneity is important in practice, and it is important theoretically as it is often the motivation for the endogeneity problems that concern economists. One additional advantage of the potential outcome set up is that the parameters of interest can be defined, and the assumptions stated, without reference to particular parametric models. Of particular importance in Rubin s approach is the relationship between treatment assignment and the potential outcomes. The simplest case for analysis is when assignment to treatment is randomized, and thus independent of covariates as well as the potential outcomes. In such classical randomized experiments, it is straightforward to obtain attractive estimators for the average effect of the treatment, e.g. the difference in means by treatment status. Randomized experiments have been used in some areas in economics. In the seventies, negative income tax experiments received widespread attention. In the late eighties, following an influential paper by Lalonde (1986) that concluded econometric methods were unable to replicate experimental results, more emphasis was put on experimental evaluations of labor market programs, although more recently this emphasis seems to have weakened a bit. In the last couple of years, some of the most interesting experiments have been conducted in development economics (e.g., Miguel and Kremer, 2004; Duflo, 2001; Angrist, Bettinger and Kremer, 2005; Banerjee, Duflo, Cole and Linden, 2007) and behavioral economics (e.g., Bertrand and Mullainathan, 2004). Nevertheless, experimental evaluations remain relatively rare in economics. More common is the case where economists analyze data from observational studies. Observational data generally create challenges in estimating causal effects, but in one important special case, variously referred to as unconfoundedness, exogeneity, ignorability, or selection on observables, questions regarding identification and estimation of the policy effects are fairly well understood. All these labels refer to some form of the assumption that adjusting treatment and control groups for differences in observed covariates, or pretreatment variables, remove all biases in comparisons between treated and control units. This case is of great practical relevance, with many studies relying on some form of this assumption. The semiparametric efficiency bound has been cal- [2]

6 culated for this case (Hahn, 1998) and various semi-parametric estimators have been proposed (Hahn, 1998; Heckman, Ichimura, and Todd, 1998; Hirano, Imbens and Ridder, 2003; Chen, Hong, and Tarozzi, 2005; Imbens, Newey and Ridder, 2005; Abadie and Imbens, 2006). We discuss the current state of this literature, and the practical recommendations coming out of it, in detail in this review. Without unconfoundedness there is no general approach to estimating treatment effects. Various methods have been proposed for special cases, and in this review we will discuss several of them. One approach (Rosenbaum and Rubin, 1983; Rosenbaum, 1995) consists of sensitivity analyses, where robustness of estimates to specific limited departures from unconfoundedness are investigated. A second approach, developed by Manski (1990, 2003, 2007), consists of bounds analyses, where ranges of estimands consistent with the data and the limited assumptions the researcher is willing to make, are derived and estimated. A third approach, instrumental variables, relies on the presence of additional treatments, the so-called instruments, that satisfy specific exogeneity and exclusion restrictions. The formulation of this method in the context of the potential outcomes framework is presented in Imbens and Angrist (1994) and Angrist, Imbens and Rubin (1996). A fourth approach applies to settings where, in its pure form, overlap is completely absent because the assignment is a deterministic function of covariates, but comparisons can be made exploiting contintuity of average outcomes as a function of covariates. This setting, known as the regression discontinuity design, has a long tradition in statistics (see Shadish, Campbell, and Cook, (2002), Cook (2007) for a historical perspective), but has recently been revived in the economics literature through work by VanderKlaauw (2002), Hahn, Todd, and VanderKlaauw (2000), Lee (2001), and Porter (2003). Finally, a fifth approach, referred to as difference in differences, relies on the presence of additional data in the form of samples of treated and control units before and after the treatment. An early application is Ashenfelter and Card (1985). Recent theoretical work includes Abadie (2005), Bertrand, Duflo and Mullainathan (2004), Donald and Lang (2008), and Athey and Imbens (2006). In this review we will discuss in detail some of the new methods that have been developed in this literature. We will pay particular attention to the practical issues raised by the implementation of these methods. At this stage, the literature has matured to the extent that it has much to offer the empirical researcher. Although the evaluation problem is one where identification problems are important, there is currently a much better understanding of which assumptions are most useful, as well as a better set of methods for inference given different sets of assumptions. Most of this review will be limited to settings with binary treatments. This is in keeping with the literature, which has largely focused on binary treatment case. There are some extensions of these methods to multivalued, and even continuous, treatments (e.g., Imbens, 2000; Lechner, 2001; Lechner and Miquel, 2005; Gill and Robins, 2001; Hirano and Imbens, 2004), and some of these extensions will be discussed in the current review. But the work in this area is ongoing, and much remains to be done here. The running example we will use throughout the paper is that of a job market training program. Such programs have been among the leading applications in the economics litera- [3]

7 ture, starting with Ashenfelter (1978) and including Lalonde (1986) as a particularly influential study. In such settings, a number of individuals enroll or not in a training program, with labor market outcomes, such as yearly earnings or employment status, as the main outcome of interest. An individual not participating in the program may have choosen not to do so, or may have been ineligible for various reasons. Understanding the choices made and constraints faced by the potential participants is a crucial component of any analysis. In addition to observing participation status and outcome measures, we typically observe individual background characteristics, such as education levels and age, as well as information regarding prior labor market histories, such as earnings at various levels of aggregation (e.g., yearly, quarterly or monthly). In addition, we may observe some of the constraints faced by the individuals, including measures used to determine eligibility, as well as measures of general labor market conditions in the local labor markets faced by potential participants. 2 The Rubin Causal Model: Potential Outcomes, the Assignment Mechanism, and Interactions In this section we describe the essential elements of the modern approach to program evaluation, based on the work by Rubin. Suppose we wish to analyze a job training program using observations on N individuals, indexed by i = 1,..., N. Some of these individuals were enrolled in the training program. Others were not enrolled, either because they were ineligible or chose not to enroll. We use the indicator W i to indicate whether individual i enrolled in the training program, with W i = 0 if individual i did not, and W i = 1 if individual i did, enroll in the program. We use W to denote the N-vector with i-th element equal to W i, and N 0 and N 1 to denote the number of control and treated units, respectively. For each unit we also observe a K-dimensional column vector of covariates or pretreatment variables, X i, with X denoting the N K matrix with i-th row equal to X i. 2.1 Potential Outcomes The first element of the RCM is the notion of potential outcomes. For individual i, for i = 1,..., N, we postulate the existence of two potential outcomes, denoted by Y i (0) and Y i (1). The first, Y i (0), denotes the outcome that would be realized by individual i if he or she did not participate in the program. Similarly, Y i (1) denotes the outcome that would be realized by individual i if he or she did participate in the program. Individual i can either participate or not participate in the program, but not both, and thus only one of these two potential outcomes can be realized. Prior to the assignment being determined, both are potentially observable, hence the label potential outcomes. If individual i participates in the program Y i (1) will be realized and Y i (0) will ex post be a counterfactual outcome. If, on the other hand individual i does not participate in the program, Y i (0) will be realized and Y i (1) will be the ex post counterfactual. We will denote the realized outcome by Y i, with Y the N-vector with i-th element equal to Y i. [4]

8 The preceeding discussion implies that Y i = Y i (W i )=Y i (0) (1 W i )+Y i (1) W i = { Yi (0) if W i =0, Y i (1) if W i =1. The potential outcomes are tied to the specific manipulation that would have made one of them the realized outcome. The more precise the specification of the manipulation, the more well-defined the potential outcomes are. This distinction between the pair of potential outcomes (Y i (0),Y i (1)) and the realized outcome Y i is the hallmark of modern statistical and econometric analyses of treatment effects. We offer some comments on it. The potential outcomes framework has important precursors in a variety of other settings. Most directly, in the context of randomized experiments, the potential outcome framework was introduced by Neyman (1923) to derive the properties of estimators and confidence intervals under repeated sampling. The potential outcomes framework also has important antecedents in econometrics. Specifically, it is interesting to compare the distinction between potential outcomes Y i (0) and Y i (1) and the realized outcome Y i in Rubin s approach to Haavelmo s (1943) work on simultaneous equations models (SEMs). Haavelmo discusses identification of supply and demand models. He makes a distinction between any imaginable price π as the argument in the demand and supply functions, q d (π) and q s (π), and the actual price p, which is the observed equilibrium price satisfying q s (p) =q d (p). The supply and demand functions play the same role as the potential outcomes in Rubin s approach, with the equilibrium price similar to the realized outcome. Curiously, Haavelmo s notational distinction between equilibrium and potential prices has gotten blurred in many textbook discussions of simultaneous equations. In such discussions, the starting point is often the general formulation YΓ +XB = U for N M vectors of realized outcomes Y, N L matrices of exogenous covariates X, and an N M matrix of unobserved components U. A nontrivial byproduct of the potential outcomes approach (POA) is that it forces users of SEMs to articulate what the potential outcomes are, thereby leading to better applications of SEMs. Another area where potential outcomes are used explicitly is in the econometric analyses of production functions. Like the potential outcomes framework, a production function g(x, ε) describes production levels that would be achieved for each value of a vector of inputs, some observed (x) and some unobserved (ε). Observed inputs may be choosen partly as a function of (expected) values of unobserved inputs. Potential outcomes are also used explicitly in labor market settings by Roy (1951). Roy models individuals choosing from a set of occupations. Individuals know what their earnings would be in each of these occupations and choose the occupation (treatment) that maximizes their earnings. Here we see the explicit use of the potential outcomes, combined with a specific selection/assignment mechanism, namely, choosing the treatment with the highest potential outcome. The potential outcomes framework has a number of advantages over a framework based directly on realized outcomes. The first advantage of the potential outcome framework is that it allows us to define causal effects before specifying the assignment mechanism, and without making functional form or distributional assumptions. The most common definition of the [5]

9 causal effect at the unit level is as the difference Y i (1) Y i (0), but we may wish to look at ratios Y i (1)/Y i (0), or other functions. Such definitions do not require us to take a stand on whether the effect is constant or varies accross the population. Further, defining individualspecific treatment effects using potential outcomes does not require us to assume endogeneity or exogeneity of the assignment mechanism. By contrast, the causal effects are more difficult to define in terms of the realized outcomes. Often, researchers write down a regression function Y i = α + τ W i + ε i. This regression function is then interpreted as a structural equation, with τ as the causal effect. Left unclear is whether the causal effect is constant or not, and what the properties of the unobserved component, ε i, are. The potential outcomes approach separates these issues, and allows the researcher to first define the causal effect of interest without considering probabilistic properties of the outcomes or assignment. The second advantage of the POA is that it links the analysis of causal effects to explicit manipulations. Considering the two potential outcomes forces the researcher to think about scenarios under which each outcome could be observed, that is, to consider the kinds of experiments that could reveal the causal effects. Doing so clarifies the interpretation of causal effects. For illustration, consider a couple of recent examples from the economics literature. First, consider the causal effects of gender or ethnicity on outcomes of job applications. Simple comparisons of economic outcomes by ethnicity are difficult to interpret. Are they the result of discrimination by employers, or are they the result of differences between applicants, possibly arising from discrimination at an earlier stage of life? Now, one can obtain unambiguous causal interpretations by linking comparisons to specific manipulations. A recent example is the study by Bertrand and Mullainathan (2004), who compare call-back rates for job applications submitted with names that suggest African-American or Caucasian ethnicity. Their study has a clear manipulation a name change and therefore a clear causal effect. As a second example, consider some recent economic studies that have focused on causal effects of individual characteristics such as beauty (Hamermesh and Biddle, 1994), or height. Do the differences in earnings by ratings on a beauty scale represent causal effects? One possible interpretation is that they represent causal effects of plastic surgery. Such a manipulation would make differences causal, but it appears unclear whether cross-sectional correlations between beauty and earnings in a survey from the general population represent causal effects of plastic surgery. A third advantage of the POA is that it separates the modelling of the potential outcomes from that of the assignment mechanism. Modelling the realized outcome is complicated by the fact that it combines the potential outcomes and the assignment mechanism. The researcher may have very different sources of information to bear on each. For example, in the labor market program example we can consider the outcome, say, earnings, in the absence of the program: Y i (0). We can model this in terms of individual characteristics and labor market histories. Similarly, we can model the outcome given enrollment in the program, again conditional on individual characteristics and labor market histories. Then finally we can model the probability of enrolling in the program given the earnings in both treatment arms conditional on individual characteristics. This sequential modelling will lead to a model for the realized outcome, but it may be easier than directly specifying a model for the realized outcome. A fourth advantage of the potential outcomes approach is that it allows us to formulate [6]

10 probabilistic assumptions in terms of potentially observable variables, rather than in terms of unobserved components. In this approach, many of the critical assumptions will be formulated as (conditional) independence assumptions involving the potential outcomes. Assessing their validity requires the researcher to consider the dependence structure if all potential outcomes where observed. By contrast, models in terms of realized outcomes often formulate the critical assumptions in terms of errors in regression functions. To be specific, consider again the regression function Y i = α + τ W i + ε i. Typically (conditional independence) assumptions are made on the relationship between ε i and W i. Such assumptions implicitly bundle a number of assumptions, including functional-form assumptions and substantive exogeneity assumptions. This bundling makes the plausibility of these assumptions more difficult to assess. A fifth advantage of the POA is that it clarifies where the uncertainty in the estimators comes from. Even if we observe the entire (finite) population (as is increasingly common with the growing availability of administrative data sets) so we can estimate population averages with no uncertainty causal effects will be uncertain because for each unit at most one of the two potential outcomes is observed. One may still use super population arguments to justify approximations to the finite sample distributions, but such arguments are not required to motivate the existence of uncertainty about the causal effect. 2.2 The Assignment Mechanism The second ingredient of the RCM is the assignment mechanism. This is defined as the conditional probability of receiving the treatment, as a function of potential outcomes and observed covariates. We distinguish three classes of assignment mechanisms, in order of increasing complexity of the required analysis. The first class of assignment mechanisms is that of randomized experiments. In randomized experiments, the probability of assignment to treatment does not vary with potential outcomes, and is a known function of covariates. The leading case is that of a completely randomized experiment where, in a population of N units, N 1 <Nrandomlychosen units are assigned to the treatment and the remaining N 0 = N N 1 units are in the control group. There are important variations on this example, such as pairwise randomization, where initially units are matched in pairs, and in a second stage one unit in each pair is randomly assigned to the treatment. Another variant is a general stratified experiment, where randomization takes place within a finite number of strata. In any case, there are in practice few experiments in economics, and most of those are of the completely randomized experiment variety, so we shall limit our discussion to this type of experiment. It should be noted though that if one has the opportunity to design a randomized experiment, and if pretreatment variables are available, stratified experiments are at least as good as completely randomized experiments, and typically better, in terms of expected mean squared error, even in finite samples. See Imbens, King, McKenzie and Ridder (2008) for more details. The use of formal randomization has become more widespread in the social sciences in recent years, sometimes as a formal design for an evaluation and sometimes as an acceptable way of allocating scarce resources. The analysis of such experiments is often straightforward. In practice, however, researchers have typically limited themselves to simple mean differences by assignment. Such analyses are valid, but often [7]

11 they are not the most powerful tools available to exploit the randomization. We discuss the analysis of randomized experiments, including more powerful randomization-based methods for inference, in Section 4. The second class of assignment mechanisms maintains the restriction that the assignment probabilities do not depend on the potential outcomes, or ( Y i (0),Y i (1)) Xi, W i where A B C denotes conditional independence of A and B given C. However, in contrast to randomized experiments, the assignment probabilities are no longer assumed to be a known function of the covariates. The precise form of this critical assumption, not tied to functional form or distributional assumptions, was first presented in Rosenbaum and Rubin (1983a). Following Rubin (1990) we refer to this assignment mechanism as unconfounded assignment. Somewhat confusingly, this assumption, or variations on it, are in the literature also referred to by various other labels. These include selection on observables 1, exogeneity 2, and conditional independence 3. Although the analysis of data with such assignment mechanisms is not as straightforward as that of randomized experiments, there are now many practical methods available for this case. We review them in Section 5. The third class of assignment mechanisms contains all remaining assignment mechanisms with some dependence on potential outcomes. 4 Many of these create substantive problems for the analysis, for which there is no general solution. There are a number of special cases that are by now relatively well understood, and we discuss these in Section 6. The most prominent of these cases are instrumental variables, regression discontinuity, and differences-in-differences. In addition, we discuss two general methods that also relax the unconfoundedness assumption but do not replace it with additional assumptions. The first relaxes the unconfoundedness assumption in a limited way and investigates the sensitivity of the estimates to such violations. The second drops the unconfoundedness assumption entirely and establishes bounds on estimands of interest. The latter is associated with the work by Manski (1990, 1995, 2008). 1 Although Heckman, Ichimura and Todd (1997, page 611) write that In the language of Heckman and Robb (1985) matching assumes that selection is on observables (their italics), the original definition in Heckman and Robb (1985, page 163) is not equivalent to unconfoundedness. In the context of a single cross-section version of their two equation selection model, Y = X iβ + W iα + ε i and W i =1{Z iγ + ν i > 0}, they define selection bias to refer to the case where E[ε iw i] 0, and selection-on-observables to the case where selection bias is present and caused by correlation between ε i and Z i, rather than by correlation between ε i and ν i. 2 Although X i is not exogenous for E[Y i(1) Y i(0)], according to the definitions in Engle, Hendry and Richard (1983), because knowledge of its marginal distribution contains information about E[Y i(1) Y i(0)], standard usage of the term exogenous does appear to capture the notion of unconfoundedness, e.g., Manski, Sandefur, McLanahan, and Powers (1992), and Imbens (2004). 3 E.g., Lechner, 2001; Cameron and Trivedi, This includes some mechanisms where the dependence on potential outcomes does not create any problems in the analyses. Most prominent in this category are sequential assignment mechanisms. For example, one could randomly assign the first ten units to the treatment or control group with probability 1/2. From then on one could skew the assignment probability to the treatment with the most favorable outcomes so far. For example, if the active treatment looks better than the control treatment based on the first N units, then the (N + 1)th unit is assigned to the active treatment with probability 0.8 and vice versa. Such assignment mechanisms are not very common in economics settings, and we ignore them in this discussion. [8]

12 2.3 Interactions and General Equilibrium Effects In most of the literature it is assumed that treatments received by one unit do not affect outcomes for another unit. Only the level of the treatment applied to the specific individual is assumed to potentially affect outcomes for that particular individual. In the statistics literature this assumption is referred to as the Stable-Unit-Treatment-Value-Assumption (SUTVA, Rubin, 1978). In this paper we mainly focus on settings where this assumption is maintained. In the current section we discuss some of the literature motivated by concerns about this assumption. This lack-of-interaction assumption is very plausible in many biomedical applications. Whether one individual receives or does not receive a new treatment for a stroke or not is unlikely to have a substantial impact on health outcomes for any other individual. However, there are also many cases in which such interactions are a major concern and the assumption is not plausible. Even in the early experimental literature, with applications to the effect of various fertilizers on crop yields, researchers were cognizant of potential problems with this assumption. In order to minimize leaking of fertilizer applied to one plot into an adjacent plot experimenters used guard rows to physically separate the plots that were assigned different fertilizers. A different concern arises in epidimiological applications when the focus is on treatments such as vaccines for contagious diseases. In that case, it is clear that the vaccination of one unit can affect the outcomes of others in their proximity, and such effects are a large part of the focus of the evaluation. In economic applications, interactions between individual are also a serious concern. It is clear that a labor market program that affects the labor market outcomes for one individual potentially has an effect on the labor market outcomes for others. In a world with a fixed number of jobs, a training program could only redistribute the jobs, and ignoring this constraint on the number of jobs by using a partial, instead of a general, equilibrium analysis could lead one to erroneously conclude that extending the program to the entire population would raise aggregate employment. Such concerns have rarely been addressed in the recent program evaluation literature. Exceptions include Heckman, Lochner, and Taber (1999) who provide some simulation evidence for the potential biases that may result from ignoring these issues. In practice these general equilbrium effects may, or may not, be a serious problem. The indirect effect on one individual of exposure to the treatment of a few other units is likely to be much smaller than the direct effect of the exposure of the first unit itself. Hence, with most labor market programs both small in scope and with limited effects on the individual outcomes, it appears unlikely that general equilibrium effects are substantial and they can probably be ignored for most purposes. One general solution to these problems is to redefine the unit of interest. If the interactions between individuals are at an intermediate level, say a local labor market, or a classroom, rather than global, one can analyze the data using the local labor market or classroom as the unit and changing the no-interaction assumption to require the absence of interactions among local labor markets or classrooms. Such aggregation is likely to make the no-interaction assumption more plausible, albeit at the expense of reduced precision. An alternative solution is to directly model the interactions. This involves specifying which [9]

13 individuals interact with each other, and possibly relative magnitudes of these interactions. In some cases it may be plausible to assume that interactions are limited to individuals within well-defined, possibly overlapping, groups, with the intensity of the interactions equal within this group. This would be the case in a world with a fixed number of jobs in a local labor market. Alternatively, it may be that interactions occur in broader groups but decline in importance depending on some distance metric, either geographical distance or proximity in some economic metric. The most interesting literature in this area views the interactions not as a nuisance but as the primary object of interest. This literature, which includes models of social interactions and peer effects, has been growing rapidly in the last decade, following the early work by Manski (1993). See Manski (2000) and Brock and Durlauf (2000) for recent surveys. Empirical work includes Kling, Liebman and Katz (2007), who look at the effect of households moving to neighborhoods with higher average socio-economic status; Sacerdote (2001), who studies the effect of college roommate behavior on a student s grades; Glaeser, Sacerdote and Scheinkman (1996), who study social interactions in criminal behavior; Case and Katz (1991), who look at neighbourhood effects on disadvantaged youths, Graham (2006), who infer interactions from the effect of class size on the variation in grades; and Angrist and Lang (2004), who study the effect of desegregation programs on students grades. Many identification and inferential questions remain unanswered in this literature. 3 What are We Interested In? Estimands and Hypotheses In this section we discuss some of the questions that researchers have asked in this literature. A key feature of the current literature, and one that makes it more important to be precise about the questions of interest, is the accommodation of general heterogeneity in treatment effects. In contrast, in many early studies it was assumed that the effect of a treatment was constant, implying that the effect of various policies could be captured by a single parameter. The essentially unlimited heterogeneity in the effects of the treatment allowed for in the current literature implies that it is generally not possible to capture the effects of all policies of interest in terms of a few summary statistics. In practice researchers have reported estimates of the effects of a few focal policies. In this section we describe some of these estimands. Most of these estimands are average treatment effects, either for the entire population or for some subpopulation, although some correspond to other features of the joint distribution of potential outcomes. Most of the empirical literature has focused on estimation. Much less attention has been devoted to testing hypotheses regarding the properties or presence of treatment effects. Here we discuss null and alternative hypotheses that may be of interest in settings with heterogeneous effects. Finally, we discuss some of the recent literature on decision-theoretic approaches to program evaluation that ties estimands more closely to optimal policies. [10]

14 3.1 Average Treatment Effects The econometric literature has largely focused on average effects of the treatment. The two most prominent average effects both rely on a superpopulation perspective. The sample of size N is viewed as a random sample from a large (super-)population, and interest is in the average effect in the superpopulation. The most popular one is the Population Average Treatment Effect (PATE), the population expectation of the unit-level causal effect, Y i (1) Y i (0): τ pate = E [Y i (1) Y i (0)]. If the policy under consideration would expose all units to the treatment or none at all, this is the most relevant quantity. Another popular estimand is the Population Average Treatment effect on the Treated (PATT), the average over the subpopulation of treated units: τ patt = E [Y i (1) Y i (0) W i = 1]. In many observational studies τ patt is a more interesting estimand than the overall average effect. As an example, consider the case where a well defined population was exposed to a treatment, say a job training program. There may be various possibilities for a comparison group, including subjects drawn from public use data sets. In that case it is generally not interesting to consider the effect of the program for the comparison group: for many members of the comparison group (e.g., individuals with stable, high-wage jobs) it is difficult and uninteresting to imagine their being enrolled in the labor market program. (Of course, the problem of averaging across units that are unlikely to receive future treatments can be mitigated by more carefully constructing the comparison group to be more like the treatment group, making τ pate a more meaningful parameter. See the discussion below.) A second case where τ patt is the estimand of most interest is in the setting of a voluntary program where those not enrolled will never be required to participate in the program. A specific example is the effect of serving in the military where an interesting question concerns the foregone earnings for those who served (Angrist, 1998). In practice, there is typically little motivation presented for the focus on the overall average effect or the average effect for the treated. Take a job training program. The overall average effect would be the parameter of interest if the policy under consideration is a mandatory exposure to the treatment versus complete elimination. It is rare that these are the alternatives, with more typically exemptions granted to various subpopulations. Similarly the average effect for the treated would be informative about the effect of entirely eliminating the current program. More plausible regime changes would correspond to a modest extension of the program to other jurisdictions, or a contraction to a more narrow population. A somewhat subtle issue is that we may wish to separate the extrapolation from the sample to the superpopulation fromthe problem of inference for the sample at hand. This suggests that, rather than focusing on PATE or PATT, we might first focus on the average causal effect conditional on the covariates in the sample, τ cate = 1 N N E [Y i (1) Y i (0) X i ], i=1 [11]

15 and, similarly, the average over the subsample of treated units: τ catt = 1 N 1 i W i =1 E [Y i (1) Y i (0) X i ]. If the effect of the treatment or intervention is constant (Y i (1) Y i (0) = τ for some constant τ), all four estimands, τ pate, τ patt, τ cate, and τ catt, are obviously identical. However, if there is heterogeneity in the effect of the treatment, the estimands may all be different. The difference between τ pate and τ cate (and between τ patt and τ catt ) is relatively subtle. Most estimators that are attractive for the population treatment effect are also attractive for the corresponding conditional average treatment effect, and vice versa. Therefore, we do not have to be particularly concerned with the distinction between the two estimands at the estimation stage. However, there is an important difference between the population and conditional estimands at the inference stage. If there is heterogeneity in the effect of the treatment, we can estimate the sample average treatment effect τ cate more precisely than the population average treatment effect τ pate. When one estimates the variance of an estimator ˆτ which can serve as an estimate for τ pate or τ cate one therefore needs to be explicit about whether one is interested in the variance relative to the population or to the conditional average treatment effect. We will return to this issue in Section 5. A more general class of estimands includes average causal effects for subpopulations and weighted average causal effects. Let A be a subset of the covariate space X, and let τ cate,a denote the conditional average causal effect for the subpopulation with X i A: τ cate,a = 1 N A i:x i A E [Y i (1) Y i (0) X i ], where N A is the number of units with X i A. Crump, Hotz, Imbens and Mitnik (2008a) argue for considering such estimands. Their argument is not based on the intrinsic interest of these subpopulations. Rather, they show that such estimands may be much easier to estimate than τ cate (or τ catt ). Instead of solely reporting an imprecisely estimated average effect for the overall population, they suggest it may be informative to also report a precise estimate for the average effect of some subpopulation. They then propose a particular set A for which the average effect is most easily estimable. See Section for more details. The Crump et al estimates would not necessarily have as much external validity as estimates for the overall population, but they may be much more informative for the sample at hand. In any case, in many instances the larger policy questions concern extensions of the interventions or treatments to other populations, so that external validity may be elusive irrespective of the estimand. In settings with selection on unobservables the enumeration of the estimands of interest becomes more complicated. A leading case is instrumental variables. In the presence of heterogeneity in the effect of the treatment one can typically not identify the average effect of the treatment even in the presence of valid instruments. There are two new approaches in the recent literature. One is to focus on bounds for well-defined estimands such as the average effect τ pate or τ cate. Manski (1990, 2003) developed this approach in a series of papers. An alternative is to focus on estimands that can be identified under weaker conditions than those [12]

16 requird for the average treatment effect. Imbens and Angrist (1994) show that one can, under much weaker conditions than required for identification of τ pate, identify the average effect for the subpopulation of units whose treatment status is affected by the instrument. They refer to this subpopulation as the compliers. This does not directly fit into the classification above since the subpopulation is not defined solely in terms of covariates. We discuss this estimand in more detail in Section Quantile and distributional Treatment Effects and other estimands An alternative class of estimands concerns quantile treatment effects. These have only recently been studied and applied in the economics literature, although they were introduced in the statistics literature in the seventies. Doksum (1974) and Lehman (1974) define τ q = F 1 1 (q) F (q), (1) Y (1) Y (0) as the q-th quantile treatment effect. There are some important issues in interpreting these quantile treatment effects. First, note that these quantiles effects are defined as differences between quantiles of the two marginal potential outcome distributions, rather than as quantiles of the unit level effect, τ q = F 1 Y (1) Y (0) (q). (2) In general the quantile of the difference, τ q, differs from the difference in the quantiles, τ q, unless there is perfect rank correlation between the potential outcomes Y i (0) and Y i (1) (the leading case of this is the constant additive treatment effect). The quantiles of the treatment effect, τ q, have received much less attention than the quantile treatment effects, τ q. The main reason is that the τ q are generally not identified without assumptions on the rank correlation between the potential outcomes, even with data from a randomized experiment. Note that this issue does not arise if we look at average effects because the mean of the difference is equal to the difference of the means: E [Y i (1) Y i (0)] = E [Y i (1)] E [Y i (0)]. A complication facing researchers interested in quantile treatment effects is that the difference in a marginal quantile, τ q, is in general not equal to the average difference in the conditional quantiles, where the latter are defined as τ q (x) =F 1 1 Y (1) X(q x) FY (0) X (q x). In other words, even if we succeed in estimating τ q (x), we cannot simply average τ q (X i ) across i to consistently estimate τ q. Bitler, Gelbach, and Hoynes (2002) estimate quantile treatment effects in a randomized evaluation of a job training program. Firpo (2006) develops methods for estimating τ q in observational studies given unconfoundedness. Abadie, Angrist and Imbens (2002) and Chernozhukov and Hansen (2005) study quantile treatment effects in instrumental variables settings. [13]

17 3.3 Testing The literature on hypothesis testing in program evaluation is relatively limited. Most of the testing in applied work has focused on the null hypothesis that the average effect of interest is zero. Because many of the commonly used estimators for average treatment effects are asymptotically normally distributed with zero asymptotic bias, it follows that standard confidence intervals (the point estimate plus or minus a constant times the standard error) can be used for testing such hypotheses. However, there are other interesting hypotheses to consider. One question of interest is whether there is any effect of the program, that is whether the distribution of Y i (1) differs from that of Y i (0). This is equivalent to the hypothesis that not just the mean, but all moments, are identical in the two treatment groups. Abadie (2002) studies such tests in the settings with randomized experiments as well as settings with instrumental variables using Kolmogorov-Smirnov type testing procedures. A second set of questions concerns treatment effect heterogeneity. Even if the average effect is zero, it may be important to establish whether a targeted implementation of the intervention, with only those who can expect to benefit from the intervention assigned to it, could improve average outcomes. In addition, in cases where there is not sufficient information to obtain precise inferences for the average causal effect τ pate, it may still be possible to establish whether there are any subpopulations with an average effect positive or different from zero, or whether there are subpopulations with an average effect exceeding some threshold. It may also be interesting to test whether there is any evidence of heterogeneity in the treatment effect by observable characteristics. This bears heavily on the question whether the estimands are useful for extrapolation to other populations which may differ in terms of some observable characteristics. Crump, Hotz, Imbens and Mitnik (2008b) study these questions in settings with unconfounded treatment assignment. 3.4 Decision-theoretic Questions Recently, a small but very innovative literature has started to move away from the focus on summary statistics of the distribution of treatment effects or potential outcomes to directly address policies of interest. This is very much a literature in progress. Manski (2000, 2002, 2004), Dehejia (2005), and Hirano and Porter (2005) study the problem faced by program administrators who can assign individuals to the active treatment or to the control group. These administrators have available two pieces of information. First, covariate information for these individuals, and second, information about the efficacy of the treatment based on a finite sample of other individuals for whom both outcome and covariate information is available. The administrator may care about the entire distribution of outcomes, or solely about average outcomes, and may also take into account costs associated with participation. If the administrator knew exactly the conditional distribution of the potential outcomes given the covariate information this would be a simple problem: the administrator would simply compare the expected welfare for different rules and choose the one with the highest value. However, the administrator does not have this knowledge and needs to make a decision given uncertainty about these distributions. In these settings, it is clearly important that the statistical model allows for heterogeneity in the [14]