Causal inference using regression on the treatment variable

Transcription

1 CHAPTER 9 Causal inference using regressin n the treatment variable 9.1 Causal inference and predictive cmparisns S far, we have been interpreting regressins predictively: given the values f several inputs, the fitted mdel allws us t predict y, cnsidering the n data pints as a simple randm sample frm a hypthetical infinite superppulatin r prbability distributin. Then we can make cmparisns acrss different cmbinatins f values fr these inputs. This chapter and the net cnsider causal inference, which cncerns what wuld happen t an utcme y as a result f a hypthesized treatment r interventin. In a regressin framewrk, the treatment can be written as a variable T : 1 { 1 if unit i receives the treatment T i = 0 if unit i receives the cntrl, r, fr a cntinuus treatment, T i = level f the treatment assigned t unit i. In the usual regressin cntet, predictive inference relates t cmparisns between units, whereas causal inference addresses cmparisns f different treatments if applied t the same units. Mre generally, causal inference can be viewed as a special case f predictin in which the gal is t predict what wuld have happened under different treatment ptins. We shall discuss this theretical framewrk mre thrughly in Sectin 9.2. Causal interpretatins f regressin cefficients can nly be justified by relying n much stricter assumptins than are needed fr predictive inference. T mtivate the detailed study f regressin mdels fr causal effects, we present tw simple eamples in which predictive cmparisns d nt yield apprpriate causal inferences. Hypthetical eample f zer causal effect but psitive predictive cmparisn Cnsider a hypthetical medical eperiment in which 100 patients receive the treatment and 100 receive the cntrl cnditin. In this scenari, the causal effect represents a cmparisn between what wuld have happened t a given patient had he r she received the treatment cmpared t what wuld have happened under cntrl. We first suppse that the treatment wuld have n effect n the health status f any given patient, cmpared with what wuld have happened under the cntrl. That is, the causal effect f the treatment is zer. Hwever, let us further suppse that treated and cntrl grups systematically differ, with healthier patients receiving the treatment and sicker patients receiving 1 We use a capital letter fr the vectr T (vilating ur usual rule f reserving capitals fr matrices) in rder t emphasize the treatment as a key variable in causal analyses, and als t avid ptential cnfusin with t, which we smetimes use fr time. 167

2 168 CAUSAL INFERENCE USING DIRECT REGRESSION Distributin f measurements after eperiment Previus if assigned the if assigned the health status cntrl cnditin treatment Pr Fair Gd Figure 9.1 Hypthetical scenari f zer causal effect f treatment: fr any value f previus health status, the distributins f ptential utcmes are identical under cntrl and treatment. Hwever, the predictive cmparisn between treatment and cntrl culd be psitive, if healthier patients receive the treatment and sicker patients receive the cntrl cnditin. Distributin f measurements after eperiment Previus if assigned the if assigned the health status cntrl cnditin treatment Pr Fair Gd Figure 9.2 Hypthetical scenari f psitive causal effect f treatment: fr any value f previus health status, the distributins f ptential utcmes are centered at higher values fr treatment than fr cntrl. Hwever, the predictive cmparisn between treatment and cntrl culd be zer, if sicker patients receive the treatment and healthier patients receive the cntrl cnditin. Cmpare t Figure 9.1. the cntrl. This scenari is illustrated in Figure 9.1, where the distributin f utcme health status measurements is centered at the same place fr the treatment and cntrl cnditins within each previus health status categry (reflecting the lack f causal effect) but the heights f each distributin reflect the differential prprtins f the sample that fell in each cnditin. This scenari leads t a psitive predictive cmparisn between the treatment and cntrl grups, even thugh the causal effect is zer. This srt f discrepancy between the predictive cmparisn and the causal effect is smetimes called self-selectin bias, r simply selectin bias, because participants are selecting themselves int different treatments. Hypthetical eample f psitive causal effect but zer psitive predictive cmparisn Cnversely, it is pssible fr a truly nnzer treatment effect t nt shw up in the predictive cmparisn. Figure 9.2 illustrates. In this scenari, the treatment has a psitive effect fr all patients, whatever their previus health status, as displayed

3 CAUSAL INFERENCE AND PREDICTIVE COMPARISONS 169 by utcme distributins that fr the treatment grup are centered ne pint t the right f the crrespnding (same previus health status) distributins in the cntrl grup. S, fr any given unit, we wuld epect the utcme t be better under treatment than cntrl. Hwever, suppse that this time, sicker patients are given the treatment and healthier patients are assigned t the cntrl cnditin, as illustrated by the different heights f these distributins. It is then pssible t see equal average utcmes f patients in the tw grups, with sick patients wh received the treatment canceling ut healthy patients wh received the cntrl. Previus health status plays an imprtant rle in bth these scenaris because it is related bth t treatment assignment and future health status. If a causal estimate is desired, simple cmparisns f average utcmes acrss grups that ignre this variable will be misleading because the effect f the treatment will be cnfunded with the effect f previus health status. Fr this reasn, such predictrs are smetimes called cnfunding cvariates. Adding regressin predictrs; mitted r lurking variables The preceding theretical eamples illustrate hw a simple predictive cmparisn is nt necessarily an apprpriate estimate f a causal effect. In these simple eamples, hwever, there is a simple slutin, which is t cmpare treated and cntrl units cnditinal n previus health status. Intuitively, the simplest way t d this is t cmpare the averages f the current health status measurements acrss treatment grups nly within each previus health status categry; we discuss this kind f subclassificatin strategy in Sectin Anther way t estimate the causal effect in this scenari is t regress the utcme n tw inputs: the treatment indicatr and previus health status. If health status is the nly cnfunding cvariate that is, the nly variable that predicts bth the treatment and the utcme and if the regressin mdel is prperly specified, then the cefficient f the treatment indicatr crrespnds t the average causal effect in the sample. In this eample a simple way t avid pssible misspecificatin wuld be t discretize health status using indicatr variables rather than including it as a single cntinuus predictr. In general, then, causal effects can be estimated using regressin if the mdel includes all cnfunding cvariates (predictrs that can affect treatment assignment r the utcme) and if the mdel is crrect. If the cnfunding cvariates are all bserved (as in this eample), then accurate estimatin cmes dwn t prper mdeling and the etent t which the mdel is frced t etraplate beynd the supprt f the data. If the cnfunding cvariates are nt bserved (fr eample, if we suspect that healthier patients received the treatment, but n accurate measure f previus health status is included in the mdel), then they are mitted r lurking variables that cmplicate the quest t estimate causal effects. We cnsider these issues in mre detail in the rest f this chapter and the net, but first we will prvide sme intuitin in the frm f an algebraic frmula. Frmula fr mitted variable bias We can quantify the bias incurred by ecluding a cnfunding cvariate in the cntet where a simple linear regressin mdel is apprpriate and there is nly ne cnfunding cvariate. First define the crrect specificatin as y i = β 0 + β 1 T i + β 2 i + ɛ i (9.1)

4 170 CAUSAL INFERENCE USING DIRECT REGRESSION where T i is the treatment and i is the cvariate fr unit i. If instead the cnfunding cvariate, i, is ignred, ne can fit the mdel y i = β0 + β1t i + ɛ i What is the relatin between these mdels? T understand, it helps t define a third regressin, i = γ 0 + γ 1 T i + ν i If we substitute this representatin f int the riginal, crrect, equatin, and rearrange terms, we get y i = β 0 + β 2 γ 0 +(β 1 + β 2 γ 1 )T i + ɛ i + β 2 ν i (9.2) Equating the cefficients f T in (9.1) and (9.2) yields β1 = β 1 + β2γ 1 This crrespndence helps demnstrate the definitin f a cnfunding cvariate. If there is n assciatin between the treatment and the purprted cnfunder (that is, γ 1 = 0) r if there is n assciatin between the utcme and the cnfunder (that is, β 2 = 0) then the variable is nt a cnfunder because there will be n bias (β 2γ 1 =0). This frmula is cmmnly presented in regressin tets as a way f describing the bias that can be incurred if a mdel is specified incrrectly. Hwever, this term has little meaning utside f a cntet in which ne is attempting t make causal inferences. 9.2 The fundamental prblem f causal inference We begin by cnsidering the prblem f estimating the causal effect f a treatment cmpared t a cntrl, fr eample in a medical eperiment. Frmally, the causal effect f a treatment T n an utcme y fr an bservatinal r eperimental unit i can be defined by cmparisns between the utcmes that wuld have ccurred under each f the different treatment pssibilities. With a binary treatment T taking n the value 0 (cntrl) r 1 (treatment), we can define ptential utcmes, yi 0 and yi 1 fr unit i as the utcmes that wuld be bserved under cntrl and treatment cnditins, respectively. 2 (These ideas can als be directly generalized t the case f a treatment variable with multiple levels.) The prblem Fr smene assigned t the treatment cnditin (that is, T i =1),yi 1 is bserved and yi 0 is the unbserved cunterfactual utcme it represents what wuld have happened t the individual if assigned t cntrl. Cnversely, fr cntrl units, yi 0 is bserved and yi 1 is cunterfactual. In either case, a simple treatment effect fr unit i can be defined as treatment effect fr unit i = yi 1 yi 0 Figure 9.3 displays hypthetical data fr an eperiment with 100 units (and thus 200 ptential utcmes). The tp panel displays the data we wuld like t be able t see in rder t determine causal effects fr each persn in the dataset that is, it includes bth ptential utcmes fr each persn. 2 The wrd cunterfactual is smetimes used here, but we fllw Rubin (1990) and use the term ptential utcme because sme f these ptential data are actually bserved.

5 THE FUNDAMENTAL PROBLEM OF CAUSAL INFERENCE 171 (Hypthetical) cmplete data: Observed data: Pre-treatment Treatment Ptential Treatment inputs indicatr utcmes effect Unit, i X i T i yi 0 yi 1 yi 1 yi Pre-treatment Treatment Ptential Treatment inputs indicatr utcmes effect Unit, i X i T i yi 0 yi 1 yi 1 yi ?? ?? ? 102? ? 111? ? 114? Figure 9.3 Illustratin f the fundamental prblem f causal inference. Fr each unit, we have bserved sme pre-treatment inputs, and then the treatment (T i = 1) r cntrl (T i =0)is applied. We can then bserve nly ne f the ptential utcmes, (yi 0,yi 1 ). As a result, we cannt bserve the treatment effect, yi 1 yi 0, fr any f the units. The tp table shws what the cmplete data might lk like, if it were pssible t bserve bth ptential utcmes n each unit. Fr each pair, the bserved utcme is displayed in bldface. The bttm table shws what wuld actually be bserved. The s-called fundamental prblem f causal inference is that at mst ne f these tw ptential utcmes, yi 0 and yi 1, can be bserved fr each unit i. The bttm panel f Figure 9.3 displays the data that can actually be bserved. The yi 1 values are missing fr thse in the cntrl grup and the yi 0 values are missing fr thse in the treatment grup. Ways f getting arund the prblem We cannt bserve bth what happens t an individual after taking the treatment (at a particular pint in time) and what happens t that same individual after nt taking the treatment (at the same pint in time). Thus we can never measure a causal effect directly. In essence, then, we can think f causal inference as a predictin f what wuld happen t unit i if T i =0rT i =1.Itisthuspredictive inference in the ptential-utcme framewrk. Viewed this way, estimating causal effects requires ne r sme cmbinatin f the fllwing: clse substitutes fr the ptential utcmes, randmizatin, r statistical adjustment. We discuss the basic strategies here and g int mre detail in the remainder f this chapter and the net.

6 172 CAUSAL INFERENCE USING DIRECT REGRESSION Clse substitutes. One might bject t the frmulatin f the fundamental prblem f causal inference by nting situatins where it appears ne can actually measure bth yi 0 and y1 i n the same unit. Cnsider, fr eample drinking tea ne evening and milk anther evening, and then measuring the amunt f sleep each time. A careful cnsideratin f this eample reveals the implicit assumptin that there are n systematic differences between days that culd als affect sleep. An additinal assumptin is that applying the treatment n ne day has n effect n the utcme n anther day. Mre pristine eamples can generally be fund in the natural and physical sciences. Fr instance, imagine dividing a piece f plastic int tw parts and then epsing each piece t a crrsive chemical. In this case, the hidden assumptin is that pieces are identical in hw they wuld respnd with and withut treatment, that is, y1 0 = y0 2 and y1 1 = y1 2. As a third eample, suppse yu want t measure the effect f a new diet by cmparing yur weight befre the diet and yur weight after. The hidden assumptin here is that the pre-treatment measure can act as a substitute fr the ptential utcme under cntrl, that is, yi 0 = i. It is nt unusual t see studies that attempt t make causal inferences by substituting values in this way. It is imprtant t keep in mind the strng assumptins ften implicit in such strategies. Randmizatin and eperimentatin. A different apprach t causal inference is the statistical idea f using the utcmes bserved n a sample f units t learn abut the distributin f utcmes in the ppulatin. The basic idea is that since we cannt cmpare treatment and cntrl utcmes fr the same units, we try t cmpare them n similar units. Similarity can be attained by using randmizatin t decide which units are assigned t the treatment grup and which units are assigned t the cntrl grup. We will discuss this strategy in depth in the net sectin. Statistical adjustment. Fr a variety f reasns, it is nt always pssible t achieve clse similarity between the treated and cntrl grups in a causal study. In bservatinal studies, units ften end up treated r nt based n characteristics that are predictive f the utcme f interest (fr eample, men enter a jb training prgram because they have lw earnings and future earnings is the utcme f interest). Randmized eperiments, hwever, can be impractical r unethical, and even in this cntet imbalance can arise frm small-sample variatin r frm unwillingness r inability f subjects t fllw the assigned treatment. When treatment and cntrl grups are nt similar, mdeling r ther frms f statistical adjustment can be used t fill in the gap. Fr instance, by fitting a regressin (r mre cmplicated mdel), we may be able t estimate what wuld have happened t the treated units had they received the cntrl, and vice versa. Alternately, ne can attempt t divide the sample int subsets within which the treatment/cntrl allcatin mimics an eperimental allcatin f subjects. We discuss regressin appraches in this chapter. We discuss imbalance and related issues mre thrughly in Chapter 10 alng with a descriptin f ways t help bservatinal studies mimic randmized eperiments. 9.3 Randmized eperiments We begin with the cleanest scenari, an eperiment with units randmly assigned t receive treatment and cntrl, and with the units in the study cnsidered as a randm sample frm a ppulatin f interest. The randm sampling and randm

7 RANDOMIZED EXPERIMENTS 173 treatment assignment allw us t estimate the average causal effect f the treatment in the ppulatin, and regressin mdeling can be used t refine this estimate. Average causal effects and randmized eperiments Althugh we cannt estimate individual-level causal effects (withut making strng assumptins, as discussed previusly), we can design studies t estimate the ppulatin average treatment effect: average treatment effect = avg (yi 1 y0 i ), fr the units i in a larger ppulatin. The cleanest way t estimate the ppulatin average is thrugh a randmized eperiment in which each unit has a psitive chance f receiving each f the pssible treatments. 3 If this is set up crrectly, with treatment assignment either entirely randm r depending nly n recrded data that are apprpriately mdeled, the cefficient fr T in a regressin crrespnds t the causal effect f the treatment, amng the ppulatin represented by the n units in the study. Cnsidered mre bradly, we can think f the cntrl grup as a grup f units that culd just as well have ended up in the treatment grup, they just happened nt t get the treatment. Therefre, n average, their utcmes represent what wuld have happened t the treated units had they nt been treated; similarly, the treatment grup utcmes represent what might have happened t the cntrl grup had they been treated. Therefre the cntrl grup plays an essential rle in a causal analysis. Fr eample, if n 0 units are selected at randm frm the ppulatin and given the cntrl, and n 1 ther units are randmly selected and given the treatment, then the bserved sample averages f y fr the treated and cntrl units can be used t estimate the crrespnding ppulatin quantities, avg(y 0 )andavg(y 1 ), with their difference estimating the average treatment effect (and with standard errr s 2 0 /n 0 + s 2 1 /n 1; see Sectin 2.3). This wrks because the yi 0 s fr the cntrl grup are a randm sample f the values f yi 0 in the entire ppulatin. Similarly, the yi 1 s fr the treatment grup are a randm sample f the y1 i s in the ppulatin. Equivalently, if we select n 0 + n 1 units at randm frm the ppulatin, and then randmly assign n 0 f them t the cntrl and n 1 t the treatment, we can think f each f the sample grups as representing the crrespnding ppulatin f cntrl r treated units. Therefre the cntrl grup mean can act as a cunterfactual fr the treatment grup (and vice versa). What if the n 0 +n 1 units are selected nnrandmly frm the ppulatin but then the treatment is assigned at randm within this sample? This is cmmn practice, fr eample, in eperiments invlving human subjects. Eperiments in medicine, fr instance, are cnducted n vlunteers with specified medical cnditins wh are willing t participate in such a study, and eperiments in psychlgy are ften cnducted n university students taking intrductry psychlgy curses. In this case, causal inferences are still justified, but inferences n lnger generalize t the entire ppulatin. It is usual instead t cnsider the inference t be apprpriate t a hypthetical superppulatin frm which the eperimental subjects were drawn. Further mdeling is needed t generalize t any ther ppulatin. A study 3 Ideally, each unit shuld have a nnzer prbability f receiving each f the treatments, because therwise the apprpriate cunterfactual (ptential) utcme cannt be estimated fr units in the crrespnding subset f the ppulatin. In practice, if the prbabilities are highly unequal, the estimated ppulatin treatment effect will have a high standard errr due t the difficulty f reliably estimating such a rare event.

8 174 CAUSAL INFERENCE USING DIRECT REGRESSION Test scres in cntrl classes Test scres in treated classes Grade 1 mean = 69 sd = 13 mean = 77 sd = Grade 2 mean = 93 sd = 12 mean = 102 sd = Grade 3 mean = 106 sd = 7 mean = 107 sd = Grade 4 mean = 110 sd = 7 mean = 114 sd = Figure 9.4 Pst-treatment test scres frm an eperiment measuring the effect f an educatinal televisin prgram, The Electric Cmpany, n children s reading abilities. The eperiment was applied n a ttal f 192 classrms in fur grades. At the end f the eperiment, the average reading test scre in each classrm was recrded. in which causal inferences are merited fr a specific sample r ppulatin is said t have internal validity, and when thse inferences can be generalized t a brader ppulatin f interest the study is said t have eternal validity. We illustrate with a simple binary treatment (that is, tw treatment levels, r a cmparisn f treatment t cntrl) in an educatinal eperiment. We then briefly discuss mre general categrical, cntinuus, and multivariate treatments. Eample: shwing children an educatinal televisin shw Figure 9.4 summarizes data frm an educatinal eperiment perfrmed arund 1970 n a set f elementary schl classes. The treatment in this eperiment was epsure t a new educatinal televisin shw called The Electric Cmpany. In each f fur grades, the classes were randmized int treated and cntrl grups. At the end f the schl year, students in all the classes were given a reading test, and the average test scre within each class was recrded. Unfrtunately, we d nt have data n individual students, and s ur entire analysis will be at the classrm level. Figure 9.4 displays the distributin f average pst-treatment test scres in the cntrl and treatment grup fr each grade. (The eperimental treatment was applied t classes, nt t schls, and s we treat the average test scre in each class as

9 RANDOMIZED EXPERIMENTS 175 a single measurement.) We break up the data by grade fr cnvenience and because it is reasnable t suppse that the effects f this shw culd vary by grade. Analysis as a cmpletely randmized eperiment. The eperiment was perfrmed in tw cities (Fresn and Yungstwn). Fr each city and grade, the eperimenters selected a small number f schls (10 20) and, within each schl, they selected the tw prest reading classes f that grade. Fr each pair, ne f these classes was randmly assigned t cntinue with its regular reading curse and the ther was assigned t view the TV prgram. This is called a paired cmparisns design (which in turn is a special case f a randmized blck design, with eactly tw units within each blck). Fr simplicity, hwever, we shall analyze the data here as if the treatment assignment had been cmpletely randmized within each grade. In a cmpletely randmized eperiment n n units (in this case, classrms), ne can imagine the units mied tgether in a bag, cmpletely mied, and then separated int tw grups. Fr eample, the units culd be labeled frm 1 t n, and then permuted at randm, with the first n 1 units receiving the treatment and the thers receiving the cntrl. Each unit has the same prbability f being in the treatment grup and these prbabilities are independent f each ther. Again, fr the rest f this chapter we pretend that the Electric Cmpany eperiment was cmpletely randmized within each grade. In Sectin 23.1 we return t the eample and present an analysis apprpriate t the paired design that was actually used. Basic analysis f a cmpletely randmized eperiment When treatments are assigned cmpletely at randm, we can think f the different treatment grups (r the treatment and cntrl grups) as a set f randm samples frm a cmmn ppulatin. The ppulatin average under each treatment, avg(y 0 ) and avg(y 1 ), can then be estimated by the sample average, and the ppulatin average difference between treatment and cntrl, avg(y 1 ) avg(y 0 ) that is, the average causal effect can be estimated by the difference in sample averages, ȳ 1 ȳ 0. Equivalently, the average causal effect f the treatment crrespnds t the cefficient θ in the regressin, y i = α + θt i + errr i. We can easily fit the fur regressins (ne fr each grade) in R: fr (k in 1:4) { display (lm (pst.test ~ treatment, subset=(grade==k))) } The estimates and uncertainty intervals fr the Electric Cmpany eperiment are graphed in the left panel f Figure 9.5. The treatment appears t be generally effective, perhaps mre s in the lw grades, but it is hard t be sure given the large standard errrs f estimatin. Rcde Cntrlling fr pre-treatment predictrs In this study, a pre-test was given in each class at the beginning f the schl year (befre the treatment was applied). In this case, the treatment effect can als be estimated using a regressin mdel: y i = α+θt i +β i +errr i n the pre-treatment predictr. 4 Figure 9.6 illustrates fr the Electric Cmpany eperiment. Fr each 4 We avid the term cnfunding cvariates when describing adjustment in the cntet f a randmized eperiment. Predictrs are included in this cntet t increase precisin. We epect

10 176 CAUSAL INFERENCE USING DIRECT REGRESSION Subppulatin Regressin n treatment indicatr Regressin n treatment indicatr, cntrlling fr pre test Grade 1 Grade 2 Grade 3 Grade 4 Figure 9.5 Estimates, 50%, and 95% intervals fr the effect f the Electric Cmpany televisin shw (see data in Figures 9.4 and 9.6) as estimated in tw ways: first, frm a regressin n treatment alne, and secnd, als cntrlling fr pre-test data. In bth cases, the cefficient fr treatment is the estimated causal effect. Including pre-test data as a predictr increases the precisin f the estimates. Displaying these cefficients and intervals as a graph facilitates cmparisns acrss grades and acrss estimatin strategies (cntrlling fr pre-test r nt). Fr instance, the plt highlights hw cntrlling fr pre-test scres increases precisin and reveals decreasing effects f the prgram fr the higher grades, a pattern that wuld be mre difficult t see in a table f numbers. Sample sizes are apprimately the same in each f the grades. The estimates fr higher grades have lwer standard errrs because the residual standard deviatins f the regressins are lwer in these grades; see Figure 9.6. grade 1 grade 2 grade 3 grade 4 pst test, y i pst test, y i pst test, y i pst test, y i pre test, i pre test, i pre test, i pre test, i Figure 9.6 Pre-test/pst-test data fr the Electric Cmpany eperiment. Treated and cntrl classes are indicated by circles and dts, respectively, and the slid and dtted lines represent parallel regressin lines fit t the treatment and cntrl grups, respectively. The slid lines are slightly higher than the dtted lines, indicating slightly psitive estimated treatment effects. Cmpare t Figure 9.4, which displays nly the pst-test data. grade, the difference between the regressin lines fr the tw grups represents the treatment effect as a functin f pre-test scre. Since we have nt included any interactin in the mdel, this treatment effect is assumed cnstant ver all levels f the pre-test scre. Fr grades 2 4, the pre-test was the same as the pst-test, and s it is n surprise that all the classes imprved whether treated r nt (as can be seen frm the plts). Fr grade 1, the pre-test was a subset f the lnger test, which eplains why the pre-test scres fr grade 1 are s lw. We can als see that the distributin f psttest scres fr each grade is similar t the net grade s pre-test scres, which makes sense. In any case, fr estimating causal effects (as defined in Sectin 9.2) we are interested in the difference between treatment and cntrl cnditins, nt in the simple imprvement frm pre-test t pst-test. The pre-pst imprvement is nt a them t be related t the utcme but nt t the treatment assignment due t the randmizatin. Therefre they are nt cnfunding cvariates.

11 RANDOMIZED EXPERIMENTS 177 causal effect (ecept under the assumptin, unreasnable in this case, that under the cntrl there wuld be n change frm pre-pst change). In the regressin y i = α + θt i + β i + errr i (9.3) the cefficient fr the treatment indicatr still represents the average treatment effect, but cntrlling fr pre-test can imprve the efficiency f the estimate. (Mre generally, the regressin can cntrl fr multiple pre-treatment predictrs, in which case the mdel has the frm y i = α + θt i + X i β + errr i, r alternatively α can be remved frm the equatin and cnsidered as a cnstant term in the linear predictr Xβ.) The estimates fr the Electric Cmpany study appear in the right panel f Figure 9.5. It is nw clear that the treatment is effective, and it appears t be mre effective in the lwer grades. A glance at Figure 9.6 suggests that in the higher grades there is less rm fr imprvement; hence this particular test might nt be the mst effective fr measuring the benefits f The Electric Cmpany in grades 3 and 4. It is nly apprpriate t cntrl fr pre-treatment predictrs, r, mre generally, predictrs that wuld nt be affected by the treatment (such as race r age). This pint will be illustrated mre cncretely in Sectin 9.7. Gain scres An alternative way t specify a mdel that cntrls fr pre-test measures is t use these measures t transfrm the respnse variable. A simple apprach is t subtract the pre-test scre, i, frm the utcme scre, y i, thereby creating a gain scre, g i. Then this scre can be regressed n the treatment indicatr (and ther predictrs if desired), g i = α + θt i + errr i. (In the simple case with n ther predictrs, the regressin estimate is simply ˆθ =ḡ T ḡ C, the average difference f gain scres in the treatment and cntrl grups.) In sme cases the gain scre can be mre easily interpreted than the riginal utcme variable y. Using gain scres is mst effective if the pre-treatment scre is cmparable t the pst-treatment measure. Fr instance, in ur Electric Cmpany eample it wuld nt make sense t create gain scres fr the classes in grade 1 since their pre-test measure was based n nly a subset f the full test. One perspective n this mdel is that it makes an unnecessary assumptin, namely, that β = 1 in mdel (9.3). On the ther hand, if this assumptin is clse t being true then θ may be estimated mre precisely. One way t reslve this cncern abut misspecificatin wuld simply be t include the pre-test scre as a predictr as well, g i = α + θt i + γ i + errr i. Hwever, in this case, ˆθ, the estimate f the cefficient fr T, is equivalent t the estimated cefficient frm the riginal mdel, y i = α + θt i + β i + errr i (see Eercise 9.7). Mre than tw treatment levels, cntinuus treatments, and multiple treatment factrs Ging beynd a simple treatment-and-cntrl setting, multiple treatment effects can be defined relative t a baseline level. With randm assignment, this simply fllws general principles f regressin mdeling. If treatment levels are numerical, the treatment level can be cnsidered as a cntinuus input variable. T cnceptualize randmizatin with a cntinuus treatment variable, think f chsing a randm number that falls anywhere in the cntinuus range. As with regressin inputs in general, it can make sense t fit mre cmpli-

12 178 CAUSAL INFERENCE USING DIRECT REGRESSION grade 1 grade 2 grade 3 grade 4 pst test, y i pre test, i pst test, y i pre test, i pst test, y i pre test, i pst test, y i pre test, i Figure 9.7 Pre-test/pst-test data fr the Electric Cmpany eperiment. Treated and cntrl classes are indicated by circles and dts, respectively, and the slid and dtted lines represent separate regressin lines fit t the treatment and cntrl grups, respectively. Fr each grade, the difference between the slid and dtted lines represents the estimated treatment effect as a functin f pre-test scre. cated mdels if suggested by thery r supprted by data. A linear mdel which estimates the average effect n y fr each additinal unit f T is a natural starting pint, thugh it may need t be refined. With several discrete treatments that are unrdered (such as in a cmparisn f three different srts f psychtherapy), we can mve t multilevel mdeling, with the grup inde indicating the treatment assigned t each unit, and a secnd-level mdel n the grup cefficients, r treatment effects. We shall illustrate such mdeling in Sectin 13.5 with an eperiment frm psychlgy. We shall fcus mre n multilevel mdeling as a tl fr fitting data, but since the treatments in that eample are randmly assigned, their cefficients can be interpreted as causal effects. Additinally, different cmbinatins f multiple treatments can be administered randmly. Fr instance, depressed individuals culd be randmly assigned t receive nthing, drugs, cunseling sessins, r a cmbinatin f drugs and cunseling sessins. These cmbinatins culd be mdeled as tw treatments and their interactin r as fur distinct treatments. The assumptin f n interference between units Our discussin s far regarding estimatin f causal effects using eperiments is cntingent upn anther, ften verlked, assumptin. We must assume als that the treatment assignment fr ne individual (unit) in the eperiment des nt affect the utcme fr anther. This has been incrprated int the stable unit treatment value assumptin (SUTVA). Otherwise, we wuld need t define a different ptential utcme fr the i th unit nt just fr each treatment received by that unit but fr each cmbinatin f treatment assignments received by every ther unit in the eperiment. This wuld enrmusly cmplicate even the definitin, let alne the estimatin, f individual causal effects. In settings such as agricultural eperiments where interference between units is t be epected, it can be mdeled directly, typically using spatial interactins. 9.4 Treatment interactins and pststratificatin Interactins f treatment effect with pre-treatment inputs Once we include pre-test in the mdel, it is natural t allw it t interact with treatment effect. The treatment is then allwed t affect bth the intercept and the slpe f the pre-test/pst-test regressin. Figure 9.7 shws the Electric Cmpany

13 TREATMENT INTERACTIONS AND POSTSTRATIFICATION 179 data with separate regressin lines estimated fr the treatment and cntrl grups. As with Figure 9.6, fr each grade the difference between the regressin lines is the estimated treatment effect as a functin f pre-test scre. We illustrate in detail fr grade 4. First, we fit the simple mdel including nly the treatment indicatr: lm(frmula = pst.test ~ treatment, subset=(grade==4)) cef.est cef.se (Intercept) treatment n = 42, k = 2 residual sd = 6.0, R-Squared = 0.09 The estimated treatment effect is 3.7 with a standard errr f 1.8. We can imprve the efficiency f the estimatr by cntrlling fr the pre-test scre: lm(frmula = pst.test ~ treatment + pre.test, subset=(grade==4)) cef.est cef.se (Intercept) treatment pre.test n = 42, k = 3 residual sd = 2.2, R-Squared = 0.88 The new estimated treatment effect is 1.7 with a standard errr f 0.7. In this case, cntrlling fr the pre-test reduced the estimated effect. Under a clean randmizatin, cntrlling fr pre-treatment predictrs in this way shuld reduce the standard errrs f the estimates. 5 (Figure 9.5 shws the estimates fr the Electric Cmpany eperiment in all fur grades.) Cmplicated arise when we include the interactin f treatment with pre-test: lm(frmula = pst.test ~ treatment + pre.test + treatment:pre.test, subset=(grade==4)) cef.est cef.se (Intercept) treatment pre.test treatment:pre.test n = 42, k = 4 residual sd = 2.1, R-Squared = 0.89 The estimated treatment effect is nw , which is difficult t interpret withut knwing the range f. Frm Figure 9.7 we see that pre-test scres range frm apprimately 80 t 120; in this range, the estimated treatment effect varies frm = 5 fr classes with pre-test scres f 80 t = 1 fr classes with pre-test scres f 120. This range represents the variatin in estimated treatment effects as a functin f pre-test scre, nt uncertainty in the estimated treatment effect. T get a sense f the uncertainty, we can plt the estimated treatment effect as a functin f, verlaying randm simulatin draws t represent uncertainty: R utput R utput R utput 5 Under a clean randmizatin, cntrlling fr pre-treatment predictrs in this way des nt change what we are estimating. If the randmizatin was less than pristine, hwever, the additin f predictrs t the equatin may help us cntrl fr unbalanced characteristics acrss grups. Thus, this strategy has the ptential t mve us frm estimating a nncausal estimand (due t lack f randmizatin) t estimating a causal estimand by in essence cleaning the randmizatin.

14 180 CAUSAL INFERENCE USING DIRECT REGRESSION treatment effect in grade 4 treatment effect pre test Figure 9.8 Estimate and uncertainty fr the effect f viewing The Electric Cmpany (cmpared t the cntrl treatment) fr furth-graders. Cmpare t the data in the rightmst plt in Figure 9.7. The dark line here the estimated treatment effect as a functin f pretest scre is the difference between the tw regressin lines in the grade 4 plt in Figure 9.7. The gray lines represent 20 randm draws frm the uncertainty distributin f the treatment effect. Rcde Rcde Rcde lm.4 <- lm (pst.test ~ treatment + pre.test + treatment:pre.test, subset=(grade==4)) lm.4.sim <- sim (lm.4) plt (0, 0, lim=range (pre.test[grade==4]), ylim=c(-5,10), lab="pre-test", ylab="treatment effect", main="treatment effect in grade 4") abline (0, 0, lwd=.5, lty=2) fr (i in 1:20){ curve (lm.4.sim$beta[i,2] + lm.4.sim$beta[i,4]*, lwd=.5, cl="gray", add=true)} curve (cef(lm.4)[2] + cef(lm.4)[4]*, lwd=.5, add=true) This prduces the graph shwn in Figure 9.8. Finally, we can estimate a mean treatment effect by averaging ver the values f in the data. If we write the regressin mdel as y i = α+θ 1 T i +β i +θ 2 T i i +errr i, then the treatment effect is θ 1 +θ 2, and the summary treatment effect in the sample is 1 n n i=1 (θ 1 + θ 2 i ), averaging ver the n furth-grade classrms in the data. We can cmpute the average treatment effect as fllws: n.sims <- nrw(lm.4.sim$beta) effect <- array (NA, c(n.sims, sum(grade==4))) fr (i in 1:n.sims){ effect[i,] <- lm.4.sim$beta[i,2] + lm.4.sim$beta[i,4]*pre.test[grade==4] } avg.effect <- rwmeans (effect) The rwmeans() functin averages ver the grade 4 classrms, and the result f this cmputatin, avg.effect, is a vectr f length n.sims representing the uncertainty in the average treatment effect. We can summarize with the mean and standard errr: print (c (mean(avg.effect), sd(avg.effect))) The result is 1.8 with a standard deviatin f 0.7 quite similar t the result frm the mdel cntrlling fr pre-test but with n interactins. In general, fr a linear regressin mdel, the estimate btained by including the interactin, and then averaging ver the data, reduces t the estimate with n interactin. The mtivatin fr including the interactin is thus t get a better idea f hw the treatment effect varies with pre-treatment predictrs, nt simply t estimate an average effect.

15 OBSERVATIONAL STUDIES 181 Pststratificatin We have discussed hw treatment effects interact with pre-treatment predictrs (that is, regressin inputs). T estimate an average treatment effect, we can pststratify that is, average ver the ppulatin. 6 Fr eample, suppse we have treatment variable T and pre-treatment cntrl variables 1, 2, and ur regressin predictrs are 1, 2,T,and the interactins 1 T and 2 T, s that the linear mdel is: y = β 0 +β 1 1 +β 2 2 +β 3 T +β 4 1 T +β 5 2 T + errr. The estimated treatment effect is then β 3 + β β 5 2, and its average, in a linear regressin, is simply β 3 + β 4 μ 1 + β 5 μ 2,whereμ 1 and μ 2 are the averages f 1 and 2 in the ppulatin. These ppulatin averages might be available frm anther surce, r else they can be estimated using the averages f 1 and 2 in the data at hand. Standard errrs fr summaries such as β 3 + β 4 μ 1 + β 5 μ 2 can be determined analytically, but it is easier t simply cmpute them using simulatins. Mdeling interactins is imprtant when we care abut differences in the treatment effect fr different grups, and pststratificatin then arises naturally if a ppulatin average estimate is f interest. 9.5 Observatinal studies In thery, the simplest slutin t the fundamental prblem f causal inference is, as we have described, t randmly sample a different set f units fr each treatment grup assignment frm a cmmn ppulatin, and then apply the apprpriate treatments t each grup. An equivalent apprach is t randmly assign the treatment cnditins amng a selected set f units. Either f these appraches ensures that, n average, the different treatment grups are balanced r, t put it anther way, that the ȳ 0 and ȳ 1 frm the sample are estimating the average utcmes under cntrl and treatment fr the same ppulatin. In practice, hwever, we ften wrk with bservatinal data because, cmpared t eperiments, bservatinal studies can be mre practical t cnduct and can have mre realism with regard t hw the prgram r treatment is likely t be administered in practice. As we have discussed, hwever, in bservatinal studies treatments are bserved rather than assigned (fr eample, cmparisns f smkers t nnsmkers), and it is nt at all reasnable t cnsider the bserved data under different treatments as randm samples frm a cmmn ppulatin. In an bservatinal study, there can be systematic differences between grups f units that receive different treatments differences that are utside the cntrl f the eperimenter and they can affect the utcme, y. In this case we need t rely n mre data than just treatments and utcmes and implement a mre cmplicated analysis strategy that will rely upn strnger assumptins. The strategy discussed in this chapter, hwever, is relatively simple and relies n cntrlling fr cnfunding cvariates thrugh linear regressin. Sme alternative appraches are described in Chapter In survey sampling, stratificatin refers t the prcedure f dividing the ppulatin int disjint subsets (strata), sampling separately within each stratum, and then cmbining the stratum samples t get a ppulatin estimate. Pststratificatin is the analysis f an unstratified sample, breaking the data int strata and reweighting as wuld have been dne had the survey actually been stratified. Stratificatin can adjust fr ptential differences between sample and ppulatin using the survey design; pststratificatin makes such adjustments in the data analysis.

16 182 CAUSAL INFERENCE USING DIRECT REGRESSION Subppulatin Estimated effect f supplement, cmpared t replacement Grade 1 Grade 2 Grade 3 Grade 4 Figure 9.9 Estimates, 50%, and 95% intervals fr the effect f The Electric Cmpany as a supplement rather than a replacement, as estimated by a regressin n the supplement/replacement indicatr als cntrlling fr pre-test data. Fr each grade, the regressin is perfrmed nly n the treated classes; this is an bservatinal study embedded in an eperiment. Electric Cmpany eample Here we illustrate an bservatinal study fr which a simple regressin analysis, cntrlling fr pre-treatment infrmatin, may yield reasnable causal inferences. The educatinal eperiment described in Sectin 9.3 actually had an embedded bservatinal study. Once the treatments had been assigned, the teacher fr each class assigned t the Electric Cmpany treatment chse t either replace r supplement the regular reading prgram with the Electric Cmpany televisin shw. That is, all the classes in the treatment grup watched the shw, but sme watched it instead f the regular reading prgram and thers gt it in additin. 7 The simplest starting pint t analyzing these bservatinal data (nw limited t the randmized treatment grup) is t cnsider the chice between the tw treatment ptins replace r supplement t be randmly assigned cnditinal n pre-test scres. This is a strng assumptin but we use it simply as a starting pint. We can then estimate the treatment effect by regressin, as with an actual eperiment. In the R cde, we create a variable called supp that equals 0 fr the replacement frm f the treatment, 1 fr the supplement, and NA fr the cntrls. We then estimate the effect f the supplement, as cmpared t the replacement, fr each grade: Rcde fr (k in 1:4) { k <- (grade==k) & (!is.na(supp)) lm.supp <- lm (pst.test ~ supp + pre.test, subset=k) } The estimates are graphed in Figure 9.9. The uncertainties are high enugh that the cmparisn is incnclusive ecept in grade 2, but n the whle the pattern is cnsistent with the reasnable hypthesis that supplementing is mre effective than replacing in the lwer grades. Assumptin f ignrable treatment assignment As ppsed t making the same assumptin as the cmpletely randmized eperiment, the key assumptin underlying the estimate is that, cnditinal n the cnfunding cvariates used in the analysis (here as inputs in the regressin analysis), the distributin f units acrss treatment cnditins is, in essence, randm 7 This prcedural detail reveals that the treatment effect fr the randmized eperiment is actually mre cmplicated than described earlier. As implemented, the eperiment estimated the effect f making the prgram available, either as a supplement r replacement fr the current curriculum.

17 OBSERVATIONAL STUDIES 183 (in this case, pre-test scre) with respect t the ptential utcmes. T help with the intuitin here, ne culd envisin units being randmly assigned t treatment cnditins cnditinal n the cnfunding cvariates; hwever, f curse, n actual randmized assigment need take place. Ignrability is ften frmalized by the cnditinal independence statement, y 0,y 1 T X. This says that the distributin f the ptential utcmes, (y 0,y 1 ), is the same acrss levels f the treatment variable, T, nce we cnditin n cnfunding cvariates X. This assumptin is referred t as ignrability f the treatment assignment in the statistics literature and selectin n bservables in ecnmetrics. Said anther way, we wuld nt necessarily epect any tw classes t have had the same prbability f receiving the supplemental versin f the treatment. Hwever, we epect any tw classes at the same levels f the cnfunding cvariates (that is, pre-treatment variables; in ur eample, average pre-test scre) t have had the same prbability f receiving the supplemental versin f the treatment. A third way t think abut the ignrability assumptin is that it requires that we cntrl fr all cnfunding cvariates, the pre-treatment variables that are assciated with bth the treatment and the utcme. If ignrability hlds, then causal inferences can be made withut mdeling the treatment assignment prcess that is, we can ignre this aspect f the mdel as lng as analyses regarding the causal effects cnditin n the predictrs needed t satisfy ignrability. Randmized eperiments represent a simple case f ignrability. Cmpletely randmized eperiments need nt cnditin n any pre-treatment variables this is why we can use a simple difference in means t estimate causal effects. Randmized eperiments that blck r match satisfy ignrability cnditinal n the design variables used t blck r match, and therefre these variables need t be included when estimating causal effects. In the Electric Cmpany supplement/replacement eample, an eample f a nnignrable assignment mechanism wuld be if the teacher f each class chse the treatment that he r she believed wuld be mre effective fr that particular class based n unmeasured characteristics f the class that were related t their subsequent test scres. Anther nnignrable assignment mechanism wuld be if, fr eample, supplementing was mre likely t be chsen by mre mtivated teachers, with teacher mtivatin als assciated with the students future test scres. Fr ignrability t hld, it is nt necessary that the tw treatments be equally likely t be picked, but rather that the prbability that a given treatment is picked shuld be equal, cnditinal n ur cnfunding cvariates. 8 In an eperiment, ne can cntrl this at the design stage by using a randm assignment mechanism. In an bservatinal study, the treatment assignment is nt under the cntrl f the statistician, but ne can aim fr ignrability by cnditining in the analysis stage n as much pre-treatment infrmatin in the regressin mdel as pssible. Fr eample, if teachers mtivatin might affect treatment assignment, it wuld be advisable t have a pre-treatment measure f teacher mtivatin and include this as an input in the regressin mdel. This wuld increase the plausibility f the ignrability assumptin. Realistically, this may be a difficult characteristic t 8 As further clarificatin, cnsider tw participants f a study fr which ignrability hlds. If we define the prbability f treatment participatin as Pr(T =1 X), then this prbability must be equal fr these tw individuals. Hwever, suppse there eists anther variable, w, that is assciated with treatment participatin (cnditinal n X) but nt with the utcme (cnditinal n X). We d nt require that Pr(T =1 X, W ) be the same fr these tw participants.

18 184 CAUSAL INFERENCE USING DIRECT REGRESSION Pst test scre Pst test scre Pre test scre Pre test scre Figure 9.10 Hypthetical befre/after data demnstrating the ptential prblems in using linear regressin fr causal inference. The dark dts and line crrespnd t the children wh received the educatinal supplement; the lighter dts and line crrespnd t the children wh did nt receive the supplement. The dashed lines are regressin lines fit t the bserved data. The mdel shwn in the right panel allws fr an interactin between receiving the supplement and pre-test scres. measure, but ther teacher characteristics such as years f eperience and schling might act as partial pries. In general, ne can never prve that the treatment assignment prcess in an bservatinal study is ignrable it is always pssible that the chice f treatment depends n relevant infrmatin that has nt been recrded. In an educatinal study this infrmatin culd be characteristics f the teacher r schl that are related bth t treatment assignment and t pst-treatment test scres. Thus, if we interpret the estimates in Figure 9.9 as causal effects, we d s with the understanding that we wuld prefer t have further pre-treatment infrmatin, especially n the teachers, in rder t be mre cnfident in ignrability. If we believe that treatment assignments depend n infrmatin nt included in the mdel, then we shuld chse a different analysis strategy. We discuss sme ptins at the end f the net chapter. Judging the reasnableness f regressin as a mdeling apprach, assuming ignrability Even if the ignrability assumptin appears t be justified, this des nt mean that simple regressin f ur utcmes n cnfunding cvariates and a treatment indicatr is necessarily the best mdeling apprach fr estimating treatment effects. There are tw primary cncerns related t the distributins f the cnfunding cvariates acrss the treatment grups: lack f cmplete verlap and lack f balance. Fr instance, cnsider ur initial hypthetical eample f a medical treatment that is suppsed t affect subsequent health measures. What if there were n treatment bservatins amng the grup f peple whse pre-treatment health status was highest? Arguably, we culd nt make any causal inferences abut the effect f the treatment n these peple because we wuld have n empirical evidence regarding the cunterfactual state. Lack f verlap and balance frces strnger reliance n ur mdeling than if cvariate distributins were the same acrss treatment grups. We prvide a brief illustratin in this chapter and discuss in greater depth in Chapter 10. Suppse we are interested in the effect f a supplementary educatinal activity (such as viewing The Electric Cmpany) that was nt randmly assigned. Suppse, hwever, that nly ne predictr, pre-test scre, is necessary t satisfy ignrability that is, there is nly ne cnfunding cvariate. Suppse further, thugh, that thse individuals wh participate in the supplementary activity tend t have higher pre-