TECHNICAL WORKING PAPER SERIES INFERENCE WITH DIFFERENCE IN DIFFERENCES WITH A SMALL NUMBER OF POLICY CHANGES. Timothy Conley Christopher Taber

Transcription

1 TECHNICAL WORKING PAPER SERIES INFERENCE WITH DIFFERENCE IN DIFFERENCES WITH A SMALL NUMBER OF POLICY CHANGES Tmothy Conley Chrstoher Taber Techncal Workng Paer 32 htt:// NATIONAL BUREAU OF ECONOMIC RESEARCH 050 Massachusetts Avenue Cambrdge, MA 0238 June 2005 We thank Federco Band, Alan Bester, Phl Cross, Chrs Hansen, Rosa Matzkn, Bruce Meyer, and Jeff Russell for helful comments and Aroo Chaterjee and Nathan Hedren for research assstantsh. All errors are our own. Conley gratefully acknowledges fnancal suort from the NSF (SES ) and from the IBM Cororaton Faculty Research Fund at the Unversty of Chcago Graduate School of Busness. Taber gratefully acknowledges fnancal suort from the NSF (SES ). The vews exressed heren are those of the author(s) and do not necessarly reflect the vews of the Natonal Bureau of Economc Research by Tmothy Conley and Chrstoher Taber. All rghts reserved. Short sectons of text, not to exceed two aragrahs, may be quoted wthout exlct ermsson rovded that full credt, ncludng notce, s gven to the source.

2 Inference wth Dfference n Dfferences wth a Small Number of Polcy Changes Tmothy Conley and Chrstoher Taber NBER Techncal Workng Paer No. 32 July 2005 JEL No. ABSTRACT Dfference n dfferences methods have become very oular n aled work. Ths aer rovdes a new method for nference n these models when there are a small number of olcy changes. Ths stuaton occurs n many mlementatons of these estmators. Identfcaton of the key arameter tycally arses when a grou changes some artcular olcy. The asymtotc aroxmatons that are tycally emloyed assume that the number of cross sectonal grous, N, tmes the number of tme erods, T, s large. However, even when N or T s large, the number of actual olcy changes observed n the data s often very small. In ths case, we argue that ont estmators of treatment effects should not be thought of as beng consstent and that the standard methods that researchers use to erform nference n these models are not arorate. We develo an alternatve aroach to nference under the assumton that there are a fnte number of olcy changes n the data, usng asymtotc aroxmatons as the number of non-changng grous gets large. In ths stuaton we cannot obtan a consstent ont estmator for the key treatment effect arameter. However, we can consstently estmate the fnte-samle dstrbuton of the treatment effect estmator, u to the unknown arameter tself. Ths allows us to erform hyothess tests and construct confdence ntervals. For exostonal and motvatonal uroses, we focus on the dfference n dfferences case, but our aroach should be arorate more generally n treatment effect models whch emloy a large number of controls, but a small number of treatments. We demonstrate the use of the aroach by analyzng the effect of college mert ade rograms on college attendance. We show that n some cases the standard aroach can gve msleadng results. Tmothy Conley Unversty of Chcago GSB 0 E 58 th Street Chcago, IL tconley@gsb.uchcago.edu Chrstoher Taber Deartment of Economcs Nortwestern Unversty 2003 Sherdan Road Evanston, IL and NBER ctaber@northwestern.edu

3 Introducton Dfference n dfferences methods have become very oular n aled work. These models are tycally qute easy to mlement and to nterret. However, erformng nference wth these models s often dffcult. The goal of ths aer s to address one artcular asect that s lkely to be very mortant n many mlementatons of these estmators. Identfcaton of the key arameter often arses when a grou changes some artcular olcy. We use the notaton N 0 to refer to the number of treatment grous that change ther olcy n the data and N to refer to the number of control grous who do not change ther olcy. The asymtotc aroxmatons that are tycally emloyed assume that the number of both grous, N 0 and N, are large. However, even when the total number of grous s large, the number of actual olcy changes observed n the data s often very small. In ths case, we argue that ont estmators of treatment effects should not be thought of as beng consstent and that the standard methods that researchers use to erform nference n these models are not arorate. We develo an alternatve aroach to nference under the assumton that N 0 s fnte, usng asymtotc aroxmatons that let N grow large. Whle our ont estmator of the treatment effect arameter s not consstent, we can consstently estmate ts fnte-samle dstrbuton u to the true value of the arameter tself. Ths allows us to test thehyothessthatthsarametertakesonanygvenvalueandtoconstructaconfdence nterval for t by nvertng a test statstc. For exostonal and motvatonal uroses, we focus on the dfference n dfferences case, but our aroach s arorate more generally n treatment effect models n whch there are a large number of controls, but a small number of treatments. Our aroach s related to a large body of exstng work on dfference and dfference models and nference n more general grou effect models. It s comlementary to tycal aroaches focusng on stuatons where the number of treatment and control grous, N 0 and N, are both large (e.g. Moulton, 990) or both small (e.g. Donald and Lang, 2002). Our aroach s n the srt of comarsons of changes n treatment grous to control grous often done by careful aled researchers. Anderson and Meyer (2000) rovde a nce examle of the tye of queston for whch our methodology s artcularly well suted. They examne the effect of changes n unemloyment nsurance ayroll n Washngton state See for examle Angrst and Krueger (999) and Meyer (995) for overvews of dfference n dfference methods. Wooldrge (2003) rovdes a concse survey of grou effect models.

4 onanumberofoutcomesusngadfference n dfferences aroach wth all other states reresentng the control grous. In addton to standard analyss, they comare the change n the olcy n Washngton state to the dstrbuton of changes across other states durng the same erod n tme n order determne whether t s an outler consstent wth a olcy effect. Ths alcaton of exact nference s very much n the srt of our aroach. Our aroach can also be thought of as a generalzaton/formalzaton of other exact nference tye rocedures lke the lacebo laws exerments that Bertrand, Duflo, and Mullanathan (2004) use to obtan crtcal values for hyotheses testng under a artcular null hyothess about the dstrbuton of the treatment ndcator. 2 There are so many examles of dfference-n-dfferences-style emrcal work that we do not attemt to survey them. Bertrand, Duflo, and Mullanathan (2004) rovde a nce overvew. However, we wll menton a few examles for whch our aroach seems arorate. As mentoned above, Anderson and Meyer (2000) look at changes n Washngton state usng other states as controls. Another examle s the effects of mert ad rograms on college attendance. For examle, n some of her secfcatons Dynarsk (2004) dentfes the effect usng a olcy change from a sngle state (Georga). Fnally, Gruber, Levne, and Stager (999) use comarsons between the fve treatment states that legalzed aborton ror to Roe v. Wade versus the remanng states. One can also fnd many studes whch use a small number of both treatments and controls. However, f there exst grou tme effects, the usual aroach for nference s narorate. An alternatve samle desgn s to collect many control grous. One could then use our methods for arorate nference. For examle Card and Krueger (994) examne the mact of the New Jersey mnmum wage law change on emloyment n the fast food ndustry. Ther samle desgn ncludes only one control grou (eastern Pennsylvana), but they could have collected data from many control states to contrast wth the avalable treatment state. Another famous examle s Card (990) who examnes the effect of the Marel Boatlft on 2 Bertrand, Duflo, and Mullanathan (2004) concern themselves rmarly wth seral correlaton and mostly use a standard asymtotc aroach, but at one ont also dscuss an exact test usng a lacebo laws exerment. The lacebo laws exerment of Bertrand et. al. recovers the exact dstrbuton of a treatment effect arameter (condtonal on state and tme fxed effects) for grou-tme aggregates under a artcular null hyothess. Our thought exerment s somewhat dfferent as we use the control grous to obtan a consstent estmate of the dstrbuton of a treatment effect arameter, whch s then used to conduct small samle nference for the treatment grou. Our setu allows for a rcher set of models n terms of regressors and unobservable structure; secal cases of our setu wll result n nference analogous to that obtaned va the Bertrand et. al. smulaton. 2

5 the Mam labor market. He uses four comarson ctes as controls, but could have used many addtonal ctes. The closest analog to our aroach to nference n econometrcs s work on testng for structural breaks. In artcular, work on testng for end-of-samle stablty/structural breaks such as that by, e.g., Dufor, Ghysels, and Hall (994) and Andrews (2003) s qute related to our basc aroach. These authors consder the roblem of testng for a structural break over a fxed and erhas very short nterval at the end of a samle, analogous to our N 0 observatons on olcy changers. They develo tests that are asymtotcally vald as the number of observatons before the otental break ont grows, holdng fxed the number of onts after the break ont. Ths s analogous to our takng large N lmts wth fxed N 0. Asymtotcally vald crtcal values for these tests rely on usng the tme san before the otental break to get consstent estmates of the dstrbuton of a test statstc formed from data durng the fxed end-of-samle nterval. Andrews accomlshes ths va a rocedure akn to subsamlng and Dufor, Ghysels, and Hall (994) use sem-nonarametrc densty estmators. Agan, our method for constructng nterval estmates s roughly analogous n that we use consstent model estmates obtaned from the N non-changers to characterze the small-samle dstrbuton of the treatment arameter. Basc Model and Problem We consder a case n whch we have reeated cross secton data 3 from dfferent grous (e.g. U.S. states) and tme erods. To gve the man ntuton for the result consder a smle verson of the model wth an ndvdual, wth outcome Y who s n grou j(), and observed at tme t(). We model hs outcome as Y αd j()t() + θ j() + γ t() + η j()t() + ε () where d jt s the olcy varable of nterest. 4 The arameter θ j s a fxed effect for grou j,..., N 0 + N that wll be common to grou j across tme, γ t s a tme effectthats common across all grous but vares across tme t,..., T, η jt s a grou tme random effect that vares across grous and tme, and ε s an ndvdual secfc error term. We assume that ε s..d. wth E(ε )0and that t s ndeendent of all other terms n the 3 Extenson of these results to anel data s straght forward. We assume throughout that we are usng cross sectonal data to economze already comlcated notaton. 4 We focus on lnear models, but extensons to nonlnear models seem feasble combnng the aroach here wth Athey and Imbens (2002). 3

6 model. Let M(j, t) be the set of ndvduals observed n grou j at tme t and M(j, t) denote the number of ndvduals n ths set. We assume throughout ths aer that T s fxed. The rmary goal s to estmate the treatment arameter α. Intal work usng ths model gnored η jt whch leads to the classc dfference n dfferences estmator. In ths case one can obtan a consstent estmate of α usng only two grous and two tme erods. In artcular assume that η jt 0 for all j, t (2) and denote the two grous j {0, } and two tme erods t {0, }. Suose further that the olcy varable s bnary, and for grou 0, there s no change n the treatment (d 00 d 0 0), but for grou the treatment s enacted between the erods zero and one (d 0 0,d ). We defne the notaton Y jt and ε jt to denote the averages of Y and ε P across all the ndvduals n grou j at tme t, (.e. Y jt M(j,t) M(j,t) Y ). The classc dfference n dfferences estmator s: bα DD Y Y 0 Y 0 Y 00 (α + θ + γ θ γ 0 ) (θ 0 + γ θ 0 γ 0 )+(ε ε 0 ) (ε 0 ε 00 ) (α + γ γ 0 ) (γ γ 0 )+(ε ε 0 ) (ε 0 ε 00 ) α +(ε ε 0 ) (ε 0 ε 00 ) α. Thegrouandtmeeffects of course dro out due to the dfferencng, wth large samles wthn each grou/tme the ε terms vansh, and f (2) holds bα DD s a consstent estmator of α as M(j, t) gets large for each grou/erod. In the ast decade or so, researchers have recognzed that (2) s an extremely strong assumton and they have tred to account for η effects n estmaton (see e.g. Moulton, 990). It s easy to show that two grou/two tme erod dfferences n dfference s not consstent wthout assumng (2). In that case bα DD α +(η η 0 ) (η 0 η 00 )+ (ε ε 0 ) (ε 0 ε 00 ) α +(η η 0 ) (η 0 η 00 ). The term nvolvng (η η 0 ) (η 0 η 00 ) does not vansh as the number of observed ndvduals at each grou/tme erod ncreases. Our focus s on analogs of ths stuaton 4

7 where a fxed number of grous wth olcy changes mly that the randomness due to η cannot be elmnated by cross-grou averagng. 5 Many emrcal economsts recognze ths roblem and augment ther natural exerment by collectng data from addtonal grous that do not exerence treatment changes and/or addtonal tme erods. For smlcty, assume that only the frst grou exerences a treatment change after erod t, so the bnary treatment ndcator for grou one can be wrtten as: d t (t>t ) (where ( ) s the ndcator functon) and for all other grous d jt d jτ for all t and τ. Also to kee the exoston smle, assume that all cell szes are the same ( M(j, t) m). Note that d jt for control grou j could be all zeros or all ones. Consder estmatng the model () by usng fxed effects regresson, controllng for grou and tme effects through dummy varables. Let bα FE be the regresson estmate of α. It s straght forward to show that ths canbewrttenasadfference of dfferences " # bα FE α + (η T t t + ε t ) t (η t t + ε t ) (3) tt + t à t! ηjt + ε (N ) (T t jt ηjt + ε ) (N ) t jt j2 tt + The terms nvolvng ε jt wll all vansh as wthn-grou samle szes grow (.e. m ). If E(η jt d jt )0then ths yelds an unbased estmate of α. However, bα FE s not consstent as ³ P the number of grous grows snce the term n brackets aroaches T T t tt + η P t t t t η t as ether m or N get large. Ths roblem s rarely acknowledged n emrcal work and researchers often gnore t when calculatng standard errors. In ractce, f the error terms are truly normally dstrbuted, standard methods wll yeld the correct standard errors (f degree of freedom correctons are used, see Donald and Lang, 200). However, f the dstrbuton of η jt s suffcently dfferent from normal, the standard errors may be very msleadng. The examle resented n equaton (3) consdered the case of a sngle treatment grou. Clearly the same roblem holds when the number of treatment grous s small. 6 The goal of 5 Of course wth access to many grous that exerence a olcy change, averagng across grous can yeld a consstent estmator of α under sutable assumtons about η. 6 Clearly the recse samle sze that consttutes small s an emrcal queston that s beyond the scoe of ths aer. 5 j2 t

8 ths aer s to show that even though one can not obtan consstent estmates of α n these cases, t s stll ossble to erform nference. We assume that there are a fnte number of olcy changes n the data N 0, but aroxmate the dstrbuton of our estmator of α takng lmts as the number of control grous (N ) gets large. The remander of ths aer s organzed nto four sectons. In Secton 2, we resent regresson models for both grou and ndvdual-level data. In each case we show how to erform nference about the arameter α. Extensons to lmted deendent varables are dscussed n Secton 3. Secton 4 of the aer rovdes an llustratve examle alcaton estmatng the effect of mert ad rograms uon college attendance. Fnally, Secton 5 offers bref conclusons. 2 Models Ths secton resents two models. In the frst, we assume that we have one observaton er grou tme cell (e.g. data that s collected at the state year level). In the second, we allow multle observatons er grou tme. For the second model we focus on aroxmatons n whch the number of ndvduals n a grou tme cell remans fxed, sutable for alcatons where at least some of the grous are small. 2. Model We start by dscussng the analog of equaton () defned at the grou tme level and allowng for regressors. We assume that Y jt αd jt + jtβ 0 + θ j + γ t + η jt. (4) Note that we no longer restrct d jt to be bnary. The crucal assumton for dfference n dfferences s that changes n η jt are unrelated to moston of the treatment. In order to erform nference n our case, we also assume that (η j,..., η jt ) s ndeendent and dentcally dstrbuted across grous. Wthn a grou, we allow arbtrary correlaton over tme. Assumton. j,η j,..., jt,η jt s ndeendent and dentcally dstrbuted across unts; η j,..., η jt s ndeendent of (dj,..., d jt ) and ( j,..., jt ) and has a bounded densty and bounded suort; and all random varables have fnte second moments. 6

9 The key roblem motvatng our aroach s that for many grous there s lttle varaton n d jt. Followng the notaton n the ntroducton, defne N 0 as the number of grous for whch d jt changes durng the samle erod and let N reresent the number of remanng grous. We wll refer to the N 0 changers as treatment grous and the remanng non-changng grous as controls. Wthout loss of generalty, defne the ndex j so that the j,..., N 0 reresents the observatons for whch d jt changes at some tme t and j N 0 +,..., N 0 + N reresents the observatons for whch d jt s unchanged for the whole samle. Thus f j>n 0 then for any t,..., T, d jt d j. We treat N 0 and T as fxed, takng lmts as N grows large. We are assumng throughout that at least one grou changes ts olcy so that N 0. For any random varable Z jt, defne Z j T Z t t Z jt N + N 0 Z T N + N 0 +N 0 j t Z jt +N 0 j fz jt Z jt Z j Z t + Z Z jt The essence of dfference n dfferences s that we can rewrte regresson model (4) as ey jt αd e jt + e jtβ 0 + eη jt. (5) One can then estmate α by regressng ey jt on d e jt and e jt. Let bα and β b denote the OLS estmates of α and β n (5). We need an assumton to guarantee that after takng out tme and fxed effects, e jt s not collnear. Assumton.2 N + N 0 where Σ x s fnte and of full rank. +N 0 j t e jt e 0 jt Σ x In Prooston. we show that OLS yelds a consstent estmator of β and we derve the lmtng dstrbuton of bα. 7

10 Prooston. Under Assumtons.-.2, as N. bβ β (bα α) j t j t djt d j ηjt η j djt d j 2 In the exresson above, η jt η j aears rather than the orgnal resdual eηjt. Ths results because both η t and η converge n robablty to zero as N gets large. Thefactthatbα s not consstent does not revent us from conductng nference about thetruevalueofα. The dfference between bα and α deends on two varables: d jt and ηjt η j. The djt areobservableandthedstrbutonof η jt η j can be estmated from the control grous, j>n 0. Therefore, we can estmate the asymtotc (N ) condtonal dstrbuton of (bα α) gven d jt for the treatment grous. We state ths as Prooston.2 below. Estmaton of the dstrbuton of bα allows hyothess testng on α and constructon of confdence ntervals for (bα α). To see how the dstrbuton of η jt η j can be estmated, consder estmaton of the resdual for a member of the control grou (.e. j>n 0 ), ey jt e jtˆβ 0 e jt(ˆβ 0 β)+ η jt η j η t + η η jt η j hence the dstrbuton of η jt η j s trvally dentfed usng resduals for grous j >N 0. From ths t s straght forward to show how to estmate the asymtotc dstrbuton of bα u to α. Let Γ(a) lm Pr((bα α) <a {d jt,j,..,n 0,t,...,T}). N We wll estmate Γ(a) wth the analogous emrcal dstrbuton of resduals from the control grous. For the N 0 case we can estmate Γ(a) usng More generally µ bγ (a) N bγ (a) 0 +N N N0 0 +N N 0 + N N N0 N 0 + t dt d ³ e Y t e 0 tˆβ 2 <a. t dt d ³ j t djt d j ey j t e 0 j tˆβ 2 <a. j t djt d j 8

11 Prooston.2 Under Assumtons. and.2, b Γ(a) converges unformly to Γ(a). To see the usefulness of ths result, frst consder testng the null hyothess H 0 : α α 0 condtonng on the observed sequence d jt,j,.., N 0,t,..., T. We could defne an aroxmate 95% accetance regon by A b, ba 2 ³ as the maxmum value of A lower and mnmum value of A uer such that bγ (A uer α 0 ) bγ (A lower α 0 ) h Then we reject f bα s outsde A b, ba 2. Under the null hyothess, the rejecton robablty wll converge to 5% as N.Wedefne an aroxmate confdence nterval of α as the set of α 0 for whch we do not reject the null hyothess. As N,the coverage robablty of ths nterval wll converge to 95%. 2.2 Model 2 Now we augment the model to allow for ndvdual data. Snce dfference-n-dfferences methods are most commonly used wth reeated cross-secton data, we let ndex an ndvdual whosobservedwthnasnglegrouatasngletmeerod. Asnthentroducton,we use the notaton j() to reresent the grou to whch ndvdual belongs, and t() to reresent the tme erod n whch we observe ndvdual. We also contnue to assume that the data come from reeated cross sectons so that we only observe ndvdual durng one tme erod. We see no reason why extenson to anel data would be roblematc. Our model s analogous to () wth the addton of regressors: Y αd j()t() + 0 β + θ j() + γ t() + η j()t() + ε. (6) Gven that the model s defned somewhat dfferently than n the revous secton, we need to modfy the assumtons slghtly: Assumton 2. η jt, { : M(j, t)} ª T s..d. across grous t s..d. wthn grou for all j and t,and all second moments exst. Furthermore the dstrbuton of η j,..., η jt s ndeendent of (d j,..., d jt ) and { : M(j, t)} T t and has a bounded densty and bounded suort. 9

12 We add the addtonal assumton that Assumton 2.2 ε s..d. across ndvduals and s ndeendent of (d jt,,η jt ) and E(ε )0. We use notaton analogous to the above for Model. Frst, we modfy the notaton for averages across tme wthn a grou. For a generc varable Z defne P t M(j,t) Z j Z t M(j, t). Snce n general, the number of ndvduals vares across (j, t) cells, dervaton of the dfference n dfferences oerator requres addtonal notaton. We need to formally defne the full set of ndcators for grous {g } N 0+N and tme erods, { τ } T τ so that g ( j()) (7) τ (τ t()). (8) Further defne G and P as the vectors of these dummy varables, G g g 2... g No+N, 0 (9) P 2... T, 0. (0) Then for any ndvdual-secfc random varable Z, let ez be the resdual from a lnear regresson of Z on {g } N 0+N and { τ } T τ. That s 0 N G 0 +N ez Z Gh Gh P P h P h j t h M(j,t) 0 N 0 +N j t h M(j,t) Gh P h Z h. We need a regularty condton to guarantee enough degrees of freedom that regressons uon tme and grou ndcators can be run. Assumton 2.3 +N t P M(j,t) P P 0 j j +N j Ω t M(j, t) P t M(j,t) P G 0 where Ω s of full rank. ³ PN0 +N j P t M(j,t) G P G 0 N0 +N j t P M(j,t) G P 0 j t M(j, t) 0

13 Under ths condton, we can rewrte the model as: ey α e d j()t() + e 0 β + eη j()t() + eε. () We estmate α and β n equaton () by OLS, lettng bα and β b denote the corresondng estmators. Ths requres the usual OLS rank condton stated as Assumton 2.4 P N +N 0 j P N +N 0 j where Σ x s fnte and of full rank. P e t M(j,t) e 0 t M(j, t) Σx When each (j, t) cell has a large samle, nference n model () can be conducted n essentally the same manner as for Model snce averagng wthn tme grou cells effectvely elmnates eε. For the sake of comleteness, n the Aendx, we resent a consstency result for β b and the dstrbuton of bα, when M(j, t) and N grow. However, we focus on the fxed- M(j, t) casebecauseweantcatethattwllbemore arorate for a majorty of alcatons. Ths s because large M(j, t) aroxmatons must work n all grou/tme erod cells-not just on average n order for the resultng aroxmaton for the dstrbuton of (bα α) to erform well. There wll routnely be substantal heterogenety n M(j, t) across grous, e.g. states, wth the smallest M(j, t) erhas best consdered a small rather than large samle. For examle, n our llustratve examle alcaton usng states as grous, M(j, t) ranges from 383 to 5. We characterze the fxed M(j, t) case n the followng manner: Assumton 2.5 For each j,.., N 0 + N, M(j, t) for t,..., T s fxed and fnte. In addton, ( M(j, t), t,..., T ) s ndeendent and dentcally dstrbuted across j for j>n 0 and jontly ndeendent of η and ε. Note that we have assumed that M(j, t) s..d. for j>n 0, butweallowthedstrbuton of M(j, t) for j N 0 to dffer from the dstrbuton of M(j, t) for j>n 0. For examle, f larger states were lkely to mlement olcy changes earler, the dstrbuton of M(j, t) for j N 0 would stochastcally domnate the dstrbuton of M(j, t) for j>n 0. Prooston 2. rovdes a statement of consstency for β b as N grows large and the asymtotc dstrbuton of (bα α).

14 Prooston 2. Under Assumtons , bβ β (bα α) P ³ T P j t M(j,t) djt d j (ηjt η j + ε ε j ) t M(j, t) 2 d jt d j j as N. Analogous to model, the exresson for (bα α) nvolves (η jt η j + ε jt ε j ) rather than eηj()t() + eε. To see why, consder the regresson of ηj()t() +ε on grou and tme ndcators. The coeffcent on each grou ndcator converges to η j + ε j whle the coeffcents on the tme ndcators converge to zero snce these random varables both have exectaton zero. Anumberofdfferent otons are avalable for estmatng the dstrbuton of (bα α). In rncle, wth enough grous, one could smly estmate the dstrbuton of resduals condtonal on the values of M(j, t) for the treatment states. We susect that ths rocedure would not work well n most alcatons snce the number of control grous s lkely not large enough for ths to be a useful aroxmaton. Instead we take advantage of our model s structure to estmate the dstrbuton of comonents of (η jt η j + ε jt ε j ). More secfcally defne v Y 0 β (2) αd j()t() + γ t() + θ j() + η j()t() + ε η j()t() + ε. Note that snce we are usng control grous only, the term n brackets s constant across ndvduals wthn the same tme and grou and s ndeendent of ε. Our goal s to smulate the dstrbuton of (η jt η j ) and (ε ε j ). Note that snce αd j()t() + θ j() does not vary across tme wthn a grou and γ t does not vary across grous, knowledge of the jont dstrbuton of η jt s suffcent for knowledge of (η jt η j ). If we have a consstent estmate of the dstrbuton of ε, we can consstently estmate the dstrbuton of (ε ε j ). Thus our goal s to obtan consstent estmates of the dstrbuton of η jt and the dstrbuton of ε. Ths s a standard deconvoluton roblem. We wll frstshowthatthese dstrbutons are dentfed makng use of a well known result. We reort Theorem 2.. n Prakasa Rao (992) whch he attrbutes to Kotlarsk (967) as Theorem

15 Theorem 2.2 (Kotlarsk, Prakasa Rao) Suose that, 2,and 3 real valued random varables. Defne are ndeendent Z 3 Z f the characterstc functon of (Z,Z 2 ) does not vansh then the jont dstrbuton of (Z,Z 2 ) determnes the dstrbutons of (, 2, 3 ) u to a change of the locaton. To aly the theorem we need one addtonal assumton. Assumton 2.6 The characterstc functons of ε and η jt do not vansh. Gventhat,wecanshowdentfcaton of the dstrbuton of bα. Prooston 2.3 Under Assumtons , the dstrbuton of (bα α) s dentfed from knowledge of d jt and M(j, t) from the treatment grous and the jont dstrbuton of v for the control grous. Manyotonsareavalabletoestmatethedstrbutonsofε and η jt. In ths secton we resent one ossble estmator whch s erhas the most common way to estmate ths tye of mxture model n economcs. We derve a seve estmator assumng that (η j,..., η jt ) has fnte suort. Ths aroach s most commonly assocated wth Heckman and Snger (984). We roose to estmate the model n two stes. Frst we run the fxed effects model (). We can construct the resdual for each ndvdual n the control set bv Y 0 β b (3) ³ 0 β β b + η j()t() + ε. Our goal s to searately estmate the dstrbuton of ε from η jt. We arameterze η jt to take on K values wth each value takng the value η (κ) wth robablty P (κ ) for κ,..,k. We let ε be a mxture of normals that take on K 2 values wth mean and standard devaton (µ (κ2),σ) wth robablty P (κ 2) 2 for κ 2,..., K 2. The objectve functon s Ã! 0 +N K TY Y K 2 log bv η (κ ) t µ (κ 2) φ P (κ ) P (κ 2) 2 (4) σ jn 0 + κ t M(j,t) κ 2 3

16 where σ s resecfed. Asymtotcally we allow K and K 2 to grow wth the samle sze whch s why we nterret ths model as a seve model. Showng consstency of ths estmator s a straghtforward alcaton of seve methodology, but nvolves ntroducng much new notaton. Snce ths s only one of numerous estmaton otons and to avod ntroducng ths notaton n the text, we leave the detals of the estmaton to the Aendx Secton A.7 where we show ths rovdes a consstent estmator of the two dstrbuton functons. Wth consstent estmates of dstrbutons of ε and the η jt n hand, we can smulate the dstrbuton of (bα α) for any hyotheszed value of α. 3 Emrcal Examle: The Effect of Mert-Ad Programs on Schoolng Decsons 3. Mert-Ad Programs In the last ffteen years a number of states have adoted mert-based ad rograms. These rograms are run at the state level and rovde subsdes for tuton and fees to students who meet certan mert-based crtera. The largest and robably the best known rogram s the Georga HOPE (Helng Outstandng Puls Educatonally) scholarsh whch started n 993. Ths rogram rovdes full tuton as well as some fees to elgble students who attend n-state ublc colleges. 7 Elgblty for the rogram requres mantanng a 3.0 grade ont average durng hgh school. A number of revous aers have examned the effect of HOPE and other mert based ad rograms. 8 Gventhelargeamountofrevousworkon ths subject, we leave full dscusson of the detals of these rograms to these other aers and focus on our methodologcal contrbuton. Our work most closely relates to Dynarsk (2004) by focusng on the effects of HOPE and other mert ad rograms on college enrollment of 8 and 9 year olds usng the October CPS from However, our analyss dffers from hers n several ways. Perhas most mortantly, we use all states as controls whle she just uses those from the South. Of course her aer s a more comlete emrcal analyss whle our rmary goal s to demonstrate the use of our method. Durng the tme erod, ten dfferent states ntated mert-ad rograms. We 7 A subsdy for rvate colleges s also art of the rogram. 8 Examles nclude Dynarsk (2000, 2004), Cornwell, Mustard, and Srdhar (2003), Cornwell, Lee, and Mustard, (2003), Cornwell, Ledner, and Mustard (2003), Bugler, Henry, and Rubensten (999), Berker(200), Bugler and Henry (997,998), Henry and Rubensten (2002). 4

17 use two secfcatons wth the frst focusng on the HOPE rogram alone. In ths case, we gnore data from the other nne treatment states and use 4 controls (40 states lus the dstrct of Columba). In the second case, we study the effect of mert-based rograms together and use all 5 unts. 9 The deendent varable n our model s a dummy varable reresentng whether the ndvdual s currently enrolled n college. Gven that we obtan multle observatons of ndvduals n the same state at the same tme, Model 2 s arorate. However, snce our deendent varable s bnary we modfy our aroach somewhat to deal wth bnary deendent varables. We dscuss ths aroach n secton 3.2. For estmaton we assume that the number of ndvduals n a state year s large and resent these results n secton 3.3. In secton 3.4 we treat grou sze as fxed. We control for race and gender throughout. 3.2 Lmted Deendent Varable Models Snce our college attendance deendent varable s dscrete, the analyss above can not be aled drectly. In ths Subsecton, we dscuss an extenson of Model 2 to handle lmted deendent varables. We redefne the model lettng the regresson equaton defne a latent varable Y where the researcher observes only an ndcator of ts sgn: Y. and Y αd j()t() + β 0 + θ j() + γ t() + η j()t() + ε (5) Y (Y > 0). (6) For comutatonal smlcty, we assume that the dstrbuton of ε s known wth logstc dstrbuton Λ. Wefrst dscuss the natural extenson to the case n whch M(j, t). We then turn to the dscusson of the more dffcult case where M(j, t) s fnte. Consderthecasenwhch M(j, t). As n secton 2.2, defne η jt αd jt + θ j + γ t + η jt, so that t ncororates all of the grou tme varaton. Then we can wrte Pr(Y,j(),t()) Λ( 0 β + η jt). 9 Note that these mert rograms are qute heterogeneous. Ths exercse does not necessarly mean that we are assumng that the mact of all of these rograms s the same. One could nterret ths as estmaton of a weghted average of the treatment effects. Alternatvely, we can thnk of ths as a test of the jont null hyothess that all of the effects are zero. Our methods could be extended to ncororate heterogeneous effects n whch case one could look at comlcated jont tests of the effects of the rograms. 5

18 Snce M(j, t), ths s a standard dscrete choce model and we can obtan consstent estmates of β and η jt for each j and t by maxmum lkelhood where η jt can be estmated as the coeffcent on grou tme dummy varables (a strategy analogous to that n Amemya, 978). Alternatvely, we could relax the assumton that ε s logstc and use a semarametrc estmator. Havng obtaned consstent estmates of η jt we are essentally n the condtons of Model and can aly the methodology n that Secton usng η jt as the deendent varable. When M(j, t) s assumed fxed, we can no longer obtan consstent estmates of η jt n ths model and thus can not use the Model methodology. To comlcate thngs further, the fxed effects θ j cannot be dfferenced out n ths nonlnear model. Tycal solutons to the resence of fxed effects lke Chamberlan s (980) condtonal logt model or the fxed effects maxmum score estmator (Mansk, 987) could be used to estmate β, but ths s not enough to erform hyothess tests on α whch essentally requre estmaton of the jont dstrbuton of η j,..., η jt. Thus, n order to obtan estmates of the dstrbuton of the error term we use somewhat stronger assumtons. We have defned η jt so that Y 0 β + η j()t() + ε, (7) and we assume that for the control grous, η j()t() s ndeendent of. 0 As long as the suort of β 0 s suffcently large we can dentfy the jont dstrbuton of (η j+ε,..., η jt +ε) u to scale. Gven that ths jont dstrbuton s dentfed for varous values of M(j, t), onecanuseanargumentanalogoustothatntheroofofrooston2.5toshowhowto dentfy the margnal dstrbuton of ε and the jont dstrbuton of (η j,..., η jt). Gven knowledge of, and the dstrbuton of η j and ε, for any α, we can smulate the condtonal dstrbuton of Y gven and d j()t(). Ths allows us to dentfy the dstrbuton of any test statstc that s a functon of observed varables, u to the arameter α. Thus, we can obtan nterval estmates by nvertng ateststatstc. Frst, wemustchooseatest statstc that deends on (Y,,d j()t() ). Sncewehaveestmatedamodelthatgvesusthe 0 Note that αd jt s art of η jt so that t seems as f we are assumng that d jt s ndeendent of. In our examle ths s not the case because d jt 0for all of the control states (n all tme erods). In other cases, one may want to modfy ths assumton to allow for deendence. Cameron and Taber (998) dscuss dentfcaton of anel data logt models wth unobserved heterogenety. Ths model s more comlcated n that η s a vector, but ths does not substantally comlcate the analyss. 6

19 dstrbuton of Y condtonal on,d jt, and α; we can smulate the dstrbuton of the test statstc under any null hyothess α α 0. The queston then becomes whch test statstc we should use. A natural choce would be the dfference-n-dfference arameter from a lnear robablty model. That s we can estmate the lnear regresson model Y ad j()t() + 0 b + G 0 j()c + P 0 t()f + e (8) whch has grou effects and tme effects. Here we use dfferent notaton than n the models above because the true structural model s (7) whle (8) reresents a reduced form regresson equaton for whch the arameters are defned by the lnear rojecton. We can then use the estmated value of a (call t ba) as the test statstc tself. Gven our estmated model and a null hyothess on α, we can smulate the dstrbuton of ba. Whle the estmator s not a standard fxed effect estmator, t stll embodes the central dea behnd dfference n dfferences; we would reject the null hyothess that α 0when the dfference between the retreatment and osttreatment outcomes s substantally dfferent than what one mght redct based on varaton from the control samle. Anumberof dfferent otons exst for estmaton of (8). For our alcaton the most convenent was to frst run the regresson model usng only the control states to roduce consstent estmates of b and f (call these estmates b and f b ). We then estmate α by ³ runnng a (state) fxed effect regresson of Y 0b Pt() 0 f b on d j()t(). The advantage of ths aroach s that when we smulate the dstrbuton of the test statstc we only need to smulate the error dstrbuton for the treatments whch s all that we need n the second stage of ths rocedure. 3.3 Confdence Interval Estmaton under Standard Aroach and Large Grou Szes We comare three estmaton aroaches n ths subsecton: lnear robablty estmators wth both oulaton weghtng across grous and equal weghtng across grous, and a logt estmator. For each estmator, we comare nterval estmates for the treatment arameter usng our methods to those obtaned under the tycal aroaches allowng clusterng by grou and grou-by-tme. To obtan oulaton-weghted estmates, we estmate equaton (5) va OLS usng all 34,902 observatons. These results are resented n the frst column of Table. The de- 7

20 endent varable s a dummy varable for college enrollment and the samle only ncludes ndvduals aged 8 and 9. The ont estmates suggests that the HOPE scholarsh ncreased schoolng enrollment of students who lve n Georga by about seven ercentage onts. Interval estmates of the HOPE effect are resented n the second anel of the table. The frst clusters by state and year, allowng the error terms of ndvduals wthn the same state and year to be arbtrarly correlated wth each other. One can see that the coeffcent s hghly sgnfcant. We next cluster by state whch allows for seral correlaton n η jt. Bertrand et. al (2004) dscuss a case n whch accountng for seral correlaton can lead to standard errors to ncrease, but n our case we fnd the ooste. The standard errors fall substantally when one clusters by state. Clearly one should be worred about the asymtotc assumtons underlyng these routne confdence nterval estmates. The key assumton justfyng them s that the number of states that change status s large, but only one state (Georga) contrbutes to the estmate of the treatment effect. The estmated confdence ntervals usng our method are resented n the last row of Column. These confdence ntervals are formed by nvertng the test statstc (ˆα α 0 ) usng our large-samle aroxmaton for ts dstrbuton. (For detals see Aendx Secton A.4). These confdence ntervals are substantally dfferent from those obtaned wth tycal methods. The confdence nterval ncreases by a factor of about 3 and the coeffcent s not sgnfcant. To see why, n Fgure we dslay the estmated dstrbuton of (bα α) under the null hyothess that the true value of α s zero (after usng a kernel smoother). Ths dstrbuton s estmated from the other 4 states. It aears very dfferent from normal so t s not surrsng that the asymtotc aroxmaton s very dfferent. In the second column we resent lnear robablty estmates resultng from a commonly used two-ste aroach (Amemya 978). In the frst stage we regress schoolng on the ndvdual s and on the full nteracted state year dummes. In a second stage we regress the redcted state year dummes on the HOPE ndcator controllng for state dummes and year dummes (searately). These results are resented n the second column and are remarkably close to the frst. The dfference between these estmates and those n the frst column s that the states are equally weghted whle n the frst column they are oulatonweghted. Fnally we resent a logt verson of the model. The estmates n the thrd column were obtaned n exactly the same manner as n the second column, excet that n the frst stage 8

21 we run a logt model of the school dummy on our 0 s and state year dummy varables. In the second stage we once agan regress the state year dummes on the hoe ndcator controllng for state dummes and year dummes (searately). Thus the redcted arameter has the nterretaton of a logt ndex. The attern s very smlar. In all three cases the HOPE varable becomes margnally nsgnfcant when we use our aroach even though the varable s hghly sgnfcant usng standard methods. To dslay the magntude of the rogram mact we calculate a 95% confdence nterval for changes n college attendance robablty for a artcular ndvdual. We consder an ndvdual (wthout the treatment) whose logt ndex uts hs robablty of college attendance at the samle uncondtonal average attendance robablty of 45% (.e. an ndvdual wth a logt ndex of -.20). The bracketed ntervals reorted n column three are 95% confdence ntervals for the change n attendance robablty for our reference ndvdual. 2 In Table 2 we resent results estmatng the effect of mert ad usng all ten states who added rograms durng ths tme erod. The format of the table s dentcal to Table. There are a few notable features of the table. Frst, the weghtng matters substantally as the effect s much smaller when we weght all the states equally as oosed to the oulaton weghted estmates. Second, n contrast to Table, the confdencentervals arequtesmlar when we cluster by state comared to clusterng by state year. Most mortantly our aroach changes the confdence ntervals substantally, but less dramatcally than n Table. 3.4 Confdence Interval Estmaton assumng Small Grou Szes We next turn to the case n whch M(j, t) s fxed. Gven that we have 34,902 observatons one may wonder why we are worred about the number of ndvduals n the samle not beng substantally hgh. The roblem s for the asymtotc aroxmaton n Model 2 to work well we need that the asymtotc aroxmaton works well n all states tme erods not just on average. The largest s Calforna n 99 wth 383 eole whle the smallest s New Hamshre n 992 wth 5 eole. One very well mght exect that ndvdual comonents contrbute a substantal amount to the varance of the state comonent for the smaller states. Thswouldleadthevaranceof theeffect to be substantally larger for the smaller states 2 These confdence ntervals for changes n attendance robabltes are calculated drectly from the 95% CI for α. Secfcally, when the CI for α s [c,c 2 ], we reort an nterval for the change n redcted robablty for our reference ndvdual of: (Λ(.2+c ) 45%) to (Λ(.2+c 2 ) 45%). 9

22 than the larger ones nvaldatng the revous exercse. The deconvoluton we dscuss n Secton 2 requred that ε be ndeendent of η. Ths s not ossble n a lnear robablty model snce the deendent varable must be one or zero. We nstead use logt model (5)-(6). We erform nference n ths model n three stages. Frst we obtan consstent estmates of β usng Chamberlan s (986) fxed effect logt model usng state year fxed effects. Second we estmate the jont dstrbuton of eη j u to a locaton normalzaton. Fnally, after choosng a test statstc, we smulate the dstrbuton of the test statstc from the estmated model. The frst stage s straghtforward, so we now descrbe the second. We use a Heckman and Snger (994) style nonarametrc maxmum lkelhood method analogous to that n (4). The Log-lkelhood takes the form 0 +N L TY Y log Λ( 0 β b ³ + η t) Y Λ( 0 β b Y + η t) µ. jn 0 + t I(j,t) We maxmze ths lkelhood n terms of the η t and µ arameters. In ractce we use L3 and we have 2 years of data. 3 That yelds 68 arameters. 4 Naturally, local otma are a roblem n these cases so we randomly selected many dfferent startng values to search for a global otma. 5 Gven the number of arameters and ther lmted nterretaton we do not reort these numbers. The next goal s to obtan a confdence nterval for α. We argue n secton 3.2 that a natural choce for a test statstc s the coeffcent n the dfference n dfference model. Followng the dscusson there, we can wrte the test statstc as t j Pk M(j,t) d e j(k)t(k) ³Y 0b P 0 b t() f τ P e. t j k M(j,t) d 2 j(k)t(k) We frst estmate τ usng the actual data. Once we have estmated the data generaton model, we can use t smulate the dstrbuton of τ under the null hyothess α α 0.Notethatτ wll vary n these smulatons both because of heterogenety n η and because M(j, t) s fnte. We reject the null hyothess f τ s less than the quantle or greater than the quantle of ths smulated dstrbuton. The confdence ntervals s the set of arameters for whch the null hyothess s not rejected. 3 We exermented wth alternatve values, and the results are not senstve to the choce. 4 That s 3 2 η t arameters, and 2 µ arameters (snce robabltes must add to one). 5 Many n ths case was We found that ths rocedure ran surrsngly fast takng only about two days to comlete all 5000 otmzatons on a lnux machne. 20

23 In Table 3 we resent confdence ntervals constructed usng ths aroach. The results are smlar, but not dentcal to those n Tables and 2. The confdence nterval for the HOPE rogram s slghtly bgger than those n the thrd column of Table. The nterval for all mert rograms s smlar n sze but skewed slghtly to the left of that n Table 2. For ths treatment effect, a one sded test robably s erhas most nterestng. At the 5% level a one-sded test rejects the null hyothess of no effect. One may worry that the model we have estmated s too stylzed or too flexble to aroxmate the data well. To examne ths, we tred the followng exerment somewhat lke the lacebo law used n Bertrand, Duflo, and Mullanathan. We use all 4 of our control states and construct the test statstc that we used for Georga for testng the null hyothess that α 0. That s, for each of the 4 control states n turn, we act as f the HOPE rogram were oeratng n the state after 993 and used the remanng 40 states as controls. For each alternate retend treatment state we calculate the -value for the test that α 0usng our method. Snce ths null hyothess s true by constructon, these -values should have a unform [0,] dstrbuton. We lot the dstrbuton of -values n Fgure 2. We resent a hstogram of the values and along the horzontal axs lot the actual -values. The ft of the model looks surrsngly strong n the sense that the -values are sread throughout the dstrbuton. Ths logt aroach wth ths test statstc s not the only way to obtan confdence ntervals for α, and s almost certanly not the most effcent, but t aears to work well. 4 Conclusons The man goal of ths aer s to construct a method to erform nference for dfferencen-dfferences models when the number of olcy changes observed n the data s small. We argue that ont estmates of treatment effects should not be thought of as beng consstent and that the standard methods that researchers use to erform nference n these models are not arorate. The man contrbuton of our work s to show how to erform nference under the assumton that there are a fnte number of olcy changes n the data, usng asymtotc aroxmatons as the number of control grous gets large. In ths case, we cannot obtan a consstent ont estmator for the key arameter but are able to consstently estmate ts dstrbuton, u to the unknown arameter tself. Ths allows us to erform nference on the key arameter and construct confdence ntervals. 2

24 We develo ths methodology n a number of dfferent cases. Model consders a regresson model n whch one observes grou tme level data. Model 2 extend the dea to cases n whch we observe ndvdual level data. Wthn Model 2 we focus on the case n whch the number of observatons n a grou/tme cell s fxed. We demonstrate the methodology by alyng t the study of the effects of mert-ad rograms on schoolng. We thnk ths alcaton s a good examle of a stuaton wth a few treatment grous changng olcy and many controls wth unchanged olces. To accommodate our artcular examle, we extend the methodology to a logt model. Our emrcal results suggest that conventonal methods understate the magntude of the standard errors consderably. However, we stll fnd evdence of a ostve effect of mert ad rograms. We thnk our combnaton of large and small samle nference wll be arorate n many other stuatons as well. For examle, n alcatons studyng the effect of a law change n a small number of states usng other states as controls. Whle we have focused on dfference n dfferences estmators, our aroach s more general and s straghtforward to extend to any tye of regresson model n whch there are a large number of control observatons, but only a small number of treatments. 22

25 5 References Amemya, Takesh A Note on a Random Coeffcents Model Internatonal Economc Revew Vol. 9 (3), 978, Anderson, Patrca and Bruce Meyer, The Effects of the Unemloyment Insurance Payroll Tax on Wages, Emloyment, Clams, and Denals Journal of Publc Economcs 78:,2000, Andrews, Donald W.K. End-of-SamleTests Econometrca 7(6) , Angrst, Joshua, and Alan Krueger, Emrcal Strateges n Labor Economcs, Handbook of Labor Economcs, 999, Elsever, New York, Ashenfelder and Card eds. Athey, Susan, and Gudo Imbens, Identfcaton and Inference n Nonlnear Dfference n Dfference Models, unublshed manuscrt, Berker, Al Murat, The Imact of Mert-Based Ad on College Enrollment: Evdence from HOPE-Lke Scholarsh Programs, unublshed manuscrt, Mchgan State Unversty, 200. Bertrand, Maranne, Esther Duflo, and Sendhl Mullanathan, How Much Should We Truce Dfferences-n-Dfferences Estmates?, Quarterly Journal of Economcs 9:, Bugler, Danel and Gary Henry, Evaluatng the Georga HOPE Scholarsh Program: Imact on Students Attendng Publc Colleges and Unverstes, unublshed manuscrt, Coucl for School Performance, Georga State Unversty, 997. Bugler, Danel and Gary Henry, An Evaluaton of Georga s HOPE Scholarsh Program: Imact of College Attendance and Performace, unublshed manuscrt, Councl for School Performance, Georga State Unversty, 998. Bugler, Danel, Gary Henry, and Ross Rubensten, An Evaluaton of Georga s HOPE Scholarsh Program: Effects of HOPE on Grade Inflaton, Academc Performance and College Enrollment, Councs for School Performace, Georga State Unversty, Atlanta, GA, 999. Cameron, Stehen and Chrstoher Taber, Evaluaton and Identfcaton of Semarametrc Maxmum Lkelhood Models of Dynamc Dscrete Choce, unublshed manuscrt, Northwestern Unversty, 998. Card, Davd, The Imact of the Marel Boatlft on the Mam Labor Market, Industral and Labor Relatons Revew, January, 990. Card, Davd, and A. Krueger, Mnmum Wages and Emloyment: A Case Study of the Fast-Food Industry n New Jersey and Pennsylvana, Amercan Economc Revew, Setember, 994. Chamberlan, Gary, Analyss of Covarance wth Qualtatve Data, Revew of Economc Studes, 47 (), 980, Cornwell, Chrstoher, Kyung Hee Lee, and Davd Mustard, The Effects of Mert-Based Fnancal Ad on Academc Choces n College,, unublshed manuscrt, Unversty of Georga,