Spurious Significance of reatment Effects in Overfitte Fixe Effect Moels Albrecht Ritschl LSE an CEPR March 2009 Introuction Evaluating subsample means across groups an time perios is common in panel stuies that evaluate the treatment effects of training programs, labor market policies, currency unions etc. Comparison of means between treate an non-treate groups may occur along the time axis (fixe effects, FE), the cross section (poole OLS, IV) or in a combination of the two (ifference in ifferences, DiD, see Ashenfelter (978), Ashenfelter an Car (985)), epening on the choice of ientifying assumptions about selectivity an common trens (see Heckman, Lalone, an Smith (999)). Despite their wiesprea use in evaluation stuies, FE an DiD estimators have acquire a reputation for generating spuriously low stanar errors on the estimate treatment effect. Bertran, Duflo, an Mullainathan (2004) survey empirical applications of the DiD estimator, an fin that much of this phenomenon can be attribute to autocorrelation. In a simulate ataset, they show that many stanar methos for ealing with autocorrelation yiel ownwar biase stanar errors on the treatment effect coefficient. he following note argues that spurious significance of treatment effects in panels may also occur in the absence of autocorrelation. his phenomenon arises in overfitte FE an DiD moeling of within-group comparisons. Overfitting in such moels occurs if observation-specific iniviual fixe effects (IFE) are specifie, although the comparison woul be ientifie by group-specific fixe effects. In evaluation stuies, ientifying the average treatment effect on the treate through a withingroup estimator woul require a group fixe effect on the treate (FE), see e.g. Angrist an Pischke (2009). Yet specifying an overfitte regression with IFE instea may seem innocuous to the applie researcher, as the coefficient estimates on the treatment effect uner both fixe effect specifications are ientical. Moreover, stanar software packages provie easy to use options for iniviual fixe effects, making an overfitte specification seem attractive. However, while the estimate treatment effect uner IFE an FE is the same, its estimate stanar error is not. Overfitting through IFE leas to spurious precision of the estimate treatment effect coefficient. he resulting bias is relate to the reuction in the resiual sum of squares inuce by employing IFE instea of FE. Uner ieal conitions where all Financial support from Deutsche Forschungsgemeinschaft uner SFB649 Economic Risk at Humbolt University Berlin is gratefully acknowlege.
other regressors are uncorrelate to the treatment an the fixe effects, this relation is strictly proportional. he rest of this note is structure as followe. he next section provies the setup. Section (3) presents the result. Section (4) conclues. 2 A Minimal an an Overfitte Setup Consier a ata panel with n observation units in the cross section an time perios. In this panel, enote by Y n the epenent variable. Z is a matrix of characteristics of interest, as well as any time fixe effects, while X inclues the regression constant an/or a suitably chosen matrix of either iniviual or group fixe effects. A policy treatment is applie to some observation units y i uring treatment perio τ {s,..., s+τ}. reatment uring perio t τ is inicate by a (n )-vector of ummy variables t, which are equal to one if unit i is uner treatment at time t, an zero otherwise. tr( ) < n is the number of observation units i in the treatment group. Accoringly, n is the number of observation units in the non-treate group. A stanar linear panel moel of this treatment effect problem is: Y (XD)β + Zγ + v () where v N(0, σ 2 v) an where D (0... s... τ... 0) is a ummy vector capturing the policy treatment in τ perios. Fixe effects estimation of moels like () is a popular (yet problematic) attempt to ensure the exogeneity of D with respect to the isturbance term v. o focus on the essentials, consier an ieal regression in which any characteristics inclue in Z are orthogonal to the fixe effects X an the treatment ummy D. Define the etrene variable y M z Y with M z I Z(Z Z) Z, where the influence on Y of any such characteristics, as well as any time fixe effects inclue in Z has been remove 2. As M z XD XD if Z XD 0, the moel becomes a Least Squares Dummy Variables (LSDV) regression on the fixe effect terms an the treatment ummy only: y (XD)β + u (2) where X is a suitably chosen matrix of fixe effects, D (0...... τ... 0) is a ummy vector capturing the policy treatment in τ perios, an u N(0, σu). 2 Uner iniviual fixe effects, X consists of stacke (n n) ientity matrices: X I I n n. I n n n n Uner the alternative assumption of a group fixe effect on the treate, matrix X takes the form: 2 ime fixe effects woul be orthogonal to X. heir inclusion in Z makes the FE an DiD estimators in y ientical. 2
X G.. n 2 Note that the column imension of X G is 2 as oppose to n in X I. LSDV estimation of (2) uner the two ifferent fixe effect specifications yiels: ˆβ I (X I D] X I D]) X I D] y ˆΩ ˆβ,I ˆσ 2 u,i (XI D] X I D]) (3) ˆβ G (X G D] X G D]) X G D] y ˆΩ ˆβ,G ˆσ 2 u,g (XG D] X G D]) (4) Let b I be the n+th (i.e., last) element of ˆβ I, an b G be the 3r (i.e., last) element of ˆβ G. b I an b G are the coefficients on the treatment ummy uner Iniviual Fixe Effects (I) an the Group Fixe Effect on the reate (G), respectively. Likewise, let ˆσ 2 (b I ) ˆΩ ˆβ,I,(n+,n+) an ˆσ 2 (b G ) ˆΩ ˆβ,G,(3,3) be the estimate variances of these coefficients, with Sn+,n+ I X I D] X I D]) n+,n+ an S3,3 G X G D] X G D]) 3,3 as the pertaining elements of the matrix inverses in (3) an (4), respectively. 3 Spurious Significance uner Overfitting Consier a treatment effect moel as in eq. (2), in which the enogenous variable has been etrene from any time effects, an in which any further characteristics are orthogonal to the fixe effects an treatment ummy, an have been eliminate as well. Estimation uner the alternatives of Iniviual Fixe Effects (IFE) an Fixe Effects on the reate (FE) as in (3) yiels ientical coefficient estimates on the treatment effects. However, the estimate variances on these coefficients in (4) iffer, owing to the presence of unnecessary ummy variables in the IFE specification that artificially increases the fit of the regression. his is expresse in the following Proposition. In a treatment effect moel as in eq. (2), the estimate variance of the treatment effect coefficient is ownwar biase uner IFE relative to FE. he bias is equal to the ratio of the estimate resiual variances uner IFE an FE: ˆσ 2 (b I ) ˆσ2 u,i. ˆσ 2 (b G ) ˆσ u,g 2 Proof. It suffices to show that Sn+,n+ I S3,3, G i.e. the last elements on the main iagonal of the inverte prouct sum matrices in eqs. (3) an (4) are ientical. By elementary operations, X I X I I n n. Hence, uner Iniviual Fixe Effects: ( ) X I D] X I I D] n n τ τ τ where, as efine further above, tr( ) tr( ). Inverting this partitione matrix, we fin for the (n+,n+)-element of the inverse: (X I D] X I D]) n+,n+ (τ τ τ) 3 τ( τ) (5)
Uner Fixe Effects on the reate, the prouct sum matrix becomes: ( ) n X G X G Hence, n τ X G D] X G D] τ τ τ τ Inverting this partitione matrix, we fin for the element (3,3) of the inverse: Using this becomes: (X G D] X G D]) 3,3 τ τ( (X G X G ) (X G D] X G D]) 3,3 τ τ τ τ 2 (n ) )(X G X G ) τ ( ) n ( ) (n ) τ( ) τ n ( )] (6) ( )] ( )] (n ) τ( ) 0 τ (n ) ] (7) is equal to (5), which completes the proof. τ( τ) (7) In applie work, the possible correlation of aitional characteristics Z with XD means the above relation no longer obtains exactly. Unless, however, this correlation amounts to near-collinearity, its effect is small relative to the overfitting effect escribe in the proposition. 4 Conclusion Applications of fixe effect an ifference in ifferences estimators sometimes employ iniviual, observation-unit specific fixe effects when group-specific fixe effects woul suffice for ientification. his note has examine the properties of ifference in ifferences estimators of treatment effects uner two ifferent fixe effects specifications. It shows that overfitting uner iniviual, observation-unit specific fixe effects generates lower stanar errors on the treatment effect coefficient than estimation uner a minimal specification with group specific effects. Depening on the correlation with other regressors, this bias grows at or near the relative ecrease 4
of the resiual sum of squares as the number of overfitte fixe effects increases. In large samples, which are frequent in evaluation stuies, this overfitting bias may lea to substantial unerestimation of the stanar errors on treatment effect coefficients, an hence to substantial false positives. References Angrist, J., an S. Pischke (2009): Mostly Harmless Econometrics. Princeton University Press. Ashenfelter, O. (978): Estimating the Effect of raining Programs on Earnings, Review of Economics an Statistics, 60, 47 57. Ashenfelter, O., an D. Car (985): Using the Longituinal Structure of Earnings to Estimate the Effect of raining Programs, Review of Economics an Statistics, 64, 648 660. Bertran, M., E. Duflo, an S. Mullainathan (2004): How Much Shoul We rust the Differences-in-Differences Estimator?, Quarterly Journal of Economics, 9, 249 275. Heckman, J., R. Lalone, an J. Smith (999): he Economics an Econometrics of Active Labor Market Programs, in Hanbook of Labor Economics, e. by O. Ashenfelter, an D. Car, pp. 865 2033. Elsevier. 5