Sectin 7 Mdel Assessment This sectin is based n Stck and Watsn s Chapter 9. Internal vs. external validity Internal validity refers t whether the analysis is valid fr the ppulatin and sample being studied. External validity refers t whether these results can be generalized t ther ppulatins: is the ppulatin frm which the sample is drawn representative f a larger ppulatin abut which inference is sught? Ecnmic vs. statistical significance: Even if t > 2, the effect may be t small t be ecnmically imprtant. Beta cefficients are used t give the number f standard deviatins that y changes when x increases by ne standard deviatin. Marginal effects in standard deviatins can be mre useful than marginal effect in units. External validity External validity is related t Assumptin #0. But in this case, the questin is nt whether all sample bservatins fllw the same mdel but rather d the sample bservatins fllw the same mdel as the mre general ppulatin. Or, alternatively, are they drawn frm a sub-ppulatin that has characteristics that wuld make the cefficients (r specificatin) different? All ppulatins have sub-ppulatins that vary in their characteristics. If ur sampling prcess is based n a particular sub-ppulatin, we must wrry abut the generalizability f ur results, which is external validity: Can perfrm an internally valid analysis f an idisyncratic sub-ppulatin that wuld nt generalize t thers. Example: Nel s wrk measuring the value f tree canpy r walkability in Prtland. D results generalize t ther cities r d Prtlanders value these characteristics mre (r less) than peple in ther cities. There are n direct statistical tests fr external validity (unless yu have data drawn frm a brader ppulatin, in which case yu prbably shuld have used it t begin with). It is a usually a matter f judgment. One way that sme peple try t assess external validity is t split the sample in half, estimate ver ne sample, then assess the predictins fr the ther sample. If predictins are gd, then bth halves f the sample may fllw same mdel. ~ 71 ~
This is useless if bth halves f the sample are drawn frm a subppulatin that is idisyncratic, thugh. Meta-analysis: Ding a study where each data pint is an ecnmetric result. Direct estimatin f mapping frm assumptin space t cnclusins Internal validity Given the ppulatin frm which the sample is drawn, are the assumptins underlying the estimatrs valid? Omitted variables They are always there. Omitted variables bias the cefficient estimatrs fr any included variables that are crrelated with them. In a strict sense, nearly every ecnmetric regressin is biased because f this. What variables are mst bviusly mitted? What variables in the equatin wuld be crrelated with them? Hw des this missin bias the included cefficients? Prxy variables are bservable variables that are crrelated with unbserved variables that shuld be included. Prxy variables are legitimate if we are nt particularly interested in the effect f the variable fr which they prxy. Can t interpret the cefficient n the prxy directly as the cefficient n the mitted variable. OK if the difference between the true variable and the prxy is uncrrelated with included variables. Panel data can help if unbserved variables vary acrss units but nt ver time r ver time but nt acrss units. Misspecificatin f functinal frm Can use RESET test t explre whether quadratics are useful. If yu knw what alternative functinal frms might be mre apprpriate, yu can test them. Measurement errr (errrs-in-variables bias) Measurement errr in dependent variable Suppse that the true dependent variable is y but that we instead bserve y y, where i is a randm measurement errr. i i i The estimated mdel, then is y x e ~ 72 ~. i 1 2 i i i As lng as the measurement errr in y ( ) is uncrrelated with x, there is n bias in the estimatr f 2. The SER will be an estimate f the
standard deviatin f the cmpsite errr term e +, but therwise OLS is fine. Measurement errr in regressr Suppse that the dependent variable is measured accurately but that we measure x with errr: x x. i i i. The estimated mdel is y x e i 1 2 i i 2 i Because is part f x and therefre crrelated with it, the cmpsite errr term is nw crrelated with the actual regressr, meaning that b 2 is biased and incnsistent. 2 x If e and are independent and nrmal, then plim b2 2 2 2. x The estimatr is biased tward zer. If mst f the variatin in x cmes frm x, then the bias will be small. As the variance f the measurement errr grws in relatin t the variatin in the true variable, the magnitude f the bias increases. As a wrst-case limit, if the true x desn t vary acrss ur sample f bservatins and all f the variatin in ur measure x is randm nise, then the expected value f ur cefficient is zer. Best slutin is getting a better measure. Alternatives are instrumental variables r direct measurement f degree f measurement errr. Fr example, if an alternative, precise measure is available fr sme arguably randm sub-sample f bservatins, then we can calculate the variance f the true variable and the variance f the measurement errr and crrect the estimate. Sample selectin bias Few samples are truly randm draws frm full ppulatin. Instead, they are draws (randm r nt) frm sme sub-ppulatin: Many hmeless are uncunted in census N wage data n thse wh d nt wrk Plls miss peple with n listed phne number Crss-cuntry regressins are ften limited t the cuntries fr which gd data are available (which is nt a randm sample f cuntries) If sample selectin is related t x, then we have issues f external validity (d estimates apply t missed sub-ppulatin) but nt internal validity. Results may be valid fr the sub-ppulatin fr which they are estimated. ~ 73 ~
If sample selectin is related t y (r, specifically, t e), then we are nt drawing randmly frm the ppulatin distributin f the errr term (as we assume) and ur results will be biased. There are methds f cping with sample-selectin bias. Imputing values fr missing wage data t allw inclusin f full sample Simultaneity bias (reverse r bidirectinal causality) If changes in y (presumably due t changes in e) cause x t change, then x and e will be crrelated and OLS estimates will be biased and incnsistent. Fr example, fr many years macrecnmists estimated Keynesian cnsumptin functins by OLS: Ct 0 1 GDPt ut. (There are time-series prblems with this regressin that we will study later.) Fr nw, nte that if aggregate demand affects utput, then GDP in each year is C + I + G + NX, s a psitive shck t cnsumptin (a psitive e) increases GDP. Because the regressin is crrelated with the errr term, OLS estimates f 1 were biased and incnsistent. (But they lked gd and had ridiculusly high R 2 values, s they persisted fr many years despite the prtests f ecnmetricians.) The usual crrectin is t use an instrumental-variables (tw-stage least squares) estimatr. Heterskedasticity Heterskedasticity (as we will discuss sn) causes OLS t be inefficient (relative t WLS), but it is still unbiased and cnsistent. The classical standard errrs will be biased under heterskedasticity, but we can use White s rbust cvariance matrix estimatr, which we ve talked abut earlier. Using rbust errrs is the mst cmmn crrectin fr heterskedasticity. Autcrrelatin If errr terms f different bservatins are crrelated, then OLS is als inefficient (relative t a crrected GLS estimatr), but is unbiased and cnsistent. Autcrrelatin can be spatial: Unmeasured neighbrhd characteristics (mitted variables) that cause huses that are clse tgether t be mre r less valuable. Autcrrelatin is ubiquitus in time-series data: This perid s errr term is nearly always related t last perid s. (Unmeasured mitted variables are themselves crrelated ver time.) Again, standard errrs are biased, but White s heterskedastic-cnsistent standard errrs dn t help here. ~ 74 ~
There are estimated standard errrs that are rbust t autcrrelatin. (Use hac ptin in Stata.) Alternatively, ne can try t mdel the autcrrelatin and transfrm the mdel int ne that has n autcrrelatin (GLS). Examples include AR(1) mdels in time series and mdeling spatially crrelated errrs in crss-sectin mdels. Validity in frecasting/predictin Regressin mdels may be valid fr frecasting even if their cefficients are nt unbiased r cnsistent. Suppse that we knw that x is measured with errr. We can still use a regressin f y n x t predict the utcme f a particular measured x even thugh the estimated cefficient is a biased estimatr fr the effect f x. That is because we have crrectly estimated the relatinship between the nisy x and y. We wuld nt get reliable estimates if ur predictin questin relied n the true x rather than the nisy x. We ften build mdels with nisy data r prxy variables t get predictins f anther variable. The biggest questin in frecasting is external validity: des the mdel that applies t the sample yu used fr estimatin als apply t the bservatin fr which yu want a frecast? ~ 75 ~