Ordinary Least Squares at advanced level

Transcription

1 Ordary Last Squars at advacd lvl. Rvw of th two-varat cas wth algbra OLS s th fudamtal tchqu for lar rgrssos. You should by ow b awar of th two-varat cas ad th usual drvatos. I ths txt w ar gog to rvw th OLS usg matrx algbra, whch s th rght tool to hav a mor gralzd multvarat) vw of th OLS mthodology. I th stadard two-varat cas w had th followg modl for th populato: Y = + X +.) whr s th rror, X s th xplaatory varabl, Y s th dpdt varabl ad btas dots th populato coffcts. hs s calld th Populato Rgrsso Equato. What w hav s oly a sampl sampl obsrvatos ar dotd by lowrcas lttrs): y = + x + u.) whr u s th rsdual whch s our stmat of th rror. hs s th Sampl Rgrsso Equato. y ad x ar hc a sampl draw from th populato Y ad X ad th bta hats ar our stmats of th btas populato paramtrs) from th sampl. W arrvd at th OLS stmats of th bta coffcts by th last squars prcpl SSR sum of squars rsduals): ) = =.3). h frst ordr codtos for a mmum rqurs that: SSR = u = y x = u ) = y x = u= u= u= Eu ) =.4) othr = = = = words, th rsdual of th OLS wll always hav zro ma, provdd w clud a trcpt or costat trm. = u ) = y x x = ux = ux = ux = Eux ) =.5) hs s = = = = th sampl vrso of th orthogoalty or xogty codto. h OLS wll always assum that th rsduals ad th xplaatory varabls ar ucorrlatd. If ths s ot tru, th OLS s basd. From th abov codtos: ) y x = y x = = y x = y x.6) = = = = = ) = = y x x = yx x x = = = = = = = Ex x = yx x Ex y) y x.7) ) x x

2 W ca ow substtut our stmat for th trcpt to th abov xprsso to arrv at th OLS stmator for th slop coffct. Ex y) y x Ex ) x x ) ) ) x y y x x x y y x x y = xy = = = = = = = x x x x x x) x x) = = =.8) hs ar all quvalt ad w ca us ay of ths as w plas. You should b abl to rproduc abov drvato wthout ay dffculty bfor advacg furthr. If you do ot udrstad a stp, you wll fd plty of hlp o th trt. h da s ot that you mmorz th drvatos but that you arrv at vald rsults startg out from th sam assumptos, othr words you ca rproduc th drvatos. Lt m shar my xprc wth you: you compltly udrstad somthg oly f you ca drv t. Kowg th bg pctur oly, wll hlp you to advac fastr tally, but t wll hold you back from som pot o. Lt us look at th proprts of th OLS ow usg th two-varat cas!. Larty: Frst of all w wll show why th rgrsso as.) s lar. If th paramtrs ca b xprssd as a lar combato or wghtd avrag of th obsrvatos of th dpdt varabl, w call th rgrsso lar. W tak o vrso of th stmator: = = x ) x y = = x x) = ky.9), whr k = = x x) x x).). So vry sgl obsrvato of th dpdt varabl s gog to affct our stmat of th slop paramtr by a uqu wght. h wght dpds o th varac of th xplaatory varabl ad th dvato of th xplaatory varabl at obsrvato from th ma. Proprts of k x x) = k = =, = = x x) = = = = x ) x x kx = = x x) k x x) = x x) x x) x x) = = = = = = =, k x x) = x x) x x) = = = x x) x x). Ubasdss: A stmator s ubasd f ts xpctd valu quals th populato paramtr. hat s E ) =. If thr s a dffrc th that dffrc s calld th bas. k,

3 W show th ubasdss frst. h trck s that sc th populato rgrsso fucto or th Data Gratg Procss) s Y = + X +, hc y = + x +. hs ca b substtutd to th stmator to drv th rlatoshp btw th populato paramtr ad our stmat. ) ) ) + + ) ) x x y y x x x x x x = = = = = = + = + =.) x x) x x) x x) = = = x E ) = + hc, f th rror ad th xplaatory varabls ar ucorrlatd, th OLS stmator s x ubasd orthogoalty codto aga). h: E ) = Smlarly, o = y x = ) x hc o) ) ubasd too. k E = E x, so f s ubasd, s 3. Effccy: hs cocpt ca oly b udrstood a rlatv ss. A stmator s gog to hav a stadard dvato calld stadard rror) sc wth ay w sampl draw from th sam populato, you ar gog to obta dffrt stmats for th populato paramtr. If you hav two altratv stmators, th o wth lowr stadard rror s calld mor ffct. hr s a lowr lmt of stadard rrors, gv by th Cramér Rao lowr boud othr words, o stmator ca hav lss varac tha ths lmt). If th stmator s varac s at th lowr boud, th w call t ffct. W ca xprss th varac of th OLS stmator for as follows: = E E ) ) = E k =.) = = = If th orthogoalty codto holds, E k E k E = ad w ca trat t as costat ad brg t frot of th th rror s homoscdastc th ) summa sg. sc th cross products ar all zro. If = E k E ) E k = = = = = x x) = = x x).3) h problm s that w do ot kow th stadard dvato of th rror. But w ca us th rsdual u varac or ma sum of squars rsdual) as stmator of th rror varac. =.

4 u But why dos our OLS stmator for th populato rror varac quals a two-varat rgrsso? Evryo sms to accpt ths, yt oly rarly s t drvd algbracally t s qut smpl to do wth matrx algbra, though). Lt us s th drvato! Frst, you wll d som ucomfortabl algbra to xprss th rsdual as fucto of th rror. u = y x = + x + y x) x = + x + + x + ) + x x = o o o o ) ) = x x No w tak th squar rsdual: ) ) ) ) ) ) u = + x x x x ad th sum of squard rsduals s th: = = = = ) ) ) ) ) ) u = + x x x x Now w d th xpctato of th sum of squard rsduals. ) ) ) ) ) ) ) E u = E + E x x E x x = = = = = x x x x ) ) = ) ) = = = x x) x x) = ) + x x = = = ) =.4) whr w mad us of th followg: = = = E ) E E ) = = = = = = = = = E ) E E ) ) = = = whr th covrso: = s tru udr th assumpto of o autocorrlato. = = h stadard rror of th costat trm ca b drvd as follows:

5 ) = x+ = x k hc o = o = + x k x k = + x k x k = = = = ) E o ) ) ad = = + = prvous quato gvs us: x x x) Lt us rmmbr: x = x x = substtutg ths to th E x x x o = + = ) ) = = x x) x x) = =.5) h Cramér-Rao lowr boud h Cramér-Rao lowr boud for th varac of a stmator θ ) s xprssd as: θ, whr θ I θ ) dots th populato paramtr to b stmatd. Iθ) s th Fschr formato whch s dfd as: x, θ) x, θ) I θ ) = E = E θ θθ sgl paramtr to stmat., whr x, θ ) s th log-lklhood fucto, ad w hav a For xampl f th dpdt varabl Y follows a ormal dstrbuto ad w stmat ts populato ma oly µ ). h th log-lklhood fucto s: Y X,, ) = l π ) = l π ) Y µ ) Y X, µ ) = Y µ ) = µ Y Y Y, also kow as th scor fucto. X, µ ) = =, hc var µ Y ). You should rmmbr that wh w th sampl µ µ Y Y Y ma s usd as a stmator for th populato ma, ts stadard rror was: y = hc th sampl ma s at th Cramér-Rao lowr boud ad s a ffct stmator of th populato ma. Actually, th stadard dvato s also a paramtr to stmat, but ths s dpdt of th ma, so I dsrgard t ow.

6 What f, as usually th cas, w hav a vctor of paramtrs to stmat.. multpl paramtrs)? h w hav th Fschr formato matrx. h,j th lmt of whch s: Iθ), j x, θ) xθ, ) xθ, ) = E = E. θ θ j θ θ j For xampl f w hav th PRF.) ad s assumd to b ormally dstrbutd, th X,, ) = l π ) l ) Y X ) π = X,, ) = Y X ) X X, X, ) X,, ) X =, = X,, ) = Y X ) = X,, ) =, X,, ) X =, X Iθ) = X X Iθ) X X ) ) X X X X = X X X) X X ) Hc: X ) X X ad X X), whch qual th varacs.3),.4) of th OLS stmats udr xogty, homoscdastcty ad o autocorrlato assumptos. h Gauss Markov thorm If th followg codtos ar mt:. j j, j= k Y = + X + th modl s lar). E ) = 3. Var) = < homoscdastcty) 4. Cov, ) =, j o autocorrlato) j 5. Cov X j,) = for ay Xj xogty) h th OLS s th bst lar ubasd stmator or BLUE. Bst rfrs to th fact that ts stadard rrors ar o th Cramér-Rao lowr boud, hc w caot hav ay stmator wth a lowr stadard rror. hs s cor rsult statstcs. Obsrv that th ormalty of th

7 rror trm s ot rqurd, v though t s customary to lst amog th assumptos of th Classcal Lar Modl, but w usd t to drv th Cramér-Rao Lowr Boud. Yt, sc th coffcts ar calculatd as th wghtd sum of obsrvatos draw from th sam probablty dstrbuto y), thr dstrbuto should covrg to th ormal dstrbuto accordg to th Ctral Lmt horm CL).. OLS wth matrx algbra Lt us df th followg lar modl th populato: y X X k y = X y X X k + or = +.) y X X k k w ca stmat th vctor of coffcts ) usg th last squars prcpl. h vctor dots th vctor of rrors: = y X. Hc th sum of squars rsdual SSE) s ) ) SSE = = = + = + uu y X y X y y y X X y X X y y X y X X.), whch s a scalar. Hr w mad us of th fact that X y = y X, sc thy ar scalars thr dmso s x). h Frst Ordr Codto of a xtrmum rqurs that: uu = Xy+ =.3) or = Xy, whch s calld th ormal qauto Whr I mad us of th followg ruls: Ax x Ax = A, = x A+ A ) x x x Ax ad f A s symmtrc : = xa x h vctor of btas s hc: = ) - Xy.4) w ca furthr dffrtat.5) by ordr to chck th Scod Ordr Codto ad obta: that uu = >, that s, w dd hav a mmum. From ths pot t follows must b vrtbl, hc t must b of full rak. hs s oly possbl f th matrx X has full

8 colum rak,.., our xplaatory varabls ar larly dpdt. ths s th codto of o multcollarty). It wll mak our lf much asr f w troduc two mportat matrcs. h frst s th projcto matrx - somtms rfrrd to as th hat matrx) P): P = X) X.6), th scod s th ahlator - matrx M): M = I P = I X) X.7). hs matrcs ar squar matrcs x), symmtrc, that s P= P ad M = M ad dmpott,.., PP = P ad MM = M Proof: PP = X) X X) X = X) X = P ) ) MM = I P I P = I P+ PP= I P+ P= I P= M h projcto matrx projcts y oto a colum vctor spac dfd by th xplaatory varabls X. hat s: - Py = X) X y = X = y.8) Hc th projcto matrx cotas th wghts ad plays th sam rol as th wghts.). - My = I X) X ) y = y X = u.9) - - MX = I X) X ) X = X X) X X =.) ad My = M X + ) = M = u.). Ubasdss: W ca prov th ubasdss of th OLS stmator as follows: ) ) ) ) = - = = + Xy X X X.) - = + ) E X ).3). hat s, f E X ) = E ) ubasd. Effccy: xogty) th OLS stmats ar Frst w d th varac of th OLS stmator wth ubasdss assummd. h varac of th stmator s th: ) ) ) ) ) ) = E ) = E = XE ) X.4) If th rror s homoscdastc, ad ot autocorraltd ths s a wak vrso of th codto of dtcally ad dpdtly dstrbutd rrors) th E ) = = I Hc.3) ca b wrtt a much smplr form: = ) -..

9 Yt, w do ot kow th rror varac, oly th rsdual varc. W cahowvr stablsh th rlatoshp asly. Usg.): u u = M M = M.5) Sc ths s a scalar, ts valu wll b qual to ts trac: tr u u) = tr M) for th trac thr xsts a rul rgardg cyclc prmutatos, amly that tr ABCD) = tr DABC) = tr CDAB ) =....6) usg ths ruls w obta that: ) tr uu ) = tr M = tr M) = tr M ).7) But what s th trac of th ahlator matrx? h trac of a x dtty matrx s, ad th trac of th projcto matrx quals th rak of th matrx X), whch s k. tr X) X ) = tr X X) X X) = tr I ) = k. Hc: tr M) = tr I ) tr P ) = k.8). k u =.9) Hr w rcvd th sam rsult for k< paramtrs to b stmatd as.4) for k k=. h ffct of addtoal xplaatory varabls o th coffct Lt assum that w hav two sts of rgrssors, X ad X. If w rgrss y o both sts of varabls: y = X + Xγ + u h rsdual wll b: u= yx Xγ h sum of squar rsduals s: ) ) uu= y X γ X yx Xγ = = yy yx yxγ Xy γ γxy γ+ γγ whch w sk to mmz by choosg th coffct vctors: uu = Xy+ + = = ) ) γ Xy γ.) ad uu = Xy+ + = = ) ) γ γ Xy.) γ Xy Or =.) whch s th st of ormal quatos. γ Xy Hc w ca s that th coffcts a multvarat rgrsso wll rflct th ffct of th corrlato amog th dffrt rgrssors. If, ad oly f th two sts of rgrssors wr ucorrlatd, that s, ) ) = = could w xpct that th coffct from a rgrsso of y o X would yld th sam bta coffcts as ). h Frsch-Waugh horm also kow as th Frsch-Waugh-Lovll horm)

10 Lt us substtut.) to.)! ) ) Xy γ Xy = γ Xy ) ) + = Xy γ γ Xy Lt us df th projcto matrx for th colum vctor spac spad by X : ) a ahlator matrx: M = I P XPy XPXγ + γ = Xy P= X X ad XMXγ = XMy = ) γ XMX XMyor, du to dmpotc ad symmtry ) γ = XMMX XMMy What s My? It s th rsdual from a rgrsso of y o X oly. Smlarly, MX s th st of rsduals from th rgrssos of all colums of X o X. h ffct of X o th coffct vctor γ s ttd out or partald out. Frsch-Waugh thorm stats that th coffcts from a multvarat rgrsso ar dtcal from a two-varat rgrsso whr th ffct of all othr varabls s ttd out. h coffcts from a multvarat rgrsso hc ca b trprtd as th partal ffct of th varabl qusto o th dpdt varabl, that s, wth all othr ffcts rmovd. But what s th practcal mportac of ths aothr cor rsult statstcs?. h da of ctrs parbus s ctral th mthodology of coomcs, for xampl comparatv statcs. I comparatv statcs w aalyz th ffct of a sgl varabl or paramtr o th outcom varabl wth all othr factors fxd. Hc multvarat rgrssos ar obvous ways to drctly masur such rlatoshps.. Hav you vr cosdrd what th rght way s to rgrss y o x wh you kow that sasoal ffcts ar prst? Should you rgrss y o x wth sasoal dumms cludd, or rathr should you frst dsasoalz y ad x dvdually, ad rgrss th dsasoalzd y o th dsasoalzd x? Frsch ad Waugh hav a good ws to you. It s th sam.

11 Practcal xampl W hav data o th salary of mploys ad thr ducato yars of ducato) ad xprc yars) Ramatha data6-4.gdt Grtl). Frst w stmat th ffct of both ducato ad ag o th logarthm of salary a thr-varat rgrsso. Modl : OLS, usg obsrvatos -49 Dpdt varabl: l_wage Now w ar gog to partal out th ffct of xprc o ducato. Frst w rgrss th log wag o xprc ad sav th rsdual rs).

12 h w rgrss th ducato o xprc ad sav th rsdual rs). Whch dd yld th sam coffct as th ducato has th multvarat rgrsso.