Eco 388 R. Butler 06 rev Lecture 5 Multvarate I. Parttoed Regresso ad Partal Regresso Table : Projectos everywhere P = ( ) ad M = I ( ) ad s a vector of oes assocated wth the costat term Sample Model Regresso Fdg the mea: = α + μ Smple regresso: = β 0 + β + μ Geeral regresso: = β + μ Cetered R: ad s devated from meas = β + μ M = M β + μ So we ca square both sdes, smplfy Frsch theorem for = β + β + μ, where we have parttoed the s to two arbtrary subsets, Normal Equato (mmze resdual vector by choosg parameter values to make the resdual vector orthogoal to lear combatos of the rght had sde predctor varables) μ = 0, or ( α ) = 0, or (y α ) = 0, or (y ) = (α ), or y =α, or y =α μ = 0 ad μ = 0, has order x, or ( β 0 β ) = 0, ( β 0 β ) = 0, (y β 0 β x ) = 0, x (y β 0 β x ) = 0, see chapter of text ad appedx A for soluto μ = 0, has order xk, k equatos k ukows: ( β ) = 0 Where M are devatos from the meas, so (M ) (M ) = (M β + μ ) (M β + μ ) = (M β ) (M β ) + (μ μ ) + (M β ) μ + ((M β ) μ ) the last two r.h.s. terms=0 (why?), so (y y ) = ((y y ) + (y y )) = (y y ) + (y y ), or SST= SSM + SSR Prelmary cleasg of (of correlatos): M = ( ) part of ucorrel wth Prelmary cleasg of (of correlatos): M = ( ) part of that s ucorrel wth Regressg cleased o cleased : β = ((M ) (M )) (M ) (M ) = ( M ) ( M ) Resultg parameter estmator α = β = (x x )(y y )/ (x x ) β 0 = y β x β = ( ) cetered R = SSM SSR = - SST SST Symmetry ad dempotecy of M esure the last equvaleces Notes Decompose vector to ts mea, α =, ad devatos from ts mea μ = (order x) β = rato of the cov of (x,y) to varace of x; β 0 s the mea of y oce the effect of x s take out. β j geeralzes the smple model: rato of partal covar to partal varaces I our class, almost everythg s doe wth the cetered R ; adj- R compares two regressos wth the same, but wth dfferet s Eve for subsets of varables (ot just a sgle varable), the respectve coeffcets are the cleased (of corr w/ ) partal correl of ad
Ths last result s partalg or ettg out the effect of. For ths reaso, the coeffcets a multple regresso are ofte called the partal regresso coeffcets. Based o Frsch Theorem, we also have the followg results: II. Goodess of Ft ad the Aalyss of Varace R aga, wth more detals o the dervato (the coeffcet of determato, ths s actually the cetered R-square, whe we talk about R-square ormal ecoometrese, we mea the cetered R-square)-- the proporto of the varato the depedet (respose) varable that s 'explaed' by the varato the depedet varables. The dea s that the varato the depedet varable, (called the total sum of squares, SST= ( ), ca be dvded up to two parts: ) a part that s explaed by the regresso model (the sum of squares explaed by the regresso model, SSM) ad ) a part that s uexplaed by the model ( sum of squared resduals, SSR). Some of the math s gve Wooldrdge, chapter where he has used the followg algebra result: (A B ) (A A B B ) = Where ˆ ˆ ˆ A A B B. The step that s omtted s showg that ( ˆ ) uˆ =0. Ths bols dow pretty quckly to ˆ uˆ uˆ = ˆ u ˆ 0, so you just have to show that ˆ u ˆ = 0 order to prove the result. It s zero because OLS resduals are costructed to be ucorrelated wth predcted values of, sce resduals are chose so that they are ucorrelated wth (or orthogoal to) the depedet varables, they wll be orthogoal to ay lear combato of the depedet varables cludg the predcted value of. So a very reasoable measure of goodess of ft would be the explaed sum of squares (by the regresso) relatve to the total sum of squares. I fact, ths s just what R s: R SSM / SST SSR / SST It s the amout of varace the depedet varable that s explaed by the regresso model (amely, explaed by all of the depedet varables). R = s a perfect ft (determstc relatoshp), R = 0 dcates o relatoshp betwee the ad the slope regressors the data matrx. The dark sde of R : R ca oly rse whe aother varable s added as dcated above. We ca get to the same result more quckly by applyg the M orthogoal projecto operator (ths takes devatos from the mea, as seem above) to the stadard sample regresso fucto. Adjusted R-Squared ad a Measure of Ft Oe problem wth R s, t wll ever decrease whe aother varable s added to a regresso equato. To correct ths, the adjusted R (deoted as adj-r ) was created by Thel (oe of my teachers at the Uv of Chcago): ˆ' ˆ /( k) adj R ' M /( ) Wth the coecto betwee R ad adj-r beg adj R ( ) ( R ( k) )
The adjusted R may rse or fall wth the addto of aother depedet varable, so t s used to compare models wth the same depedet varable wth sample sze fxed, ad the umber of rght had sde varables chagg. [[[[ Do ou Wat a Whole Hershey Bar? Well, cosder the followg dagram for a smple regresso model whe aswerg these three multple choce questos:. The sum of the squared dffereces betwee where the sold bar ( ) hts the axs ad the dotted les ht the axs: a. Is the SST (sum of squares total) b. ( ) c. A estmate of the varace of (whe dvdg by -) d. all of the above. The sum of the square of the dstaces represeted by the short, thck les are a. SST (sum of squares total) b. SSE (explaed sum of squares) c. SSR (resdual sum of squares) d. oe of the above 3. The sum of squares metoed questo wll equal the sum of squares metoed whe ˆ, ˆ both must be equal to zero a. 0 b. ˆ s equal to zero, regardless of the tercept coeffcets value ˆ 0 c. s equal to zero, regardless of the slope coeffcet value d. whe me at BU are ecouraged to wear beards ]]]]] III. Matrx form of Assumptos I ad II: Assumpto I: = β + μ Assumpto II: ( ) exsts (so we ca solve for uque values of the least squares estmator, β ) Now: Assumpto III, the most mportat assumpto--the error term μ has a expected value of zero, for ay values of the depedet varables. E(μ ) = 0. Wth radom samplg, ths assumpto of zero codtoal mea for the error allows us to treat the s as f they were fxed umbers (fxed repeated samples), smplfyg the dervato of ubased estmated regresso parameters. Prove E( ˆ )=, wth just these frst three assumptos (remember: start wth defto, make substtutos that relate to model assumptos, smplfy, ad take expected values): 3
ˆ ( ' ) ' ( ' ) '( u) ( ' ) ' ( ' ) ( ' ) ' u Now take expectato of both sdes (codtog o ) to get E ( ˆ) ( ' ) ' E( u ) ( ' ) 0 Note o the E( )=0 assumpto. Ths s actually stroger tha eeded for most proofs ad theorems that we wll employ: What s eeded that E( μ) = 0 where we clude the tercept (a vector of oes) amog the depedet varables. Hece E( μ) = 0 mples the followg two results: ( ) 0 ad Cov( j, ) 0 j,..., k What extra does E( μ) = 0 gve us? Not oly that E(μ) = 0 but also that s ucorrelated wth ay fucto of (ot just ucorrelated wth that the Cov(, ) 0 assumpto gve us). Ths s equvalet to sayg that j we have fully accouted for the mpact of o, or equvaletly, that the fuctoal form of E( ) s properly specfed. IV. A Implcato of Model assumptos so far A. Omtted varable bas: True Model: = β + Zα + μ Here we let the omtted varable be deoted by Z. But we omt the Z varable (we assume that just oe varable s omtted ths example, so that s a scalar--a sgle, real umber but the otato also apples the more geeral case) from the specfcato, ad estmate stead: u Wll the omsso of Z from the estmated model, affect the bas of the ˆ vector? Gve the omsso of Z, the estmated coeffcets (whch we wll deote by ~ sce the model s msspecfed) wll be β = ( ) = ( ) (β + Zα + μ) = β + ( ) Zα + ( ) μ Now takg the expectato of both sdes (codtog o ) we have E(β ) = β + ( ) Zα + ( ) E(μ ) = β + ( ) Zα Hece, the bas from omttg a varable our model s equal to ( ) Z α Ths cossts of two parts. The frst correspods to the regresso of Z o (( ) Z); f there are four slope regressors, the the regresso of Z o would produce the partal correlato of each of those regressors wth respect to Z. Each of these partal correlatos would the be multpled by to get the bas preset for each elemet of ˆ. Smply stated, the bas would have the same sg as the product of the partal correlato tmes the omtted coeffcet. For example, f wages were regressed o o-fatal jury rsk (ad other worker s characterstcs), but omtted workers compesato beeft geerosty, there mght be a bas. The sg of the bas would equal: Sg(bas) = Sg(partal correlato betwee jury rsk ad beefts * ), where would be the coeffcet of jury beeft geerosty the wage equato. We would expect to be egatve ths cotext. Further, t may be that hgher beefts are pad where there s greater jury rsk, so the partal correlato would be postve. The et bas, uder these crcumstaces, would be egatve (( + )*( -) =( -)). If we omt beefts geerosty from the equato, but we would be less lkely to fd a compesatg wage sce our estmated coeffcet would be based dowwards. ' u Fally, we see that there are two cases where there s o bas: a. whe the correlato betwee Z ad s zero, ad b. whe Z has a coeffcet of 0 (so t ca safely be omtted ayway). 4
[[[TIME TO PLA: DO OU WANT A WHOLE HERSE BAR? :. To say that ˆ s ubased meas: a. t does t md who moves to ts eghborhood b. t always equals the populato parameter vector c. ts mea equals the populato parameter vector d. there are o omtted varables the specfcato. omtted varable bas: a. wll always be a problem whe we omt varables the model b. your cyclcal bout wth amesa caused you to forget your mother s brthday c. equals the regresso of the omtted varable o the cluded varables,, tmes the coeffcet of the omtted varable d. oe of the above 3. the proof of ubasedess for ˆ depeds o: a. the tercept havg a value of 0 b. the ormalty of the error terms c. the resduals summg to zero d. oe of the above]]]]]] Appedx o Frsch (or Partal Dervatve) Theorem A Pcture of the theorem three dmesos The partal dervatve theorem for regresso models says that for the followg two regressos: ˆ ˆ ˆ ad M ˆ ˆ M ` M, t s the case that ˆ ˆ ad ˆ ˆ M, where the varables the secod regresso are resduals from regressg ad o respectvely: ˆ M ad ˆ M. So frst, we show just the frst regresso: O Upper sde vew of regresso Top vew of the, plae Now we do the secod regresso (usg the resduals from the regressos) by tself (essetally decomposg ad terms of ) 5
O Upper sde vew of regresso Top vew of the, plae Ad fally we add them together so that equvaleces become apparet obvously ˆ ˆ ˆ ˆ M by specto of the graphs below, ad by the law of smlar tragles. I fact, that s all the partal dervatve (Frsch) theorem really s just a restatemet of the Pythagorea theorem ad elemetary propertes of geometry ( Eucldea space). O Upper sde vew of regresso Top vew of the, plae Geeral Results o the Frsch Theorem Apples wheever there are two or more regressors (rght had sde terms) our model. Cosder a arbtrary dvso of the regressors ad wrte them as 6) resduals Let s the projecto of to the space orthogoal to, or other words, the resduals of the regresso of oto (ths s aalogous to the decomposto of to the part explaed by the whole vector, =P + M, but t s a dfferet decomposto, as s lower dmeso tha, ad M s hgher dmeso tha M): exp part by laed of y to orthogoal part of y Smlarly, f has dmeso -x-z ad has dmeso -x-(k-z), are the resduals from regresso(s) of o ( are the depedet varable(s) ad are the depedet varables), the dmeso of beg -x- ad the dmeso of -x-z so that has dmeso -x-z. The cosder aother regresso: 7) resduals 6
The the Frsch Theorem says that a) the estmate of from equato (6) wll be detcal to the estmate of from equato (7), b) the resduals from these two equatos wll also be detcal, ad c) ˆ s the estmate of the effect of o, after the fluece of has bee factored out, d) ad as a result of a ad b, we have ' M ' M ' P or M M M P, or M M P M. M Applyg ths result to the case where s just the tercept (the vector of oes we deote as ), ad as all the regressors, cludg the tercept, the result (d) becomes ' M ' M ' P or M M M P M, or M P M, M that s, varato (where varato s calculato by devatg from ts mea usg a vector of oes, M) ca be explaed by varato due to varato the s (more specfcally, that part of the s ot assocated wth the costat, PM ) ad the varato that s uexplaed by all the s (cludg the tercept, M ). Equvaletly, the total sum of squares for (SST) equals the part of the varatos explaed by varatos the model (SSM, where the s are the model, ad M devates from ther meas, or takes varatos of ) ad the part left uexplaed, captured by the sum of squared resduals (SSR). APPENDI: Projecto Vew of Omtted Varable Bas. Z Z Estmato Model (leavg out Z): Projecto Vew. The ) ( Z ( Z) Z Now ad let Z So that But So That s, whe we omt Z, our estmators for the effect of s based by. The bas s, aga, 0 f 0 or f Z 0 whch would be the case f ad Z were orthogoal (ucorrected). 7