Multiple Regression Analysis

Size: px

Start display at page:

Download "Multiple Regression Analysis"

Marybeth Chapman
5 years ago
Views:

//04 CDS M Phl Old Least Squares (OLS) Vjayamohaa Plla N CDS M Phl Vjayamoha CDS M Phl Vjayamoha Multple Regresso

.. β x + u Multple Regresso Aalyss Geeral form of the multple lear regresso model: Defe colum vectors of observatos o

.. + β x + u Ths ca be expressed as βj j y x + u j 4 Multple Regresso Aalyss Multple Regresso Aalyss y β + β x + β x +.

1 //04 CDS M Phl Old Least Squares (OLS) Vjayamohaa Plla N CDS M Phl Vjayamoha CDS M Phl Vjayamoha Multple Regresso Aalyss y β 0 + β x + β x +... β x + u Multple Regresso Aalyss Geeral form of the multple lear regresso model: Defe colum vectors of observatos o y ad K varables. y f,..., x ) + u ( x, x y β + β x + β x β x + u Ths ca be expressed as βj j y x + u j 4 Multple Regresso Aalyss Multple Regresso Aalyss y β + β x + β x β x + u,, y + βx + βx + βx βx u y + βx + βx + βx βx u y + βx + βx + βx βx u. y β x + β x + β x β x + u y x y x y y x x x... x Xβ + u x s a colum of oes: x β u x β + u x β u K varables ad costat [ x L x ] T [ L ] T 5 6

//04 March 04 The Classcal Assumptos A. Learty Learty of the parameters (ad dsturbace) Sometmes models whch appear to be o-lear ca be estmated usg the least-squares squares procedure.

2 //04 March 04 The Classcal Assumptos A. Learty Learty of the parameters (ad dsturbace) Sometmes models whch appear to be o-lear ca be estmated usg the least-squares squares procedure. y exp β β x exp u l(y) β + β l(x ) + u Vjayamoha: CDS MPhl: 7 The Classcal Assumptos A. E(u) 0 The dsturbace term has a zero expectato u u E(u) E M u 0 0 M 0 8 The Classcal Assumptos The Classcal Assumptos A. Nostochastc Regressors X s a o-stochastc x matrx. That s, t cossts of a set of fxed umbers. What does Nostochastc Mea? Assumpto A comes two forms: a wea verso ad a strog verso. Strog Verso: The explaatory varables should be o - stochastc. Wea Verso: The explaatory varables are radom but dstrbuted depedetly of the error term. 9 0 The Classcal Assumptos The Classcal Assumptos: Homoscedastcty What does Nostochastc Mea? Varace The term ostochastc essetally meas that the varables are determed outsde the cotext of the regresso (ths s aother reaso why we use the term depedet whe referrg to explaatory varables)

3 //04 The Classcal Assumptos The Classcal Assumptos: Sphercal dsturbaces A4. Sphercal Dsturbaces Var(u) E(u) T E(u u ) E(uu ) M E(u u ) T E(uu ) σ I E(uu ) E(u) M E(u u ) L E(u u) L E(uu) O M L E(u ) σ 0 M 0 0 σ M 0 L L O L 0 0 M σ T Var(u) E(uu ) σ I E(u ) E(uu ) L E(uu ) T E(u u ) E(u ) E(u u ) E(uu ) L M M O M E(u u ) E(u u ) L E(u ) Sphercal dsturbaces () ad Var(u) E(u ) σ,..., σ 0 0 σ M M 0 0 L 0 L 0 O M L σ σ I homosedastcty σ I () Cov(u,u j) E(u,u j) 0 j No autocorrelato 4 The Classcal Assumptos: Homoscedastcty The Classcal Assumptos: Homoscedastcty f(y x) y f(y x).. E(y x) β 0 + β x Homoscedastcty... E(y x) β0 + β x Heteroscedastcty x x x x x x x 5 6 The Classcal Assumptos The Classcal Assumptos A5. Idetfablty A5. Idetfablty No exact lear relatoshps amog the varables (o perfect multcollearty). No exact lear relatoshps amog the varables (o perfect multcollearty). Specfcally, X s x K wth ra K. (X has full (colum) ra) the colums of X are learly depedet. Implct wth ths assumpto are the requremets of more observatos tha varables (mcroumerosty) ad suffcet varablty the values of the regressors. No-sgular X 7 8

4 //04 The Classcal Assumptos A6. Normalty Fal assumpto: the dsturbaces are ormally dstrbuted. u ~ N (0, σ I ) useful for the purposes of statstcal ferece The Classcal / Old / Ordary Least squares (OLS) Sample couterpart of the -varable regresso model: y X + û Where βˆ ad û are the sample couterparts of β ad u. OLS ams to mmse the dfferece betwee a observed value of y ad ts predcted value. but ot ecessary for aalysg the propertes of the estmators. the error û y ŷ be the least. ŷ X 9 0 The Classcal / Old / Ordary Least squares (OLS) The Classcal / Old / Ordary Least squares (OLS) Specfcally, the problem s to fd a estmator that mmses the error sum of squares: û û (y ŷ)(y ŷ) y y + ˆXXˆ β β ˆXy β A ecessary codto for a mmum s that the frstorder codtos equal zero. ûû XXˆ β Xy 0 β ˆ Therefore, rearrage to gve the ormal equato ( XX)ˆ β Xy ˆ (XX) β Xy The Classcal / Old / Ordary Least squares (OLS) The Classcal / Old / Ordary Least squares (OLS) ˆ β (XX) Xy For a -varable model, ˆ Cov(X, y) β ; Var(X) > 0 Var(X) X T X s vertble provded X has full ra. (Why?) ûû XX β β ˆ ˆ X T X s postve defte for a mmum. 4 4

5 //04 Statstcal Propertes of Least Squares Estmator Mea of βˆ (XX) Xy β + (XX) Xu E (ˆ) β β Property of ubasedess Statstcal Propertes of Least Squares Estmator Varace of βˆ Var (ˆ) β E(ˆ [ β β)( β) ] Notg that ˆ β β (XX) Xu the Var(ˆ) β E[(XX) XuuX(XX) ] Var(ˆ) β σ (XX) [ey assumptos: E(u) 0 ad o-stochastc regressors] 5 [ey assumptos: Var(u) E(u σ ad ) Cov(u,u j) E(u,u j) 0 for all, j. 6 Statstcal Propertes of Least Squares Estmator Varace of βˆ Var(ˆ) β σ (XX) I the scalar case Var(ˆ β) Mmum Varace (best) property σ x Statstcal Propertes of Least Squares Estmator Ubased Estmator of σ Var(ˆ) β σ (XX) For statstcal ferece we requre a estmate of Varace of βˆ ad therefore, σ. A ubased estmator of σ s: û ˆσ s Its square root the stadard error of the regresso 7 8 ( y y) ( ŷ y) û s s OLS: Goodess of Ft What proporto of the total varato of y s accouted for by the varato X? I terms of sums of squares TSS RSS + ESS ( y y) ( ŷ y) + the totalsumof squares(tss) the(explaed) regresso sum of squares(rss) s the errorsumof squares(ess) û 9 OLS: Measure of Goodess of Ft Coeffcet of determato: or ESS R TSS RSS R TSS R ever decreases whe a ew X varable s added to the model Ths ca be a dsadvatage whe comparg models What s the et effect of addg a ew varable? 0 5

6 //04 OLS: Measure of Goodess of Ft What s the et effect of addg a ew varable? We lose a degree of freedom whe a ew X varable s added Dd the ew X varable add eough explaatory power to offset the loss of oe degree of freedom? OLS: Measure of Goodess of Ft Adjusted R Shows the proporto of varato Y explaed by all X varables adjusted for the umber of X varables used R adj ESS/( ) TSS/( ) ( R ) (where sample sze, umber of parameters) OLS: Measure of Goodess of Ft OLS: Measure of Goodess of Ft R adj Adjusted R ESS/( ) ( R ) TSS/( ) (where sample sze, umber of parameters) Pealze excessve use of umportat depedet varables Smaller tha r Useful comparg amog models The relatoshp betwee R ad R R ( R ) + R Two other measures of ft are the Schwartz crtero (SC) ESS SC l + l ad the Aae formato crtero (AIC) ESS AIC l + 4 we ow that the expected value s I order to mae meagful ferece the varable must be ormally dstrbuted. E (ˆ) β β Oe of the assumptos troduced above was: Ad we ow that a sutable estmate of the varace of the slope parameter s Var(ˆ) β σ (XX) Var(ˆ) β s (XX) u~n(0, σ I ) Ths meas that u has a multvarate ormal dstrbuto. û ˆ s σ (If t s ot, the must be large.e. becomes ormally dstrbuted by vrtue of the cetral lmt theorem

7 //04 Ths mples that the samplg error ( βˆ - β) s related to u as follows: ˆ β (XX) β Xy β (XX) X(Xβ + u) β (XX) Xu I hypothess testg we are cocered about testg whether a partcular fdg s compatble wth some stated hypothess or ot. A commo hypothess test s formulated as whch mples that the samplg error s also multvarate ormally dstrbuted: (ˆ β β)~n(0, σ (XX) ) H 0 : β H : β β β 7 8 Usg our earler result, f we were loog at the th dagoal elemet of (X T X) - we obta ˆ ( β β)~n(0, σ ((XX) ) ) where ((XX) ) s the (,) elemet of (X T X) - If we do ot ow σ, t s atural to replace t wth the estmate s. Ths geerates the t-rato If we defe the rato z by dvdg z β β σ ((XX) ) where z ~N(0,) by ts stadard devato ˆ β β β t T s ((X X) ) SE(ˆ β) where SE( βˆ ) deotes the stadard error of βˆ. Ths s dstrbuted as t c α/;(-) A Example: Ivestmet Fucto A Example: Ivestmet Fucto Theory of the behavour of vestors: They care oly about real terest rates. Hypothess: Real I f(real gdp, a terest rate (the 90-day T-bll rate), flato (chage the log of CPI), real dsposable persoal come, tred) Equal creases terest rates ad rate of flato would have o depedet effect o vestmet. The test statstc s: So the Ho : β + β 0. ˆ + β 0 t SE(ˆ β ˆ + β) 4 4 7

8 //04 A Example: Ivestmet Fucto Testg Lear Restrctos As well as testg restrctos o dvdual regresso coeffcets we may wsh to test lear combatos of them. For example SE(.) [ (.78 x 0 6 )] / t % crtcal value: t(0 5).96 (a) (b) (c) H0 : β + β H0 : H β : β β 4... β 0 β β Testg Lear Restrctos: The F Test The t-dstrbuto ca be used to test a sgle ull hypothess If we wat to coduct a jot test the we ca o loger use the t dstrbuto For example, suppose we wat to test whether all the explaatory varables the modelare sgfcatly dfferet from zero Testg Lear Restrctos: The F Test F-Test of Etre Equato ( Testg the Jot Sgfcace of the Explaatory Varables ) H 0 : β β β 0 Equvaletly: H 0 : R 0 H : H 0 ot true (at least oe of the β s ozero) Caot say: f the coeffcets are dvdually sgfcat ths meas they must be jotly sgfcat Testg Lear Restrctos: The F Test F-Test of Etre Equato ( Testg the Jot Sgfcace of the Explaatory Varables ) H 0 : β β β 0 Equvaletly: H 0 : R 0 H : H 0 ot true (at least oe of the β s ozero) RSS F ESS or R F R 47 Testg Lear Restrctos: Multple Restrcto F Test Ths tme suppose we wsh to test whether a subset of regresso coeffcets are zero. I ths case y + X + û [ X X ] + û H 0 : β 0 X H : H 0 ot true 48 8

//04 y Testg Lear Restrctos: Multple Restrcto F Test [ X X ] + û ˆ H β 0 : β 0 H : H 0 ot true X + X + û To fd the chage the ft of a multple regresso whe a addtoal varable x s added to a model that

9 //04 y Testg Lear Restrctos: Multple Restrcto F Test [ X X ] + û ˆ H β 0 : β 0 H : H 0 ot true X + X + û To fd the chage the ft of a multple regresso whe a addtoal varable x s added to a model that already cotas K varables: J lear restrcto Testg Lear Restrctos: Multple Restrcto F Test Thus the urestrcted regresso s: y [ X X ] y X Ad the restrcted regresso: + û + X + û y [ X X ] y X H 0 : β 0 H : H 0 ot true + û β ˆ + û 0 * * Testg Lear Restrctos: Multple Restrcto F Test The resduals from the restrcted regresso: û y X * whereas the resduals from the urestrcted regresso: û y X X Essetally we wsh to see whether the reducto the ESS s 'large eough' to suggest that X s sgfcat. 5 Testg Lear Restrctos: Multple Restrcto F Test Resduals from the restrcted regresso: û* y Xˆ β Resduals from the urestrcted regresso: û y X The test statstc s T T (û* û* û û)/j F J ; K T û û/( ) or (R R* )/J F ( R )/( ) X 5 Testg Lear Restrctos: Multple Restrcto F Test A Example: Producto Fucto A Example: Producto Fucto Cobb-Douglas: ly α + β ll + βlk + u Traslog: Urestrcted model: K 6 ly α + β ll + βlk + β(ll) + β4(lk) + β5lllk + u Traslog fucto relaxes the CD assumpto of a utary elastcty of substtuto CD obtaed by the restrcto: β β β J

10 //04 Estmated producto Fuctos A Example: Producto Fucto CD or Traslog? The F-statstc for the hypothess of CD model: Ho: CD model s approprate T T (û* û* û û)/j F T û û/( ) ( )/ F / Crtcal value: F(,).07 Cocluso? March 04 Vjayamoha: CDS MPhl: March 04 A Example: Producto Fucto CD fucto: Costat Returs to Scale? Hypothess of costat returs to scale: Equvalet to a restrcto that the two coeffcets of CD fucto sum to uty Ho: β + β F-test wth J ad K ( ) F 0.57 t (0.0096) Crtcal value: F(,4) 4.6 Vjayamoha: CDS MPhl: Cocluso? 57 A Example: Producto Fucto CD fucto: Costat Returs to Scale? Hypothess of costat returs to scale: Ho: β + β Ca have t-test: ˆ + β t SE(ˆ β ˆ + β ) ( ) t 0.40 / [ (0.0096)] (F / ) Crtcal value: rule of thumb Cocluso? 58 To test structural chage tme seres data Savgs Icome Savgs ad persoal dsposable come (bllos of dollars) US,

//04 Savgs 50 00 50 00 500 000 500 000 500 Icome 970 98 Savgs 50 00 50 00 Pot of structural brea: 98 (assumed) 000 000 4000 5000 6000 Icome Savgs 50 00 50 00 50 00 98 995 6 000 000 000 4000 5000

The pooled regresso restrcted regresso, obtaed uder the restrctos that a a ( a 0 ) ad b b ( b 0 ).. Get the restrcted ESS from t (ESS R ): ( û* û* ) 6 Steps.

11 //04 Savgs Icome Savgs Pot of structural brea: 98 (assumed) Icome Savgs Icome Steps. Ru the two sub-perod regressos ad the fullperod regresso Full-perod: S t a 0 + b 0 Y t 6 ( + ) Sub-perod : S t a + b Y t Sub-perod : S t a + b Y t 4. The pooled regresso restrcted regresso, obtaed uder the restrctos that a a ( a 0 ) ad b b ( b 0 ).. Get the restrcted ESS from t (ESS R ): ( û* û* ) 6 Steps. Ru the two sub-perod regressos ad the fullperod regresso Full-perod: S t a 0 + b 0 Y t 6 ( + ) Sub-perod : S t a + b Y t Sub-perod : S t a + b Y t 4 4. The sub-perod regressos urestrcted regressos. 5. Get the urestrcted ESS from them: ( ûû) : ESS UR ESS + ESS. Steps Sub-perod : S t a + b Y t Sub-perod : S t a + b Y t 4 4. The sub-perod regressos urestrcted regressos. 5. Get the urestrcted ESS from them: ( ûû) : ESS UR ESS + ESS. T T (û* û* û û)/ F T j; ( ) + ( ) û û/( ) 6 64 Pooled Regresso Result: Sub-perod : Restrcted ESS: ( û* û* ) 48. Urestrcted ESS

//04 Sub-perod : 98 995 T T (û* û* û û)/ F T û û/( ) j; ( ) + ( ) (48. 790.54)/ F 0.69 790.54/ Crtcal value: F(,) at % α 5.

12 //04 Sub-perod : T T (û* û* û û)/ F T û û/( ) j; ( ) + ( ) ( )/ F / Crtcal value: F(,) at % α 5.7 Urestrcted ESS Urestrcted ESS ESS + ESS ( ûû) H0 a a a 0 ad b b b 0 : Parameter stablty Cocluso? 67 68

Ordinary Least Squares Regression. Simple Regression. Algebra and Assumptions.

Ordinary Least Squares Regression. Simple Regression. Algebra and Assumptions. Ordary Least Squares egresso. Smple egresso. Algebra ad Assumptos. I ths part of the course we are gog to study a techque for aalysg the lear relatoshp betwee two varables Y ad X. We have pars of observatos