Chapter 3 Multiple Linear Regression Model

Transcription

1 Chapter 3 Multple Lear Regresso Model We cosder the problem of regresso whe study varable depeds o more tha oe explaatory or depedet varables, called as multple lear regresso model. Ths model geeralzes the smple lear regresso two ways. It allows the mea fucto E( y ) to deped o more tha oe explaatory varables ad to have shapes other tha straght les, although t does ot allow for arbtrary shapes. The lear model: Let y deotes the depedet (or study) varable that s learly related to depedet (or explaatory) varables X, X,..., X through the parameters,,..., ad we wrte y X X... X. Ths s called as the multple lear regresso model. The parameters,,..., are the regresso coeffcets assocated wth X, X,..., X respectvely ad s the radom error compoet reflectg the dfferece betwee the observed ad ftted lear relatoshp. There ca be varous reasos for such dfferece, e.g., ot effect of those varables ot cluded the model, radom factors whch ca ot be accouted the model etc. Note that the th regresso coeffcet represets the expected chage y per ut chage th depedet varable X. Assumg E( ), E( y). X Lear model: A model s sad to be lear whe t s lear parameters. I such a case y (or equvaletly E( y) ) X should ot deped o ay 's. For example ) y X s a lear model as t s lear s parameter. ) y X ca be wrtte as log y log log X y x Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

2 ) whch s lear s parameter ad, but olear s varables y log y, x log x. So t s a lear model. y X X s lear parameters, ad but t s olear s varables X. So t s a lear model v) y X s olear parameters ad varables both. So t s a olear model. v) y X v) s olear parameters ad varables both. So t s a olear model. y X X X 3 3 s a cubc polyomal model whch ca be wrtte as y X X X 3 3 whch s lear parameters,,, 3 ad lear varables a lear model. X X, X X, X X. So t s 3 3 Example: The come ad educato of a perso are related. It s expected that, o a average, hgher level of educato provdes hgher come. So a smple lear regresso model ca be expressed as come educato. Not that reflects the chage s come wth respect to per ut chage s educato ad reflects the come whe educato s zero as t s expected that eve a llterate perso ca also have some come. Further ths model eglects that most people have hgher come whe they are older tha whe they are youg, regardless of educato. So wll over-state the margal mpact of educato. If age ad educato are postvely correlated, the the regresso model wll assocate all the observed crease come wth a crease educato. So better model s come educato age. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

3 Ofte t s observed that the come teds to rse less rapdly the later earg years tha s early years. To accommodate such possblty, we mght exted the model to come educato age age 3 Ths s how we proceed for regresso modelg real lfe stuato. Oe eeds to cosder the expermetal codto ad the pheomeo before tag the decso o how may, why ad how to choose the depedet ad depedet varables. Model set up: Let a expermet be coducted tmes ad the data s obtaed as follows: Observato umber Respose y y y y Explaatory varables X X X x x x x x x x x x Assumg that the model s y XX... X, the -tuples of observatos are also assumed to follow the same model. Thus they satsfy y x x... x y x x... x y x x... x. These equatos ca be wrtte as y x x x y x x x y x x x or y X. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 3

4 I geeral, the model wth explaatory varables ca be expressed as y X where y ( y, y,..., y )' s a vector of observato o study varable, x x x x x x X x x x s a matrx of observatos o each of the explaatory varables, (,,..., )' s a vector of regresso coeffcets ad (,,..., )' s a vector of radom error compoets or dsturbace term. If tercept term s preset, tae frst colum of X to be (,,,). Assumptos multple lear regresso model Some assumptos are eeded the model y X for drawg the statstcal fereces. The followg assumptos are made: () E( ) () E ( ') I () Ra( X ) (v) (v) X s a o-stochastc matrx ~ (, ) N I. These assumptos are used to study the statstcal propertes of estmator of regresso coeffcets. The followg assumpto s requred to study partcularly the large sample propertes of the estmators (v) X ' X lm exsts ad s a o-stochastc ad osgular matrx (wth fte elemets). The explaatory varables ca also be stochastc some cases. We assume that X s o-stochastc uless stated separately. We cosder the problems of estmato ad testg of hypothess o regresso coeffcet vector uder the stated assumpto. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 4

5 Estmato of parameters: A geeral procedure for the estmato of regresso coeffcet vector s to mmze M( ) M( y x x... x ) for a sutably chose fucto M. Some examples of choce of M are M ( x) x M ( x) x M ( x) x p, geeral. We cosder the prcple of least square whch s related to estmato for the estmato of parameters. M ( x) x ad method of maxmum lelhood Prcple of ordary least squares (OLS) Let B be the set of all possble vectors. If there s o further formato, the B s -dmesoal real Eucldea space. The obect s to fd a vector b' ( b, b,..., b ) from B that mmzes the sum of squared devatos of ' s,.e., ( ) ' ( )'( ) S y X y X for gve y ad X. A mmum wll always exst as S( ) s a real valued, covex ad dfferetable fucto. Wrte S( ) y' y' X ' X ' X ' y. Dfferetate S( ) wth respect to S( ) X ' X X ' y S( ) X ' X (atleast o-egatve defte). The ormal equato s S( ) X ' Xb X ' y Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 5

6 where the followg result s used: Result: If f ( z) Z' AZ s a quadratc form, Z s a m vector ad A s ay m m symmetrc matrx the F( z) Az. z Sce t s assumed that ra ( X) ormal equato s ( ' ) ' b X X X y whch s termed as ordary least squares estmator (OLSE) of. (full ra), the X ' X s postve defte ad uque soluto of Sce S( ) s at least o-egatve defte, so b mmze S( ). I case, X s ot of full ra, the b ( X ' X) X ' y I ( X ' X) X ' X where ( X ' X ) s the geeralzed verse of X ' X ad s a arbtrary vector. The geeralzed verse ( X ' X ) of X ' X satsfes Theorem: X ' X( X ' X) X ' X X ' X X( X ' X) X ' X X X ' X( X ' X) X ' X ' () Let ŷ Xb be the emprcal predctor of y. The ŷ has the same value for all solutos b of X ' Xb X ' y. () S( ) attas the mmum for ay soluto of X ' Xb X ' y. Proof: () Let b be ay member b ( X ' X) X ' y I ( X ' X) X ' X. Sce X ( X ' X) X ' X X, so the Xb X ( X ' X ) X ' y X I ( X ' X ) X ' X = X ( X ' X) X ' y whch s depedet of. Ths mples that ŷ has same value for all soluto b of X ' Xb X ' y. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 6

7 () Note that for ay, S( ) yxb X( b) yxb X( b) ( yxb)( yxb) ( b) X ' X( b) ( b) X( yxb) ( yxb)( yxb) ( b) X ' X( b) (Usg X ' Xb X ' y) ( yxb)( yxb) S( b) y' y y' Xbb' X ' Xb y' yb' X ' Xb y' y yˆ' yˆ. Ftted values: If ˆ s ay estmator of for the model y X, the the ftted values are defed as ŷ Xˆ where ˆ s ay estmator of. I case of ˆ b, yˆ Xb X X X Hy X y ( ' ) ' where H X X X X ( ' ) ' s termed as Hat matrx whch s () symmetrc () dempotet (.e., HH H) ad () tr H tr X X X X tr X X X X tr I. ( ) ' ' ( ' ) Resduals The dfferece betwee the observed ad ftted values of study varable s called as resdual. It s deoted as e y ~ yˆ y yˆ yxb yhy ( I H) y Hy where H I H. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 7

8 Note that () () H s a symmetrc matrx H s a dempotet matrx,.e., HH ( I H)( I H) ( I H) H ad () trh tri trh ( ). Propertes of OLSE () Estmato error: The estmato error of b s b ( X ' X) X ' y ( X ' X) X '( X ) ( X ' X) X ' () Bas Sce X s assumed to be ostochastc ad E( ) Eb X X X E. ( ) ( ' ) ' ( ) Thus OLSE s a ubased estmator of. () Covarace matrx The covarace matrx of b s Vb ( ) Eb ( )( b )' E ( X ' X) X ' ' X( X ' X) ( X ' X) X ' E( ') X( X ' X) ( X ' X) X ' IX( X ' X) ( X ' X). Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 8

9 (v) Varace The varace of b ca be obtaed as the sum of varaces of all b, b,..., b whch s the trace of covarace matrx of b. Thus Var( b) tr V ( b) Eb ( ) Var( b ). Estmato of The least squares crtero ca ot be used to estmate E( ), so we attempt wth resduals e to estmate e y yˆ yx( X ' X) X ' y [ I X( X ' X) X '] y Hy. Cosder the resdual sum of squares SS res e ee ' ( yxb)'( yxb) y'( I H)( I H) y y'( I H) y yhy '. because as follows: does ot appear S( ). Sce Also SS ( y Xb)'( y Xb) re s y' y b' X ' yb' X ' Xb y' yb' X ' y (Usg X ' Xb X ' y) SS re s y ' Hy ( X )' H( X ) ' H (Usg HX ) Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 9

10 Sce ~ N(, I). So y ~ N( X, I). Hece Thus yhy ' ~ ( ). EyHy [ ' ] ( ) or yhy ' E or E MS res where MS res SS res s the mea sum of squares due to resdual. Thus a ubased estmator of ˆ MS s (say) res s whch s a model depedet estmator. Varace of ŷ The varace of ŷ s V( yˆ ) V( Xb) XV ( b) X ' X ( X ' X) X ' H. Gauss-Marov Theorem: The ordary least squares estmator (OLSE) s the best lear ubased estmator (BLUE) of. Proof: The OLSE of s b ( X ' X) X ' y whch s a lear fucto of y. Cosder the arbtrary lear estmator b a' y of lear parametrc fucto ' where the elemets of a are arbtrary costats. The for b, E( b ) E( a' y) a' X Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

11 ad so b s a ubased estmator of ' whe ( ) ' ' Eb ax a' X '. Sce we wsh to cosder oly those estmators that are lear ad ubased, so we restrct ourselves to those estmators for whch a' X '. Further Var( a ' y) a ' Var( y) a a ' a Var( ' b) ' Var( b) axx ' ( ' X) Xa '. Cosder Var( a ' y) Var( ' b) a ' a a ' X ( X ' X ) X ' a a' I X( X ' X) X ' a a'( I H) a. Sce ( I H ) s a postve sem-defte matrx, so Var( a ' y) Var( ' b). Ths reveals that f b s ay lear ubased estmator the ts varace must be o smaller tha that of b. Cosequetly b s the best lear ubased estmator, where best refers to the fact that b s effcet wth the class of lear ad ubased estmators. Maxmum lelhood estmato: I the model y X, t s assumed that the errors are ormally ad depedetly dstrbuted wth costat varace or ~ N(, I). The ormal desty fucto for the errors s f ( ) exp,,...,.. The lelhood fucto s the ot desty of,,..., gve as Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

12 L (, ) f( ) exp / ( ) exp ' / ( ) exp ( y X )'( y X ). / ( ) Sce the log trasformato s mootoc, so we maxmze l L(, ) stead of L(, ). l L(, ) l( ) ( yx )'( y X ). The maxmum lelhood estmators (m.l.e.) of ad are obtaed by equatg the frst order dervatves of l L(, ) wth respect to ad to zero as follows: l L(, ) X '( yx ) ( yx )'( yx ). ( ) l L(, ) The lelhood equatos are gve by X ' X X ' y ( )'( ). yx yx Sce ra( X ), so that the uque mle of ad are obtaed as ( X ' X) X ' y ( )'( ). y X y X Further to verfy that these values maxmze the lelhood fucto, we fd l L(, ) X ' X l L(, ) 4 6 ( yx )'( yx ) ( ) l L(, ) X '( yx ). 4 Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

13 Thus the Hessa matrx of secod order partal dervatves of l L(, ) wth respect to ad s l L(, ) l L(, ) l L(, ) l L(, ) ( ) whch s egatve defte at ad these values.. Ths esures that the lelhood fucto s maxmzed at Comparg wth OLSEs, we fd that () OLSE ad m.l.e. of are same. So m.l.e. of s also a ubased estmator of. () OLSE of s s whch s related to m.l.e. of as s. So m.l.e. of s a based estmator of. Cosstecy of estmators () Cosstecy of b : X ' X Uder the assumpto that lm elemets), we have X ' X lm. lm Vb ( ) lm exsts as a ostochastc ad osgular matrx (wth fte Ths mples that OLSE coverges to quadratc mea. Thus OLSE s a cosstet estmator of. Ths holds true for maxmum lelhood estmators also. Same cocluso ca also be proved usg the cocept of covergece probablty. A estmator ˆ coverges to probablty f lm P ˆ for ay ad s deoted as plm( ˆ ). Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 3

14 The cosstecy of OLSE ca be obtaed uder the weaer assumpto that X ' X plm. exsts ad s a osgular ad ostochastc matrx such that X ' plm. Sce b ( X ' X) X ' So X ' X X '. X ' X X ' plm( b ) plm plm.. Thus b s a cosstet estmator of. Same s true for m.l.e. also. () Cosstecy of s Now we loo at the cosstecy of s e' e ' H s as a estmate of ' ' ( ' ) ' X X X X ' ' X X ' X X '. as Note that ' cossts of ad {,,,..., } s a sequece of depedetly ad detcally dstrbuted radom varables wth mea. Usg the law of large umbers Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 4

15 ' plm ' X X ' X X ' 'X X ' X X ' plm plm plm plm plm( s ) ( )... Thus s s a cosstet estmator of. Same hold true for m.l.e. also. Cramer-Rao lower boud Let (, )'. Assume that both ad are uow. If boud for ˆ s grater tha or equal to the matrx verse of The E( ˆ ) l L( ) I( ) E ' l L(, ) l L(, ) E E l L(, ) l L(, ) E E ( ) X ' X X '( y X ) E E 4 ( yx)' X ( yx)'( y X ) E E X ' X. 4 ( X ' X) I( ) 4 s the Cramer-Rao lower boud matrx of ad., the the Cramer-Rao lower Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 5

16 The covarace matrx of OLSEs of ad OLS ( X ' X) 4 s whch meas that the Cramer-Rao have boud s attaed for the covarace of b but ot for s. Stadardzed regresso coeffcets: Usually t s dffcult to compare the regresso coeffcets because the magtude of ˆ reflects the uts of measuremet of th explaatory varable. X For example, the followg ftted regresso model yˆ 5 X X, y s measured lters, X s lters ad X mlllters. Although ˆ ˆ but effect of both explaatory varables s detcal. Oe lter chage ether X ad X whe other varable s held fxed produces the same chage s ŷ. Sometmes t s helpful to wor wth scaled explaatory varables ad study varable that produces dmesoless regresso coeffcets. These dmesoless regresso coeffcets are called as stadardzed regresso coeffcets. There are two popular approaches for scalg whch gves stadardzed regresso coeffcets. We dscuss them as follows:. Ut ormal scalg: Employ ut ormal scalg to each explaatory varable ad study varable. So defe x x z,,,...,,,,..., s y y y s y where s x x ( ) Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 6

17 ad s y y y ( ) are the sample varaces of th explaatory varable ad study varable respectvely. All scaled explaatory varable ad scaled study varable have mea zero ad sample varace uty,.e., usg these ew varables, the regresso model becomes y z z... z,,,...,. Such ceterg removes the tercept term from the model. The least squares estmate of (,,..., )' s ˆ ( ' ) ' Z Z Z y. Ths scalg has a smlarty to stadardzg a ormal radom varable,.e., observato mus ts mea ad dvded by ts stadard devato. So t s called as a ut ormal scalg.. Ut legth scalg: I ut legth scalg, defe x x,,,..., ;,,..., S y / y y SS / T where ( ) s the corrected sum of squares for S x x th explaatory varables X ad T T ( ) s the total sum of squares. I ths scalg, each ew explaatory varable S SS y y W has mea ad legth ( ). I terms of these varables, the regresso model s o y...,,,...,. The least squares estmate of regresso coeffcet (,,..., )' s ˆ ( ' ) ' WW W y. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 7

18 I such a case, the matrx where r r3 r r r3 r WW ' r3 r3 r 3 r r r3 r u S ( SS ) ( x x )( x x ) u u ( SS ) / / WW ' s the form of correlato matrx,.e., s the smple correlato coeffcet betwee the explaatory varables X ad X. Smlarly where o W ' y ( r, r,..., r )' r y u Sy ( S SS ) y y y ( x x )( y y) u u ( S SS ) T / T s the smple correlato coeffcet betwee / th explaatory varable X ad study varable y. Note that t s customary to refer r ad r as correlato coeffcet though X ' s are ot radom varable. y If ut ormal scalg s used, the Z ' Z ( ) W ' W. So the estmates of regresso coeffcet ut ormal scalg (.e., ˆ) ad ut legth scalg (.e., ˆ) are detcal. So t does ot matter whch scalg s used, so ˆ ˆ. The regresso coeffcets obtaed after such scalg, vz., ˆ or ˆ usually called stadardzed regresso coeffcets. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 8

19 The relatoshp betwee the orgal ad stadardzed regresso coeffcets s ad / ˆ SST b,,,..., S b y b x where b s the OLSE of tercept term ad b are the OLSE of slope parameters. The model devato form The multple lear regresso model ca also be expressed the devato form. Frst all the data s expressed terms of devatos from sample mea. The estmato of regresso parameters s performed two steps: Frst step: Estmate the slope parameters. Secod step : Estmate the tercept term. The multple lear regresso model devato form s expressed as follows: Let A I ' where,,..., ' s a vector of each elemet uty. So The A. y y,,..., y y y ' y Ay yy y y, y y,..., y y '. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 9

20 Thus pre-multplcato of ay colum vector by A produces a vector showg those observatos devato form: Note that A '. ad A s symmetrc ad dempotet matrx. I the model y X, the OLSE of s ' b X X X ' y ad resdual vector s e y Xb. Note that Ae e. If the matrx s parttoed as X X X where X,,..., ' s vector wth all elemets uty, explaatory varables, 3,..., tercept term as X X X ad OLSE b b, b ' X s matrx of observatos of s sutably parttoed wth OLSE of b ad b as a vector of OLSEs assocated wth, 3,...,. The y X b X b e. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

21 Premultply by A, Ay AX b AX b Ae AX b e. Premultply by X gves X ' Ay X ' AX b X ' e X ' AX b. Sce A s symmetrc ad dempotet, AX ' Ay AX ' AX b.. Ths equato ca be compared wth the ormal equatos X ' y X ' Xb the model y X. Such a comparso yelds followg coclusos: b s the sub vector of OLSE. Ay s the study varables vector devato form. AX s the explaatory varable matrx devato form. Ths s ormal equato terms of devatos. Its soluto gves OLS of slope coeffcets as b AX ' AX AX ' Ay. The estmate of tercept term s obtaed the secod step as follows: Premultplyg y Xbe by ' gves ' y ' Xb ' e b b y X X3... X b b yb X b X... b X. 3 3 Now we expla varous sums of squares terms of ths model. The expresso of total sum of squares (TSS) remas same as earler ad s gve by TSS y ' Ay. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

22 Sce Ay AX b e y' Ay y' AX b y' e Xb e ' AX b y ' e ' X b X b e AX b X b X b e ' e b ' X ' AX b e' e TSS SS SS reg where sum of squares due to regresso s res SS b ' X ' AX b reg ad sum of squares due to resdual s SSres e' e. Testg of hypothess: There are several mportat questos whch ca be aswered through the test of hypothess cocerg the regresso coeffcets. For example. What s the overall adequacy of the model?. Whch specfc explaatory varables seems to be mportat? etc. I order the aswer such questos, we frst develop the test of hypothess for a geeral framewor, vz., geeral lear hypothess. The several tests of hypothess ca be derved as ts specal cases. So frst we dscuss the test of a geeral lear hypothess. Test of hypothess for H : R r We cosder a geeral lear hypothess that the parameters are cotaed a subspace of parameter space for whch R r, where R s ( J ) matrx of ow elemets ad r s a ( J ) vector of ow elemets. I geeral, the ull hypothess H : R r s termed as geeral lear hypothess ad Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur

23 H : R r s the alteratve hypothess. We assume that ra ( R) J,.e., full ra so that there s o lear depedece the hypothess. Some specal cases ad terestg example of H : R r are as follows: () H : Choose J, r, R [,,...,,,,...,] where occurs at the th posto s R. Ths partcular hypothess explas whether () H 3 4 H 3 4 : or : Choose J, r, R [,,,,,...,] X has ay effect o the lear model or ot. () H : or H :,... Choose J, r (,) ', R.... (v) H : 354 Choose J, r, R,,,5,... (v) H :... 3 J r (,,...,)' R I.... ( ) Ths partcular hypothess explas the goodess of ft. It tells whether has lear effect or ot ad are they of ay mportace. It also tests that X, X3,..., X have o fluece the determato of y. Here s excluded because ths volves addtoal mplcato that the mea level of y s zero. Our ma cocer s to ow whether the explaatory varables helps to expla the varato y aroud ts mea value or ot. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 3

24 We develop the lelhood rato test for H : R r. Lelhood rato test: The lelhood rato test statstc s max L(, y, X) L( ) max (,,, ) ˆ( ) L y X R r L where s the whole parametrc space ad s the sample space. ˆ If both the lelhood are maxmzed, oe costraed ad the other ucostraed, the the value of the ucostraed wll ot be smaller tha the value of the costraed. Hece. Frst we dscus the lelhood rato test for a smpler case whe R I ad r, e..,. Ths wll gve as better ad detaled uderstadg for the mor detals ad the we geeralze t for R r, geeral. Lelhood rato test for H : Let the ull hypothess related to vector s H : where s specfed by the vestgator. The elemets of ca tae o ay value, cludg zero. The cocered alteratve hypothess s H :. Sce ~ N(, I) y X y N X I,so ~ (, ). Thus the whole parametrc space ad sample space are ad respectvely gve by : (, ) :,,,,..., : (, ):,. The ucostraed lelhood uder. L(, y, X) exp ( y X)'( y X) / ( ). Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 4

25 Ths s maxmzed over whe ( X ' X) X ' y ( yx )'( yx ). where ad lelhood fucto. are the maxmum lelhood estmates of ad ˆ( ) max,, ) L L y X ( yx )'( yx ) exp ( yx )'( yx ) ( yx )'( y X ) / exp. / ( ) ( yx )'( yx ) The costraed lelhood uder s L L y X y X y X ( ) whch are the values maxmzg the ˆ( ) max (,,, ) / exp ( )'( ). Sce s ow, so the costraed lelhood fucto has a optmum varace estmator ( yx )'( y X ) / exp Lˆ( ) / ) /. ( ) ( yx)'( yx The lelhood rato s / exp( / ) / Lˆ( ) ( ) ( yx )'( yx ) L ˆ( ) / exp( / ) / ( ) ( yx )'( yx ) ( yx)'( yx) ( yx )'( yx ) / / / / / Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 5

26 where ( yx)'( yx) ( yx )'( yx ) s the rato the quadratc forms. Now we smplfy the umerator as follows: ( yx)'( yx) ( yx ) X( ) ( yx ) X( ) ( yx )'( yx ) ( )' X ' X( ). ( y X )'( y X ) y' I X( X ' X) X ' X( ) ( )' X ' X( ) Thus ( yx )'( yx ) ( )' X ' X( ) ( yx )'( yx ) ( )' X ' X( ) ( yx )'( yx ) ( )' X ' X( ) or ( yx )'( yx ) where. Dstrbuto of rato of quadratc forms Now we fd the dstrbuto of the quadratc forms volved s to fd the dstrbuto of as follows: ( yx )'( yx ) e' e y' I X( X ' X) X ' y yhy ' ( X )' H( X ) ' H (usg HX ) ( ) ˆ Result: If Z s a radom vector that s dstrbuted as N(, I ) ad A s ay symmetrc dempotet matrx of ra p the matrx of ra q, the Z' BZ ~ ( q) Z' AZ p ~ ( ). If B s aother symmetrc dempotet. If AB the Z ' AZ s dstrbuted depedetly of Z ' BZ. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 6

27 So usg ths result, we have yhy ' ( ) ˆ ~ ( ). Further, f H s true, the ad we have the umerator. Rewrtg the umerator, geeral, we have ( )' X ' X( ) ' X( X ' X) X ' X( X ' X) X ' ' X( X ' X) X ' ' H where H s a dempotet matrx wth ra. Thus usg ths result, we have ' H ' X '( X ' X) X ' ~ ( ). Furthermore, the product of the quadratc form matrces the umerator ( ' H ) ad deomator ( ' H ) of s ad hece the I X X X X X X X X X X X X X X X X X X X X ( ' ) ' ( ' ) ' ( ' ) ' ( ' ) ' ( ' ) ' radom varables umerator ad deomator of are depedet. Dvdg each of the radom varable by ther respectve degrees of freedom ( )' X ' X( ) ( ) ˆ ( )' X ' X( ) ˆ ( yx)'( yx) ( yx )'( yx ) ˆ ~ F (, ) uder H. Note that ( y X)'( y X) : Restrcted error sum of squares ( y X )'( y X ) : Urestrcted error sum of squares Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 7

28 Numerator : Dfferece betwee the restrcted ad urestrcted error sum of squares. The decso rule s to reect H : at level of sgfcace wheever F (, ) where F (, ) s the upper crtcal pots o the cetral F -dstrbuto wth ad degrees of freedom. Lelhood rato test for H : R r The same logc ad reasos used the developmet of lelhood rato test for H : ca be exteded to develop the lelhood rato test for H : R r as follows. (, ) :,,,,..., (, ):, R r,. Let ( X ' X) X ' y.. The ER ( ) R V( R ) ER( )( )' R' RV ( ) R ' RX ( ' X) R'. Sce ~ N, ( X ' X) so R ~ N R, R( X ' X) R' R r R R R N R X X R ( ) ~, ( ' ) '. There exsts a matrx Q such that ( ' ) ' ' R X X R QQ ad the ( ) (, ). Therefore uder QR b N I H : R r, so Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 8

29 ' ( R r)' QQ'( R r) ( R r)' R( X ' X) R' ( R r) = whch s obtaed as ( )' R' R( X ' X) R' R( ) ' X ( X ' X) R' R( X ' X) R' R( X ' X) X ' ~ ( J ). whch s the assocated degrees of freedom. X ( X ' X) R' R( X ' X) R' R( X ' X) X ' s a dempotet matrx ad ts trace s J Also, rrespectve of whether H s true or ot, e' e ( yx)'( yx) y' Hy ( ) ˆ ~ ( ). Moreover, the product of quadratc form matrces of ee ' ad R R X X R R s zero mplyg that both the quadratc forms are depedet. So ( )' ' ( ' ) ' ( ) terms of lelhood rato test statstc J ( ) ˆ R r)' R( X ' X) R' R r J ˆ ~ F( J, ) uder H. ( R r)' R( X ' X) R' ( R r) So the decso rule s to reect H wheever F ( J, ) freedom. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur where F ( J, ) s the upper crtcal pots o the cetral F dstrbuto wth J ad ( ) degrees of 9

30 Test of sgfcace of regresso (Aalyss of varace) If we set R[ I ], r, the the hypothess H : R r reduces to the followg ull hypothess: H 3 :... agast the alteratve hypothess H : for at least oe,3,..., Ths hypothess determes f there s a lear relatoshp betwee y ad ay set of the explaatory varables X, X3,..., X. Notce that X correspods to the tercept term the model ad hece x for all,,...,. Ths s a overall or global test of model adequacy. Reecto of the ull hypothess dcates that at least oe of the explaatory varables amog X, X3,..., X. cotrbutes sgfcatly to the model. Ths s called as aalyss of varace. Sce ~ N(, I), so y ~ N( X, I) b X X X y N X X ( ' ) ' ~, ( ' ). SS res Also ˆ ( y yˆ)'( y yˆ) y' I X( X ' X) X ' y yhy ' yy ' bx ' ' y. Sce - ( X ' X) X ' H, so b ad ˆ are depedetly dstrbuted. Sce yhy ' ' H ad H s a dempotet matrx, so SS ~, re s ( ).e., cetral dstrbuto wth ( ) degrees of freedom. Partto X [ X, X ] where the submatrx X, X,..., X X cotas the explaatory varables 3 ad partto [, ] where the subvector cotas the regresso coeffcets,,...,. 3 Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 3

31 Now partto the total sum of squares due to ysas ' SS T y ' Ay SS reg SS res where SS b ' X ' AX b s the sum of squares due to regresso ad the sum of squares due to resduals s gve by reg SS ( y Xb)'( y Xb) res yhy ' SS T SS re g. Further SSre g ' X' AX ' X' AX ~,.e., o-cetral dstrbuto wth o cetralty parameter, SST ' X ' AX ' X' AX ~,.e., o-cetral dstrbuto wth o cetralty parameter. Sce X H, so SSregad SSresare depedetly dstrbuted. The mea squares due to regresso s SSre g MSre g ad the mea square due to error s The MS res SS re s. MSreg ' X' AX ~ F, MSres whch s a o-cetral F -dstrbuto wth (, ) degrees of freedom ad ocetralty parameter ' X' AX. Uder H : 3..., MS F MS reg res ~ F., The decso rule s to reect at level of sgfcace wheever F F (, ). Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 3

32 The calculato of F -statstc ca be summarzed the form of a aalyss of varace (ANOVA) table gve as follows: Source of varato Sum of squares Degrees of freedom Mea squares F Regresso Error Total SS re g SS re s SS T MSreg SSr e g / MSres SSr e s /( ) F Reecto of H dcates that t s lely that atleast oe (,,..., ). Test of hypothess o dvdual regresso coeffcets I case, f the test aalyss of varace s reected, the aother questo arses s that whch of the regresso coeffcets s/are resposble for the reecto of ull hypothess. The explaatory varables correspodg to such regresso coeffcets are mportat for the model. Addg such explaatory varables also creases the varace of ftted values ŷ, so oe eed to be careful that oly those regressors are added that are of real value explag the respose. Addg umportat explaatory varables may crease the resdual mea square whch may decrease the usefuless of the model. To test the ull hypothess H : versus the alteratve hypothess H : has already bee dscussed s the case of smple lear regresso model. I preset case, f H s accepted, t mples that the explaatory varable X ca be deleted from the model. The correspodg test statstc s b t ~ t( ) uder H se( b ) where the stadard error of OLSE b of s se( b ) ˆ C where C deotes the th dagoal elemet of X X correspodg to ( ' ) b. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 3

33 The decso rule s to reect H at level of sgfcace f t t., Note that ths s oly a partal or margal test because ˆ depeds o all the other explaatory varables X ( that are the model. Ths s a test of the cotrbuto of X gve the other explaatory varables the model. Cofdece terval estmato The cofdece tervals multple regresso model ca be costructed for dvdual regresso coeffcets as well as otly. We cosder both of them as follows: Cofdece terval o the dvdual regresso coeffcet: Assumg ' s are detcally ad depedetly dstrbuted followg y ~ N( X, I) b~ N(, ( X ' X) ). Thus the margal dstrbuto of ay regresso coeffcet estmate b N C ~ (, ) N(, ) y X, we have where C s the th dagoal elemet of X X. ( ' ) Thus b t ~ t( ) uder H,,,... ˆ C SSre s y' y b' X ' y where ˆ. So the ( )% cofdece terval for (,,..., ) s obtaed as follows: b Pt t,, ˆ C Pb ˆ ˆ t C b t C.,, So the cofdece terval s b ˆ ˆ t C, b t C.,, Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 33

34 Smultaeous cofdece tervals o regresso coeffcets: A set of cofdece tervals that are true smultaeously wth probablty ( ) are called smultaeous or ot cofdece tervals. It s relatvely easy to defe a ot cofdece rego for multple regresso model. Sce ( b)' X ' X( b) ~ F, MSres ( b)' X ' X( b) P F (, ). MSres So a ( )% ot cofdece rego for all of the parameters s ( b )' X ' X( b ) F (, ) MS res whch descrbes a ellptcally shaped rego. Coeffcet of determato ( R ) ad adusted R Let R be the multple correlato coeffcet betwee y ad X, X,..., X. The square of multple correlato coeffcet ( R ) s called as coeffcet of determato. The value of R commoly descrbes that how well the sample regresso le fts to the observed data. Ths s also treated as a measure of goodess of ft of the model. Assumg that the tercept term s preset the model as the where y X 3X 3... X u,,,..., R SS SS SS SS re g T ee ' ( y y) res T Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 34

35 SSre s: sum of squares due to resduals, SS T : total sum of squares SS re g : sum of squares due to regresso. R measure the explaatory power of the model whch tur reflects the goodess of ft of the model. It reflects the model adequacy the sese that how much s the explaatory power of explaatory varable. Sce ee y IX X X X y yhy ' ' ( ' ) ' ', ( y y) y y, where y y ' y wth,,..., ', y y, y,..., y ' Thus ( y y) y' y ' yy' y' y y' ' y y' y y' ( ' ) ' y y' I ( ' ) ' y y' Ay where So R A I ( ' ). ' yhy. y' Ay ' The lmts of R. R are ad,.e., R dcates the poorest ft of the model. R dcates the best ft of the model R.95 dcates that 95% of the varato y s explaed by good. R. I smple words, the model s 95% Smlarly ay other value of R betwee ad dcates the adequacy of ftted model. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 35

36 Adusted R If more explaatory varables are added to the model, the rrelevat, the R wll stll crease ad gves a overly optmstc pcture. R creases. I case the varables are Wth a purpose of correcto overly optmstc pcture, adusted R, deoted as R or ad R s used whch s defed as SSres /( ) R SST /( ) ( R ). We wll see later that ( ) ad ( ) are the degrees of freedom assocated wth the dstrbutos of SS res ad SS T. Moreover, the quattes SS r e s ad SS T varaces of e ad y s the cotext of aalyss of varace. are based o the ubased estmators of respectve The adusted R wll decle f the addto f a extra varable produces too small a reducto ( R ) to compesate for the crease s. Aother lmtato of adusted R s that t ca be egatve also. For example f R 3,,.6, the R whch has o terpretato. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 36

37 Lmtatos. If costat term s abset the model, the egatve. Some ad-hoc measures based o the lterature. R ca ot be defed. I such cases, R ca be R for regresso le through org have bee proposed Reaso that why R s vald oly lear models wth tercept term: I the model y X, the ordary least squares estmator of s ftted model as y Xb( y Xb) Xb e where e s the resdual. Note that where ŷ yly Xbely yˆ ely b ( X ' X) X ' y. Cosder the Xb s the ftted value ad l (,,...,)' s a vector of elemets uty. The total sum of squares TSS ( y y) s the obtaed as TSS ( y ly)'( y ly) [( yˆly) e]'[( yˆly) e] ( yˆly)'( yˆly) e' e( yˆly)' e SS ( )' (because ˆ reg SSres Xb ly e y Xb) SS SS yl' e (because X ' e). reg res The Fsher Cochra theorem requres TSS SSreg SSres to hold true the cotext of aalyss of varace ad further to defe the R. I order that TSS SSreg SSres holds true, we eed that leshould ' be zero,.e. model. We show ths clam as follows: le ' = l'( y yˆ ) whch s possble oly whe there s a tercept term the Frst we cosder a o tercept smple lear regresso model y x, (,,..., ) where the parameter s estmated as b xy x geeral.. The le ' = e ( y yˆ ) ( ybx), Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 37

38 Smlarly, a o tercept multple lear regresso model y X, we fd that le ' = l'( y yˆ ) l'( X Xb) = lxb ' ( ) l', geeral. Next we cosder a smple lear regresso model wth tercept term y x, (,,..., ) sxy where the parameters ad are estmated as b y bx ad b respectvely, where s xx s ( x x)( y y), xy xx ( ), s x x x x y y. We fd that le ' = e ( y yˆ ) ( y b bx ) ( y ybx bx ) [( y y) b( x x)] ( y y) b ( x x). I a multple lear regresso model wth a tercept term y l X where the parameters ad are estmated as ˆ y bx ad le ' = l'( y yˆ ) = l'( y ˆ Xb) = l'( y y Xb Xb), = l '( y y) l'( X X) b =. b ( X ' X) X ' y, respectvely. We fd that Thus we coclude that for the Fsher Cochra to hold true the sese that the total sum of squares ca be dvded to two orthogoal compoets, vz., sum of squares due to regresso ad sum of squares due to errors, t s ecessary that le ' = l'( y yˆ ) holds ad whch s possble oly whe the tercept term s preset the model. Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 38

39 . R s sestve to extreme values, so R lacs robustess. 3. R always creases wth a crease the umber of explaatory varables the model. The ma drawbac of ths property s that eve whe the rrelevat explaatory varables are added the model, R stll creases. Ths dcates that the model s gettg better whch s ot really correct. 4. Cosder a stuato where we have followg two models: y X... X u,,,.., log y X... X v The questo s ow whch model s better? For the frst model, R ( y yˆ ) ( y y) ad for the secod model, a opto s to defe R as R (log y log yˆ ) (log y log y). As such R ad R are ot comparable. If stll, the two models are eeded to be compared, a better proposto to defe where models. R R ca be as follows: ( y ˆ atlog y ) 3 ( y y) y log y. Now R ad R o comparso may gve a dea about the adequacy of the two 3 Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 39

40 Relatoshp of aalyss of varace test ad coeffcet of determato Assumg the to be a tercept term, the for H : 3..., the F statstc aalyss of varace test s MSre g F MSres ( ) SSre g ( ) SSres SSre g SST SS SSre g SST SS re g SST R R where re g R s the coeffcet of determato. So F ad R are closely related. Whe R, the F. I lmt, whe R, F. So both F ad R vary drectly. Larger R mples greater F value. That s why the F test uder aalyss of varace s termed as the measure of overall sgfcace of estmated regresso. It s also a test of sgfcace of H,.e. y s learly related to X '. s R. If F s hghly sgfcat, t mples that we ca reect Predcto of values of study varable The predcto multple regresso model has two aspects. Predcto of average value of study varable or mea respose.. Predcto of actual value of study varable.. Predcto of average value of y We eed to predct E( y) at a gve x ( x, x,..., x )'. The predctor as a pot estmate s p xb x( X' X) X' y E( p). ' ' ' x So p s a ubased predctor for E( y ). Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 4

41 Its varace s The Var( p) E p E( y) ' pe( y) = x ( X ' X) x ' E( yˆ ) x E( y/ x ) Var y x X X x ' ' ( ˆ) ( ' ) The cofdece terval o the mea respose at a partcular pot, such as x, x,..., x ca be foud as follows: Defe x x x x The (,,..., )'. The ftted value at x s yˆ xb. ' yˆ E( y/ x) Pt t, ', ˆ x( X ' X) x ' ' Pyˆ ˆ ˆ ˆ t x( X ' X) x E( y/ x) y t x( X ' X) x.,, The ( )% cofdece terval o the mea respose at the pot x, x,..., x,.e., E( y/ x ) s ' ' yˆ t ˆ ˆ ˆ x( X ' X) x, y t x( X ' X) x.,,. Predcto of actual value of y We eed to predct y at a gve x ( x, x,..., x )'. The predctor as a pot estmate s So f p f x b ' ' E( pf ) x p s a ubased for y. It's varace s Var( p ) E ( p y)( p y)' f f f ' x( X ' X) x. The ( )% cofdece terval for ths future observato s ' ' p ˆ ˆ f t [ x( X ' X) x], pf t [ x( X ' X) x].,, Regresso Aalyss Chapter 3 Multple Lear Regresso Model Shalabh, IIT Kapur 4