Chapter 3 Asmptotc Theor ad Stochastc Regressors The ature of eplaator varable s assumed to be o-stochastc or fed repeated samples a regresso aalss Such a assumpto s approprate for those epermets whch are coducted sde the laboratores where the epermeter ca cotrol the values of eplaator varables The the repeated observatos o stud varable ca be obtaed for fed values of eplaator varables I practce, such a assumpto ma ot alwas be satsfed Sometmes, the eplaator varables a gve model are the stud varable aother model Thus the stud varable depeds o the eplaator varables that are stochastc ature Uder such stuatos, the statstcal fereces draw from the lear regresso model based o the assumpto of fed eplaator varables ma ot rema vald We assume ow that the eplaator varables are stochastc but ucorrelated wth the dsturbace term I case, the are correlated the the ssue s addressed through strumetal varable estmato Such a stuato arses the case of measuremet error models Stochastc regressors model Cosder the lear regresso model Xβ + ε where X s a ( k matr of observatos o k eplaator varables X, X,, X k whch are stochastc ature, s a ( vector of observatos o stud varable, β s a ( k vector of regresso coeffcets ad ε s the ( vector of dsturbaces Uder the assumpto ( E( ε 0, V ε I, the dstrbuto of ε, codtoal o, satsf these propertes for all all values of X where deotes the th row of X Ths s demostrated as follows: Let p( ε be the codtoal probablt dest fucto of ε gve ad ( p ε s the ucodtoal probablt dest fucto of ε The ( ( p( d E ε ε p ε dε E 0 ε ε ε ( ε Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur
( ( p( d E ( ε E ε ε p ε dε ε ε ε I case, ε ad are depedet, the p( ε p( ε Least squares estmato of parameters The addtoal assumpto that the eplaator varables are stochastc poses o problem the ordar least squares estmato of β ad wth respect β as ( b X X X ad estmator of s obtaed as ( ( s Xb Xb k The OLSE of β s obtaed b mmzg ( Xβ ( Xβ Mamum lkelhood estmato of parameters: Assumg ~ N( 0, I ε the model Xβ + ε alog wth X s stochastc ad depedet of ε, the jot probablt dest fucto ε ad X ca be derved from the jot probablt dest fucto of ad X as follows: ( ε, ( ε, ε,, ε,,,, f X f ( ε ( f f ( ( f f ( f ( f ( f (, (,,,,,,, f f ( X, Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur
Ths mples that the mamum lkelhood estmators of β ad f ( f ( ε so the wll be same as based o the assumpto that s,,,, mamum lkelhood estmators of β ( X X X ( X ( X β β β ad wll be based o ε are dstrbuted as ( N 0, So the whe the eplaator varables are stochastc are obtaed as Alteratve approach for dervg the mamum lkelhood estmates Alteratvel, the mamum lkelhood estmators of β ad probablt dest fucto of ad X ca also be derved usg the jot Note: Note that the vector ( k whch ecludes the tercept term s represeted b a uderscore ths secto to deote that t s order s Let,,,, are from a multvarate ormal dstrbuto wth mea vector µ ad covarace matr Σ, e, ~ (, N µ Σ ad the jot dstrbuto of ad µ Σ ~ N, µ Σ Σ s Let the lear regresso model s where ( β0 + β + ε,,,, s a ( k vector of observato of radom vector, β 0 s the tercept term ad β s the k vector of regresso coeffcets Further depedet of ε s dsturbace term wth ~ N ( 0, ε ad s Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 3
Suppose µ Σ ~ N, µ Σ Σ The jot probablt dest fucto of (, f µ µ, ep k Σ µ µ ( ( π Now usg the followg result, we fd based o radom sample of sze s Σ Σ : Result: Let A be a osgular matr whch parttoed sutabl as B C A, D E where E ad F B CE D are osgular matrces, the A F F CE E DF E + E DF CE Note that Thus AA A A I where Σ Σ Σ ΣΣ Σ +Σ ΣΣΣ Σ Σ Σ, The ( π { ( ( ( } µ µ µ µ f(, ep k Σ Σ + Σ Σ The margal dstrbuto of ( k varate multvarate ormal dstrbuto as ( π s obtaed b tegratg f(, over ad the resultg dstrbuto s g( ep ( µ ( µ k Σ Σ Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 4
The codtoal probablt dest fucto of gve f ( f (, ( g s {( ( } µ µ ep Σ Σ π whch s the probablt dest fucto of ormal dstrbuto wth where codtoal mea ( E( µ + µ Σ Σ ad codtoal varace ( ( Var ρ Σ Σ Σ ρ s the populato multple correlato coeffcet I the model β + β + ε, 0 the codtoal mea s ( β0 + β+ ( ε E E β + β 0 Comparg ths codtoal mea wth the codtoal mea of ormal dstrbuto, we obta the relatoshp wth β 0 ad β as follows: β Σ Σ β µ µβ 0 The lkelhood fucto of (, based o a sample of sze s µ µ L ep k Σ ( µ µ π Σ Mamzg the log lkelhood fucto wth respect to µ, µ, Σ ad Σ, the mamum lkelhood estmates of respectve parameters are obtaed as Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 5
µ µ (,,, 3 k Σ S Σ S where (, 3,, k, S s [( k- ( k-] matr wth elemets ( t ( tj j ad S s t [( k- ] vector wth elemets ( t ( t Based o these estmates, the mamum lkelhood estmators of β ad β 0 are obtaed as β S S β β β 0 β β 0 ( X X X Propertes of the estmators of least squares estmator: The estmato error of OLSE ( ( ( X X X ( X ( X X b β X X X β β + ε β X ε The assumg that ( b X X X of β s E X X X ests, we have ( ε {( ε } Eb ( β E X X X because ( X X E E X X X X 0 ( ( ε E X X X E X ad ε are depedet So b s a ubased estmator of β Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 6
The covarace matr of b s obtaed as ( ( β( β V b E b b E ( X X X εε X ( X X E E{ ( X X X εε X ( X X X} E ( X X X E( εε X ( X X X E ( X X X X ( X X E ( X X Thus the covarace matr volves a mathematcal epectato The ukow ee ˆ k ( Xb ( Xb k where e Xb s the resdual ad ( ˆ ( ˆ E E E X ee E E X k E ( Note that the OLSE ( ca be estmated b b X X X volves the stochastc matr X ad stochastc vector, so b s ot a lear estmator It s also o more the best lear ubased estmator of β as the case whe X s ostochastc The estmator of as beg codtoal o gve X s a effcet estmator Asmptotc theor: The asmptotc propertes of a estmator cocers the propertes of the estmator whe sample sze grows large For the eed ad uderstadg of asmptotc theor, we cosder a eample Cosder the smple lear regresso model wth oe eplaator varable ad observatos as ( ( β0 + β + ε, E ε 0, Var ε,,,, Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 7
The OLSE of β s b ( ( ad ts varace s ( Var ( b If the sample sze grows large, the the varace of b gets smaller The shrkage varace mples that as sample sze creases, the probablt dest of OLSE b collapses aroud ts mea because Var( b becomes zero Let there are three OLSEs b, b ad b 3 whch are based o sample szes, ad 3 respectvel such that < < sa If c ad δ are some arbtrarl chose postve costats, the the probablt that the 3, value of b les wth the terval β ± c ca be made to be greater tha ( δ for a large value of Ths propert s the cosstec of b whch esure that eve f the sample s ver large, the we ca be cofdet wth hgh probablt that b wll eld a estmate that s close to β Probablt lmt Let ˆ β be a estmator of β based o a sample of sze Let γ be a small postve costat The for large, the requremet that b takes values wth probablt almost oe a arbtrar small eghborhood of the true parameter value β s lm P ˆ β β < γ whch s deoted as plm ˆ β β ad t s sad that ˆ β coverges to β probablt The estmator ˆ β s sad to be a cosstet estmator of β A suffcet but ot ecessar codto for ˆ β to be a cosstet estmator of β s that ad lm E ˆ β β lm Var ˆ β 0 Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 8
Cosstec of estmators Now we look at the cosstec of the estmators of β ad ( Cosstec of b Uder the assumpto that X X lm ests as a ostochastc ad osgular matr (wth fte elemets, we have X X lm Vb ( lm lm 0 Ths mples that OLSE coverges to β quadratc mea Thus OLSE s a cosstet estmator of β Ths also holds true for mamum lkelhood estmators also Same cocluso ca also be proved usg the cocept of covergece probablt The cosstec of OLSE ca be obtaed uder the weaker assumpto that X X plm ests ad s a osgular ad ostochastc matr ad Sce So X ε plm 0 b β ( X X X ε X X X ε X X X ε plm( b β plm plm 0 0 Thus b s a cosstet estmator of β The same s true for mamum lkelhood estmators also Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 9
( Cosstec of s Now we look at the cosstec of s ee k ε Hε k s as a estmate of k ( X X X X εεε ε We have k εε ε X X X X ε εε Note that cossts of ε ad { ε,,,, } s a sequece of depedetl ad detcall dstrbuted radom varables wth mea Thus εε plm Usg the law of large umbers ε X X X X ε εx X X X ε plm plm plm plm s plm( ( 0 0 s s a cosstet estmator of 0 0 0 The same holds true for mamum lkelhood estmates also Asmptotc dstrbutos: Suppose we have a sequece of radom varables { α } wth a correspodg sequece of cumulatve dest fuctos { F } for a radom varable α wth cumulatve dest fucto F The α coverges dstrbuto to α f F coverges to F pot wse I ths case, F s called the asmptotc dstrbuto of α Note that sce covergece probablt mples the covergece dstrbuto, so D plm α α α α ( α ted to α dstrbuto, e, the asmptotc dstrbuto of whch s the dstrbuto of α Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur α s F 0
Note that E ( α : Mea of asmptotc dstrbuto Var( α : Varace of asmptotc dstrbuto lm E( α : Asmptotc mea lm E α lm E( α : Asmptotc varace Asmptotc dstrbuto of sample mea ad least squares estmato Let α be the sample mea based o a sample of sze Sce sample mea s a cosstet estmator of populato mea, so plm whch s costat Thus the asmptotc dstrbuto of s the dstrbuto of a costat Ths s ot a regular dstrbuto as all the probablt mass s cocetrated at oe pot Thus as sample sze creases, the dstrbuto of collapses Suppose cosder ol the oe thrd observatos the sample ad fd sample mea as 3 3 The E( ad Var 9 9 3 3 0 as 3 ( Var ( Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur
Thus plm ad has the same degeerate dstrbuto as Sce Var ( Var ( >, so s preferred over Now we observe the asmptotc behavour of ad Cosder a sequece of radom varables { α } Thus for all, we have α α ( ( ( α ( ( α ( E E 0 E E 0 Var ( α E ( 3 Var ( α E ( 3 Assumg the populato to be ormal, the asmptotc dstrbuto of s N( 0, s ( 0,3 N So ow s preferable over The cetral lmt theorem ca be used to show that asmptotcall ormal dstrbuto eve f the populato s ot ormall dstrbuted α wll have a Also, sce Z ( ~ ( 0, ( N ( ~ N 0, ad ths statemet holds true fte sample as well as asmptotc dstrbutos Cosder the ordar least squares estmate ( Xβ + ε If X s ostochastc the the fte covarace matr of b s Vb ( ( X X b X X X of β lear regresso model Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur
The asmptotc covarace matr of b uder the assumpto that lm X X Σ ests ad s osgular It s gve b X X lm ( X X lm lm 0 Σ 0 whch s a ull matr Cosder the asmptotc dstrbuto of ( b β the asmptotcall ( β ~ N( 0, Σ b ( β Σ ( b β b χ ~ k The eve f ε s ot ecessarl ormall dstrbuted, If X X s cosdered as a estmator of Σ, the X X ( β ( bβ ( bβ X X ( bβ b ( s the usual test statstc as s the case of fte samples wth b N β ( X X ~, Ecoometrcs Chapter 3 Asmptotc Theor ad Stochastc Regressors Shalabh, IIT Kapur 3