Lecture 1 Least Squares

Transcription

1 RS Lecture Lecture Least Squares What s Econometrcs? Ragnar Frsch, Econometrca Vol. No. 933 revsted Eperence has shown that each o these three vew-ponts, that o statstcs, economc theory, and mathematcs, s a necessary, but not by tsel a sucent, condton or a real understandng o the quanttatve relatons n modern economc le. It s the uncaton o all three aspects that s powerul. And t s ths uncaton that consttutes econometrcs. Mathematcal Statstcs Econometrcs Data Economc heory

2 RS Lecture What s Econometrcs? Economc heory: - he CAPM R - R = R MP - R Mathematcal Statstcs: - Method to estmate CAPM. For eample, Lnear regresson: R - R = α + R MP - R + - Propertes o b the LS estmator o - Propertes o derent tests o CAPM. For eample, a t-test or H 0 : α =0. Data: R, R, and R MP - ypcal problems: Mssng data, Measurement errors, Survvorshp bas, Auto- and Cross-correlated returns, me-varyng moments. Estmaton wo phlosophes regardng models assumptons n statstcs: Parametrc statstcs. It assumes data come rom a type o probablty dstrbuton and makes nerences about the parameters o the dstrbuton. Models are parameterzed beore collectng the data. Eample: Mamum lkelhood estmaton. Non-parametrc statstcs. It assumes no probablty dstrbuton.e., they are dstrbuton ree. Models are not mposed a pror, but determned by the data. Eamples: hstograms, kernel densty estmaton. In general, n parametrc statstcs we make more assumptons.

3 RS Lecture Least Squares Estmaton Old method: Gauss 795, 80 used t n astronomy. Idea: Carl F. Gauss , Germany here s a unctonal orm relatng a dependent varable Y and k varables. hs uncton depends on unknown parameters, θ. he relaton between Y and s not eact. here s an error,. We have observatons o Y and. We wll assume that the unctonal orm s known: y =, θ + =,,...,. We wll estmate the parameters θ by mnmzng a sum o squared errors: mn θ {S, θ =Σ } Least Squares Estmaton We wll use lnear algebra notaton. hat s, y =, θ + Vectors wll be column vectors: y, k, and are vectors: y = [y y... y ] k k = [ k k... k ] = [... ] s a k matr. Its columns are the k vectors k. It s common to treat as vector o ones: =[... ] = [... ] = ί 3

4 RS Lecture Least Squares Estmaton - Assumptons ypcal Assumptons A DGP: y =, θ + s correctly speced. For eample,, θ = A E[ ] = 0 A3 Var[ ] = σ I A4 has ull column rank rank=k-, where k. Assumpton A s called correct speccaton. We know how the data s generated. We call y =, θ + the Data Generatng Process. Note: he errors,, are called dsturbances. hey are not somethng we add to, θ because we don t know precsely, θ. No. he errors are part o the DGP. Least Squares Estmaton - Assumptons Assumpton A s called regresson. From Assumpton A we get: E[ ] = 0 E[y ]=, θ + E[ ] =, θ hat s, the observed y wll equal E[y ] + random varaton. Usng the Law o Iterated Epectatons LIE: E[] = E [E[ ]] = E [0] = 0 here s no normaton about n Cov,=0. Cov,= E[ -0 - μ ] = E[] E[] = E [E[ ]] = E [ E[ ]] = 0 usng LIE hat s, E[] = 0. 4

5 RS Lecture Least Squares Estmaton - Assumptons From Assumpton A3 we get Var[ ] = I => Var[] = I Proo: Var[] = E [Var[ ]] + Var [E[ ]] = I. hs assumpton mples homoscedastcty E[ ] = or all. no seral/cross correlaton E[ j ] = 0 or j. From Assumpton A4 => the k ndependent varables n are lnearly ndependent. hen, the kk matr wll also have ull rank.e., rank = k. o get asymptotc results we wll need more assumpton about. Least Squares Estmaton.o.c. Objectve uncton: S, θ =Σ We want to mnmze w.r.t to θ. hat s, mn θ {S, θ =Σ = Σ [y -, θ] } d S, θ/d θ = - Σ [y -, θ], θ.o.c. - Σ [y -, θ LS ], θ LS = 0 Note: he.o.c. delver the normal equatons. he soluton to the normal equaton, θ LS, s the LS estmator. he estmator θ LS s a uncton o the data y,. 5

6 RS Lecture CLM - OLS Suppose we assume a lnear unctonal orm or, θ: A DGP: y =, θ + = + Now, we have all the assumptons behnd classcal lnear regresson model CLM: A DGP: y = + s correctly speced. A E[ ] = 0 A3 Var[ ] = σ I A4 has ull column rank rank=k, where k. Objectve uncton: S, θ =Σ = = y- y- Normal equatons: - Σ [y -, θ LS ], θ LS = - y- b =0 Solvng or b b = - y CLM - OLS Eample: One eplanatory model. A DGP: y = + + Objectve uncton: S, θ =Σ = Σ y - - F.o.c. equatons, unknowns: : - Σ y -b -b - = 0 Σ y b b = 0 : - Σ y -b -b - = 0 Σ y b b = 0 From : Σ y Σ b b Σ = 0 b = -b From : Σ y -b Σ b Σ = 0 or, more elegantly, b =, b = 6

7 RS Lecture OLS Estmaton: Second Order Condton OLS estmator: b = - y Note: b = OLS. Ordnary LS. Ordnary=lnear b s a lnear uncton o the data y,. y-b = y - - y = e = 0 e. Q: Is b s a mnmum? We need to check the s.o.c. y - by - b y - b b y - by - b y - by - b = b bb b column vector = row vector = OLS Estmaton: Second Order Condton... n n n K n n n ee... K = bb n n n K K K I there were a sngle b, we would requre ths to be postve, whch t would be = 0. he matr counterpart o a postve number s a postve dente matr. n 7

8 RS Lecture OLS Estmaton: Second Order Condton = n n n K K... K K K... K = n... K... k n = n n n K K n n n... K n... K = Denton: A matr A s postve dente pd z A z >0 or any z. In general, we need egenvalues to check ths. For some matrces t s easy to check. Let A =. hen, z A z = z z = v v >0. s pd b s a mn! OLS Estmaton - Propertes he LS estmator o LS when, θ = s lnear s b = - y b s a lnear uncton o the data y,. b = - y = - + = + - Under the typcal assumptons, we can establsh propertes or b. E[b ] = E[ ] + E[ - ] = Var[b ] = E[b-b- ] = - E[ ] - = σ - We can show that b s BLUE or MVLUE. Proo. Let b* = Cy lnear n y E[b* ]= E[Cy ]=E[C + ]= unbased C=I Var[b* ]= E[b*- b*- ] = E[C C ] = σ CC 8

9 RS Lecture OLS Estmaton - Propertes Var[b* ]= σ CC Now, let D = C - - note D=0 hen, Var[b* ] = σ D+ - D + - = σ DD + σ - = Var[b ] + σ DD. hs result s known as the Gauss-Markov theorem. 3 I we make an addtonal assumpton: A5 ~d N0, σ I we can derve the dstrbuton o b. Snce b = + -, we have that b s a lnear combnaton o normal varables b ~d N, σ - OLS Estmaton - Varance Eample: One eplanatory model. A DGP: y = + + Var[b ] = σ - σ = Var[b ] = σ Var[b ] = σ σ / σ 9

10 RS Lecture Algebrac Results Important Matrces Resdual maker M = I - - My = y - - y = y b = e resduals M = 0 - M s symmetrc M = M - M s dempotent M*M = M - M s sngular M - does not est. => rankm=-k M does not have ull rank. We have already proven ths result. Algebrac Results Important Matrces Projecton matr P = - Py = - y = b =ŷ tted values Py s the projecton o y nto the column space o. PM = MP = 0 Projecton matr P = - P s symmetrc P = P - P s dempotent P*P = P - P s sngular P - does not est. rankp=k 0

11 RS Lecture Algebrac Results Dsturbances and Resduals In the populaton: E[ ]= 0. In the sample: e = y-b = y- - y = / e = 0. We have two ways to look at y: y = E[y ] + = Condtonal mean + dsturbance y = b + e = Projecton + resdual Results when Contans a Constant erm Let the rst column o be a column o ones. hat s = [ί,,, K ] hen, Snce e = 0 e = 0 the resduals sum to zero. Snce y = b + e ί y = ί b + ί e = ί b y b hat s, the regresson lne passes through the means. Note: hese results are only true contans a constant term!

12 RS Lecture OLS Estmaton Eample n R Eample: 3 Factor Fama-French Model: Returns <- read.csv"c:/class/r/dis-k_capm.csv",head=rue,sep="," y <- Returns$IBM r <- Returns$RF y <- y - r <- Returns$Mkt_RF <- Returns$SMB 3 <- Returns$HML <- length 0 <- matr,, <- cbnd0,,,3 k <- ncol b <- solvet%*% %*% t%*%y # b = - y OLS regresson e <- y - %*%b # regresson resduals, e RSS <- as.numercte%*%e # RSS Sgma <- as.numercrss/-k # Estmated σ = s See Chapter Var_b <- Sgma*solvet%*% # Estmated Var[b ] = s - SE_b <- sqrtdagvar_b # SE[b ] OLS Estmaton Eample n R > RSS [].488 > SE_reg [] > tb 3 [,] > SE_b Note: You should get the same numbers usng R s lnear model command, lm use summary. to prnt results: t <- lmy~ - summaryt

13 RS Lecture Frsch-Waugh 933 heorem Contet: Model contans two sets o varables: = [ [,tme] [ other varables] ] = [ ] Regresson model: Ragnar Frsch y = + + populaton = b + b + e sample OLS soluton: b = - y y y Problem n 933: Can we estmate wthout nvertng the k +k k +k matr? he F-W theorem helps reduce computaton, by gettng smpled algebrac epresson or OLS coecent, b. F-W: Parttoned Soluton Drect manpulaton o normal equatons produces b = y y =[, ] so and = y= y b y b = = b y b b y b b y ==> b y- b = y - b 3

14 RS Lecture F-W: Parttoned Soluton Drect manpulaton o normal equatons produces b = - y - b => Regresson o y - b on Note: = 0 b = - y Use o the parttoned nverse result produces a undamental result: What s the southeast element n the nverse o the moment matr? - [ ] -, Wth the parttoned nverse, we get: b = [ ] -, y + [ ] -, y F-W: Parttoned Soluton Recall rom the Lnear Algebra Revew:. Y R I YR. 0 [ I YY Y Y ] R 3. 0 where D [ Y YY YY I 0 Y 0 I R I YY R R I 0 Y 4. 0 I Y Y Y ] I Y Y Y D D D YY Y Y Y Y Y Y 0 0 I 0 D 0 I D Y D 4

15 RS Lecture 5 hen, ] [ ] [ ] [ where. Inverse.Matr M D I D D D D D he algebrac result s: [ ] -, = -D - [ ] -, = D = [ M ] - Interestng cases: s a sngle varable & = ί. hen, contnung the algebrac manpulaton: b = [ ] -, y + [ ] -, y = [ M ] - M y F-W: Parttoned Soluton hen, contnung the algebrac manpulaton: b = [ M ] - M y = [ M M ] - M M y = [* * ] - * y* where Z*= M Z = resduals rom a regresson o Z on. hs s Frsch and Waugh s result - the double resdual regresson. We have a regresson o resduals on resduals! Back to orgnal contet. wo ways to estmate b : Detrend the other varables. Use detrended data n the regresson. Use all the orgnal varables, ncludng constant and tme trend. Detrend: Compute the resduals rom the regressons o the varables on a constant and a tme trend. F-W: Parttoned Soluton - Results

16 RS Lecture Frsch-Waugh Result: Implcatons FW result: b = [* * ] - * y* = [ M ] - M y = [ M M ] - M M y Implcatons - We can solate a sngle coecent n a regresson. - It s not necessary to partal the other sout o y M s dempotent - Suppose. hen, we have the orthogonal regresson b = - y b = - y Frsch-Waugh Result: Implcatons Eample: De-mean Let = ί P = ί ί ί - ί = ί - ί = ίί / M z = z - ίί z / = z ί z M demeans z = [ M M ] - M M y b Note: We can do lnear regresson on data n mean devaton orm. 6

17 RS Lecture Applcaton: Detrendng G and PG Eample taken rom Greene G: Consumpton o Gasolne PG: Prce o Gasolne Applcaton: Detrendng Y Y: Income Y* = Y ******* * Year 7

18 RS Lecture Applcaton: Detrended Regresson Regresson o detrended Gasolne M G on detrended Prce o Gasolne M PG detrended Income M Y Goodness o Ft o the Regresson Ater estmatng the model, we would lke to judge the adequacy o the model. here are two ways to do ths: - Vsual: plots o tted values and resduals, hstograms o resduals. - Numercal measures: R, adjusted R, AIC, BIC, etc. Numercal measures. In general, they are smple and easy to compute. We call them goodness-o-t measures. Most popular: R. Denton: Varaton In the contet o a model, we consder the varaton o a varable as the movement o the varable, usually assocated wth movement o another varable. 8

19 RS Lecture Goodness o Ft o the Regresson n otal varaton = y - y = ym 0 y. = where M 0 = I ίί ί - ί = the M de-meanng matr. Decomposton o total varaton assume = ί a constant. y = b + e, so M 0 y = M 0 b + M 0 e = M 0 b + e devatons rom means ym 0 y = b M 0 M 0 b + ee = bm 0 b + ee. M 0 s dempotent & e M 0 = 0 SS = SSR + RSS SS: otal sum o squares SSR: Regresson Sum o Squares also called ESS: eplaned SS RSS: Resdual Sum o Squares also called SSE: SS o errors A Goodness o Ft Measure SS = SSR + RSS We want to have a measure that descrbes the t o a regresson. Smplest measure: the standard error o the regresson SER SER = RSS/-k =>SER depends on unts. Not good! R-squared R = SSR/SS + RSS/SS R = SSR/SS = Regresson varaton/otal varaton R = bm 0 b/ym 0 y = - ee/ym 0 y = ŷ - ί y ŷ - ί y / y - ί y y - ί y =[ ŷ ŷ y ]/[y y y ] 9

20 RS Lecture A Goodness o Ft Measure R = SSR/SS = bm 0 b/ym 0 y = - ee/ym 0 y Note: R s bounded by zero and one only : a here s a constant term n --we need e M 0 =0! b he lne s computed by lnear least squares. Addng regressors R never alls when regressors say z are added to the regresson. R z R * R ryz r yz : partal correlaton coecent between y and z. Problem: Judgng a model based on R tends to over-ttng. A Goodness o Ft Measure Comparng Regressons - Make sure the denomnator n R s the same -.e., same let hand sde varable. Eample, lnear vs. lnear. Loglnear wll almost always appear to t better because takng s reduces varaton. Lnear ransormaton o data - Based on, b = - y. Suppose we work wth * =H, nstead H s not sngular. P*y= *b*= HH H - Hy recall ABC - =C - B - A - = HH - - H - Hy = - y =Py same t, same resduals, same R! 0

21 RS Lecture Adjusted R-squared R s moded wth a penalty or number o parameters: Adjusted-R = - [-/-k] - R = - [-/-k] RSS/SS = - [RSS/-k] [-/SS] mamzng <=> mnmzng [RSS/-k]= s Degrees o reedom --.e., -k-- adjustment assumes somethng about unbasedness. ncludes a penalty or varables that do not add much t. Can all when a varable s added to the equaton. It wll rse when a varable, say z, s added to the regresson and only the t-rato on z s larger than one n absolute value. Adjusted R-squared hel 957 shows that, under certan assumptons an mportant one: the true model s beng consdered, we consder two lnear models M : y = β + ε M : y = β + ε and choose the model wth smaller s or, larger Adusted R, we wll select the true model, M, on average. In ths sense, we say that mamzng Adusted R s an unbased model-selecton crteron. In the contet o model selecton, the Adusted R s also reerred as hel s normaton crtera.

22 RS Lecture Other Goodness o Ft Measures here are other goodness-o-t measures that also ncorporate penaltes or number o parameters degrees o reedom. Inormaton Crtera - Amemya: [ee/ K] + k/ - Akake Inormaton Crteron AIC AIC = -/ln L k L: Lkelhood normalty AIC = lne e/ +/ k +constants - Bayes-Schwarz Inormaton Crteron BIC BIC = -/ ln L [ln/] k normalty AIC = lne e/ +[ln/] k +constants OLS Estmaton Eample n R Eample: 3 Factor Fama-French Model contnuaton: Returns <- read.csv"c:/class/r/dis-k_capm.csv",head=rue,sep="," b <- solvet%*% %*% t%*%y e <- y - %*%b RSS <- as.numercte%*%e R <- - as.numercrss/as.numercty%*%y Adj_R_ <- - -/-k*-r AIC <- RSS/+*k/ # b = - y OLS regresson # regresson resduals, e # RSS # R-squared # Adjusted R-squared # AIC under normalty.e., under A5. >Adj_R_ [] >AIC []

23 RS Lecture OLS Estmaton Eample n R > RSS [].488 > R [] > SE_reg [] > tb 3 [,] > SE_b > tt_b 3 [,] Note: You should get the same numbers usng R s lnear model command use summary. to prnt: t <- lmy~ - summaryt Mamum Lkelhood Estmaton MLE We wll assume the errors,, ollow a normal dstrbuton: A5 ~N0, σ I hen, we can wrte the jont pd o y as yt L y, y,..., y / ep[, y ep / akng s, we have the lkelhood uncton ln L t t ln ln t / ] ep[ y t t ] 3

24 RS Lecture Mamum Lkelhood Estmaton MLE Let θ =β,σ. hen, we want to Ma ln L y, ln hen, the.o.c.: ln L ln L y y 0 4 y y 0 y y Note: he.o.c. delver the normal equatons or β! he soluton to the normal equaton, β MLE, s also the LS estmator, b. hat s, ˆ MLE b y Nce result or b: ML estmators have very good propertes! ˆ MLE ee ML: Score and Inormaton Matr Denton: Score or ecent score L n S S θ s called the score o the sample. It s the vector o partal dervatves the gradent, wth respect to the parameter θ. I we have k parameters, the score wll have a k dmenson. Denton: Fsher normaton or a sngle sample: E I Iθ s sometmes just called normaton. It measures the shape o the θ. 4

25 RS Lecture 5 he concept o normaton can be generalzed or the k-parameter case. In ths case: hs s kk matr. I L s twce derentable wth respect to θ, and under certan regularty condtons, then the normaton may also be wrtten as9 Iθ s called the normaton matr negatve Hessan. It measures the shape o the lkelhood uncton. θ I θ θ θ θ θ L - E L L E I θ θ L L E ML: Score and Inormaton Matr Propertes o S θ: E[S θ]=0. 0 ] [ S E d d d d n L S ML: Score and Inormaton Matr

26 RS Lecture 6 Var[S θ]= niθ ] [ ] [ ate theaboventegraloncemore: Lets derent 0 n I nvar S Var I E E d d d d d d d ML: Score and Inormaton Matr 3 I S θ are..d. wth nte rst and second moments, then we can apply the CL to get: S n θ = Σ S θ N0, niθ. Note: hs an mportant result. It wll drve the dstrbuton o ML estmators. a ML: Score and Inormaton Matr

27 RS Lecture 7 Agan, we assume: akng s, we have the lkelhood uncton: he score uncton s rst dervatves o L wrt θ=β,σ : ML: Score and Inormaton Matr Eample 0, ~ 0, ~ N or N y I ε ε β y β ] [ ln 4 L εε ε / ln L ln ln ln ln β y β y L hen, we take second dervatves to calculate Iθ: : L 4 ln L / ln ] [ ] [ ln L ε ε εε εε ] ln [ L E I ML: Score and Inormaton Matr Eample hen,

28 RS Lecture ML: Score and Inormaton Matr In dervng propertes and, we have made some mplct assumptons, whch are called regularty condtons: θ les n an open nterval o the parameter space, Ω. he st dervatve and nd dervatves o θ w.r.t. θ est. L θ can be derentated w.r.t. θ under the ntegral sgn. v E[S θ ]>0, or all θ n Ω. v L θ can be derentated w.r.t. θ under the ntegral sgn. Recall: I S θ are..d. and regularty condtons apply, then we can apply the CL to get: S θ a N0, niθ ML: Cramer-Rao nequalty heorem: Cramer-Rao nequalty Let the random sample,..., n be drawn rom a pd θ and let =,..., n be a statstc such that E[]=uθ, derentable n θ. Let bθ= uθ - θ, the bas n. Assume regularty condtons. hen, [ u ] Var ni Regularty condtons: [ b ] ni θ les n an open nterval Ω o the real lne. For all θ n Ω, δ θ/δθ s well dened. 3 L θd can be derentated wrt. θ under the ntegral sgn 4 E[Sθ ]>0, or all θ n Ω 5 L θd can be derentated wrt. θ under the ntegral sgn 8

29 RS Lecture ML: Cramer-Rao nequalty [ u ] Var ni [ b ] ni he lower bound or Var s called the Cramer-Rao CR lower bound. Corollary: I s an unbased estmator o θ, then Var ni Note: hs theorem establshes the superorty o the ML estmate over all others. he CR lower bound s the smallest theoretcal varance. It can be shown that ML estmates acheve ths bound, thereore, any other estmaton technque can at best only equal t. Propertes o ML Estmators Ecency. Under general condtons, we have that MLE ^ Var MLE [ ni ] he rght-hand sde s the Cramer-Rao lower bound CR-LB. I an estmator can acheve ths bound, ML wll produce t. ^ Consstency. We know that E[S θ]=0 and Var[S θ]= Iθ. he consstency o ML can be shown by applyng Khnchne s LLN to S, θ and then to S n θ=σ S, θ. hen, do a st -order aylor epanson o S n θ around S S ˆ n * S S ˆ n n n S n θ and MLE n ˆMLE * S ˆ n MLE n MLE ˆMLE * ˆ MLE - θ converge together to zero.e., epectaton. n 9

30 RS Lecture Propertes o ML Estmators 3 heorem: Asymptotc Normalty Let the lkelhood uncton be L,, n θ. Under general condtons, the MLE o θ s asymptotcally dstrbuted as ˆ MLE a N,[ ni ] Sketch o a proo. Usng the CL, we ve already establshed S n θ p N0, niθ. hen, usng a rst order aylor epanson as beore, we get * S S ˆ n n / n / MLE n n Notce that E[S n θ]= -Iθ. hen, apply the LLN to get S n θ n */n p -Iθ. usng θ n * p θ. Now, algebra and Slutzky s theorem or RV get the nal result. Propertes o ML Estmators 4 Sucency. I a sngle sucent statstc ests or θ, the MLE o θ must be a uncton o t. hat s, ˆMLE depends on the sample observatons only through the value o a sucent statstc. 5 Invarance. he ML estmate s nvarant under unctonal transormatons. hat s, ˆMLE s the MLE o θ and gθ s a uncton o θ, then g s the MLE o gθ. ˆMLE 30