Asymptotic Distribution of M-estimator

Transcription

1 Asymtotic Distributio of M-estimator The followig toics are covered today: Today we briefly covered global ad local cosistecy ad asymtotic distributio of geeral M-estimators, icludig maximum likelihoodml ad geeralized method of mometsgmm. We also dicuss briefly quatile regressio ad the issue of asymtotic efficiecy. The relevat readig for today s class is Ch 4 of Amemiya ad Sectio -6 of the Newey ad McFadde Chater i Vol 4 of Hadbook of Ecoometrics. Cosistecy: cotiued from last time. There is a distictio betwee global cosistecy ad local cosistecy. Assumig the arameter sace Θ is comact.. Global CoditioThm 4.. i Amemiya, 06: su θ Θ Q θ Q θ overset 0. Ad Q θ < Q θ 0 for θ θ 0. The there is ˆθ θ 0 for ˆθ argmax θ Θ Q θ, the arameter vector that globally maximized the samle objective fuctio. 2. Local CoditioThm 4..2 i Amemiya, 0: Assumig there exists a small eighborhood Q N aroud θ 0 such that su θ θ N Qθ 0, ad that Q θ > Q θ 0 for θ θ 0 ad θ N. The if we use ˆΘ to deote the set of θ for which Qθ 0, the the claim is that some oit i ˆΘ will be a cosistet estimate for θ 0, although which oit is the cosistet estimate is ot kow i advace, i.e., ɛ > 0, lim P if θ ˆΘ θ θ 0 > ɛ 0. Practically, for the local cosistecy coditio, you oly eed to check two roerties, Qθ 0 0 ad 2 egative defiite. 2 Qθ 0 Cosistecy for MLE: Read sec Ch4 i Amemiya. Let L y,..., y, θ be the JOINT desity for the data y,..., y. The Q θ log L y,..., y, θ.. For the articular case of iid data, there is Q θ t log f y t, θ. If θ 0 is idetified i the sese that for θ θ 0, there is a ositive robabiity of y t uder θ 0 where f y t, θ f y t, θ 0, the the followig holds f y; θ E log f y; θ E log f y; θ 0 < log E f y; θ 0 log f y; θ dy log 0. where the secod iequality comes from the Jese s iequality lus the idetificatio coditio. Usually, to justify su θ Θ Q θ 0, we eed some domiace coditio like E su θ Θ log f y; θ <, see i articular Thm 4.2. i 6 i Amemiya, which we discuss i the last ote whe alyig stochastic equicotiuity. However, the coditio that E su θ Θ log f y; θ < will be violated if the suort of the y t deeds o the arameters, for examle if y t is distributed i θ 0,. because for θ > θ 0, there is be some y such that f y, θ 0, i which case log f y, θ so that the domiace coditios fails. However, as log as you imosed that costrait that θ mi y,..., y i maximizig Q θ to obtai ˆθ, recogizig Q θ is udefied for θ < mi y,..., y. This is for two reasos, because mi y,..., y a.s. θ 0, as we discussed i last lecture for the uiform case, so that ay θ < θ 0 will be ruled out by the data evetually. 2 The if we restricted ourselve to the subset of the arameter sace for which θ θ 0, the both E su θ θ0 log f y; θ < ad E log f y; θ < E log f y; θ 0 will still be true. So the rule is that you always get cosistet estimate from maximum likelihood estimatio, eve if you have a arameter-deedet suort of the data, as log as you icororate the costraits defied by the data o the arameter sace roerly ito otimizig Q θ.

2 2. I the geeral case whe y t is ot iid, there is still E log L y,..., y ; θ EL y,..., y ; θ 0 by the same alicatio of Jese s equality. However, to justify the strict iequality < is harder, see 5-6 Amemiya. y t u 2 / 2σ 2 + λ 2πσ2 ex y t u 2 2 / 2σ 2 2 ]. Set 3. Sometimes the local coditio may be imortat eve whe the global coditio fails, esecially whe Θ is ot comact. Look at the mixture of ormal examle i 9 Examle i Amemiya, where y t λn µ, σ 2 + λ N µ2, σ2 2. The likelihood fuctio is L λ t 2πσ ex u y or ay y t for that matter, ad let σ 0but ever touch 0, the L icreases to without boud. You ca t do this with the omixture model where there is just y t N µ, σ 2, because the other terms t will go to 0 at a exoetial rate, fast that the liear rate at which the first t goes to. I the mixture model, the existece of the mixture art for λ revets the other t terms from decreasig to 0, thus creatig the above roblem. However, a local root of the log likelihood fuctio ca still be cosistet, see 20 Amemiya. Cosistecy for GMM: Read Newey ad McFadde 232. Now Q θ g θ W g θ, for g θ t g z t, θ, ad W is the ositive defiite weightig matrix. If su θ Θ g θ Eg z t, θ 0 ad Eg z t, θ 0 iff θ θ 0. The ˆθ argmax θ Q θ 0. Note that if W is ot full rak, you ca essetially dro momet coditios util it is full rak. Idetificatio i Liear Models: Global idetificatio i oliear GMM model is usually difficult ad is usually assumed. However, idetificatio i liear models usually reduces to coditio that the samle var-cov matrix for regressors is full rak, this is Ex t x t for iid models, ad lim T t x tx t for fixed regressors. For least Square, t y t x tβ 2 E y x β 2. Iff Ex t x t full rak E y x β 2 E y x β 0 2 E x β β 0 ] 2 β β 0 Ex t x t β β 0 > 0 if β β 0 Quatile Regressio: The idetifyig assumtio for quatile regressio models is to assume that the coditioal τth quatile of y t give x t is a liear regressio fuctio x tβ 0, i.e. P r y t x tβ 0 x t F y x tβ 0 x t τ. The τ 2th quatile is the media. The followig momet coditios are satisfied for a liear model idetified by coditioal quatile assumtio: E τ y x tβ 0 x t Ex t τ P r y t x tβ 0 x t 0. Motivated by the oulatio momet coditio, i the samle we look for ˆβ such that 0 t x t τ y t x ˆβ t t x t τ y > x ˆβ t τ y t x ˆβ ] t. Itegrate this first order coditio back to obtai the covex objective fuctio Q β for quatile regressio: Note x + max 0, x ad x mi 0, x. Q β τ y t x tβ] y t x tβ t τ y t x tβ + + τ y x β ] To thik of relatig the derivative of this objective fuctio to the momet coditio above, thik of x+ x x x > 0 ad x x < 0, so that y x β + y x β > 0 x t, ad y x β y x β 0 x t etc. Whe τ 2, Q β t y t x tβ becomes the Least Absolute Deviatio LAD regressio, which looks for the coditioal media. t 2

3 To show formally that Ex t x t imlies global cosistecy for the liear quatile regressio model, cosider for su θ Θ Q β Q θ 0 where Q β E τ y x β y x β there is Q β Q β 0 E τ y x β y x β τ y < x β 0 y x β 0 ] Eτ x β 0 x β + E y x β y x β 0 y x β] + E x β x β 0 y < x β 0 E y x β y x β 0 y < x β] x β 0 E x x β y x β f y x dy Assumig that the set of x t ad β is bouded, ad that δ > 0 such that f y y x > δ uiformly i x for all y x β 0 δ, so that M large such that Mδ >> x β β 0 for all x ad β, we ca cotiue the iequalities as: x β 0 E x x β y x β f y x dy E x 0 E x 0 x β β 0 x β β 0 M 0 u x β β 0 f u x du E x β β 0 u x β β 0 f u x du δ M M 2 E x β β 0 2 u x β β 0 f u x du δ M 2 β β 0 Exx β β 0 > 0 for β β 0 if Exx osigular The objective fuctio Q β for quatile regressio has two features, the objective fuctio is covex so that oitwise covergece is sufficiet for uiform covergece over ay comact set Θ ad the arameter sace does ot have to be comact. More o this below. 2 No momet coditios are eeded for y t to obtai oitwise covergece, this is doe by subtractig Q β 0, a fuctio that does ot deed o β from Q β, ad show that Q β Q β 0 Q β Q β 0. The reaso that this scalig is useful is because the absolute value is a orm for which we ca aly the triagular iequality: a b a b, which allows us to write for the LAD, y t x tβ y t x tβ 0 t x tβ x tβ 0 t x t ˆβ ˆβ 0 Similarly for the geeral τth quatile regressio, we ca use a + b + a b ad a b a b to ut a boud o: τ y t x tβ + + τ y x β ] τ y t x tβ τ y x β 0 ] t τ x β x β 0 + τ x β x β 0 x β x β 0 x t ˆβ ˆβ 0 t We ca t do the same trick for least square because the square is ot a orm for which the triagular iequality is ot alicable. Cocavity ad Nocomact arameter set: whe Q θ is cocave for maximizatio roblemor covex for miimizatio roblem, the two good features are available,. If Q θ is cocave i θ almost surely, the oitwise covergece of Q θ to Q θ imlies uiform covergece over ay comact set Θ. 2. It is oly eccessary for Q θ to be uiquely maximized at θ 0 over some eighborhood aroud θ 0 to obtai global cosistecy. t t t 3

4 For details read Newey ad McFadde 233, for the basic ituitio thik of the simle icture we draw i class. Asymtotic Normality: The Geeral Framework: Read Amemiya, Thm 4..3,. Everythig is just some form of first order Taylor Exasio: Q ˆθ 0 Q θ 0 + ˆθ 2 Q θ θ0 ˆθ 2 Q θ Q θ 0 LD 2 Q θ 0 Q θ 0 d θ0 θ N 0, A BA 2 Q θ 0 Q θ 0 A θ B V ar Qθ log Lθ 2 Qθ 2 log Lθ 0 Maxmimum Likelihood Estimator:. θ. Note that the followig iformatio matrix for the log likelihood fuctio: E 2 log Lθ 0 log Lθ0 E which results from: 0 E log L θ 0 E L θ 0 L θ L θ L θ dy dy L θ dy 0. L θ log Lθ 0, wheever iterchage of itegratio ad differetiatio is justifiedby DOM for examle. Furthermore differetiate the leftmost equality: 0 log L θ E log L θ L θ dy 2 log L θ log L θ log L θ L θ dy + L θ dy E 2 log L θ log L θ log L θ + E So that i this case we have A B, ad therefore ˆθ θ0 d N 0, A N 0, lim E 2 log Lθ ]. This is true for either iid data or heterogeuous ad deedet data, as log as regularity coditios for LLN ad CLT are satisfied, as log as L reresets the joit desity for the data. I the case where the data is iid, this would simly to E 2 fy;θ log fy;θ log fy;θ E. However, there are times whe iterchagig itegratio ad differetiatio is ot ossible, ad there will be a extra term that breaks the iequality. For examle, if y θ,, the 0 θ f y dy f θ + θ f y log f y; θ dy f θ + E log fy;θ which leads to E f θ, which may or may ot be 0 deedig o whether f θ 0. GMM: Q θ g θ W g θ, g θ t g z t, θ. Let G θ gθ. Because i this case Q θ is already a quadratic form i g θ, it is ossible to avoid takig 2d order exasio of Q θ by takig st order exasio of g θ, thus avoidig imosig regularity coditio o 2 Qθ, Read Newey ad McFadde 245. Write for θ θ 0, ˆθ ] : Ĝ G ˆθ, G G θ, 4

5 G EG θ 0, Ω E g z, θ 0 g z, θ 0 : 0 Ĝ W g ˆθ Ĝ W g θ 0 + G ˆθ θ0 Ĝ W g θ 0 + Ĝ W G ˆθ θ0 ˆθ θ0 Ĝ W G Ĝ W g θ 0 LD G W G G W g θ 0 LD G W G G W N 0, Ω N 0, G W G G W ΩW G W G. Efficiet choice of W Ω or W Ω, i which case d ˆθ θ0 N 0, G Ω G. 2. Whe G is ivertible, W is irrelevat, d ˆθ θ0 N 0, G ΩG N 0, G Ω G 3. Sometimes Ω is kow u to a scalar, e.g. i homoscedastic regressio models, Ω ag, i which case we have G Ω ad d ˆβ β0 N 0, ag. Cosider the followig examles, 4. Least squarels: y x β +ε, g z, θ x y xβ. G Exx, Ω Eε 2 xx. If E ε 2 x ] σ 2, the Ω σ 2 Exx Ω σ 2 G ad d ˆβ β0 N 0, σ 2 Exx. Otherwise we must use d ˆβ β0 N 0, Exx Eε 2 xx Exx, the so-called White s heteroscedasticity cosistecy stadard error. 5. Weighted LS: g z t, β Eε 2 x y x β. G E Eε 2 x xx N 0, G. Ω d ˆβ β0 6. Liear 2SLS: g z, β z y xβ, G Ezx, Ω Eε 2 zz, W Ezz, so that N 0, V for V Exz Ezz Ezx ] Exz Ezz Eε 2 zz Ezz Ezx Exz Ezz Ezx ]. Oly whe i homoscedastic models, where Eε 2 zz σ 2 Ezz, the variace simlies to V σ 2 Exz Ezz Ezx ]. ˆβ β0 d 7. Liear 3SLS: g z, β z y xβ, G Ezx, Ω Eε 2 zz, W Eε 2 zz, so that ˆβ β0 8. MLE as GMM: g z, θ So that ˆθ θ d N 0, V for V σ 2 Exz Eε 2 zz Ezx ]. log fz,θ, G E 2 log fz,θ d N 0, G N 0, Ω. Ω E log fz,θ fz,θ, W is irrelevat. 9. GMM agai: Although there ca be more momet coditios i g z, θ tha the dimesio of θ, it is always ossible to take liear combiatios of the momet coditios so that there are exactly the same umber of g z, θ as the umber of θ. I articular, take h z, θ G W g z, θ ad use h z, θ as the ew momet coditios defiig the GMM estimate. The ˆθ argmax θ ] ] h z t, θ h z t, θ t t 5

6 is asymtotically equivalet to ˆθ argmax θ g W g. The it is also ossible to write G E mz,θ G W G, Ω Eh z, θ h z, θ G W ΩW G. 0. Quatile Regressio as GMM: For y x β + µ, P r u 0 x τ. ˆβ argmi β Q β τ y t x tβ y t x tβ, t roughly the first order coditio hold: t τ y x ˆβ t x t o. Therefore roughly seakig, g z, β τ y x β x, ad G E gz,β E y x β, W is irrelevat. The roblem here is that it is ot ossible to differetiate y x β. The quick ad dirty way to get aroud with this roblem ad obtai the result is to reted that we ca take exectatio before takig differetiatio. The formal way to justify doig this is the theory of emirical rocess, which you could read i great detail from the Adrews Chater 37 i the Hadbook, if you are iterested. Proceedig with the quick ad dirty way, G E y x β x ExF y x β x Ex F y x β x Ef y x β x xx Ef u 0 x xx. O the other had, use the fact that coditioal o x, τ y x β 0 τ u 0 is ] a Beroulli r.v. which τ w.. τ ad τ w.. τ, there is E τ y x β 0 2 x τ τ, so that Ω Eg z, β 0 g z, β 0 EE τ y x β 0 2] xx τ τ Exx. The fial coclusio is d ˆβ β0 N 0, τ τ Ef u 0 x xx ] Exx Ef u 0 x xx ]. I homoscedastic models where f 0 x f 0, this simlies to. Cosistet estimatio of G ad Ω: ˆβ β0 d N 0, τ τ f0 Exx. a G is the easy art, estimate by G. gz t,ˆθ t. For osmooth roblems as quatile regressio, use umerical derivative of the objective fuctio Qˆθ+2h +Q ˆθ 2h 2Qˆθ to aroximate G. Require h o ad h o. Essetially, h eeds to go to 0 slower tha ˆβ to β 0 i order to smooth out the oise of ˆβ i estimatig β 0. b For statioary data, heteroscedasticity ad deedece will oly affect estimatio of Ω. For ideedet data, use White s heteroscedasticity-cosistet estimate of Ω, for deedet data, use Newey-West s autocorrelatio-cosistet estimate of Ω. For more detail, read Newey ad McFadde Iteratio ad Oe Ste Estimatio: Read Amemiya 37-4, Newey ad McFadde Startig from a iitial guess θ, There are two ways of iteratio to obtai the ext roud guess θ.. Newto-Rahso, Use quadratic aroximatio for Q θ: Q θ Q θ Q θ + θ θ + θ 2 θ 2 Q θ θ θ θ 0. Q θ 2 θ Q θ + θ θ 0 θ θ 2 Q θ Q 4h 2 6

7 2. Gauss-Newto, use liear aroximatio for the first-order coditio, e.g. GMM: Q θ g θ + G θ θ W g θ + G θ θ G W g θ + G W G θ θ 0. θ θ G G W GW g θ 3. If the iitial guess is a cosistet estimate, e.g. β β0 O, the θ LD θ0 ˆθ θ0, for ˆθ argmax θ Q θ. More iteratio will ot icrease first-order asymtotic efficiecy: a For Newto-Rahso: 2 θ 3 ` θ θ0 θ θ0 4 2 Q 5 θ θ0 Q θ 2 θ Q» 5 Q θ 0 + θ 2 Q θ θ0 0 2 θ 3 2 θ Q 5 2 Q θ C A θ θ0 4 2 Q 5 b For Gauss-Newto: ` θ θ0 θ θ0 Q θ 0 G G W G W hg θ i θ 0 + G θ0 o + ˆθ θ0 I G W G «G W G θ G θ0 G W G W g θ 0 o + θ θ0 Ifluece Fuctio: Read Newey ad McFadde 242, z t is called a ifluece fuctio whe ˆθ θ0 t φ z t + o, ad Eφ z t 0, Eφ z t φ z t <. With slight abuse otatio, you ca thik of ˆθ θ0 distributed as a vector φ z t that is ormal N 0, Ω Eφφ. Examles iclude:. For MLE, φ z t E 2 l fy t,θ 0. Or ] l fyt,θ0 E l fyt,θ 0 l fy t,θ 0 ] l fyt,θ0 2. For GMM, φ G W G G W g z t, θ 0, or φ E h h zt, θ 0 for h z t, θ 0 G W g z t, θ Quatile Regressio: φ z t Ef 0 x xx ] τ u 0 x t. Ifluece fuctio reresetatio is articularly used for discussio of asymtotic efficiecy, two ste or multiste estimatio, etc. Asymtotic Efficiecy: Read Amemiya 23-25, 46. Newey ad McFadde Suerefficiet estimator: suose θ 0 is just a scalar, there is some estimate ˆθ such that d ˆθ θ N 0, V for all θ. Now defie { ˆθ θ if ˆθ /4 0 if ˆθ < /4 7

8 Questio: Show that if θ 0 0, the θ d θ 0 N 0, 0. But if θ 0 0, the θ θ 0 LD d ˆθ θ0 N 0, V. θ does better tha ˆθ at θ 0 0 ad does o worse tha ˆθ at ay other oit. This tye of behavior eeds to be ruled out by regularity coditios which requires covergece of ˆθ θ0 to be locally uiform i θ 0, for details read the Newey990 article. ˆθ is regular if for ay data geerated by θ θ 0 + δ/, for δ 0, ˆθ θ0 has a limit distributio that does ot deed o δ. I articular, θ is ot regular accordig to this defiitio, you ca verify that the limitig distributio deeds o whether δ For regular estimators that have a ifluece fuctio reresetatio, idexed by τ: ˆθ τ θ0 LD φ z, τ N 0, Eφ τ φ τ, a eccessary coditio for ˆθ τ beig efficiet tha ay other ˆθ τ, meaig that it has a smaller var-cov matrix, is that Cov φ z, τ φ z, τ, φ z, τ 0 for all τ icludig τ itself. Thik about the simle icture draw i class, the basic geometric ituitio is that ˆθ τ ˆθ τ φ z, τ φ z, τ should be orthogoal to ˆθ τ φ z, τ, i order for ˆθ τ φ z, τ to have the shortest distace from the origi to the subsace saed by the radom variables of all ossible regular estimators ˆθ τ. You ca verify that the followig are equivalet: Cov φ z, τ φ z, τ, φ z, τ 0 Cov φ z, τ, φ z, τ V ar φ z, τ Eφ z, τ φ z, τ Eφ z, τ φ z, τ 3. If Cov φ z, τ φ z, τ, φ z, τ 0, the you ca fid φ z, τ with a smaller variace tha φ z, τ by rojectig φ z, τ oto the φ z, τ φ z, τ, ad set φ z, τ equal to the residual of this least square rojectio. See Amemiya P46 for the formula. 4. Newey s efficiecy framework: Most of the estimators that ca be classified ito the GMM framework have a ifluece fuctio with the form: φ z, τ D τ m z, τ. a For the class of GMM estimator idexed by τ W the weightig matrix, give a vector g z, θ 0, has the form above with D τ D W G W G ad m z, τ m z, W G W g z, θ 0. b Cosider MLE amog the class of GMM estimators with all ossible g z, θ 0 ad ossible W, so that τ idexes ay vector of momet fuctio havig the same dimesio as θ. I this case, D τ D h E h ad m z, τ h z t, θ For this articular case where φ z, τ D τ m z, τ, the coditio Eφ z, τ φ z, τ Eφ z, τ φ z, τ for τ idexig a efficiet estimators becomes: D τ Em z, τ m z, τ D τ D τ Em z, τ m z, τ D τ It is easy to check that if τ is such that D τ Em z, τ m z, τ for all τ, the both sides above are the same D τ. Examles: a GMM with otimal weightig matrix: Remember D τ D W G W G, Em z, τ m z, τ Em z, W m z, W G W Ω W G. 8

9 So that you wat 0 D τ Em z, τ m z, τ G W G G W Ω W G G W G Ω W G, which is satisfied by settig W Ω. b MLE better tha ay GMM: For D τ E h, m z, τ h z, θ 0, the otimal choice of h is oe for which D τ E hz,θ 0 Eh z, θ 0 h z, θ 0. You ca verify that this is satisfied if h z, θ 0 l fy,θ 0, the score fuctio for MLE. This is due to the geeralized iformatio matrix equality, which says that 0 Eh z, θ 0 h z, θ f z, θ dz h z, θ l f z, θ f z, θ dz + h z, θ f z, θ dz E h z, θ 0 + Eh z, θ 0 l f z, θ 0 c For the examle o efficiet istrumet i oliear GMM model, look at Newey ad McFadde You ca ractice by lookig at the revious examles of WLS, 2SLS, 3SLS i this setu. Two Ste Estimator: Read Sec 6 i Newey ad McFadde, the ifluece fuctio is articularly coveiet, simly do Taylor exasio ad lug i the ifluece fuctio reresetatio:. Geeral Framework: You have a first ste estimator where ˆγ γ 0 t φ z t + o. You estimate ˆθ by Qˆθ,ˆγ qz t,ˆθ,ˆγ t 0. For otatioal coveiece let h z, θ, γ qzt,θ,γ. Let also H z, θ, γ hz,θ,γ ad Γ z, θ, γ hz,θ,γ γ. Also let H EH z t, θ 0, γ 0, Γ EΓ z, θ 0, γ 0, h h θ 0, γ 0. The just taylor exad i the usual way: h z t, ˆθ, ˆγ ˆθ θ0 0 h θ0, ˆγ + H θ, ˆγ ˆθ θ0 0. H θ, ˆγ] h θ0, ˆγ LD H h θ0, γ 0 + Γ θ0, γ ] ˆγ γ 0 ] LD H h + Γ φ z t + o So that d ˆθ θ0 N 0, V for LD H ] h + Γ φ z t V H E h + Γφ h + φ Γ H H Ehh + Ehφ Γ + ΓEφh + ΓEφφ Γ H 2. GMM both first stage ˆγ ad secod stage ˆθ: Now φ M m z, for some momet coditio m z, γ. h θ, ˆγ G W g z, θ, ˆγ so that H G W G, Γ G W g γ G W G γ for G γ g γ. Plug these ito the above geeral case: V G W G G W ΩW G + G W Egφ G γw G + G W G γ E φg W G + G W G γ Eφφ G γw G ] G W G If W I, ad G is ivertible, the this simlies to V G Ω + Egφ G γ + G γ Eφg + G γ Eφφ G γ] G. The same formula ca be obtaied by Newey s stackig momet aroach, for details read sec 6 of Newey McFadde chater. 9

10 3. Agai if you have trouble differetiatig gθ,γ or gθ,γ γ, the simly take exectatio before differetiatio, just relace H ad Γ by Egθ,γ ad gθ,γ γ. A good exercise to try a γ ad/or θ that results from a quatile regressio estimate. 0