Stat210B: Theoretical Statistics Lecture Date: April 19, 2007 Lecture 27: Optimal Estimators ad Fuctioal Delta Method Lecturer: Michael I. Jorda Scribe: Guilherme V. Rocha 1 Achievig Optimal Estimators From the last class, we kow that the best limitig distributio we ca hope for a parameter ψ is N0, ψ I ψ T. The et questio to ask is whether such boud ca be achieved. This is the theme of our et result: Lemma 1. Lemma 8.14 i va der Vaart, 1998 Assume that the eperimet P : Θ is differetiable i quadratic mea at 0 with o-sigular Fisher iformatio matri I. Let ψ be differetiable at 0. Let T be a estimator sequece i the eperimets P : Rk such that: T ψ = 1 ψ I 1 l X i + o P 1, the T is the best regular estimator for ψ at. Coversely, every best regular estimator sequece satisfies this epasio. Proof. Let, := 1 l X i. We kow that, coverges i distributio to a with a N0,I distributio. From Theorem 7.2 i va der Vaart 1998 we kow that: dp + h log = h T 1 2 ht I h + o P 1. dp Usig Slutsky s theorem, we get: T ψ log dp + h dp [ N ψ I 1 h T 1 2 ht I h 0 1 2 ht I h ] [ ψ, I 1 ψ h T ψ T ψ h h T I h ] Usig Le Cam s third lemma we ca coclude that the sequece T ψ uder + h coverges i distributio to N ψ h, ψ I 1 ψ T. Sice ψ is differetiable, we have that ψ + h ψ ψ h as. We coclude that, uder + h, T ψ + h does ot ivolve h, that is, T is regular. 1
2 Lecture 27: Optimal Estimators ad Fuctioal Delta Method To prove the coverse, let T ad S be a two best regular estimator sequeces. Alog subsequeces, it ca be show that: S ψ + h [ ] + h S ψ h T ψ + h T ψ h for a radomized estimator S,T i the limitig eperimet. Because S ad T are best regular, S ad T are best equivariat-i-law. Thus S = T = ψ X almost surely ad, as a result, S T coverges i distributio to S T = 0. As a result, every two best regular estimator sequeces are asymptotically equivalet. To get the result, apply this coclusio to T ad: S = ψ + 1 ψ I 1, Remarks o Theorem 8.14: From theorem 5.39, a Maimum Likelihod Estimator ˆ satisfies: ˆ = 1 I 1 l X i + o P 1 uder regularity coditios. It follows that MLEs are asymptotically efficiet. This result ca be eteded to a trasforms of the MLE ψ for a differetiable ψ by usig the delta method ad observig that ψˆ satisfies the epasio i lemma 8.14. Lemma 8.14 suggests that Rao score fuctios leadig to tests costructed from the scores are asymptotically efficiet. 2 Fuctioal Delta Method The fuctioal delta method aims at etedig the delta method to a oparametric cotet. The high level idea is to iterpret a statistic as a fuctioal φ mappig from the space of probability distributios D to the real lie R ad use a otio of derivative of this fuctioal to obtai the asymptotic distributio of φˆf. We have prove before that: sup ˆF F p 0, Gliveko-Catelli Theorem F ˆF F GF, Dosker Theorem where ˆF is the empirical distributio fuctio based i samples ad G F is the Browia Bridge. Our goal ow is to fid coditios o the fuctioals φ so we ca eted the above modes of covergece to φˆf. As we will see i detail below, cosistecy of the sequece φˆf follows easily from assumig φ to be cotiuous with respect to the supremum orm: this a atural etesio of the cotiuous mappig theorem. The geeralizatio of the delta method to fuctioals is more ivolved as differet otios of differetiability of fuctioals eist. Before we jump ito the cotiuous mappig theorem ad the fuctioal delta method, a few eamples of statistical fuctioals are i order.
Lecture 27: Optimal Estimators ad Fuctioal Delta Method 3 2.1 Eamples of statistical fuctioals The mea: µf = XdFX; The variace: VarF = X µf 2 dfx; Higher order momets: µ k F = X µf k dfx; The Kolmogorov-Smirov statistics: KF = sup F F 0, where F 0 is a fied hypothesized distributio; The Crámer-vo Mises statistics: CF = F F 0 2 df 0, where F 0 is a fied hypothesized distributio; V-statistics: φf = E F TX 1,X 2,...,X p where X 1,X 2,...,X p are idepedet copies of F- distributed radom variables; Quatile fuctioal: φf = F 1 p = if { : F p}; 2.2 Cosistecy of statistical fuctioals Oe possible assumptio to esure that φˆf coverges to φf is cotiuous with respect to the supremum orm. Formally, this is defied as: Defiitio 2. Cotiuity of a fuctioal Let D be the space of distributios ad φ : D R. We say φ is cotiuous with respect to the supremum orm at F if: sup F F 0 φf φf. The et result is a etesio of the cotiuous mappig theorem to fuctioals ad ca be used to establish the cosistecy of statistical fuctioals. Theorem 3. Cotiuous Mappig Theorem for Statistical Fuctioals Let φ : D R be a cotiuous fuctioal at F. It follows that: If F F p 0, the φf φf p 0. Proof. From cotiuity of φ with respect to the sup orm, we have that for every ε > 0, there eists δ > 0 such that: Hece, F F δ φf φf ε. 0 P φf φf > ε P F F > δε 0, where the last covergece is due to F F p 0 by hypothesis.
4 Lecture 27: Optimal Estimators ad Fuctioal Delta Method 2.2.1 Eamples of cotiuous statistical fuctioals The followig two fuctioals are cotiuous with respect to the supremum orm: φf = Fa: The distributio fuctio evaluated at a poit. To establish the cotiuity of this fuctio, otice that for a sequece of distributios such that F F 0: 0 F a Fa sup F F 0 φf = F F 0 2 df 0 : The Crámer-vo Mises fuctioal. Agai, take a sequece of distributios such that F F 0. We have: 0 φf φf = [F 2 F 2 + 2F 0 F F ] df 0 F F 2F 0 F F df 0 }{{} 2 2sup F F df 0 0 2.3 Limitig distributio of statistical fuctioals Recall our goal to determie the limitig distributio of φˆf. Heuristically, we might hope to derive it if we ca fid a liear φ F resultig i a approimatio of the sort: φˆf φf = φ FˆF F + some residual = φ FĜ + some residual As before, we must keep track of the behavior of the residual term as grows. The fact that φ operates o the ifiite dimesioal space of distributio fuctios will require us to be more careful ad resort to some cocepts of fuctioal aalysis. Namely, we will be lookig at the otios of Gateau ad Hadamard derivatives. We start by cosiderig Gateau derivatives. To see how they ca be used, otice that from liearity of φ F, we have that: φ FĜ = 1 φ Fδ Xi F, where δ Xi is the distributio fuctio cocetratig all mass at X i. For each of the terms i the sum, we ca cosider a directioal derivative of φ i the directio δ Xi F. That is the Gateau derivative which i this case is defied as: φ Fδ Xi F = d dt [φ1 tf + tδ X i ]. t=0 The epressio for φ F δ X i F above is kow i the robust statistics literature as the ifluece fuctio: IF φ,f = d dt [φ1 tf + tδ ]. t=0
Lecture 27: Optimal Estimators ad Fuctioal Delta Method 5 To some etet, it measures how much the statistic φf is affected by addig a ew observatio at to the sample. A related cocept is the gross-error sesitivity defied as: For robustess, γ must be bouded. γ = supif φ,f. Goig back to the approimatio of φˆf φf, we ow write: φˆf φf = 1 IF φ,f X i + R. We ow have: E F φ F δ Xi F = φ F δ PdP = 0, If we assume: Var F φ F δ Xi F = IF φ,f 2 df <. ad the residual term R ca be cotrolled somehow more o this i later classes, a cetral limit theorem will hold for the radom variable φ F δ X i F ad we ca epect that: φˆf φf F N0,λ 2. As is the case i multivariate calculus, the eistece of Gateau directioal derivative does ot esure that the residual of the approimatio is well behaved differetiability. I the et classes, we will study the residual term more closely. Comig up et I the et classes, we will: make this heuristic of the fuctioal delta method more precise; look more closely at the otio of Hadamard derivative; use Hadamard derivative to determie coditios that esure the fuctioal delta method works Refereces va der Vaart, A. W. 1998. Asymptotic Statistics. Cambridge Uiversity Press, Cambridge.