Last Lecture Biostatistics 602 - Statistical Iferece Lecture 17 Asymptotic Evaluatio of oit Estimators Hyu Mi Kag March 19th, 2013 What is a Bayes Risk? What is the Bayes rule Estimator miimizig square error loss? What is the Bayes rule Estimator miimizig absolte error loss? What are the tools for provig a poit estimator is cosistet? Ca a biase estimator be cosistet? Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 1 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 2 / 33 Bayes Estimator base o absolute error loss Asymptotic Evaluatio of oit Estimators Suppose that Lθ, ˆθ = θ ˆθ The posterior expecte loss is E[Lθ, ˆθx] = θ ˆθx πθ xθ Ω = E[ θ ˆθ X = x] = ˆθ ˆθ E[Lθ, ˆθx] = Therefore, ˆθ is posterior meia θ ˆθπθ xθ + ˆθ πθ xθ ˆθ ˆθ θ ˆθπθ xθ πθ xθ = 0 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 3 / 33 Whe the sample size approaches ifiity, the behaviors of a estimator are ukow as its asymptotic properties Defiitio - Cosistecy Let W = W X 1,, X = W X be a sequece of estimators for τθ We say W is cosistet for estimatig τθ if W τθ uer θ for every θ Ω W τθ coverges i probability to τθ meas that, give ay ϵ > 0 lim τθ ϵ = 0 lim τθ < ϵ = 1 Whe W τθ < ϵ ca also be represete that W is close to τθ Cosistecy implies that the probability of W close to τθ approaches to 1 as goes to Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 4 / 33
Tools for provig cosistecy Theorem for cosistecy Use efiitio complicate Chebychev s Iequality r W τθ ϵ = rw τθ 2 ϵ 2 E[W τθ] 2 ϵ 2 = MSEW ϵ 2 = Bias2 W + VarW ϵ 2 Nee to show that both BiasW a VarW coverges to zero Theorem 1013 If W is a sequece of estimators of τθ satisfyig lim > BiasW = 0 lim > VarW = 0 for all θ, the W is cosistet for τθ Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 5 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 6 / 33 Weak Law of Large Numbers Cosistet sequece of estimators Theorem 552 Let X 1,, X be ii raom variables with EX = µ a VarX = σ 2 < The X coverges i probability to µ ie X µ Theorem 1015 Let W is a cosistet sequece of estimators of τθ Let a, b be sequeces of costats satisfyig 1 lim a = 1 2 lim b = 0 The U = a W + b is also a cosistet sequece of estimators of τθ Cotiuous Map Theorem If W is cosistet for θ a g is a cotiuous fuctio, the gw is cosistet for gθ Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 7 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 8 / 33
Example - Expoetial Family Cosistet estimator of rx c roblem ii Suppose X 1,, X Expoetialβ 1 ropose a cosistet estimator of the meia 2 ropose a cosistet estimator of rx c where c is costat rx c = c 0 1 β e x/β x = 1 e c/β As X is cosistet for β, 1 e c/β is cotiuous fuctio of β By cotiuous mappig Theorem, gx = 1 e c/x is cosistet for rx c = 1 e c/β = gβ Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 9 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 10 / 33 Cosistet estimator of rx c - Alterative Metho Cosistecy of MLEs Defie Y i = IX i c The Y i ii Beroullip where p = rx c Y = 1 Y i = 1 IX i c is cosistet for p by Law of Large Numbers Theorem 1016 - Cosistecy of MLEs ii Suppose X i fx θ Let ˆθ be the MLE of θ, a τθ be a cotiuous fuctio of θ The uer regularity coitios o fx θ, the MLE of τθ ie τˆθ is cosistet for τθ Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 11 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 12 / 33
Cetral Limit Theorem Defiitio: A statistic or a estimator W X is asymptotically ormal if W τθ N 0, νθ for all θ where stas for coverge i istributio τθ : asymptotic mea νθ : asymptotic variace We eote W AN τθ, νθ Cetral Limit Theorem ii Assume X i fx θ with fiite mea µθ a variace σ 2 θ X AN µθ, σ2 θ X µθ Theorem 5517 - Slutsky s Theorem If X X, Y 1 Y X ax 2 X + Y X + a N 0, σ 2 θ a, where a is a costat, Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 13 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 14 / 33 Example - Estimator of rx c Example Defie Y i = IX i c The Y i ii Beroullip where p = rx c Y = 1 is cosistet for p Therefore, 1 Y i = 1 IX i c AN = = AN IX i c EY, VarY p, p1 p Let X 1,, X be ii samples with fiite mea µ a variace σ 2 Defie By Cetral Limit Theorem, S 2 = 1 1 X µ X µ σ X i X 2 X AN N 0, σ 2 N 0, 1 µ, σ2 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 15 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 16 / 33
Example cot Delta Metho X µ S = σ S X µ We showe previously S 2 σ 2 S σ σ/s 1 Therefore, By Slutsky s Theorem X µ N 0, 1 S σ Theorem 5524 - Delta Metho Assume W AN θ, νθ If a fuctio g satisfies g θ 0, the gw AN gθ, [g θ] 2 νθ Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 17 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 18 / 33 Delta Metho - Example ii X 1,, X Beroullip where p 1 2, we wat to kow the asymptotic istributio of X1 X By cetral limit Theorem, X p N 0, 1 p1 p X AN p, Defie gy = y1 y, the X1 X = gx g y = y y 2 = 1 2y p1 p By Delta Metho, gx = X1 X AN gp, [g 2 p1 p p] 2 p1 p = AN p1 p, 1 2p Give a statistic W X, for example X, s 2 X, e X W τθ N 0, νθ for all θ W AN τθ, νθ Tools to show asymptotic ormality 1 Cetral Limit Theorem 2 Slutsky Theorem 3 Delta Metho Theorem 5524 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 19 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 20 / 33
Usig Cetral Limit Theorem Usig Slutsky Theorem X AN µθ, σ2 θ where µθ = EX, a σ 2 θ = VarX For example, i orer to get the asymptotic istributio of 1 X2 i, efie Y i = X 2 i, the 1 X 2 i = 1 AN AN Y i = Y EY, VarY EX 2, VarX2 Whe X X, Y a, the 1 Y X ax 2 X + Y X + a Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 21 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 22 / 33 Usig Delta Metho Theorem 5524 Example Assume W AN θ, νθ gw AN If a fuctio g satisfies g θ 0, the gθ, [g θ] 2 νθ roblem X 1,, X ii N µ, σ 2 µ 0 Fi the asymptotic istributio of MLE of µ 2 Solutio 1 It ca be easily show that MLE of µ is X 2 By the ivariace property of MLE, MLE of µ 2 is X 2 3 By cetral limit theorem, we kow that X AN µ, σ2 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 23 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 24 / 33
Solutio cot Asymptotic Relative Efficiecy ARE 4 Defie gy = y 2, a apply Delta Metho g y = 2y X 2 AN gµ, [g µ] 2 σ2 AN µ 2, 2µ 2 σ2 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 25 / 33 If both estimators are cosistet a asymptotic ormal, we ca compare their asymptotic variace Defiitio 10116 : Asymptotic Relatve Efficiecy If two estimators W a V satisfy [W τθ] [V τθ] N 0, σ 2 W N 0, σ 2 V The asymptotic relative efficiecy ARE of V with respect to W is AREV, W = σ2 W σ 2 V If AREV, W 1 for every θ Ω, the V is asymptotically more efficiet tha W Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 26 / 33 Example Solutio - Asymptotic Distributio of V roblem ii Let X i oissoλ cosier estimatig rx = 0 = e λ Our estimators are W = 1 V = e X IX i = 0 V X = e X, by CLT, X AN EX, VarX/ AN λ, λ/ Defie gy = e y, the V = gx a g y = e y By Delta Metho V = e X AN gλ, [g λ] 2 λ AN e λ, e 2λ λ Determie which oe is more asymptotically efficiet estimator Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 27 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 28 / 33
Solutio - Asymptotic Distributio of W Solutio - Calculatig ARE Defie Z i = IX i = 0 By CLT, W = 1 IX i = 0 = Z Z i BeroulliEZ EZ = rx = 0 = e λ VarZ = e λ 1 e λ W = Z AN EZ, VarZ/ AN e λ, e λ 1 e λ AREW, V = = = = e 2λ λ/ e λ 1 e λ / λ e λ 1 e λ λ e λ 1 λ 1 + λ + λ2 2 + λ3 3! + 1 1 λ 0 Therefore W = 1 IXi = 0 is less efficiet tha V MLE, a ARE attais maximum at λ = 0 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 29 / 33 Defiitio : Asympotic Efficiecy for ii samples A sequece of estimators W is asymptotically efficiet for τθ if for all θ Ω, W τθ N 0, [τ θ] 2 Iθ W AN τθ, [τ θ] 2 Iθ [ { } 2 Iθ = E log fx θ θ] θ [ ] 2 = E log fx θ θ if iterchageability hols θ2 Note: [τ θ] 2 Iθ is the C-R bou for ubiase estimators of τθ Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 31 / 33 Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 30 / 33 of MLEs Theorem 10112 Let X 1,, X be ii samples from fx θ Let ˆθ eote the MLE of θ Uer same regularity coitios, ˆθ is cosistet a asymptotically ormal for θ, ie ˆθ θ 1 N 0, for every θ Ω Iθ A if τθ is cotiuous a ifferetiable i θ, the ˆθ θ N 0, [τ θ] Iθ = τˆθ AN τθ, [τ θ] 2 Iθ Agai, ote that the asymptotic variace of τˆθ is Cramer-Rao lower bou for ubiase estimators of τθ Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 32 / 33
Toay Cetral Limit Theorem Slutsky Theorem Delta Metho Asymptotic Relative Efficiecy Next Lecture Hypothesis Testig Hyu Mi Kag Biostatistics 602 - Lecture 16 March 19th, 2013 33 / 33