Background 1. Cramer-Rao inequality

Size: px

Start display at page:

Download "Background 1. Cramer-Rao inequality"

Michael Watson
5 years ago
Views:

1 Bro-06, Lecture, D/Stt/Bro-06/tex wwwmstqueesuc/ blevit/ Observtio vector, dt: Bckgroud Crmer-Ro iequlity X (X,, X X R Here dt c be rel- or vector vlued, compoets of the dt c be idepedet or depedet, ideticlly distributed or ot; it c eve be rdom process Deote the desity of the dt by f(x θ, x X, θ Θ (, b R Assume tht f(x θ > 0, x X, f(x θ dx f(x piecewise cotiuosly differetible X Note o termiology: outside Sttistics, better kow kotio of Iformtio is due to C Sho (bout 950 However, The Fisher iformtio itroduced erlier (90-0 is more useful i Sttistics Score fuctio: Expected scores re zero: l(x θ : θ log f(x θ θ f(x θ f(x θ E X θ θ log f(x θ 0 Ideed, 0 d dθ X d f(x θ dx d dθ X dθ 0 f(x θ dx f(x θ dx l(x θf(x θ dx E X θ l(x θ θ X The Fisher iformtio I X (θ, cotied i the dt X bout the prmeter θ, for give θ: X I X (θ Vr X θ θ log f(x θ E X θ θ log f(x θ ( ( log f(x θ f(x θ dx f(x θ θ dx θ f(x θ The Crmer-Ro iequlity: For y ubised estimtor ψ ψ(x of ψ(θ, EX θ ( ψ(x ψ(θ, E X θ ( θ θ (ψ (θ I X (θ Exmple X (X,, X iid, X f(x θ The Fisher Iformtio cotied i y X i idividully:

2 I(θ Vr Xi θ ( θ log f(x i θ ( ( I X (θ Vr X θ θ log f(x θ f ( X θ Vr X θ i Vr Xi θ i ( θ log f(x i θ I(θ θ log f(x i θ Exmple X (X,, X iid, X N (θ, σ f(x i θ (x πσ e i θ σ log f(x i θ log (x i θ, πσ σ θ log f(x i θ X i θ σ ( Xi θ I(θ Vr Xi θ σ σ I X (θ σ Thus for y ubised estimtor θ of θ, The MLE is the best ubised estimtor: E X θ ( θ θ I X (θ σ ˆθ X + + X, E X θ ˆθ θ E X θ (ˆθ θ σ I X (θ A ubised estimtor trets ll θ R fily, or eqully But pose ext tht θ Θ (, b R estimtor θ does ot eed to be ubised We c cosider the mximl risk: d the miimx risk: r( θ, Θ E X θ ( θ θ r(θ if θ Miimx estimtors re oly kow i cse b 0 E X θ ( θ θ Elemets of the Decisio Theory Let λ(θ > 0, θ (, b, be weight fuctio: b λ(θ dθ

3 the verge, or Byes, risk of etimtor θ is R( θ, λ : b E X θ ( θ θ λ(θ dθ E X θ ( θ θ b ie the mximl risk does ot exceed the Byes risk of estimtor Also r( θ, Θ if θ R( θ, λ R(λ λ(θ dθ E X θ ( θ θ r( θ, Θ, d therefore, r(θ if r( θ, Θ R(λ θ Thus, the miimx risk is t lest s lrge s the Byes risk The Wld priciple: typiclly r(θ R(λ λ I the Byesi settig: λ(θ is viewed s the prior desity of θ, f X,θ (x, θ f(x θλ(θ is viewed s the joit desity of X d θ, R( θ, λ E θ ( E X θ ( θ(x θ E X,θ ( θ(x θ, ccordig to the Lw of Totl Expecttio Sice here we re delig simulteously with the coditiol d verge (Byes risk, d sice bove we hve defied the coditiol Fisher iformtio, give θ, it suggests itself turlly to defie the Byes, or verge, Fisher iformtio cotied i X, bout prmeter θ: By the Lw of Totl Expecttio, I X E θ { I X E X,θ θ log f(x θ E X θ θ log f(x θ } E θ (I X (θ b I X (θλ(θ dθ Kowig the prior distributio of θ lso provides some iformtio bout θ Agi the expected scores re zero: E θ ( d dθ log λ(θ 0 We c mesure this iformtio by the Fisher iformtio correspodig to the prior desity: I λ E θ ( d dθ log λ(θ Now we c defie the Totl Fisher iformtio joitly cotied i the dt, X, d i the prior distributio of θ: ( I E Xθ log (f(x θλ(θ θ Fisher iformtio is dditive : I I X + I λ I X (θλ(θ dt + I λ 3

4 Ideed, Corollry I E Xθ θ log (f(x θλ(θ E Xθ θ log f X(X θ + θ log λ(θ ( ( E θx θ log f X(X θ + E θ θ log λ(θ + E Xθ θ log f(x θ θ ( I X + I λ + E θ θ log λ(θe X θ log f(x θ I X + I λ θ X (X,, X iid f(x θ Cosider the mout of iformtio cotied i every sigle observtio idividully: I(θ E Xi θ ( θ log f(x i θ log λ(θ the totl Fisher iformtio I I X + I λ I(θλ(θ dθ + I λ I prticulr, i the cse we hve X (X,, X iid N (θ, σ, I σ + I λ v Trees iequlity: Byesi Crmer-Ro lower boud; see [], [] Assume tht λ(θ is piece-wise cotiuously differetible d λ( λ(b 0 for y estimtor ψ ψ(x of ψ(θ (cf the Crmer-Ro lower boud Corollry Uder the bove ssumptios, R( ψ, λ (E θψ (θ I E X θ ( ψ ψ(θ (E θψ (θ I Proof Let us tke look t the followig itegrl J : ( ψ(x ψ(θ (f(x θλ(θ dθdx θ ( ( ψ(x ψ(θf(x θλ(θ b θ ( ψ(x ψ(θf(x θλ(θ dθ dx ( ψ (θf(x θπ(θ dθ dx ψ (θλ(θ dθ E θ ψ (θ 4

5 O the other hd, E θ ψ (θ J ( ψ(x θ ψ(θ (f(x θλ(θ f(x θλ(θ dθdx f(x θλ(θ [ ( ] E Xθ ( ψ(x ψ(θ log (f(x θλ(θ θ By the Cuchy-Schwrz iequlity: (E θ ψ (θ E Xθ ( ψ(x ψ(θ E Xθ Corollry ( log (f(x θλ(θ IE θ θx ( ψ(x ψ(θ IR( ψ, λ X (X,, X iid f(x θ, θ (, b, for y piece-wise cotiuously differetible prior desity λ(θ > 0, θ (, b, λ( λ(b 0, d y estimtor ψ of ψ(θ, ( b ψ (θλ(θ dθ I prticulr, if mx E X θ( ψ ψ(θ b I(θλ(θ dθ + I λ X (X,, X iid N (θ, σ, θ (, b mx E X θ( θ θ + I σ λ Note Of the two terms pperig i the deomitor of the v Trees iequlity, the first oe is essetilly determied by the Fisher iformtio I(θ, wheres the prior kowledge tht θ (, b is mily cptured by the secod term i the deomitor The followig mkes this depedece more explicit Suppose (, b (ϑ δ, ϑ + δ where ϑ ( + b/ d δ (b / Let (resclig λ(θ ( θ ϑ δ λ 0 δ where λ 0 (θ is desity o (,, with the correspodig Fisher iformtio I λ ϑ+δ ϑ δ (λ (θ λ(θ dθ I λ0 (λ 0(θ λ 0 (θ ϑ+δ ( ( λ θ ϑ δ 0 δ ϑ δ λ ( θ ϑ δ 0 δ dθ < dθ δ (λ 0(θ λ 0 (θ dθ δ I λ0 Corollry X (X,, X iid f(x θ, θ (, b, for y piece-wise cotiuously differetible prior desity λ 0 (θ > 0, θ (,, λ 0 (± 0, d y estimtor ψ of ψ(θ, ( b ψ (θλ(θ dθ mx E X θ( ψ ψ(θ b I(θλ(θ dθ + δ I λ0 5

6 I prticulr, if X (X,, X iid N (θ, σ, θ (, b mx E X θ( θ θ + I λ 0 σ δ Oe c further improve the bove lower boud by miimizig the Fisher iformtio I λ0 Thus cosider the followig miimiztio problem: Deote I λ0 I λ0 mi, λ 0 (θ dθ λ 0 (θ ω (θ, ω(θ > 0, θ (,, ω(± 0 (λ 0(θ λ 0 (θ dθ This is costried miimiztio problem: (ω (θω(θ ω (θ dθ 4 (ω (θ dθ mi (ω (θ dθ mi, ω (θ dθ Euler-Lgrge equtio: ω (θ + cω(θ 0, ω(± 0 where c is the Lgrge multiplier The oly positive solutio of this equtio is ( πθ ω(θ cos Thus the desity λ 0 miimizig I λ0 d (the lest fvorble desity is ( πθ λ 0 (θ cos, θ I λ0 π Corollry 3 X (X,, X iid f(x θ, θ (, b, ssumig λ(θ is the lest fvorble desity o (, b (ϑ δ, ϑ + δ, for y estimtor ψ of ψ(θ, I prticulr, if for y estimtor θ of θ mx E X θ( ψ ψ(θ ( b ψ (θλ(θ dθ b I(θλ(θ dθ + π δ X (X,, X iid N (θ, σ, θ (, b mx E X θ( θ θ 6 + π σ δ

7 Applictios Asymptotic lower boud Let X (X,, X iid f(x θ Let I(θ be the Fisher iformtio cotied i y X i idividully, for y prior desity λ o [, b], d y estimtor θ, Thus E X θ ( θ θ θ (,b lim Also, for y (ϑ δ, ϑ + δ (, b lim θ (,b θ (,b E X θ ( θ θ lim b I(θλ(θ dθ + I λ E X θ ( θ θ θ (ϑ δ,ϑ+δ b I(θλ(θ dθ + I λ b I(θλ(θ dθ E X θ ( θ θ ϑ+δ ϑ δ I(θλ(θ dθ Therefore, ssumig tht I(θ is cotiuous fuctio, d lettig δ 0, we get, for y < ϑ < b, lim E( θ θ lim θ [,b] δ 0 θ [ϑ δ,ϑ+δ] I(θ I(ϑ Filly, for y (, b, d y sequece of estimtors θ, lim θ (,b E X θ ( θ θ θ (,b I(θ I prticulr, if X,, X iid N (θ, σ, the for y y sequece of estimtors θ d y < b, lim E( θ θ σ θ (,b For y (, b, equlity is chieved i this iequlity, for the MLE ˆθ Hece ˆθ is loclly symptoticlly miimx, mog ll possible estimtors (ot oly ubised Thus, we still c write σ ( + o( E( θ θ σ if we gree to iterprete this i the loclly symptoticlly miimx (LAM sese This is the best possible d complete chrcteriztio of the symptotic optimlity, mog ll possible estimtors Note, tht vlid poit-wise lower boud, i the clss of ll estimtors, is impossible becuse for y θ there exists estimtor which, for this prticulr θ, hs zero me squred error However, this result is, i certi sese trivil, becuse it does ot reflect the ifluece of the prior kowledge tht θ (, b o the optiml estimtor This is becuse such prior iformtio effects oly the secod order optiml properties Applictio Secod order miimx estimtio Let X,, X N (θ, σ, I(θ σ θ ( δ, δ for y estimtor θ E X θ ( θ θ θ ( δ,δ σ + π δ σ + π σ δ ( σ π σ δ 7

8 This lower boud is symptoticlly exct (chievble to the order A estimtor chievig this boud is secod order symptoticlly miimx Note tht lettig δ we get E X θ ( θ θ σ θ (, Thus ˆθ X is miimx i R, for y This rises ext questio whether itis lso dmissible Applictio 3 Admissibility Defiitio θ is dmissible if there is o other estimtor θ such tht E X θ ( θ θ E X θ ( θ θ, for ll θ Θ, E X θ ( θ θ < E X θ ( θ θ, for some θ Θ For y the MLE ˆθ X is dmissible Let us proof it i the cse σ, The geerl cse c be either treted quite similrly, or directly reduced to it by sufficiecy E X θ (ˆθ θ Let λ(θ be prior desity o (, Accordig to the v Trees iequlity, for y estimtor θ, Suppose there is estimtor θ such tht for ll θ E X θ ( θ θ ( θ < + I λ E X θ ( θ θ E X θ (ˆθ θ, wheres for some θ 0 E X θ( θ θ 0 < E X θ(ˆθ θ Sice E X θ ( θ θ is cotiuous fuctio of θ (why?, there exist ε > 0 such tht E X θ ( θ θ ε, θ (θ 0 ε, θ + ε, or equivletly, E X θ ( θ θ ε (θ0 ε,θ 0 +ε(θ, θ R (3 Cosider the followig eve o-egtive cotiuous fuctio θ ( θ b g(θ θ + b b 0 θ > + b where is lrge eough so tht Sice (θ 0 ε, θ 0 + ε (, g(θ dθ b (θ b dθ + dθ + 0 b 0 (θ b3 3b 8 g(θ dθ +b ( ( b3 + b, 3b 3

9 we defie prior desity s: λ(θ g(θ ( + b 3 µ ( θ b µ + b b 0 µ > + b We c evlute directly (λ I λ λ dθ +b 0 (λ λ dθ +b (λ λ dθ Sice for θ + b, we fid λ(θ (θ b ( +, b λ (θ λ b 3 b ( + b 3 b (λ (θ λ(θ (θ b (+ b 3 b 4 (θ b (+ b 3 b I λ 4 ( + b 3 b ( + b 3 b Now by (3 E X θ ( θ θ λ(θ dθ ε θ0 +ε [θ 0 ε, θ 0 + ε]λ(µ dµ ε θ 0 ε dµ ( + b 3 ε ( + b 3 O the other hd from ( E X θ ( θ θ λ(θ dθ 4 I λ + I λ ( + b b 3 These two iequlitites re comptible oly if ε ( + b 4 ( + b b ε 4 b 3 3 Choosig b lrge eough, we get cotrdictio! REFERENCES [] R Gill d B Levit, Mth Meth Sttist, v (996 [] v Trees, Detectio, Estimtio d Modultio Theory (968, vol I 9

Review of the Riemann Integral

Review of the Riemann Integral Chpter 1 Review of the Riem Itegrl This chpter provides quick review of the bsic properties of the Riem itegrl. 1.0 Itegrls d Riem Sums Defiitio 1.0.1. Let [, b] be fiite, closed itervl. A prtitio P of