Cramér-Rao Inequality: Let f(x; θ) be a probability density function with continuous parameter

Size: px

Start display at page:

Download "Cramér-Rao Inequality: Let f(x; θ) be a probability density function with continuous parameter"

Opal McDonald
5 years ago
Views:

1 WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION STEVEN J. MILLER Abstract. W invstigat a on-paramtr family of probability dnsitis (rlatd to th Parto distribution, which dscribs many natural phnomna) whr th Cramér-Rao Inquality provids no information.. Cramér-Rao Inquality Cramér-Rao Inquality: Lt f(x; θ) b a probability dnsity function with continuous paramtr θ. Lt X,..., X n b indpndnt random variabls with dnsity f(x; θ), and lt Θ(X,..., X n ) b an unbiasd stimator of θ. Assum that f(x; θ) satisfis two conditions: () w hav [ Θ(x,..., x n ) ] n f(x i ; θ) i i Θ(x,..., x n ) n i f(x i; θ) n ; (.) (2) for ach θ, th varianc of Θ(X,..., X n ) is finit. Thn var( Θ) [ ( ) 2 ], (.2) ne log f(x;θ) whr E dnots th xpctd valu with rspct to th probability dnsity function f(x; θ). For a proof, s for xampl [CaB]. Th xpctd valu in (.2) is calld th information numbr or th Fishr information of th sampl. Dat: March 7, Mathmatics Subjct Classification. 62B0 (primary), 62F2, 60E05 (scondary). Ky words and phrass. Cramér-Rao Inquality, Parto distribution, powr law.

2 2 STEVEN J. MILLER As variancs ar non-ngativ, th Cramér-Rao inquality (quation (.2)) provids no usful bounds on th varianc of an unbiasd stimator if th information is infinit, as in this cas w obtain th trivial bound that th varianc is gratr than or qual to zro. W find a simpl on-paramtr family of probability dnsity functions (rlatd to th Parto distribution) that satisfy th conditions of th Cramér-Rao inquality, but th xpctation (i.., th information) is infinit. Explicitly, our main rsult is Thorm: Lt f(x; θ) x θ log 3 x if x 0 othrwis, (.3) whr is chosn so that f(x; θ) is a probability dnsity function. Th information is infinit whn θ. Equivalntly, th Cramér-Rao inquality yilds th trivial (and uslss) bound that Var( Θ) 0 for any unbiasd stimator Θ of θ whn θ. In 2 w analyz th dnsity in our thorm in grat dtail, driving ndd rsults about and its drivativs as wll as discussing how f(x; θ) is rlatd to important distributions usd to modl many natural phnomna. W show th information is infinit whn θ in 3, which provs our thorm. 2. An Almost Parto Dnsity Considr f(x; θ) x θ log 3 x if x 0 othrwis, (2.) whr is chosn so that f(x; θ) is a probability dnsity function. Thus x θ log 3 x. (2.2)

3 WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION 3 W chos to hav log 3 x in th dnominator to nsur that th abov intgral convrgs, as dos log x tims th intgrand; howvr, th xpctd valu (in th xpctation in (.2)) will not convrg. For xampl, /x log x divrgs (its intgral looks lik log log x) but /x log 2 x convrgs (its intgral looks lik / log x); s pags of [Rud] for mor on clos squncs whr on convrgs but th othr dos not. This distribution is clos to th Parto distribution (or a powr law). Parto distributions ar vry usful in dscribing many natural phnomna; s for xampl [DM, N, NM]. Th inclusion of th factor of log 3 x allows us to hav th xponnt of x in th dnsity function qual and hav th dnsity function dfind for arbitrarily larg x; it is also ndd in ordr to apply th Dominatd Convrgnc Thorm to justify som of th argumnts blow. If w rmov th logarithmic factors, thn w obtain a probability distribution only if th dnsity vanishs for larg x. As log 3 x is a vry slowly varying function, our distribution f(x; θ) may b of us in modling data from an unboundd distribution whr on wants to allow a powr law with xponnt, but cannot as th rsulting probability intgral would divrg. Such a situation occurs frquntly in th Bnford Law litratur; s [Hi, Rai] for mor dtails. W study th varianc bounds for unbiasd stimators Θ of θ, and in particular w show that whn θ thn th Cramér-Rao inquality yilds a uslss bound. Not that it is not uncommon for th varianc of an unbiasd stimator to dpnd on th valu of th paramtr bing stimatd. For xampl, considr th uniform distribution on [0, θ]. Lt X dnot th sampl man of n indpndnt obsrvations, and Y n max i n X i b th largst obsrvation. Th xpctd valu of 2X and n+ n Y n ar both θ (implying ach is an unbiasd stimator for θ); howvr, Var(2X) θ 2 /3n and Var( n+ n Y n) θ 2 /n(n + ) both dpnd on θ, th paramtr bing stimatd (s, for xampl, pag 324 of [MM] for ths calculations). Lmma 2.. As a function of θ [, ), is a strictly incrasing function and a 2. It has a on-sidd drivativ at θ, and d (0, ).

4 4 STEVEN J. MILLER Proof. W hav x θ log 3 x. (2.3) Whn θ w hav a [ ] x log 3, (2.4) x which is clarly positiv and finit. In fact, a 2 bcaus th intgral is x log 3 x log 3 x d log x 2 log 2 x 2 ; (2.5) though all w nd blow is that a is finit and non-zro, w hav chosn to start intgrating at to mak a asy to comput. It is clar that is strictly incrasing with θ, as th intgral in (2.4) is strictly dcrasing with incrasing θ (bcaus th intgrand is dcrasing with incrasing θ). W ar lft with dtrmining th on-sidd drivativ of at θ, as th drivativ at any othr point is handld similarly (but with asir convrgnc argumnts). It is tchnically asir to study th drivativ of /, as d a 2 θ d (2.6) and x θ log 3 x. (2.7) Th rason w considr th drivativ of / is that this avoids having to tak th drivativ of th rciprocals of intgrals. As a is finit and non-zro, it is asy to pass to d θ. Thus w hav d θ lim h 0 + h lim h 0 + [ x +h log 3 x x h h x log 3 x x h x log 3 x. (2.8) ]

5 WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION 5 W want to intrchang th intgration with rspct to x and th limit with rspct to h abov. This intrchang is prmissibl by th Dominatd Convrgnc Thorm (s Appndix A for dtails of th justification). Not x h lim log x; (2.9) h 0 + h xh on way to s this is to us th limit of a product is th product of th limits, and thn us L Hospital s rul, writing x h as h log x. Thrfor d θ x log 2 x ; (2.0) as this is finit and non-zro, this complts th proof and shows d θ (0, ). Rmark 2.2. W s now why w chos f(x; θ) /x θ log 3 x instad of f(x; θ) /x θ log 2 x. If w only had two factors of log x in th dnominator, thn th on-sidd drivativ of at θ would b infinit. Rmark 2.3. Though th actual valu of d θ dos not mattr, w can comput it quit asily. By (2.0) w hav d θ log x x log 2 x log 2 x d log x. (2.) Thus by (2.6), and th fact that a 2 (Lmma 2.), w hav d a 2 θ d θ 4. (2.2)

6 6 STEVEN J. MILLER 3. Computing th Information [ ( ) ] 2 W now comput th xpctd valu, E log f(x;θ) ; showing it is infinit whn θ complts th proof of our main rsult. Not log f(x; θ) log θ log x + log log 3 x log f(x; θ) d log x. (3.) By Lmma 2. w know that d [ ( ) ] 2 log f(x; θ) E is finit for ach θ. Thus [ ( ) ] 2 d E log x ( ) 2 d log x x θ log 3 x. (3.2) If θ > thn th xpctation is finit and non-zro. W ar lft with th intrsting cas whn θ. As d θ is finit and non-zro, for x sufficintly larg (say x x for som x, though by Rmark 2.3 w s that w may tak any x 4 ) w hav As a 2, w hav da θ log x a θ 2. (3.3) [ ( ) ] 2 log f(x; θ) E θ 2 x x ( ) 2 log x a x 2 2x log x 2 log log x x log 3 x log x d log x x. (3.4) Thus th xpctation is infinit. Lt Θ b any unbiasd stimator of θ. If θ thn th Cramér-Rao Inquality givs var( Θ) 0, (3.5)

7 WHEN THE CRAMÉR-RAO INEQUALITY PROVIDES NO INFORMATION 7 which provids no information as variancs ar always non-ngativ. Appndix A. Applying th Dominatd Convrgnc Thorm W justify applying th Dominatd Convrgnc Thorm in th proof of Lmma 2.. S, for xampl, [SS] for th conditions and a proof of th Dominatd Convrgnc Thorm. Lmma A.. For ach fixd h > 0 and any x, w hav and log x x log 3 x x h h x h log x, (A.) is positiv and intgrabl, and dominats ach xh h x h x log 3 x. Proof. W first prov (A.). As x and h > 0, not x h. Considr th cas of /h log x. Sinc x h < + x h 2x h, w hav x h hx h < 2xh hx h 2 h 2 log x. (A.2) W ar lft with th cas of /h > log x, or h log x <. W hav x h h log x (h log x) n n! h log x < h log x n0 (h log x) n n n n! (h log x) n (n )! h log x h log x. (A.3) This, combind with h log x < and x h yilds x h hx h < h log x h log x. (A.4) It is clar that log x x log 3 x is positiv and intgrabl, and by L Hospital s rul (s (2.9)) w hav that x h lim h 0 + h x h x log 3 x x log 2 x. (A.5)

8 8 STEVEN J. MILLER Thus th Dominatd Convrgnc Thorm implis that x h lim h 0 + h x h (th last quality is drivd in Rmark 2.3). x log 3 x x log 2 x (A.6) Acknowldgmnts I would lik to thank Alan Landman for many nlightning convrsations. Rfrncs [CaB] G. Caslla and R. Brgr, Statistical Infrnc, 2nd dition, Duxbury Advancd Sris, Pacific Grov, CA, [DM] D. Dvoto and S. Martinz, Truncatd Parto Law and orsiz distribution of ground rocks, Mathmatical Gology 30 (998), no. 6, [Hi] T. Hill, A statistical drivation of th significant-digit law, Statistical Scinc 0 (996), [MM] I. Millr and M. Millr, John E. Frund s Mathmatical Statistics with Applications, svnth dition, Prntic Hall, [N] M. E. J. Nwman, Powr laws, Parto distributions and Zipfs law, Contmporary Physics 46 (2005), no. 5, [NM] M. Nigrini and S. J. Millr, Bnford s Law applid to hydrology data rsults and rlvanc to othr gophysical data, prprint. [Rai] R. A. Raimi, Th first digit problm, Amr. Math. Monthly 83 (976), no. 7, [Rud] W. Rudin, Principls of Mathmatical Analysis, third dition, Intrnational Sris in Pur and Applid Mathmatics, McGraw-Hill Inc., Nw York, 976. [SS] E. Stin and R. Shakarchi, Ral Analysis: Masur Thory, Intgration, and Hilbrt Spacs, Princton Univrsity Prss, Princton, NJ, Dpartmnt of Mathmatics, Brown Univrsity, 5 Thayr Strt, Providnc, RI addrss: sjmillr@math.brown.du

BINOMIAL COEFFICIENTS INVOLVING INFINITE POWERS OF PRIMES. 1. Statement of results

BINOMIAL COEFFICIENTS INVOLVING INFINITE POWERS OF PRIMES DONALD M. DAVIS Abstract. If p is a prim and n a positiv intgr, lt ν p (n dnot th xponnt of p in n, and u p (n n/p νp(n th unit part of n. If α