SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker

Transcription

1 SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker CHAPTER 9. POINT ESTIMATION 9. Covergece i Probability. The bases of poit estimatio have already bee laid out i previous chapters. I chapter 5 i particular, we proved the law of large umbers ad Beroulli s theorem, which are results about covergece i probability. From time to time we have also metioed i a iformal way about a statistic, i.e., a kow fuctio of several observable radom variables, beig a ubiased estimate of some parameter. I this chapter, we shall deal with questios about poit estimatio i a more structured way. We shall itroduce i this sectio the otio of covergece i probability. De itio. If X, X, X 2; are radom variables, we say that the sequece fx g coverges i probability to X if, for every > 0, the sequece of umbers fp ([j X X j ])g coverges to 0 as! i.e., P ([j X X j ])! 0 as!. We frequetly deote this by X! P X. I sectio 5:2 we proved the law of large umbers. I our preset laguage this ca be expressed as follows: if fx g is a sequece of idepedet ad idetically distributed radom variables with ite commo secod momet, the X X i! P E(X ) as!. i= As a special corollary of the law of large umbers we obtaied Beroulli s theorem which i our preset laguage states: if for each, fs g is Bi(; p), the S!P p as!. Example: For = 2; 3;, let X be a radom variable with absolutely cotiuous distributio fuctio with desity give by 8 < if 0 x f X (x) = : if < x 0 if x < 0 or x >. 26

2 Let > 0 be arbitrary, with <. The for >, we have 0 P ([j X 0 j ]) = P ([X ]) P ([X ]) =! 0 as! 0. Thus, for this example, X! P 0. The followig remark is a immediate cosequece of the de itio. Remark. If X, X, X 2; are radom variables, the sequece fx g coverges i probability to X if ad oly if the sequece of di ereces, fx Xg, coverges i probability to 0. The followig theorem is beyod the scope of this course ad is give as a faith theorem. We have proved special cases of it i sectio 2:. Theorem. If f : R! R is a fuctio which is cotiuous over R, ad if X ; ; X are radom variables de ed o the same probability space, the f(x ; ; X ) is a radom variable. The followig theorem ca be stated more geerally, amely over R, but the statemet ad proof for = 2 ca easily be exteded. Theorem 2. If f : R 2! R is a fuctio which is cotiuous at the poit (a; b) 2 R 2, ad if fx g ad fy g are sequeces of radom variables de ed o the same probability space such that X! P a ad Y! P b, the the sequece of radom variables ff(x ; Y )g coverges i probability to f(a; b) as!, i.e., f(x ; Y )! P f(a; b) as!. Proof: Sice by hypothesis f : R 2! R is cotiuous at (a; b), the by the de itio of cotiuity, give > 0, there exists a > 0 such that [j X a j< ] \ [j Y b j< ] [j f(x ; Y ) f(a; b) j< ]. By the DeMorga formula, we take complemets of both sides to get [j f(x ; Y ) f(a; b) j ] [j X a j ] [ [j Y b j ]. Takig the probabilities of both sides, ad usig Boole s iequality, we obtai 0 P ([j f(x ; Y ) f(a; b) j ]) P ([j X a j ]) + P ([j Y b j ]). Sice, by hypothesis, P ([j X a j ])! 0 ad P ([j Y b j ])! 0, it follows that P ([j f(x ; Y ) f(a; b) j ])! 0 as!; 27

3 i.e., f(x ; Y )! P f(a; b) as!. De itio. If X is a observable radom variable, we say it is a ubiased estimate of the costat if E(X) =. The ext de itio appears to be a little odd, but this is the way a lot of mathematical statisticias speak ad write. De itio. If fx g is a i ite sequece of observable radom variables, the we say X is a cosistet estimate of if X! P as!. There are may examples of ubiased estimates. If X is Bi(; p), the E(X ) = p, or E(X =) = p. Thus X = is a ubiased estimate of p. Also, by Beroulli s theorem ad the de itio of covergece i probability give above, X = is a cosistat estimate of p. Here is a example where we ca obtai three di eret cosistet estimates of a particular parameter. Let X be a radom variable with absolutely cotiuous distributio with desity give by if < x < + f X (x) = 0 otherwise. Let X, X 2, deote a sequece of idepedet observatios o X, i.e., X, X 2, is a sequece of idepedet, idetically distributed radom variables, each with the same distributio fuctio as X, ad let U = mifx i ; i g. The for every > 0 satisfyig 0 < <, we have 0 P ([j U j ]) = P ([U + ]) = P ( T i= [X j + ]) = Q i= P ([X j + ]) = ( )! 0 as!. Thus U! P as!, so U is a cosistat estimate of. But if we also de e V = X + + X 2, we ca show that V is also a cosistet estimate of. Ideed, by the law of large umbers, sice E(X) = +, 2 P ([j V j ]) = P ([j X + + X E(X) j ])! 0 as!, for all > 0. Note also that if the observable radom variable 28

4 W is de ed by W = maxfx i ; i g, the oe ca easily prove that W! P as!. EXERCISES. The de itio of a fuctio f : R 2! R beig cotiuous at a poit (a; b) 2 R 2 is usually give as: for every > 0, there exists a umber > 0 such that j f(x; y) f(a; b) j< for all poits (x; y) 2 R 2 that satisfy j x a j< ad j y b j<. Prove that this justi es the followig statemet i the proof of theorem : Sice by hypothesis f : R 2! R is cotiuous at (a; b), the by the de itio of cotiuity, give > 0, there exists a > 0 such that [j X a j< ] \ [j Y b j< ] [j f(x ; Y ) f(a; b) j< ]. 2. Prove: if fx g ad fy g are sequeces of radom variables, if X! P a ad if Y! P b, where a ad b are costats, the X + Y! P a + b ad X Y! P ab as!. 3. Let fu g be a sequece of radom variables, where U is N (0; ), = ; 2;. Prove that U! P 0 as!. (Hit: Use Chebishev s iequality.) 4. Prove: if fv g is a sequece of radom variables, ad if V has the F ;2 distributio, = ; 2;, the V! P. 5. Let fx g be a sequece of idepedet, idetically distributed radom variables with a commo absolutely cotiuous distributio fuctio whose desity is give by e f X (x) = if x 0 0 if x < 0, where > 0 is a costat. Prove that X + X! P as!. 6. Prove that W! P as!, where W is as de ed at the ed of this sectio. 9.2 The Method of Momets. A advatage of the otio of covergece i probability to the value of a parameter of iterest, i.e., a advatage 29

5 of a cosistet estimate of the parameter, is that oe is assured that the farther out oe goes i the sequece the closer oe will get to this parameter i a certai sese. Especially if oe does ot have a ubiased estimate of the parameter but does have a cosistat estimate, the oe ca take comfort i the fact that the larger is, the closer the estimate is to the value of the parameter. A very useful method of dig cosistat estimates is by the method of momets which is developed i this sectio. We rst eed a importat lemma. Lemma. If fx g are idepedet ad idetically distributed radom variables with ite 2rth momet, where r is a positive iteger, the X Xi r! P E(X)as r!. i= Proof: We rst observe that the rth powers, fxg, r are idepedet ad idetically distributed with commo ite secod momet. Hece by the law of large umbers we obtai the coclusio. Lemma 2. If X; X ; X 2 ; are radom variables that are costat, i.e., there exists costats a; a ; a 2 ; such that P ([X = a]) = ad P ([X = a ]) = for all, ad if a! a as!, the X! P X = a as!. Proof: Let > 0 be arbitrary. Sice by hypothesis, a! a as!, there exists a positive iteger N such that j a a j< for all values of that satisfy > N. Hece P ([j X X j< ]) = for all > N, from which it follows that P ([j X X j ]) = 0 for all > N, i.e., X! P X = a as!. We shall cosider the case of a sequece of idepedet observatios X ; X 2 ; o a radom variable X. This meas that each radom variable i the sequece has the same distributio fuctio as does X ad is a idepedet observatio o the same populatio that X is. To keep thigs simple, suppose the distributio fuctio of X depeds o two ukow parameters, ad 2 ; ad suppose we wish to estimate both of these parameters with cosistet estimates. Let us suppose that this sequece of radom variables has a ite commo fourth momet. The by the above lemma, X Xi r! P E(X r )as! for r = ; 2. i= We also suppose that each of the rst two momets of X, m ad m 2, is a cotiuous fuctio ofthe ukow parameters, ( ; 2 ). Call these fuctios 220

6 m = u( ; 2 ) ad m 2 = v( ; 2 ). Further we assume that a cotiuuous iverse of this mappig exists, amely, that oe ca solve these two equatios for ad 2 as cotiuous fuctios, = g(m ; m 2 ) ad 2 = h(m ; m 2 ), P P of m ad m 2. Let us deote X = X i ad V = X 2 i. The by i= theorem i sectio 9: ad theorem above, ad g(x ; V )! P g(m ; m 2 ) = h(x ; V )! P h(m ; m 2 ) = 2 as!. This is essetially the method of momets. It should be realized that this holds for ay ite umber of parameters provided the correct commo momet is ite. Example. Let X be a radom variable with a absolutely cotiuous distributio with desity give by p Lx if 0 < x < 2=L f X (x) = 0 if x 0 or x p 2=L, where L > 0 is ukow, ad let X ; X 2 ; be idepedet observatios o X. I order to d a cosistat estimate of L, we compute E(X) = Solvig for L we obtai Z Z p 2=L xf X (x)dx = L = 0 8 9(E(X)) 2. i= Lx 2 dx = 23=2 3L =2. Sice h(x) = 8 is a cotiuous fuctio of x, it follows that a cosistet 9x 2 estimate of L is L b = 8. 9X 2 Example 2: Cosider a radom variable X with ite fourth momet ad ukow expectatio. Let X ; X 2 ; deote a sequece of idepedet observatios o X, i.e., X ; X 2 ; are idepedet observatios o the same populatio o which X is a observatio. Accordigly, X ; X 2 ; are idepedet ad idetically distributed radom variables whose commo distributio is tthe same as the distributio of X. Now let us de e X by X = X + X 22

7 for every. The by the law of large umbers proved i sectio 5:2 ad by the de itios give i sectio 9:, X! P, i.e., X is a cosistet estimate of. From ow o we shall use the otatio X to deote the arithmetic mea of the rst idepedet observatios o a radom variable X. Likewise we shall use i such a case the otatio b 2 = X (X k X ) 2 ad s 2 = k= X (X k X ) 2. k= Theorem. If X ; X 2 ; are idepedet observatios o a radom variable X which has a ite fourth momet, the both b 2 ad s 2 are cosistet estimates of V ar(x), but b 2 is ot a ubiased estimate of V ar(x). P Proof: We rst show that b 2 = X k X 2. Ideed, b 2 = i= = i= = i= k= P (Xi 2 2X X i + X 2 ) P Xi 2 2X 2 + X 2 P Xi 2 X 2. Now, we apply the law of large umbers ad theorem 2 of sectio 9: ad lemma above. Observig that the fuctio f(x; y) = y x 2 is cotiuous i (x; y), ad sice V ar(x) = E(X 2 ) (E(X)) 2, we may coclude that b 2! P V ar(x) as!. Sice s 2 = b2 ad sice by lemma 2 the costat radom variable!p, we agai obtai by theorem 2 i sectio 9:, by meas of the cotiuous fuctio g(x; y) = xy, that s 2! P V ar(x). Now we compute E(X 2 ) = E P 2 i= X2 i + P! X j X k j6=k! = 2 E(X 2 ) + P j6=k E(X j )E(X k ) = 2 (E(X 2 ) + ( )(E(X)) 2 ) = E(X2 ) + (E(X))2. 222

8 Thus, E(b 2 ) = (E(X2 ) (E(X)) 2 ) = V ar(x), which implies that b 2 is ot a ubiased estimate of V ar(x). However, sice E(s 2 ) = E(b2 ), it follows that s 2 is a ubiased estimate of V ar(x). Example 3.. Let X be a radom variable with a absolutely cotiuous distributio with desity give by f X (x) = if a < x < b b a 0 if x a or x b, where a < b are ukow costats, ad let X ; X 2 ; be idepedet observatios o X. We wish to d cosistat estimates of a ad of b by the method of momets. We compute the rst two momets of X ad obtai E(X) = a+b 2 E(X 2 ) = 3 (a2 + ab + b 2 ). Solvig for a ad b i terms of E(X) ad E(X 2 ) we obtai a = E(X) (3V ar(x)) =2, ad b = E(X) + (3V ar(x)) =2. P I example 2 we proved that s 2 = i= (X i X ) 2! P V ar(x). Sice fuctios g(x; y) = x (3y) =2 ad h(x; y) = x + (3y) =2 are cotiuous fuctios of (x; y), we obtai cosistet estimates ba ad b of a ad b respectively by ba = X (3s 2 ) =2 ad b b = X + (3s 2 ) =2. EXERCISES. All of the terms of the sequece f + : = ; 2; g may be cosidered as radom variables. Prove that this sequece coverges i probability to. 2. Let X be a radom variable with discrete desity give by P ([X = ]) = pe ( p)e!! 223 for = 0; ; 2;,

9 where 0 < p < is a ukow costat. If X ; X 2 ; are idepedet observatios o X, use the method of momets to d a cosistat estimate of p. 3. Let X be a radom variable with geometric distributio give by P ([X = ]) = ( p) p for = ; 2;, where 0 < p < is a ukow costat. If X ; X 2 ; are idepedet observatios o X, use the method of momets to d a cosistat estimate of p. 4. Suppose that X has a absolutely cotiuous distributio with desity give by x e x= if x > 0 f X (x) = (+) + 0 if x 0, where > 0 ad > are ukow costats. If X ; X 2 ; are idepedet observatios o X, use the method of momets to d cosistat estimates of ad. 5. Let X have a absolutely cotiuous distributio with desity give by = ( p q)e f X (x) = x + 2pe 2x + 3qe 3x if x > 0 = 0 otherwise. where p > 0, q > 0, ad p + q < are ukow costats. If X ; X 2 ; are idepedet observatios o X, use the method of momets to d cosistat estimates of p ad q 9.3 Maximum Likelihood Estimatio. Give a set of observable radom variables whose joit distributio depeds o oe or more ukow parameters, there is the problem of dig a fuctio of these radom variables that estimates oe or more of these parameters. This fuctio is called a estimate. What properties would we like such a estimate to have? I geeral, we wat the distributio of this estimate to be cocetrated ear the true value of the parameter, whatever value this parameter might have. We might also desire that this estimate be ubiased or that it be a cosistet estimate. A maximum likelihood estimate might have oe of these properties. However, it is sometimes easy or at least possible to obtai, so it becomes a cadidate, ad from the o oe tries to d out what ice properties it has, or whether it ca be altered to have certai ice properties. This sectio is devoted oly to discussig how to obtai these estimates. 224

10 I brief, a maximum likelihood estimate is a fuctio g of the observable radom variables X that maximizes the probability of the value of X that is observed. If the radom variables X have a joit discrete desity, a desity that is sometimes writte as f X (xj) i order to show its depedece o the parameter, a maximum likelihood estimate of is a fuctio = (X) that maximizes f X (Xj _ ). Note that x is replaced by X; this is because _ X is the value of X that you get, so f X (Xj) is the probability of gettig the value that you get. If this is di cult to uderstad, have patiece, ad wait for the examples that follow. I the case where the joit distributio is absolutely cotiuous ad has desity f X (xj), the it is loosely said that f X (xj)dx is the probability of X takig a value i a small eighborhood of x, ad so the maximum likelihood estimate of i this case is that value of that maximizes the probability of gettig close to, or i a eighborhood of, the value of X that oe observes, i.e., = (X) i this case is the value of that maximizes the joit desity. At this poit it is perhaps best to cosider examples. Example. The ormal distributio. Suppose X ; ; X is a sample of idepedet observatios o a radom variable X which is N (; 2 ), where ad 2 > 0 are ukow.we kow that their joit desity is f X ; ;X (x ; ; x ) = (2 2 ) =2 expf 2 2 X i= (X i ) 2 g. Suppose that we are oly iterested i a maximum likelihood estimate, (X ; ; X ), of. We must therefore d the value of that maximizes this joit desity. This is equivalet to dig the value (X ; ; X ) of that miimizes P i= (X i ) 2. I sectio 8:3 we foud that this value of as a fuctio of X ; ; X is (X ; ; X ) = X = X i= X i. This value of maximizes the desity o matter what the value of 2 is. So if istead lookig oly for the maximum likelihood estimate fo we are also iterested i the value of the (; 2 ) that maximizes the joit desity. To do this, let = X ad look for the value of 2 that maximizes the joit 225

11 desity whe = X. Takig the derivative with respect to 2 we d that the joit desity is maximized whe 2 = b 2 = X i= (X i X ) 2. I sectio 9:2 we proved that b 2 is a cosistet but ot a ubiased estimate of 2, while s 2 is both ubiased ad cosistet. Example 2. The uiform distributio. Let X ; ; X be a sample of idepedet observatios o a radom variable X that has the uiform distributio over [0; ], for some ukow costat > 0. Thus the commo desity of these radom variables is f X (xj) = if 0 x 0 otherwise. Sice the radom variables are idepedet, their joit desity is f X ; ;X (x ; ; x j) = if 0 x i ; i 0 otherwise. From this expressio of the joit desity, it is clear that if we take b such that b < maxfx ; ; x g, the f X ; ;X (x ; ; x j b ) = 0: Whe this joit desity is positive, the smaller oe takes b the larger the desity becomes. The smallest value that b ca be the is b = maxfx ; ; x g. Thus (X b ; ; X ) = maxfx ; ; X g is a maximum likelihood estimate of. It is ot di cult to prove that (X b ; ; X ) is ot a ubiased estimate of but is a cosistet estimate of. Example 3. The hypergeometric distributio. I a lake there are N sh, where N is ukow. The problem is to estimate the value of N. Someoe catches r sh (all at the same time), marks each with a spot ad returs all of them to the lake.after a reasoable legth of time, durig which these tagged sh are assumed to have distributed themselves at radom i the lake, someoe catches s sh (agai, all at oce). (Note: r ad s are cosidered to be xed, predetermied costats.) Amog these s sh caught 226

12 there will be X tagged sh, where X is a radom variable. The discrete desity of X is give by f X (xjn) = where x is a iteger that satis es N s N s r x r x maxf0; s N + rg x mifs; rg, ad f X (xjn) = 0 for all other values of x. The problem of dig a maximum likelihood estimate of N is to d that value b N = b N(x) of N for which f X (xjn) is maximized. I order to accomplish this we cosider the ratio R(N) = f X(xjN) f X (xjn ). For those values of N for which R(N) > we kow that f X (xjn) > f X (xjn ), ad for those values of N for which R(N) < we kow that f X (xjn) is a decreasig fuctio of N. Usig the formula for the desity of X we ote (after a certai amout of algebra) that R(N) > if ad oly if N < rs=x, ad R(N) < if ad oly if N > rs=x. We see that f X (xjn) reaches its maximum value(as a fuctio of N) whe N = [rs=x]. Thus a maximum likelihood estimate of N is b N = [rs=x]. (This is a method used i wildlife estimatio ad is usually referred to as the capture-recapture metod.) Example 4. The geeral liear model. I the geeral liear model preseted i chapter 8, Y = X + Z, we saw that the joit distributio of Y was N (X; 2 ). Thus its joit desity is f Y (y) = (2 2 ) expf =2 2 (y 2 X)t (y X)g. The problem is to d a maximum likelihood estimate of ad of 2. A value of that maximizes f Y (y), whatever the value of 2, is clearly the value of that miimizes (y X) t (y X). We solved this problem i sectio 8:, dig that b = (X t X) X t Y is the value of that miimizes (y X) t (y X). Thus the maximum likelihood estimate of is b= (X t X) X t Y. We have show that this estimate is a ubiased estimate. I order to d the joit maximum likelihood estimate of ; 2, we 227

13 substitute b = (X t X) X t Y for i the formula for the joit desity ad di eretiate with respect to 2. Upo so doig, ad solvig for 2, we obtai that the joit desity is maximized whe b = (X t X) X t Y ad 2 = b 2, where b 2 = jj Y X b jj 2. However, this value of 2 is ot a ubiased estimate of 2 ; this was show i chapter 8. is EXERCISES. Let X be a radom variable with a discrete distributio whose desity 8 < p if x = f X (x) = p if x = 0 : 0 otherwise, where 0 < p < is a ukow costat. Let X ; ; X be idepedet observatios o X. Fid the joit desity of X ; ; X, ad d the maximum likelihood estimate of p. 2. Let X ; ; X deote idepedet observatios o a radom variable whose distributio is P(), where > 0 is ukow. Fid the joit desity of X ; ; X, ad d the maximum likelihood estimate,, b of. 3. Let X ; ; X be idepedet observatios o a radom variable X that has a absolutely cotiuous distributio with desity give by ( x) f X (xj) = if x 2 (0; ) 0 otherwise, where > is a ukow costat. Fid the maximum likelihood estimate of. 4. Let X ; ; X be idepedet observatios o a radom variable X with discrete distributio, whose desity is f X (xjn) = if x N N 0 otherwise. Fid the maximum likelihood estimate of N. 5. I example 2, prove that b (X ; ; X ) is a cosistet estimate of. 228