Properties of Point Estimators and Methods of Estimation

Size: px
Start display at page:

Download "Properties of Point Estimators and Methods of Estimation"

Transcription

1 CHAPTER 9 Properties of Poit Estimators ad Methods of Estimatio 9.1 Itroductio 9. Relative Efficiecy 9.3 Cosistecy 9.4 Sufficiecy 9.5 The Rao Blackwell Theorem ad Miimum-Variace Ubiased Estimatio 9.6 The Method of Momets 9.7 The Method of Maximum Likelihood 9.8 Some Large-Sample Properties of Maximum-Likelihood Estimators (Optioal) 9.9 Summary Refereces ad Further Readigs 9.1 Itroductio I Chapter 8, we preseted some ituitive estimators for parameters ofte of iterest i practical problems. A estimator ˆθ for a target parameter θ is a fuctio of the radom variables observed i a sample ad therefore is itself a radom variable. Cosequetly, a estimator has a probability distributio, the samplig distributio of the estimator. We oted i Sectio 8. that, if E(ˆθ) = θ, the the estimator has the (sometimes) desirable property of beig ubiased. I this chapter, we udertake a more formal ad detailed examiatio of some of the mathematical properties of poit estimators particularly the otios of efficiecy, cosistecy, ad sufficiecy. We preset a result, the Rao Blackwell theorem, that provides a lik betwee sufficiet statistics ad ubiased estimators for parameters. Geerally speakig, a ubiased estimator with small variace is or ca be made to be 444

2 9. Relative Efficiecy 445 a fuctio of a sufficiet statistic. We also demostrate a method that ca sometimes be used to fid miimum-variace ubiased estimators for parameters of iterest. We the offer two other useful methods for derivig estimators: the method of momets ad the method of maximum likelihood. Some properties of estimators derived by these methods are discussed. 9. Relative Efficiecy It usually is possible to obtai more tha oe ubiased estimator for the same target parameter θ. I Sectio 8. (Figure 8.3), we metioed that if ˆθ 1 ad ˆθ deote two ubiased estimators for the same parameter θ, we prefer to use the estimator with the smaller variace. That is, if both estimators are ubiased, ˆθ 1 is relatively more efficiet tha ˆθ if V (ˆθ )>V (ˆθ 1 ). I fact, we use the ratio V (ˆθ )/V (ˆθ 1 ) to defie the relative efficiecy of two ubiased estimators. DEFINITION 9.1 Give two ubiased estimators ˆθ 1 ad ˆθ of a parameter θ, with variaces V (ˆθ 1 ) ad V (ˆθ ), respectively, the the efficiecy of ˆθ 1 relative to ˆθ, deoted eff (ˆθ 1, ˆθ ), is defied to be the ratio eff (ˆθ 1, ˆθ ) = V (ˆθ ) V (ˆθ 1 ). If ˆθ 1 ad ˆθ are ubiased estimators for θ, the efficiecy of ˆθ 1 relative to ˆθ, eff (ˆθ 1, ˆθ ), is greater tha 1 oly if V (ˆθ )>V (ˆθ 1 ). I this case, ˆθ 1 is a better ubiased estimator tha ˆθ. For example, if eff (ˆθ 1, ˆθ ) = 1.8, the V (ˆθ ) = (1.8)V (ˆθ 1 ), ad ˆθ 1 is preferred to ˆθ. Similarly, if eff (ˆθ 1, ˆθ ) is less tha 1 say,.73 the V (ˆθ ) = (.73)V (ˆθ 1 ), ad ˆθ is preferred to ˆθ 1. Let us cosider a example ivolvig two differet estimators for a populatio mea. Suppose that we wish to estimate the mea of a ormal populatio. Let ˆθ 1 be the sample media, the middle observatio whe the sample measuremets are ordered accordig to magitude ( odd) or the average of the two middle observatios ( eve). Let ˆθ be the sample mea. Although proof is omitted, it ca be show that the variace of the sample media, for large,isv (ˆθ 1 ) = (1.533) (σ /). The the efficiecy of the sample media relative to the sample mea is eff (ˆθ 1, ˆθ ) = V (ˆθ ) V (ˆθ 1 ) = σ / (1.533) σ / = 1 (1.533) = Thus, we see that the variace of the sample mea is approximately 64% of the variace of the sample media. Therefore, we would prefer to use the sample mea as the estimator for the populatio mea.

3 446 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio EXAMPLE 9.1 Let Y 1, Y,..., Y deote a radom sample from the uiform distributio o the iterval (0, θ). Two ubiased estimators for θ are ( ) + 1 ˆθ 1 = Y ad ˆθ = Y (), where Y () = max(y 1, Y,..., Y ). Fid the efficiecy of ˆθ 1 relative to ˆθ. Solutio Because each Y i has a uiform distributio o the iterval (0, θ), μ = E(Y i ) = θ/ ad σ = V (Y i ) = θ /1. Therefore, ( ) θ E(ˆθ 1 ) = E(Y ) = E(Y ) = (μ) = = θ, ad ˆθ 1 is ubiased, as claimed. Further, [ ] ( )( V (Yi ) 4 θ ) V (ˆθ 1 ) = V (Y ) = 4V (Y ) = 4 = = θ 1 3. To fid the mea ad variace of ˆθ, recall (see Exercise 6.74) that the desity fuctio of Y () is give by ( y ) ( ) 1 1 g () (y) = [F Y (y)] 1, 0 y θ, f Y (y) = θ θ Thus, E(Y () ) = θ ( ) y dy = θ, θ ad it follows that E{[( + 1)/]Y () }=θ; that is, ˆθ is a ubiased estimator for θ. Because E(Y() ) = θ ( ) y +1 dy = θ, θ + we obtai V (Y () ) = E(Y () ) [E(Y ())] = 0 [ + ( ) ] θ + 1 ad [( ) ] ( ) V (ˆθ ) = V Y () = V (Y ()) [ ( + 1) ] = ( + ) 1 θ θ = ( + ). Therefore, the efficiecy of ˆθ 1 relative to ˆθ is give by eff (ˆθ 1, ˆθ ) = V (ˆθ ) V (ˆθ 1 ) = θ /[( + )] θ /3 = 3 +. This efficiecy is less tha 1 if > 1. That is, if > 1, ˆθ has a smaller variace tha ˆθ 1, ad therefore ˆθ is geerally preferable to ˆθ 1 as a estimator of θ.

4 Exercises 447 We preset some methods for fidig estimators with small variaces later i this chapter. For ow we wish oly to poit out that relative efficiecy is oe importat criterio for comparig estimators. Exercises 9.1 I Exercise 8.8, we cosidered a radom sample of size 3 from a expoetial distributio with desity fuctio give by { (1/θ)e y/θ, 0 < y, f (y) = 0, elsewhere, ad determied that ˆθ 1 = Y 1, ˆθ = (Y 1 +Y )/, ˆθ 3 = (Y 1 +Y )/3, ad ˆθ 5 = Y are all ubiased estimators for θ. Fid the efficiecy of ˆθ 1 relative to ˆθ 5,ofˆθ relative to ˆθ 5, ad of ˆθ 3 relative to ˆθ Let Y 1, Y,..., Y deote a radom sample from a populatio with mea μ ad variace σ. Cosider the followig three estimators for μ: ˆμ 1 = 1 (Y 1 + Y ), ˆμ = 1 4 Y 1 + Y + +Y 1 ( ) Y, ˆμ 3 = Y. a b Show that each of the three estimators is ubiased. Fid the efficiecy of ˆμ 3 relative to ˆμ ad ˆμ 1, respectively. 9.3 Let Y 1, Y,..., Y deote a radom sample from the uiform distributio o the iterval (θ, θ + 1). Let ˆθ 1 = Y 1 ad ˆθ = Y () + 1. a Show that both ˆθ 1 ad ˆθ are ubiased estimators of θ. b Fid the efficiecy of ˆθ 1 relative to ˆθ. 9.4 Let Y 1, Y,..., Y deote a radom sample of size from a uiform distributio o the iterval (0, θ).ify (1) = mi(y 1, Y,..., Y ), the result of Exercise 8.18 is that ˆθ 1 = ( + 1)Y (1) is a ubiased estimator for θ.ify () = max(y 1, Y,..., Y ), the results of Example 9.1 imply that ˆθ = [( + 1)/]Y () is aother ubiased estimator for θ. Show that the efficiecy of ˆθ 1 to ˆθ is 1/. Notice that this implies that ˆθ is a markedly superior estimator. 9.5 Suppose that Y 1, Y,..., Y is a radom sample from a ormal distributio with mea μ ad variace σ. Two ubiased estimators of σ are ˆσ 1 = S = 1 (Y i Y ) 1 ad ˆσ = 1 (Y 1 Y ). Fid the efficiecy of ˆσ 1 relative to ˆσ. 9.6 Suppose that Y 1, Y,..., Y deote a radom sample of size from a Poisso distributio with mea λ. Cosider ˆλ 1 = (Y 1 + Y )/ ad ˆλ = Y. Derive the efficiecy of ˆλ 1 relative to ˆλ. 9.7 Suppose that Y 1, Y,..., Y deote a radom sample of size from a expoetial distributio with desity fuctio give by { (1/θ)e y/θ, 0 < y, f (y) =

5 448 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio I Exercise 8.19, we determied that ˆθ 1 = Y (1) is a ubiased estimator of θ with MSE(ˆθ 1 ) = θ. Cosider the estimator ˆθ = Y ad fid the efficiecy of ˆθ 1 relative to ˆθ. *9.8 Let Y 1, Y,..., Y deote a radom sample from a probability desity fuctio f (y), which has ukow parameter θ. If ˆθ is a ubiased estimator of θ, the uder very geeral coditios [ ( )] 1 V (ˆθ) I (θ), where I (θ) = E l f (Y ). θ (This is kow as the Cramer Rao iequality.) If V (ˆθ) = I (θ), the estimator ˆθ is said to be efficiet. 1 a b Suppose that f (y) is the ormal desity with mea μ ad variace σ. Show that Y is a efficiet estimator of μ. This iequality also holds for discrete probability fuctios p(y). Suppose that p(y) is the Poisso probability fuctio with mea λ. Show that Y is a efficiet estimator of λ. 9.3 Cosistecy Suppose that a coi, which has probability p of resultig i heads, is tossed times. If the tosses are idepedet, the Y, the umber of heads amog the tosses, has a biomial distributio. If the true value of p is ukow, the sample proportio Y/ is a estimator of p. What happes to this sample proportio as the umber of tosses icreases? Our ituitio leads us to believe that as gets larger, Y/ should get closer to the true value of p. That is, as the amout of iformatio i the sample icreases, our estimator should get closer to the quatity beig estimated. Figure 9.1 illustrates the values of ˆp = Y/ for a sigle sequece of 1000 Beroulli trials whe the true value of p is 0.5. Notice that the values of ˆp bouce aroud 0.5 whe the umber of trials is small but approach ad stay very close to p = 0.5asthe umber of trials icreases. The sigle sequece of 1000 trials illustrated i Figure 9.1 resulted (for larger ) i values for the estimate that were very close to the true value, p = 0.5. Would additioal sequeces yield similar results? Figure 9. shows the combied results of 50 sequeces of 1000 trials. Notice that the 50 distict sequeces were ot idetical. Rather, Figure 9. shows a covergece of sorts to the true value p = 0.5. This is exhibited by a wider spread of the values of the estimates for smaller umbers of trials but a much arrower spread of values of the estimates whe the umber of trials is larger. Will we observe this same pheomeo for differet values of p? Some of the exercises at the ed of this sectio will allow you to use applets (accessible at to explore more fully for yourself. How ca we techically express the type of covergece exhibited i Figure 9.? Because Y/ is a radom variable, we may express this closeess to p i probabilistic terms. I particular, let us examie the probability that the distace betwee the estimator ad the target parameter, (Y/) p, will be less tha some arbitrary positive real umber ε. Figure 9. seems to idicate that this probability might be 1. Exercises preceded by a asterisk are optioal.

6 9.3 Cosistecy 449 FIGURE 9.1 Values of ˆp = Y/ for a sigle sequece of 1000 Beroulli trials, p = 0.5 Estimate of p Trials FIGURE 9. Values of ˆp = Y/ for 50 sequeces of 1000 Beroulli trials, p = 0.5 Estimate of p Trials icreasig as gets larger. If our ituitio is correct ad is large, this probability, ( ) Y P p ε, should be close to 1. If this probability i fact does ted to 1 as, we the say that (Y/) is a cosistet estimator of p, or that (Y/) coverges i probability to p.

7 450 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio DEFINITION 9. The estimator ˆθ is said to be a cosistet estimator of θ if, for ay positive umber ε, or, equivaletly, lim P( ˆθ θ ε) = 1 lim P( ˆθ θ >ε)= 0. The otatio ˆθ expresses that the estimator for θ is calculated by usig a sample of size. For example, Y is the average of two observatios whereas Y 100 is the average of the 100 observatios cotaied i a sample of size = 100. If ˆθ is a ubiased estimator, the followig theorem ca ofte be used to prove that the estimator is cosistet. THEOREM 9.1 Proof A ubiased estimator ˆθ for θ is a cosistet estimator of θ if lim V (ˆθ ) = 0. If Y is ay radom variable with E(Y ) = μ ad V (Y ) = σ < ad if k is ay oegative costat, Tchebysheff s theorem (see Theorem 4.13) implies that P( Y μ > kσ) 1 k. Because ˆθ is a ubiased estimator for θ, it follows that E(ˆθ ) = θ. Let σˆθ = V (ˆθ ) deote the stadard error of the estimator ˆθ. If we apply Tchebysheff s theorem for the radom variable ˆθ, we obtai P ( ˆθ θ ) 1 > kσˆθ k. Let be ay fixed sample size. For ay positive umber ε, k = ε is a positive umber. Applicatio of Tchebysheff s theorem for this fixed ad this choice of k shows that P ( ˆθ θ ( ) >ε = P ˆθ θ [ ] ) ε 1 > σˆθ ( ) = V (ˆθ ). ε ε/σˆθ Thus, for ay fixed, σˆθ σˆθ 0 P ( ˆθ θ >ε ) V (ˆθ ) ε.

8 9.3 Cosistecy 451 If lim V (ˆθ ) = 0 ad we take the limit as of the precedig sequece of probabilities, lim (0) lim P( ˆθ θ >ε ) V (ˆθ ) lim = 0. ε Thus, ˆθ is a cosistet estimator for θ. The cosistecy property give i Defiitio 9. ad discussed i Theorem 9.1 ivolves a particular type of covergece of ˆθ to θ. For this reaso, the statemet ˆθ is a cosistet estimator for θ is sometimes replaced by the equivalet statemet ˆθ coverges i probability to θ. EXAMPLE 9. Solutio Let Y 1, Y,..., Y deote a radom sample from a distributio with mea μ ad variace σ <. Show that Y = 1 Y i is a cosistet estimator of μ. (Note: We use the otatio Y to explicitly idicate that Y is calculated by usig a sample of size.) We kow from earlier chapters that E(Y ) = μ ad V (Y ) = σ /. Because Y is ubiased for μ ad V (Y ) 0as, Theorem 9.1 establishes that Y is a cosistet estimator of μ. Equivaletly, we may say that Y coverges i probability to μ. The fact that Y is cosistet for μ, or coverges i probability to μ, is sometimes referred to as the law of large umbers. It provides the theoretical justificatio for the averagig process employed by may experimeters to obtai precisio i measuremets. For example, a experimeter may take the average of the weights of may aimals to obtai a more precise estimate of the average weight of aimals of this species. The experimeter s feelig, a feelig cofirmed by Theorem 9.1, is that the average of may idepedetly selected weights should be quite close to the true mea weight with high probability. I Sectio 8.3, we cosidered a ituitive estimator for μ 1 μ, the differece i the meas of two populatios. The estimator discussed at that time was Y 1 Y, the differece i the meas of idepedet radom samples selected from two populatios. The results of Theorem 9. will be very useful i establishig the cosistecy of such estimators. THEOREM 9. Suppose that ˆθ coverges i probability to θ ad that ˆθ coverges i probability to θ. a ˆθ + ˆθ coverges i probability to θ + θ. b ˆθ ˆθ coverges i probability to θ θ. c If θ 0, ˆθ /ˆθ coverges i probability to θ/θ. d If g( ) is a real-valued fuctio that is cotiuous at θ, the g(ˆθ ) coverges i probability to g(θ).

9 45 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio The proof of Theorem 9. closely resembles the correspodig proof i the case where {a } ad {b } are sequeces of real umbers covergig to real limits a ad b, respectively. For example, if a a ad b b the a + b a + b. EXAMPLE 9.3 Suppose that Y 1, Y,..., Y represet a radom sample such that E(Y i ) = μ, E(Yi ) = μ 4 ad E(Yi ) = μ 4 are all fiite. Show that S = 1 (Y i Y ) 1 is a cosistet estimator of σ = V (Y i ).(Note: We use subscript o both S ad Y to explicitly covey their depedece o the value of the sample size.) Solutio We have see i earlier chapters that S, ow writte as S ( ),is S = 1 ( ) ( Yi Y 1 = 1 1 Y i ) Y. The statistic (1/) Y i is the average of idepedet ad idetically distributed radom variables, with E(Yi ) = μ ad V (Yi ) = μ 4 (μ ) <. By the law of large umbers (Example 9.), we kow that (1/) Y i coverges i probability to μ. Example 9. also implies that Y coverges i probability to μ. Because the fuctio g(x) = x is cotiuous for all fiite values of x, Theorem 9.(d) implies that Y coverges i probability to μ. It the follows from Theorem 9.(a) that 1 Yi Y coverges i probability to μ μ = σ. Because /( 1) is a sequece of costats covergig to 1 as, we ca coclude that S coverges i probability to σ. Equivaletly, S, the sample variace, is a cosistet estimator for σ, the populatio variace. I Sectio 8.6, we cosidered large-sample cofidece itervals for some parameters of practical iterest. I particular, if Y 1, Y,..., Y is a radom sample from ay distributio with mea μ ad variace σ, we established that Y ± z α/ ( σ ) is a valid large-sample cofidece iterval with cofidece coefficiet approximately equal to (1 α).ifσ is kow, this iterval ca ad should be calculated. However, if σ is ot kow but the sample size is large, we recommeded substitutig S for σ i the calculatio because this etails o sigificat loss of accuracy. The followig theorem provides the theoretical justificatio for these claims.

10 9.3 Cosistecy 453 THEOREM 9.3 Suppose that U has a distributio fuctio that coverges to a stadard ormal distributio fuctio as.ifw coverges i probability to 1, the the distributio fuctio of U /W coverges to a stadard ormal distributio fuctio. This result follows from a geeral result kow as Slutsky s theorem (Serflig, 00). The proof of this result is beyod the scope of this text. However, the usefuless of the result is illustrated i the followig example. EXAMPLE 9.4 Suppose that Y 1, Y,..., Y is a radom sample of size from a distributio with E(Y i ) = μ ad V (Y i ) = σ. Defie S as S = 1 (Y i Y ). 1 Show that the distributio fuctio of ) (Y μ S coverges to a stadard ormal distributio fuctio. Solutio I Example 9.3, we showed that S coverges i probability to σ. Notice that g(x) = + x/c is a cotiuous fuctio of x if both x ad c are positive. Hece, it follows from Theorem 9.(d) that S /σ =+ S /σ coverges i probability to 1. We also kow from the cetral limit theorem (Theorem 7.4) that the distributio fuctio of U = ( Y μ σ coverges to a stadard ormal distributio fuctio. Therefore, Theorem 9.3 implies that the distributio fuctio of )/ (Y μ (S /σ ) = ( ) Y μ σ S coverges to a stadard ormal distributio fuctio. ) The result of Example 9.4 tells us that, whe is large, (Y μ)/s has approximately a stadard ormal distributio whatever is the form of the distributio from which the sample is take. If the sample is take from a ormal distributio, the results of Chapter 7 imply that t = (Y μ)/s has a t distributio with 1 degrees of freedom (df). Combiig this iformatio, we see that, if a large sample is take from a ormal distributio, the distributio fuctio of t = (Y μ)/s ca be approximated by a stadard ormal distributio fuctio. That is, as gets large ad hece as the umber of degrees of freedom gets large, the t-distributio fuctio coverges to the stadard ormal distributio fuctio.

11 454 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio If we obtai a large sample from ay distributio, we kow from Example 9.4 that (Y μ)/s has approximately a stadard ormal distributio. Therefore, it follows that [ P z α/ ( ) ] Y μ z α/ 1 α. S If we maipulate the iequalities i the probability statemet to isolate μ i the middle, we obtai P [ ( ) ( )] S S Y z α/ μ Y + z α/ 1 α. Thus, Y ± z α/ (S / ) forms a valid large-sample cofidece iterval for μ, with cofidece coefficiet approximately equal to 1 α. Similarly, Theorem 9.3 ca be applied to show that ˆp ˆq ˆp ± z α/ is a valid large-sample cofidece iterval for p with cofidece coefficiet approximately equal to 1 α. I this sectio, we have see that the property of cosistecy tells us somethig about the distace betwee a estimator ad the quatity beig estimated. We have see that, whe the sample size is large, Y is close to μ, ad S is close to σ, with high probability. We will see other examples of cosistet estimators i the exercises ad later i the chapter. I this sectio, we have used the otatio Y, S, ˆp, ad, i geeral, ˆθ to explicitly covey the depedece of the estimators o the sample size. We eeded to do so because we were iterested i computig lim P( ˆθ θ ε). If this limit is 1, the ˆθ is a cosistet estimator for θ (more precisely, ˆθ a cosistet sequece of estimators for θ). Ufortuately, this otatio makes our estimators look overly complicated. Heceforth, we will revert to the otatio ˆθ as our estimator for θ ad ot explicitly display the depedece of the estimator o. The depedece of ˆθ o the sample size is always implicit ad should be used wheever the cosistecy of the estimator is cosidered. Exercises 9.9 Applet Exercise How was Figure 9.1 obtaied? Access the applet PoitSigle at www. thomsoedu.com/statistics/wackerly. The top applet will geerate a sequece of Beroulli trials [X i = 1, 0 with p(1) = p, p(0) = 1 p] with p =.5, a sceario equivalet to successively tossig a balaced coi. Let Y = X i = the umber of 1s i the first trials ad ˆp = Y /. For each, the applet computes ˆp ad plots it versus the value of. a If ˆp 5 = /5, what value of X 6 will result i ˆp 6 > ˆp 5? b Click the butto Oe Trial a sigle time. Your first observatio is either 0 or 1. Which value did you obtai? What was the value of ˆp 1? Click the butto Oe Trial several more

12 Exercises 455 c d e times. How may trials have you simulated? What value of ˆp did you observe? Is the value close to.5, the true value of p? Is the graph a flat horizotal lie? Why or why ot? Click the butto 100 Trials a sigle time. What do you observe? Click the butto 100 Trials repeatedly util the total umber of trials is Is the graph that you obtaied idetical to the oe give i Figure 9.1? I what sese is it similar to the graph i Figure 9.1? Based o the sample of size 1000, what is the value of ˆp 1000? Is this value what you expected to observe? Click the butto Reset. Click the butto 100 Trials te times to geerate aother sequece of values for ˆp. Commet Applet Exercise Refer to Exercise 9.9. Scroll dow to the portio of the scree labeled Try differet probabilities. Use the butto labeled p = i the lower right corer of the display to chage the value of p to a value other tha.5. a b Click the butto Oe Trial a few times. What do you observe? Click the butto 100 Trials a few times. What do you observe about the values of ˆp as the umber of trials gets larger? 9.11 Applet Exercise Refer to Exercises 9.9 ad How ca the results of several sequeces of Beroulli trials be simultaeously plotted? Access the applet PoitbyPoit. Scroll dow util you ca view all six buttos uder the top graph. a b c Do ot chage the value of p from the preset value p =.5. Click the butto Oe Trial a few times to verify that you are obtaiig a result similar to those obtaied i Exercise 9.9. Click the butto 5 Trials util you have geerated a total of 50 trials. What is the value of ˆp 50 that you obtaied at the ed of this first sequece of 50 trials? Click the butto New Sequece. The color of your iitial graph chages from red to gree. Click the butto 5 Trials a few times. What do you observe? Is the graph the same as the oe you observed i part (a)? I what sese is it similar? Click the butto New Sequece. Geerate a ew sequece of 50 trials. Repeat util you have geerated five sequeces. Are the paths geerated by the five sequeces idetical? I what sese are they similar? 9.1 Applet Exercise Refer to Exercise What happes if each sequece is loger? Scroll dow to the portio of the scree labeled Loger Sequeces of Trials. a Repeat the istructios i parts (a) (c) of Exercise b What do you expect to happe if p is ot 0.5? Use the butto i the lower right corer to chage to value of p. Geerate several sequeces of trials. Commet Applet Exercise Refer to Exercises Access the applet Poit Estimatio. a b Chose a value for p. Click the butto New Sequece repeatedly. What do you observe? Scroll dow to the portio of the applet labeled More Trials. Choose a value for p ad click the butto New Sequece repeatedly. You will obtai up to 50 sequeces, each based o 1000 trials. How does the variability amog the estimates chage as a fuctio of the sample size? How is this maifested i the display that you obtaied? 9.14 Applet Exercise Refer to Exercise Scroll dow to the portio of the applet labeled Mea of Normal Data. Successive observed values of a stadard ormal radom variable ca be geerated ad used to compute the value of the sample mea Y. These successive values are the plotted versus the respective sample size to obtai oe sample path.

13 456 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio a b c Do you expect the values of Y to cluster aroud ay particular value? What value? If the results of 50 sample paths are plotted, how do you expect the variability of the estimates to chage as a fuctio of sample size? Click the butto New Sequece several times. Did you observe what you expected based o your aswers to parts (a) ad (b)? 9.15 Refer to Exercise 9.3. Show that both ˆθ 1 ad ˆθ are cosistet estimators for θ Refer to Exercise 9.5. Is ˆσ a cosistet estimator of σ? 9.17 Suppose that X 1, X,..., X ad Y 1, Y,..., Y are idepedet radom samples from populatios with meas μ 1 ad μ ad variaces σ1 ad σ, respectively. Show that X Y is a cosistet estimator of μ 1 μ I Exercise 9.17, suppose that the populatios are ormally distributed with σ 1 = σ = σ. Show that (X i X) + (Y i Y ) is a cosistet estimator of σ Let Y 1, Y,..., Y deote a radom sample from the probability desity fuctio { θy f (y) = θ 1, 0 < y < 1, 0, elsewhere, where θ>0. Show that Y is a cosistet estimator of θ/(θ + 1). 9.0 If Y has a biomial distributio with trials ad success probability p, show that Y/ is a cosistet estimator of p. 9.1 Let Y 1, Y,...,Y be a radom sample of size from a ormal populatio with mea μ ad variace σ. Assumig that = k for some iteger k, oe possible estimator for σ is give by ˆσ = 1 k k (Y i Y i 1 ). a Show that ˆσ is a ubiased estimator for σ. b Show that ˆσ is a cosistet estimator for σ. 9. Refer to Exercise 9.1. Suppose that Y 1, Y,...,Y is a radom sample of size from a Poisso-distributed populatio with mea λ. Agai, assume that = k for some iteger k. Cosider ˆλ = 1 k k (Y i Y i 1 ). a Show that ˆλ is a ubiased estimator for λ. b Show that ˆλ is a cosistet estimator for λ. 9.3 Refer to Exercise 9.1. Suppose that Y 1, Y,...,Y is a radom sample of size from a populatio for which the first four momets are fiite. That is, m 1 = E(Y 1)<, m = E(Y1 )<, m 3 = E(Y 1 3)<, ad m 4 = E(Y 1 4 )<.(Note: This assumptio is valid for the ormal ad Poisso distributios i Exercises 9.1 ad 9., respectively.) Agai, assume

14 Exercises 457 that = k for some iteger k. Cosider ˆσ = 1 k k (Y i Y i 1 ). a Show that ˆσ is a ubiased estimator for σ. b Show that ˆσ is a cosistet estimator for σ. c Why did you eed the assumptio that m 4 = E(Y 1 4)<? 9.4 Let Y 1, Y, Y 3,...Y be idepedet stadard ormal radom variables. a What is the distributio of Y i? b Let W = 1 Y i. Does W coverge i probability to some costat? If so, what is the value of the costat? 9.5 Suppose that Y 1, Y,..., Y deote a radom sample of size from a ormal distributio with mea μ ad variace 1. Cosider the first observatio Y 1 as a estimator for μ. a Show that Y 1 is a ubiased estimator for μ. b Fid P( Y 1 μ 1). c Look at the basic defiitio of cosistecy give i Defiitio 9.. Based o the result of part (b), is Y 1 a cosistet estimator for μ? *9.6 It is sometimes relatively easy to establish cosistecy or lack of cosistecy by appealig directly to Defiitio 9., evaluatig P( ˆθ θ ε) directly, ad the showig that lim P( ˆθ θ ε) = 1. Let Y 1, Y,..., Y deote a radom sample of size from a uiform distributio o the iterval (0, θ).ify () = max(y 1, Y,..., Y ), we showed i Exercise 6.74 that the probability distributio fuctio of Y () is give by 0, y < 0, F () (y) = (y/θ), 0 y θ, 1, y >θ. a b For each 1 ad every ε>0, it follows that P( Y () θ ε) = P(θ ε Y () θ + ε). Ifε>θ, verify that P(θ ε Y () θ + ε) = 1 ad that, for every positive ε<θ, we obtai P(θ ε Y () θ + ε) = 1 [(θ ε)/θ]. Usig the result from part (a), show that Y () is a cosistet estimator for θ by showig that, for every ε>0, lim P( Y () θ ε) = 1. *9.7 Use the method described i Exercise 9.6 to show that, if Y (1) = mi(y 1, Y,..., Y ) whe Y 1, Y,..., Y are idepedet uiform radom variables o the iterval (0, θ), the Y (1) is ot a cosistet estimator for θ.[hit: Based o the methods of Sectio 6.7, Y (1) has the distributio fuctio 0, y < 0, F (1) (y) = 1 (1 y/θ), 0 y θ, 1, y >θ.] *9.8 Let Y 1, Y,..., Y deote a radom sample of size from a Pareto distributio (see Exercise 6.18). The the methods of Sectio 6.7 imply that Y (1) = mi(y 1, Y,..., Y ) has the distributio fuctio give by { 0, y β, F (1) (y) = 1 (β/y) α, y >β. Use the method described i Exercise 9.6 to show that Y (1) is a cosistet estimator of β.

15 458 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio *9.9 Let Y 1, Y,..., Y deote a radom sample of size from a power family distributio (see Exercise 6.17). The the methods of Sectio 6.7 imply that Y () = max(y 1, Y,..., Y ) has the distributio fuctio give by 0, y < 0, F () (y) = (y/θ) α, 0 y θ, 1, y >θ. Use the method described i Exercise 9.6 to show that Y () is a cosistet estimator of θ Let Y 1, Y,..., Y be idepedet radom variables, each with probability desity fuctio { 3y, 0 y 1, f (y) = Show that Y coverges i probability to some costat ad fid the costat If Y 1, Y,..., Y deote a radom sample from a gamma distributio with parameters α ad β, show that Y coverges i probability to some costat ad fid the costat. 9.3 Let Y 1, Y,..., Y deote a radom sample from the probability desity fuctio f (y) = y, y, Does the law of large umbers apply to Y i this case? Why or why ot? 9.33 A experimeter wishes to compare the umbers of bacteria of types A ad B i samples of water. A total of idepedet water samples are take, ad couts are made for each sample. Let X i deote the umber of type A bacteria ad Y i deote the umber of type B bacteria for sample i. Assume that the two bacteria types are sparsely distributed withi a water sample so that X 1, X,..., X ad Y 1, Y,..., Y ca be cosidered idepedet radom samples from Poisso distributios with meas λ 1 ad λ, respectively. Suggest a estimator of λ 1 /(λ 1 +λ ). What properties does your estimator have? 9.34 The Rayleigh desity fuctio is give by ( ) y e y /θ, y > 0, f (y) = θ I Exercise 6.34(a), you established that Y has a expoetial distributio with mea θ. If Y 1, Y,..., Y deote a radom sample from a Rayleigh distributio, show that W = Y i is a cosistet estimator for θ Let Y 1, Y,... be a sequece of radom variables with E(Y i ) = μ ad V (Y i ) = σi. Notice that the σi s are ot all equal. a What is E(Y )? b What is V (Y )? c Uder what coditio (o the σi s) ca Theorem 9.1 be applied to show that Y is a cosistet estimator for μ? 9.36 Suppose that Y has a biomial distributio based o trials ad success probability p. The ˆp = Y/ is a ubiased estimator of p. Use Theorem 9.3 to prove that the distributio of

16 9.4 Sufficiecy 459 ( ˆp p)/ ˆp ˆq / coverges to a stadard ormal distributio. [Hit: Write Y as we did i Sectio 7.5.] 9.4 Sufficiecy Up to this poit, we have chose estimators o the basis of ituitio. Thus, we chose Y ad S as the estimators of the mea ad variace, respectively, of the ormal distributio. (It seems like these should be good estimators of the populatio parameters.) We have see that it is sometimes desirable to use estimators that are ubiased. Ideed, Y ad S have bee show to be ubiased estimators of the populatio mea μ ad variace σ, respectively. Notice that we have used the iformatio i a sample of size to calculate the value of two statistics that fuctio as estimators for the parameters of iterest. At this stage, the actual sample values are o loger importat; rather, we summarize the iformatio i the sample that relates to the parameters of iterest by usig the statistics Y ad S. Has this process of summarizig or reducig the data to the two statistics, Y ad S, retaied all the iformatio about μ ad σ i the origial set of sample observatios? Or has some iformatio about these parameters bee lost or obscured through the process of reducig the data? I this sectio, we preset methods for fidig statistics that i a sese summarize all the iformatio i a sample about a target parameter. Such statistics are said to have the property of sufficiecy; or more simply, they are called sufficiet statistics. As we will see i the ext sectio, good estimators are (or ca be made to be) fuctios of ay sufficiet statistic. Ideed, sufficiet statistics ofte ca be used to develop estimators that have the miimum variace amog all ubiased estimators. To illustrate the otio of a sufficiet statistic, let us cosider the outcomes of trials of a biomial experimet, X 1, X,..., X, where { 1, if the ith trial is a success, X i = 0, if the ith trial is a failure. If p is the probability of success o ay trial the, for i = 1,,...,, { 1, with probability p, X i = 0, with probability q = 1 p. Suppose that we are give a value of Y = X i, the umber of successes amog the trials. If we kow the value of Y, ca we gai ay further iformatio about p by lookig at other fuctios of X 1, X,..., X? Oe way to aswer this questio is to look at the coditioal distributio of X 1, X,..., X,giveY : P(X 1 = x 1,...,X = x Y = y) = P(X 1 = x 1,...,X = x, Y = y). P(Y = y) The umerator o the right side of this expressio is 0 if x i y, ad it is the probability of a idepedet sequece of 0s ad 1s with a total of y 1s ad ( y) 0s if x i = y. Also, the deomiator is the biomial probability of exactly y

17 460 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio successes i trials. Therefore, if y = 0, 1,,...,, p y (1 p) y ( ) P(X 1 = x 1,...,X = x Y = y) = py (1 p) = 1 ( y y if x i = y, y), 0, otherwise. It is importat to ote that the coditioal distributio of X 1, X,..., X, give Y, does ot deped upo p. That is, oce Y is kow, o other fuctio of X 1, X,...,X will shed additioal light o the possible value of p. I this sese, Y cotais all the iformatio about p. Therefore, the statistic Y is said to be sufficiet for p. We geeralize this idea i the followig defiitio. DEFINITION 9.3 Let Y 1, Y,..., Y deote a radom sample from a probability distributio with ukow parameter θ. The the statistic U = g(y 1, Y,..., Y ) is said to be sufficiet for θ if the coditioal distributio of Y 1, Y,..., Y,giveU, does ot deped o θ. I may previous discussios, we have cosidered the probability fuctio p(y) associated with a discrete radom variable [or the desity fuctio f (y) for a cotiuous radom variable] to be fuctios of the argumet y oly. Our future discussios will be simplified if we adopt otatio that will permit us to explicitly display the fact that the distributio associated with a radom variable Y ofte depeds o the value of a parameter θ.ify is a discrete radom variable that has a probability mass fuctio that depeds o the value of a parameter θ, istead of p(y) we use the otatio p(y θ). Similarly, we will idicate the explicit depedece of the form of a cotiuous desity fuctio o the value of a parameter θ by writig the desity fuctio as f (y θ) istead of the previously used f (y). Defiitio 9.3 tells us how to check whether a statistic is sufficiet, but it does ot tell us how to fid a sufficiet statistic. Recall that i the discrete case the joit distributio of discrete radom variables Y 1, Y,..., Y is give by a probability fuctio p(y 1, y,..., y ). If this joit probability fuctio depeds explicitly o the value of a parameter θ, we write it as p(y 1, y,..., y θ). This fuctio gives the probability or likelihood of observig the evet (Y 1 = y 1, Y = y,..., Y = y ) whe the value of the parameter is θ. I the cotiuous case whe the joit distributio of Y 1, Y,..., Y depeds o a parameter θ, we will write the joit desity fuctio as f (y 1, y,...,y θ). Heceforth, it will be coveiet to have a sigle ame for the fuctio that defies the joit distributio of the variables Y 1, Y,..., Y observed i a sample. DEFINITION 9.4 Let y 1, y,..., y be sample observatios take o correspodig radom variables Y 1, Y,..., Y whose distributio depeds o a parameter θ. The, if Y 1, Y,..., Y are discrete radom variables, the likelihood of the sample, L(y 1, y,..., y θ), is defied to be the joit probability of y 1, y,..., y.

18 9.4 Sufficiecy 461 If Y 1, Y,..., Y are cotiuous radom variables, the likelihood L(y 1, y,..., y θ) is defied to be the joit desity evaluated at y 1, y,..., y. If the set of radom variables Y 1, Y,..., Y deotes a radom sample from a discrete distributio with probability fuctio p(y θ), the L(y 1, y,...,y θ) = p(y 1, y,...,y θ) whereas if Y 1, Y,..., Y f (y θ), the = p(y 1 θ) p(y θ) p(y θ), have a cotiuous distributio with desity fuctio L(y 1, y,...,y θ) = f (y 1, y,...,y θ) = f (y 1 θ) f (y θ) f (y θ). To simplify otatio, we will sometimes deote the likelihood by L(θ) istead of by L(y 1, y,..., y θ). The followig theorem relates the property of sufficiecy to the likelihood L(θ). THEOREM 9.4 Let U be a statistic based o the radom sample Y 1, Y,..., Y. The U is a sufficiet statistic for the estimatio of a parameter θ if ad oly if the likelihood L(θ) = L(y 1, y,...,y θ) ca be factored ito two oegative fuctios, L(y 1, y,...,y θ) = g(u,θ) h(y 1, y,..., y ) where g(u,θ) is a fuctio oly of u ad θ ad h(y 1, y,...,y ) is ot a fuctio of θ. Although the proof of Theorem 9.4 (also kow as the factorizatio criterio) is beyod the scope of this book, we illustrate the usefuless of the theorem i the followig example. EXAMPLE 9.5 Let Y 1, Y,..., Y be a radom sample i which Y i possesses the probability desity fuctio { (1/θ)e y i /θ, 0 y i <, f (y i θ) = 0, elsewhere, where θ>0, i = 1,,...,. Show that Y is a sufficiet statistic for the parameter θ. Solutio The likelihood L(θ) of the sample is the joit desity L(y 1, y,..., y θ) = f (y 1, y,..., y θ) = f (y 1 θ) f (y θ) f (y θ) = e y 1/θ θ e y /θ θ e y /θ θ = e yi /θ θ = e y/θ θ.

19 46 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio Notice that L(θ) is a fuctio oly of θ ad y ad that if the g(y, θ)= e y/θ θ ad h(y 1, y,..., y ) = 1, L(y 1, y,..., y θ) = g(y,θ) h(y 1, y,..., y ). Hece, Theorem 9.4 implies that Y is a sufficiet statistic for the parameter θ. Theorem 9.4 ca be used to show that there are may possible sufficiet statistics for ay oe populatio parameter. First of all, accordig to Defiitio 9.3 or the factorizatio criterio (Theorem 9.4), the radom sample itself is a sufficiet statistic. Secod, if Y 1, Y,..., Y deote a radom sample from a distributio with a desity fuctio with parameter θ, the the set of order statistics Y (1) Y () Y (), which is a fuctio of Y 1, Y,..., Y, is sufficiet for θ. I Example 9.5, we decided that Y is a sufficiet statistic for the estimatio of θ. Theorem 9.4 could also have bee used to show that Y i is aother sufficiet statistic. Ideed, for the expoetial distributio described i Example 9.5, ay statistic that is a oe to oe fuctio of Y is a sufficiet statistic. I our iitial example of this sectio, ivolvig the umber of successes i trials, Y = X i reduces the data X 1, X,..., X to a sigle value that remais sufficiet for p. Geerally, we would like to fid a sufficiet statistic that reduces the data i the sample as much as possible. Although may statistics are sufficiet for the parameter θ associated with a specific distributio, applicatio of the factorizatio criterio typically leads to a statistic that provides the best summary of the iformatio i the data. I Example 9.5, this statistic is Y (or some oe-to-oe fuctio of it). I the ext sectio, we show how these sufficiet statistics ca be used to develop ubiased estimators with miimum variace. Exercises 9.37 Let X 1, X,..., X deote idepedet ad idetically distributed Beroulli radom variables such that P(X i = 1) = p ad P(X i = 0) = 1 p, for each i = 1,,...,. Show that X i is sufficiet for p by usig the factorizatio criterio give i Theorem Let Y 1, Y,..., Y deote a radom sample from a ormal distributio with mea μ ad variace σ. a If μ is ukow ad σ is kow, show that Y is sufficiet for μ. b If μ is kow ad σ is ukow, show that (Y i μ) is sufficiet for σ. c If μ ad σ are both ukow, show that Y i ad Y i are joitly sufficiet for μ ad σ. [Thus, it follows that Y ad (Y i Y ) or Y ad S are also joitly sufficiet for μ ad σ.]

20 Exercises Let Y 1, Y,..., Y deote a radom sample from a Poisso distributio with parameter λ. Show by coditioig that Y i is sufficiet for λ Let Y 1, Y,..., Y deote a radom sample from a Rayleigh distributio with parameter θ. (Refer to Exercise 9.34.) Show that Y i is sufficiet for θ Let Y 1, Y,..., Y deote a radom sample from a Weibull distributio with kow m ad ukow α. (Refer to Exercise 6.6.) Show that Y i m is sufficiet for α. 9.4 If Y 1, Y,..., Y deote a radom sample from a geometric distributio with parameter p, show that Y is sufficiet for p Let Y 1, Y,..., Y deote idepedet ad idetically distributed radom variables from a power family distributio with parameters α ad θ. The, by the result i Exercise 6.17, if α, θ > 0, { αy α 1 /θ α, 0 y θ, f (y α, θ) = If θ is kow, show that Y i is sufficiet for α Let Y 1, Y,..., Y deote idepedet ad idetically distributed radom variables from a Pareto distributio with parameters α ad β. The, by the result i Exercise 6.18, if α, β > 0, { αβ α y (α+1), y β, f (y α, β) = If β is kow, show that Y i is sufficiet for α Suppose that Y 1, Y,..., Y is a radom sample from a probability desity fuctio i the (oe-parameter) expoetial family so that { a(θ)b(y)e [c(θ)d(y)], a y b, f (y θ) = 0, elsewhere, where a ad b do ot deped o θ. Show that d(y i) is sufficiet for θ If Y 1, Y,..., Y deote a radom sample from a expoetial distributio with mea β, show that f (y β) is i the expoetial family ad that Y is sufficiet for β Refer to Exercise If θ is kow, show that the power family of distributios is i the expoetial family. What is a sufficiet statistic for α? Does this cotradict your aswer to Exercise 9.43? 9.48 Refer to Exercise If β is kow, show that the Pareto distributio is i the expoetial family. What is a sufficiet statistic for α? Argue that there is o cotradictio betwee your aswer to this exercise ad the aswer you foud i Exercise *9.49 Let Y 1, Y,..., Y deote a radom sample from the uiform distributio over the iterval (0,θ). Show that Y () = max(y 1, Y,..., Y ) is sufficiet for θ. *9.50 Let Y 1, Y,..., Y deote a radom sample from the uiform distributio over the iterval (θ 1,θ ). Show that Y (1) = mi(y 1, Y,..., Y ) ad Y () = max(y 1, Y,..., Y ) are joitly sufficiet for θ 1 ad θ. *9.51 Let Y 1, Y,..., Y deote a radom sample from the probability desity fuctio { e (y θ), y θ, f (y θ) = Show that Y (1) = mi(y 1, Y,..., Y ) is sufficiet for θ.

21 464 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio *9.5 Let Y 1, Y,...,Y be a radom sample from a populatio with desity fuctio 3y f (y θ) = θ, 0 y θ, 3 Show that Y () = max(y 1, Y,...,Y ) is sufficiet for θ. *9.53 Let Y 1, Y,...,Y be a radom sample from a populatio with desity fuctio θ f (y θ) = y, θ < y <, 3 Show that Y (1) = mi(y 1, Y,...,Y ) is sufficiet for θ. *9.54 Let Y 1, Y,..., Y deote idepedet ad idetically distributed radom variables from a power family distributio with parameters α ad θ. The, as i Exercise 9.43, if α, θ > 0, { αy α 1 /θ α, 0 y θ, f (y α, θ) = Show that max(y 1, Y,..., Y ) ad Y i are joitly sufficiet for α ad θ. *9.55 Let Y 1, Y,..., Y deote idepedet ad idetically distributed radom variables from a Pareto distributio with parameters α ad β. The, as i Exercise 9.44, if α, β > 0, { αβ α y (α+1), y β, f (y α, β) = Show that Y i ad mi(y 1, Y,..., Y ) are joitly sufficiet for α ad β. 9.5 The Rao Blackwell Theorem ad Miimum-Variace Ubiased Estimatio Sufficiet statistics play a importat role i fidig good estimators for parameters. If ˆθ is a ubiased estimator for θ ad if U is a statistic that is sufficiet for θ, the there is a fuctio of U that is also a ubiased estimator for θ ad has o larger variace tha ˆθ. If we seek ubiased estimators with small variaces, we ca restrict our search to estimators that are fuctios of sufficiet statistics. The theoretical basis for the precedig remarks is provided i the followig result, kow as the Rao Blackwell theorem. THEOREM 9.5 Proof The Rao Blackwell Theorem Let ˆθ be a ubiased estimator for θ such that V (ˆθ) <. IfU is a sufficiet statistic for θ, defie ˆθ = E(ˆθ U). The, for all θ, E (ˆθ ) = θ ad V (ˆθ ) V (ˆθ). Because U is sufficiet for θ, the coditioal distributio of ay statistic (icludig ˆθ), give U, does ot deped o θ. Thus, ˆθ = E(ˆθ U) is ot a fuctio of θ ad is therefore a statistic.

22 9.5 The Rao Blackwell Theorem ad Miimum-Variace Ubiased Estimatio 465 Recall Theorems 5.14 ad 5.15 where we cosidered how to fid meas ad variaces of radom variables by usig coditioal meas ad variaces. Because ˆθ is a ubiased estimator for θ, Theorem 5.14 implies that E(ˆθ ) = E[E(ˆθ U)] = E(ˆθ) = θ. Thus, ˆθ is a ubiased estimator for θ. Theorem 5.15 implies that V (ˆθ) = V [E(ˆθ U)] + E[V (ˆθ U)] = V (ˆθ ) + E[V (ˆθ U)]. Because V (ˆθ U = u) 0 for all u, it follows that E[V (ˆθ U)] 0 ad therefore that V (ˆθ) V (ˆθ ), as claimed. Theorem 9.5 implies that a ubiased estimator for θ with a small variace is or ca be made to be a fuctio of a sufficiet statistic. If we have a ubiased estimator for θ, we might be able to improve it by usig the result i Theorem 9.5. It might iitially seem that the Rao Blackwell theorem could be applied oce to get a better ubiased estimator ad the reapplied to the resultig ew estimator to get a eve better ubiased estimator. If we apply the Rao Blackwell theorem usig the sufficiet statistic U, the ˆθ = E(ˆθ U) will be a fuctio of the statistic U, say, ˆθ = h(u). Suppose that we reapply the Rao Blackwell theorem to ˆθ by usig the same sufficiet statistic U. Sice, i geeral, E(h(U) U) = h(u), we see that by usig the Rao Blackwell theorem agai, our ew estimator is just h(u) = ˆθ. That is, if we use the same sufficiet statistic i successive applicatios of the Rao Blackwell theorem, we gai othig after the first applicatio. The oly way that successive applicatios ca lead to better ubiased estimators is if we use a differet sufficiet statistic whe the theorem is reapplied. Thus, it is uecessary to use the Rao Blackwell theorem successively if we use the right sufficiet statistic i our iitial applicatio. Because may statistics are sufficiet for a parameter θ associated with a distributio, which sufficiet statistic should we use whe we apply this theorem? For the distributios that we discuss i this text, the factorizatio criterio typically idetifies a statistic U that best summarizes the iformatio i the data about the parameter θ. Such statistics are called miimal sufficiet statistics. Exercise 9.66 itroduces a method for determiig a miimal sufficiet statistic that might be of iterest to some readers. I a few of the subsequet exercises, you will see that this method usually yields the same sufficiet statistics as those obtaied from the factorizatio criterio. I the cases that we cosider, these statistics possess aother property (completeess) that guaratees that, if we apply Theorem 9.5 usig U, we ot oly get a estimator with a smaller variace but also actually obtai a ubiased estimator for θ with miimum variace. Such a estimator is called a miimum-variace ubiased estimator (MVUE). See Casella ad Berger (00), Hogg, Craig, ad McKea (005), or Mood, Graybill, ad Boes (1974) for additioal details. Thus, if we start with a ubiased estimator for a parameter θ ad the sufficiet statistic obtaied through the factorizatio criterio, applicatio of the Rao Blackwell theorem typically leads to a MVUE for the parameter. Direct computatio of

23 466 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio coditioal expectatios ca be difficult. However, if U is the sufficiet statistic that best summarizes the data ad some fuctio of U say, h(u) ca be foud such that E[h(U)] = θ, it follows that h(u) is the MVUE for θ. We illustrate this approach with several examples. EXAMPLE 9.6 Solutio Let Y 1, Y,..., Y deote a radom sample from a distributio where P(Y i = 1) = p ad P(Y i = 0) = 1 p, with p ukow (such radom variables are ofte called Beroulli variables). Use the factorizatio criterio to fid a sufficiet statistic that best summarizes the data. Give a MVUE for p. Notice that the precedig probability fuctio ca be writte as Thus, the likelihood L(p) is P(Y i = y i ) = p y i (1 p) 1 y i, y i = 0, 1. L(y 1, y,...,y p) = p(y 1, y,...,y p) = p y 1 (1 p) 1 y 1 p y (1 p) 1 y p y (1 p) 1 y = p y i (1 p) y i }{{} g( y i, p) }{{} 1. h(y 1, y,...,y ) Accordig to the factorizatio criterio, U = Y i is sufficiet for p. This statistic best summarizes the iformatio about the parameter p. Notice that E(U) = p,or equivaletly, E(U/) = p. Thus, U/ = Y is a ubiased estimator for p. Because this estimator is a fuctio of the sufficiet statistic Y i, the estimator ˆp = Y is the MVUE for p. EXAMPLE 9.7 Solutio Suppose that Y 1, Y,..., Y deote a radom sample from the Weibull desity fuctio, give by ( ) y e f (y θ) = y /θ, y > 0, θ Fid a MVUE for θ. We begi by usig the factorizatio criterio to fid the sufficiet statistic that best summarizes the iformatio about θ. L(y 1, y,...,y θ) = f (y 1, y,...,y θ) ( ) ( ) = (y 1 y y ) exp 1 yi θ θ ( ) ( ) = exp 1 yi (y 1 y y ). θ θ }{{} }{{} g( h(y 1,y,...,y ) yi,θ)

24 9.5 The Rao Blackwell Theorem ad Miimum-Variace Ubiased Estimatio 467 Thus, U = Y i is the miimal sufficiet statistic for θ. We ow must fid a fuctio of this statistic that is ubiased for θ. Lettig W = Yi, we have f W (w) = f ( w) d( ( ) w) ( we dw = w/θ ) ( ) ( ) 1 1 θ = e w/θ, w > 0. w θ That is, Yi has a expoetial distributio with parameter θ. Because ) it follows that ( E(Yi ) = E(W ) = θ ad E ˆθ = 1 Y i Y i = θ, is a ubiased estimator of θ that is a fuctio of the sufficiet statistic Therefore, ˆθ is a MVUE of the Weibull parameter θ. Y i. The followig example illustrates the use of this techique for estimatig two ukow parameters. EXAMPLE 9.8 Solutio Suppose Y 1, Y,..., Y deotes a radom sample from a ormal distributio with ukow mea μ ad variace σ. Fid the MVUEs for μ ad σ. Agai, lookig at the likelihood fuctio, we have L(y 1, y,...,y μ, σ ) = f (y 1, y,...,y μ, σ ) ( ) ( ) 1 = σ exp 1 (y π σ i μ) ( ) [ ( )] 1 = σ exp 1 y π σ i μ y i + μ ( ) 1 ( μ ) [ ( )] = σ exp exp 1 y π σ σ i μ y i. Thus, Y i ad Y i, joitly, are sufficiet statistics for μ ad σ. We kow from past work that Y is ubiased for μ ad [ ] S = 1 (Y i Y ) = 1 Yi Y 1 1 is ubiased for σ. Because these estimators are fuctios of the statistics that best summarize the iformatio about μ ad σ, they are MVUEs for μ ad σ.

25 468 Chapter 9 Properties of Poit Estimators ad Methods of Estimatio The factorizatio criterio, together with the Rao Blackwell theorem, ca also be used to fid MVUEs for fuctios of the parameters associated with a distributio. We illustrate the techique i the followig example. EXAMPLE 9.9 Solutio Let Y 1, Y,..., Y deote a radom sample from the expoetial desity fuctio give by ( ) 1 e f (y θ) = y/θ, y > 0, θ Fid a MVUE of V (Y i ). I Chapter 4, we determied that E(Y i ) = θ ad that V (Y i ) = θ. The factorizatio criterio implies that Y i is the best sufficiet statistic for θ. I fact, Y is the MVUE of θ. Therefore, it is temptig to use Y as a estimator of θ. But E (Y ) = V (Y ) + [E(Y )] = θ ( ) θ = θ. It follows that Y is a biased estimate for θ. However, ( ) Y + 1 is a MVUE of θ because it is a ubiased estimator for θ ad a fuctio of the sufficiet statistic. No other ubiased estimator of θ will have a smaller variace tha this oe. A sufficiet statistic for a parameter θ ofte ca be used to costruct a exact cofidece iterval for θ if the probability distributio of the statistic ca be foud. The resultig itervals geerally are the shortest that ca be foud with a specified cofidece coefficiet. We illustrate the techique with a example ivolvig the Weibull distributio. EXAMPLE 9.10 The followig data, with measuremets i hudreds of hours, represet the legths of life of te idetical electroic compoets operatig i a guidace cotrol system for missiles: The legth of life of a compoet of this type is assumed to follow a Weibull distributio with desity fuctio give by ( ) y e f (y θ) = y /θ, y > 0, θ Use the data to costruct a 95% cofidece iterval for θ.

6. Sufficient, Complete, and Ancillary Statistics

6. Sufficient, Complete, and Ancillary Statistics Sufficiet, Complete ad Acillary Statistics http://www.math.uah.edu/stat/poit/sufficiet.xhtml 1 of 7 7/16/2009 6:13 AM Virtual Laboratories > 7. Poit Estimatio > 1 2 3 4 5 6 6. Sufficiet, Complete, ad Acillary

More information

MATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4

MATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4 MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

Stat 421-SP2012 Interval Estimation Section

Stat 421-SP2012 Interval Estimation Section Stat 41-SP01 Iterval Estimatio Sectio 11.1-11. We ow uderstad (Chapter 10) how to fid poit estimators of a ukow parameter. o However, a poit estimate does ot provide ay iformatio about the ucertaity (possible

More information

7.1 Convergence of sequences of random variables

7.1 Convergence of sequences of random variables Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite

More information

Chapter 6 Principles of Data Reduction

Chapter 6 Principles of Data Reduction Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a

More information

Chapter 3. Strong convergence. 3.1 Definition of almost sure convergence

Chapter 3. Strong convergence. 3.1 Definition of almost sure convergence Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

Review Questions, Chapters 8, 9. f(y) = 0, elsewhere. F (y) = f Y(1) = n ( e y/θ) n 1 1 θ e y/θ = n θ e yn

Review Questions, Chapters 8, 9. f(y) = 0, elsewhere. F (y) = f Y(1) = n ( e y/θ) n 1 1 θ e y/θ = n θ e yn Stat 366 Lab 2 Solutios (September 2, 2006) page TA: Yury Petracheko, CAB 484, yuryp@ualberta.ca, http://www.ualberta.ca/ yuryp/ Review Questios, Chapters 8, 9 8.5 Suppose that Y, Y 2,..., Y deote a radom

More information

Random Variables, Sampling and Estimation

Random Variables, Sampling and Estimation Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig

More information

Econ 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.

Econ 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1. Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio

More information

The variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2.

The variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2. SAMPLE STATISTICS A radom sample x 1,x,,x from a distributio f(x) is a set of idepedetly ad idetically variables with x i f(x) for all i Their joit pdf is f(x 1,x,,x )=f(x 1 )f(x ) f(x )= f(x i ) The sample

More information

Topic 9: Sampling Distributions of Estimators

Topic 9: Sampling Distributions of Estimators Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be

More information

1 Introduction to reducing variance in Monte Carlo simulations

1 Introduction to reducing variance in Monte Carlo simulations Copyright c 010 by Karl Sigma 1 Itroductio to reducig variace i Mote Carlo simulatios 11 Review of cofidece itervals for estimatig a mea I statistics, we estimate a ukow mea µ = E(X) of a distributio by

More information

FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures

FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals

More information

6.3 Testing Series With Positive Terms

6.3 Testing Series With Positive Terms 6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial

More information

Econ 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara

Econ 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara Poit Estimator Eco 325 Notes o Poit Estimator ad Cofidece Iterval 1 By Hiro Kasahara Parameter, Estimator, ad Estimate The ormal probability desity fuctio is fully characterized by two costats: populatio

More information

Discrete Mathematics for CS Spring 2008 David Wagner Note 22

Discrete Mathematics for CS Spring 2008 David Wagner Note 22 CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 22 I.I.D. Radom Variables Estimatig the bias of a coi Questio: We wat to estimate the proportio p of Democrats i the US populatio, by takig

More information

This exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.

This exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam. Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the

More information

The standard deviation of the mean

The standard deviation of the mean Physics 6C Fall 20 The stadard deviatio of the mea These otes provide some clarificatio o the distictio betwee the stadard deviatio ad the stadard deviatio of the mea.. The sample mea ad variace Cosider

More information

7.1 Convergence of sequences of random variables

7.1 Convergence of sequences of random variables Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite

More information

EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY

EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY EXAMINATIONS OF THE ROYAL STATISTICAL SOCIETY GRADUATE DIPLOMA, 016 MODULE : Statistical Iferece Time allowed: Three hours Cadidates should aswer FIVE questios. All questios carry equal marks. The umber

More information

MATH 472 / SPRING 2013 ASSIGNMENT 2: DUE FEBRUARY 4 FINALIZED

MATH 472 / SPRING 2013 ASSIGNMENT 2: DUE FEBRUARY 4 FINALIZED MATH 47 / SPRING 013 ASSIGNMENT : DUE FEBRUARY 4 FINALIZED Please iclude a cover sheet that provides a complete setece aswer to each the followig three questios: (a) I your opiio, what were the mai ideas

More information

KLMED8004 Medical statistics. Part I, autumn Estimation. We have previously learned: Population and sample. New questions

KLMED8004 Medical statistics. Part I, autumn Estimation. We have previously learned: Population and sample. New questions We have previously leared: KLMED8004 Medical statistics Part I, autum 00 How kow probability distributios (e.g. biomial distributio, ormal distributio) with kow populatio parameters (mea, variace) ca give

More information

Statistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample.

Statistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample. Statistical Iferece (Chapter 10) Statistical iferece = lear about a populatio based o the iformatio provided by a sample. Populatio: The set of all values of a radom variable X of iterest. Characterized

More information

A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence

A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece,, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet as

More information

Output Analysis and Run-Length Control

Output Analysis and Run-Length Control IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%

More information

Convergence of random variables. (telegram style notes) P.J.C. Spreij

Convergence of random variables. (telegram style notes) P.J.C. Spreij Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space

More information

Lecture 12: September 27

Lecture 12: September 27 36-705: Itermediate Statistics Fall 207 Lecturer: Siva Balakrisha Lecture 2: September 27 Today we will discuss sufficiecy i more detail ad the begi to discuss some geeral strategies for costructig estimators.

More information

Infinite Sequences and Series

Infinite Sequences and Series Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet

More information

1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable

More information

Distribution of Random Samples & Limit theorems

Distribution of Random Samples & Limit theorems STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to

More information

5. Likelihood Ratio Tests

5. Likelihood Ratio Tests 1 of 5 7/29/2009 3:16 PM Virtual Laboratories > 9. Hy pothesis Testig > 1 2 3 4 5 6 7 5. Likelihood Ratio Tests Prelimiaries As usual, our startig poit is a radom experimet with a uderlyig sample space,

More information

DS 100: Principles and Techniques of Data Science Date: April 13, Discussion #10

DS 100: Principles and Techniques of Data Science Date: April 13, Discussion #10 DS 00: Priciples ad Techiques of Data Sciece Date: April 3, 208 Name: Hypothesis Testig Discussio #0. Defie these terms below as they relate to hypothesis testig. a) Data Geeratio Model: Solutio: A set

More information

It is always the case that unions, intersections, complements, and set differences are preserved by the inverse image of a function.

It is always the case that unions, intersections, complements, and set differences are preserved by the inverse image of a function. MATH 532 Measurable Fuctios Dr. Neal, WKU Throughout, let ( X, F, µ) be a measure space ad let (!, F, P ) deote the special case of a probability space. We shall ow begi to study real-valued fuctios defied

More information

7-1. Chapter 4. Part I. Sampling Distributions and Confidence Intervals

7-1. Chapter 4. Part I. Sampling Distributions and Confidence Intervals 7-1 Chapter 4 Part I. Samplig Distributios ad Cofidece Itervals 1 7- Sectio 1. Samplig Distributio 7-3 Usig Statistics Statistical Iferece: Predict ad forecast values of populatio parameters... Test hypotheses

More information

Expectation and Variance of a random variable

Expectation and Variance of a random variable Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio

More information

Unbiased Estimation. February 7-12, 2008

Unbiased Estimation. February 7-12, 2008 Ubiased Estimatio February 7-2, 2008 We begi with a sample X = (X,..., X ) of radom variables chose accordig to oe of a family of probabilities P θ where θ is elemet from the parameter space Θ. For radom

More information

4. Partial Sums and the Central Limit Theorem

4. Partial Sums and the Central Limit Theorem 1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems

More information

Mathematical Statistics - MS

Mathematical Statistics - MS Paper Specific Istructios. The examiatio is of hours duratio. There are a total of 60 questios carryig 00 marks. The etire paper is divided ito three sectios, A, B ad C. All sectios are compulsory. Questios

More information

Problem Set 4 Due Oct, 12

Problem Set 4 Due Oct, 12 EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios

More information

Sequences A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence

Sequences A sequence of numbers is a function whose domain is the positive integers. We can see that the sequence Sequeces A sequece of umbers is a fuctio whose domai is the positive itegers. We ca see that the sequece 1, 1, 2, 2, 3, 3,... is a fuctio from the positive itegers whe we write the first sequece elemet

More information

Lecture 2: Monte Carlo Simulation

Lecture 2: Monte Carlo Simulation STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?

More information

(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3

(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3 MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special

More information

Direction: This test is worth 250 points. You are required to complete this test within 50 minutes.

Direction: This test is worth 250 points. You are required to complete this test within 50 minutes. Term Test October 3, 003 Name Math 56 Studet Number Directio: This test is worth 50 poits. You are required to complete this test withi 50 miutes. I order to receive full credit, aswer each problem completely

More information

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1

EECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1 EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum

More information

Probability and statistics: basic terms

Probability and statistics: basic terms Probability ad statistics: basic terms M. Veeraraghava August 203 A radom variable is a rule that assigs a umerical value to each possible outcome of a experimet. Outcomes of a experimet form the sample

More information

Simulation. Two Rule For Inverting A Distribution Function

Simulation. Two Rule For Inverting A Distribution Function Simulatio Two Rule For Ivertig A Distributio Fuctio Rule 1. If F(x) = u is costat o a iterval [x 1, x 2 ), the the uiform value u is mapped oto x 2 through the iversio process. Rule 2. If there is a jump

More information

Statistics 511 Additional Materials

Statistics 511 Additional Materials Cofidece Itervals o mu Statistics 511 Additioal Materials This topic officially moves us from probability to statistics. We begi to discuss makig ifereces about the populatio. Oe way to differetiate probability

More information

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak

More information

Seunghee Ye Ma 8: Week 5 Oct 28

Seunghee Ye Ma 8: Week 5 Oct 28 Week 5 Summary I Sectio, we go over the Mea Value Theorem ad its applicatios. I Sectio 2, we will recap what we have covered so far this term. Topics Page Mea Value Theorem. Applicatios of the Mea Value

More information

Statistical Theory MT 2009 Problems 1: Solution sketches

Statistical Theory MT 2009 Problems 1: Solution sketches Statistical Theory MT 009 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. (a) Let 0 < θ < ad put f(x, θ) = ( θ)θ x ; x = 0,,,... (b) (c) where

More information

Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.

Resampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n. Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator

More information

An Introduction to Randomized Algorithms

An Introduction to Randomized Algorithms A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis

More information

Frequentist Inference

Frequentist Inference Frequetist Iferece The topics of the ext three sectios are useful applicatios of the Cetral Limit Theorem. Without kowig aythig about the uderlyig distributio of a sequece of radom variables {X i }, for

More information

CHAPTER 10 INFINITE SEQUENCES AND SERIES

CHAPTER 10 INFINITE SEQUENCES AND SERIES CHAPTER 10 INFINITE SEQUENCES AND SERIES 10.1 Sequeces 10.2 Ifiite Series 10.3 The Itegral Tests 10.4 Compariso Tests 10.5 The Ratio ad Root Tests 10.6 Alteratig Series: Absolute ad Coditioal Covergece

More information

Let us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.

Let us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f. Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,

More information

Estimation for Complete Data

Estimation for Complete Data Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of

More information

MATH/STAT 352: Lecture 15

MATH/STAT 352: Lecture 15 MATH/STAT 352: Lecture 15 Sectios 5.2 ad 5.3. Large sample CI for a proportio ad small sample CI for a mea. 1 5.2: Cofidece Iterval for a Proportio Estimatig proportio of successes i a biomial experimet

More information

Discrete Mathematics for CS Spring 2007 Luca Trevisan Lecture 22

Discrete Mathematics for CS Spring 2007 Luca Trevisan Lecture 22 CS 70 Discrete Mathematics for CS Sprig 2007 Luca Trevisa Lecture 22 Aother Importat Distributio The Geometric Distributio Questio: A biased coi with Heads probability p is tossed repeatedly util the first

More information

Chapter 6 Sampling Distributions

Chapter 6 Sampling Distributions Chapter 6 Samplig Distributios 1 I most experimets, we have more tha oe measuremet for ay give variable, each measuremet beig associated with oe radomly selected a member of a populatio. Hece we eed to

More information

1 Inferential Methods for Correlation and Regression Analysis

1 Inferential Methods for Correlation and Regression Analysis 1 Iferetial Methods for Correlatio ad Regressio Aalysis I the chapter o Correlatio ad Regressio Aalysis tools for describig bivariate cotiuous data were itroduced. The sample Pearso Correlatio Coefficiet

More information

Stat410 Probability and Statistics II (F16)

Stat410 Probability and Statistics II (F16) Some Basic Cocepts of Statistical Iferece (Sec 5.) Suppose we have a rv X that has a pdf/pmf deoted by f(x; θ) or p(x; θ), where θ is called the parameter. I previous lectures, we focus o probability problems

More information

The Random Walk For Dummies

The Random Walk For Dummies The Radom Walk For Dummies Richard A Mote Abstract We look at the priciples goverig the oe-dimesioal discrete radom walk First we review five basic cocepts of probability theory The we cosider the Beroulli

More information

Lecture 19: Convergence

Lecture 19: Convergence Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may

More information

Stat 319 Theory of Statistics (2) Exercises

Stat 319 Theory of Statistics (2) Exercises Kig Saud Uiversity College of Sciece Statistics ad Operatios Research Departmet Stat 39 Theory of Statistics () Exercises Refereces:. Itroductio to Mathematical Statistics, Sixth Editio, by R. Hogg, J.

More information

Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 19

Discrete Mathematics and Probability Theory Spring 2016 Rao and Walrand Note 19 CS 70 Discrete Mathematics ad Probability Theory Sprig 2016 Rao ad Walrad Note 19 Some Importat Distributios Recall our basic probabilistic experimet of tossig a biased coi times. This is a very simple

More information

Math 155 (Lecture 3)

Math 155 (Lecture 3) Math 55 (Lecture 3) September 8, I this lecture, we ll cosider the aswer to oe of the most basic coutig problems i combiatorics Questio How may ways are there to choose a -elemet subset of the set {,,,

More information

Questions and Answers on Maximum Likelihood

Questions and Answers on Maximum Likelihood Questios ad Aswers o Maximum Likelihood L. Magee Fall, 2008 1. Give: a observatio-specific log likelihood fuctio l i (θ) = l f(y i x i, θ) the log likelihood fuctio l(θ y, X) = l i(θ) a data set (x i,

More information

Statistical Theory MT 2008 Problems 1: Solution sketches

Statistical Theory MT 2008 Problems 1: Solution sketches Statistical Theory MT 008 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. a) Let 0 < θ < ad put fx, θ) = θ)θ x ; x = 0,,,... b) c) where α

More information

Binomial Distribution

Binomial Distribution 0.0 0.5 1.0 1.5 2.0 2.5 3.0 0 1 2 3 4 5 6 7 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Overview Example: coi tossed three times Defiitio Formula Recall that a r.v. is discrete if there are either a fiite umber of possible

More information

Application to Random Graphs

Application to Random Graphs A Applicatio to Radom Graphs Brachig processes have a umber of iterestig ad importat applicatios. We shall cosider oe of the most famous of them, the Erdős-Réyi radom graph theory. 1 Defiitio A.1. Let

More information

Economics Spring 2015

Economics Spring 2015 1 Ecoomics 400 -- Sprig 015 /17/015 pp. 30-38; Ch. 7.1.4-7. New Stata Assigmet ad ew MyStatlab assigmet, both due Feb 4th Midterm Exam Thursday Feb 6th, Chapters 1-7 of Groeber text ad all relevat lectures

More information

Goodness-of-Fit Tests and Categorical Data Analysis (Devore Chapter Fourteen)

Goodness-of-Fit Tests and Categorical Data Analysis (Devore Chapter Fourteen) Goodess-of-Fit Tests ad Categorical Data Aalysis (Devore Chapter Fourtee) MATH-252-01: Probability ad Statistics II Sprig 2019 Cotets 1 Chi-Squared Tests with Kow Probabilities 1 1.1 Chi-Squared Testig................

More information

The Sample Variance Formula: A Detailed Study of an Old Controversy

The Sample Variance Formula: A Detailed Study of an Old Controversy The Sample Variace Formula: A Detailed Study of a Old Cotroversy Ky M. Vu PhD. AuLac Techologies Ic. c 00 Email: kymvu@aulactechologies.com Abstract The two biased ad ubiased formulae for the sample variace

More information

32 estimating the cumulative distribution function

32 estimating the cumulative distribution function 32 estimatig the cumulative distributio fuctio 4.6 types of cofidece itervals/bads Let F be a class of distributio fuctios F ad let θ be some quatity of iterest, such as the mea of F or the whole fuctio

More information

Discrete Mathematics for CS Spring 2005 Clancy/Wagner Notes 21. Some Important Distributions

Discrete Mathematics for CS Spring 2005 Clancy/Wagner Notes 21. Some Important Distributions CS 70 Discrete Mathematics for CS Sprig 2005 Clacy/Wager Notes 21 Some Importat Distributios Questio: A biased coi with Heads probability p is tossed repeatedly util the first Head appears. What is the

More information

Exponential Families and Bayesian Inference

Exponential Families and Bayesian Inference Computer Visio Expoetial Families ad Bayesia Iferece Lecture Expoetial Families A expoetial family of distributios is a d-parameter family f(x; havig the followig form: f(x; = h(xe g(t T (x B(, (. where

More information

Lecture 10 October Minimaxity and least favorable prior sequences

Lecture 10 October Minimaxity and least favorable prior sequences STATS 300A: Theory of Statistics Fall 205 Lecture 0 October 22 Lecturer: Lester Mackey Scribe: Brya He, Rahul Makhijai Warig: These otes may cotai factual ad/or typographic errors. 0. Miimaxity ad least

More information

Parameter, Statistic and Random Samples

Parameter, Statistic and Random Samples Parameter, Statistic ad Radom Samples A parameter is a umber that describes the populatio. It is a fixed umber, but i practice we do ot kow its value. A statistic is a fuctio of the sample data, i.e.,

More information

Lecture 7: Properties of Random Samples

Lecture 7: Properties of Random Samples Lecture 7: Properties of Radom Samples 1 Cotiued From Last Class Theorem 1.1. Let X 1, X,...X be a radom sample from a populatio with mea µ ad variace σ

More information

EE 4TM4: Digital Communications II Probability Theory

EE 4TM4: Digital Communications II Probability Theory 1 EE 4TM4: Digital Commuicatios II Probability Theory I. RANDOM VARIABLES A radom variable is a real-valued fuctio defied o the sample space. Example: Suppose that our experimet cosists of tossig two fair

More information

7 Sequences of real numbers

7 Sequences of real numbers 40 7 Sequeces of real umbers 7. Defiitios ad examples Defiitio 7... A sequece of real umbers is a real fuctio whose domai is the set N of atural umbers. Let s : N R be a sequece. The the values of s are

More information

Bayesian Methods: Introduction to Multi-parameter Models

Bayesian Methods: Introduction to Multi-parameter Models Bayesia Methods: Itroductio to Multi-parameter Models Parameter: θ = ( θ, θ) Give Likelihood p(y θ) ad prior p(θ ), the posterior p proportioal to p(y θ) x p(θ ) Margial posterior ( θ, θ y) is Iterested

More information

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015

ECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015 ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],

More information

17. Joint distributions of extreme order statistics Lehmann 5.1; Ferguson 15

17. Joint distributions of extreme order statistics Lehmann 5.1; Ferguson 15 17. Joit distributios of extreme order statistics Lehma 5.1; Ferguso 15 I Example 10., we derived the asymptotic distributio of the maximum from a radom sample from a uiform distributio. We did this usig

More information

Chapter 2 The Monte Carlo Method

Chapter 2 The Monte Carlo Method Chapter 2 The Mote Carlo Method The Mote Carlo Method stads for a broad class of computatioal algorithms that rely o radom sampligs. It is ofte used i physical ad mathematical problems ad is most useful

More information

SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker

SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker CHAPTER 9. POINT ESTIMATION 9. Covergece i Probability. The bases of poit estimatio have already bee laid out i previous chapters. I chapter 5

More information

Chapter 6 Infinite Series

Chapter 6 Infinite Series Chapter 6 Ifiite Series I the previous chapter we cosidered itegrals which were improper i the sese that the iterval of itegratio was ubouded. I this chapter we are goig to discuss a topic which is somewhat

More information

Lecture 1 Probability and Statistics

Lecture 1 Probability and Statistics Wikipedia: Lecture 1 Probability ad Statistics Bejami Disraeli, British statesma ad literary figure (1804 1881): There are three kids of lies: lies, damed lies, ad statistics. popularized i US by Mark

More information

Recap! Good statistics, cont.! Sufficiency! What are good statistics?! 2/20/14

Recap! Good statistics, cont.! Sufficiency! What are good statistics?! 2/20/14 Recap Cramér-Rao iequality Best ubiased estimators What are good statistics? Parameter: ukow umber that we are tryig to get a idea about usig a sample X 1,,X Statistic: A fuctio of the sample. It is a

More information

Introductory statistics

Introductory statistics CM9S: Machie Learig for Bioiformatics Lecture - 03/3/06 Itroductory statistics Lecturer: Sriram Sakararama Scribe: Sriram Sakararama We will provide a overview of statistical iferece focussig o the key

More information

Discrete Mathematics and Probability Theory Summer 2014 James Cook Note 15

Discrete Mathematics and Probability Theory Summer 2014 James Cook Note 15 CS 70 Discrete Mathematics ad Probability Theory Summer 2014 James Cook Note 15 Some Importat Distributios I this ote we will itroduce three importat probability distributios that are widely used to model

More information

ECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors

ECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic

More information

Sampling Distributions, Z-Tests, Power

Sampling Distributions, Z-Tests, Power Samplig Distributios, Z-Tests, Power We draw ifereces about populatio parameters from sample statistics Sample proportio approximates populatio proportio Sample mea approximates populatio mea Sample variace

More information

Since X n /n P p, we know that X n (n. Xn (n X n ) Using the asymptotic result above to obtain an approximation for fixed n, we obtain

Since X n /n P p, we know that X n (n. Xn (n X n ) Using the asymptotic result above to obtain an approximation for fixed n, we obtain Assigmet 9 Exercise 5.5 Let X biomial, p, where p 0, 1 is ukow. Obtai cofidece itervals for p i two differet ways: a Sice X / p d N0, p1 p], the variace of the limitig distributio depeds oly o p. Use the

More information

62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 +

62. Power series Definition 16. (Power series) Given a sequence {c n }, the series. c n x n = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + 62. Power series Defiitio 16. (Power series) Give a sequece {c }, the series c x = c 0 + c 1 x + c 2 x 2 + c 3 x 3 + is called a power series i the variable x. The umbers c are called the coefficiets of

More information

January 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS

January 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS Jauary 25, 207 INTRODUCTION TO MATHEMATICAL STATISTICS Abstract. A basic itroductio to statistics assumig kowledge of probability theory.. Probability I a typical udergraduate problem i probability, we

More information

Lesson 10: Limits and Continuity

Lesson 10: Limits and Continuity www.scimsacademy.com Lesso 10: Limits ad Cotiuity SCIMS Academy 1 Limit of a fuctio The cocept of limit of a fuctio is cetral to all other cocepts i calculus (like cotiuity, derivative, defiite itegrals

More information

A statistical method to determine sample size to estimate characteristic value of soil parameters

A statistical method to determine sample size to estimate characteristic value of soil parameters A statistical method to determie sample size to estimate characteristic value of soil parameters Y. Hojo, B. Setiawa 2 ad M. Suzuki 3 Abstract Sample size is a importat factor to be cosidered i determiig

More information