6. Sufficient, Complete, and Ancillary Statistics

Size: px

Start display at page:

Download "6. Sufficient, Complete, and Ancillary Statistics"

Brianna Cunningham
5 years ago
Views:

1 Sufficiet, Complete ad Acillary Statistics 1 of 7 7/16/2009 6:13 AM Virtual Laboratories > 7. Poit Estimatio > Sufficiet, Complete, ad Acillary Statistics The Basic Statistical Model Cosider agai the basic statistical model, i which we have a radom experimet with a observable radom variable X takig values i a set S. Oce agai, the experimet is typically to sample objects from a populatio ad record oe or more measuremets for each item. I this case, the outcome variable has the form X = (X 1, X 2,..., X ) where X i is the vector of measuremets for the i th item. I geeral, we suppose that the distributio of X depeds o a parameter θ takig values i a parameter space Θ. The parameter θ may also be vector-valued. We will use subscripts i probability desity fuctios, expected values, etc. to deote the depedece o θ Sufficiet Statistics Let U = h(x) be a statistic takig values i a set T. Ituitively, U is sufficiet for θ if U cotais all of the iformatio about θ that is available i the etire data variable X. Formally, U is sufficiet for θ if the coditioal distributio of X give U does ot deped o θ. Sufficiecy is related to the cocept of data reductio. Suppose that X takes values i R. If we ca fid a sufficiet statistic U that takes values i R j, the we ca reduce the origial data vector X (whose dimesio is usually large) to the vector of statistics U (whose dimesio j is usually much smaller) with o loss of iformatio about the parameter θ. The followig result gives a coditio for sufficiecy that is equivalet to this defiitio. 1. Let U = h(x) be a statistic takig values i T, ad let f θ ad g θ deote the probability desity fuctios of X ad U respectively. Show that U is sufficiet for θ if ad oly if the fuctio f θ (x) g θ (h(x)), x S is idepedet of θ. Hit: The joit distributio of (X, U) is cocetrated o the set {(x, u) : (x S) ad (u = h(x))} S T. The Factorizatio Theorem The defiitio precisely captures the ituitive otio of sufficiecy give above, but ca be difficult to apply.

2 Sufficiet, Complete ad Acillary Statistics 2 of 7 7/16/2009 6:13 AM We must kow i advace a cadidate statistic U, ad the we must be able to compute the coditioal distributio of X give U. The factorizatio theorem give i the ext exercise frequetly allows the idetificatio of a sufficiet statistic from the form of the probability desity fuctio of X. 2. Let f θ deote the probability desity fuctio of X ad suppose that U = h(x) is a statistic takig values i T. Show that U is sufficiet for θ if ad oly if there exists G : T Θ R ad r : S R such that f θ (x) = G(h(x), θ) r(x), x S, θ Θ Note that r depeds oly o the data x but ot o the parameter θ. 3. Show that if U ad V are equivalet statistics ad U is sufficiet for θ the V is sufficiet for θ. S pecial Distributios We will determie sufficiet statistics for several parametric families of distributios. 4. Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Beroulli distributio with success parameter p [ 0, 1]. Thus, X i = 1 if trial i is a success, ad X i = 0 if trial i is a failure. Let Y = i =1 X i deote the umber of successes, ad recall that Y has the biomial distributio with parameters ad p. Show directly from the defiitio that Y is sufficiet for p. Specifically, show that the coditioal distributio of X give Y = k is the uiform distributio o the set of poits. {(x 1, x 2,..., x ) {0, 1} : x 1 + x x = k} The result i the previous exercise is ituitively appealig: i a sequece of Beroulli trials, all of the iformatio about the probability of success p is cotaied i the umber of successes Y. The particular order of the successes ad failures provides o additioal iformatio. Of course, the sufficiecy of Y follows more easily from the factorizatio theorem, but the coditioal distributio provides additioal isight. 5. Suppose that the distributio of X is a k-parameter expoetial familiy with the atural statistic U = h(x). Show that U is sufficiet for θ. Because of this result, U is referred to as the atural sufficiet statistic for the expoetial family. 6. Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the ormal distributio with mea μ R ad variace σ 2 ( 0, ). Show that (Y, V) is sufficiet for (μ, σ 2 ) where Y = i =1 X i ad V = i =1 Show that (M, S 2 ) is sufficiet for (μ, σ 2 ) where M is the sample mea of X ad S 2 is the sample variace of X. Hit: Use part (a) ad equivalece. X i Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Poisso distributio with

3 Sufficiet, Complete ad Acillary Statistics 3 of 7 7/16/2009 6:13 AM mea a ( 0, ). Show that Y = i =1 X i is sufficiet for 8. Suppose that X = (X 1, X 2,..., X ) is a radom sample from the gamma distributio with shape parameter k ( 0, ) ad scale parameter b ( 0, ) Show that (Y, V) is sufficiet for (k, b) where Y = i =1 X i ad V = i =1 Show that (M, U) is sufficiet for (k, b) where M is the sample (arithmetic) mea of X ad U is the sample geometric mea of X. Hit: Use part (a) ad equivalece. X i. 9. Suppose that X = (X 1, X 2,..., X ) is a radom sample from the beta distributio with left parameter a ( 0, ) ad right parameter b ( 0, ). Show that (U, V) is sufficiet for (a, b) where U = i =1 ad V = i =1 (1 X i ). 10. Suppose that X = (X 1, X 2,..., X ) is a radom sample from the Pareto distributio with shape parameter a ( 0, ). Show that U = i =1 X i is sufficiet for 11. Suppose that X = (X 1, X 2,..., X ) is a radom sample from the uiform distributio o the iterval [ 0, a] where a ( 0, ) is the ukow parameter. Show that X, = max {X 1, X 2,..., X } (the th order statistic) is sufficiet for X i Miimal S ufficiet S tatistics The etire data variable X is trivially sufficiet for θ. However, as oted above, there usually exists a statistic U that is sufficiet for θ ad has smaller dimesio, so that we ca achieve real data reductio. Naturally, we would like to fid the statistic U that has the smallest dimesio possible. I may cases, this smallest dimesio j will be the same as the dimesio k of the parameter vector θ. However, as we will see, this is ot ecessarily the case; j ca be smaller or larger tha k. Formally, suppose that a statistic U is sufficiet for θ. The U is miimally sufficiet if U is a fuctio of ay other statistic V that is sufficiet for θ. Oce agai, the defiitio precisely captures the otio of miimal sufficiecy, but is hard to apply. The followig exercise gives a equivalet coditio. 12. Let f θ deote the probability desity fuctio of X correspodig to the parameter value θ ad suppose that U = h(x) is a statistic takig values i T. Show that U is miimally sufficiet for θ if the followig coditio holds: for x S ad y S f θ (x) is idepedet of θ if ad oly if h(x) = h(y) f θ (y) Hit: If V = g(x) is aother sufficiet statistic, use the factorizatio theorem ad the coditio above to show that g(x) = g(y) implies h(x) = h(y) for x S ad y S. The coclude that U is a fuctio of V.

4 Sufficiet, Complete ad Acillary Statistics 4 of 7 7/16/2009 6:13 AM 13. Show that if U ad V are equivalet statistics ad U is miimally sufficiet for θ the V is miimally sufficiet for θ. 14. Suppose that the distributio of X is a k-parameter expoetial family with atural sufficiet statistic U = h(x). Show that U is a miimally sufficiet for θ. 15. Show that the sufficiet statistics give above for the Beroulli, Poisso, ormal, gamma, ad beta families are miimally sufficiet for the give parameters. 16. Suppose that X = (X 1, X 2,..., X ) is a radom sample from the uiform distributio o the iterval [ a, a + 1] where a ( 0, ) is the ukow parameter. Show that (X, 1, X, ), the vector cosistig of the first ad last order statistics, is miimally sufficiet for Note that we have a sigle parameter, but the miimally sufficiet statistic is a vector of dimesio 2. Properties of S ufficiet S tatistics Sufficiecy is related to several of the methods of costructig estimators that we have studied. 17. Suppose that U is sufficiet for θ ad that there exists a maximum likelihood estimator of θ. Show that there exists a M LE V that is a fuctio of U. Hit: Use the factorizatio theorem. I particular, suppose that V is the uique M LE of θ ad that V is sufficiet for θ. If U is sufficiet for θ the V is a fuctio of U by the previous exercise. Hece it follows that V is miimally sufficiet for θ. 18. Suppose that the statistic U is sufficiet for the parameter θ ad that V is a Bayes' estimator of θ. Show that V is a fuctio of U. Hit: Use the factorizatio theorem. The followig exercise gives the Rao-Blackwell theorem, amed for CR Rao ad David Blackwell. The theorem shows how a sufficiet statistic ca be used to improve a ubiased estimator. 19. Suppose that U is sufficiet for θ ad that V is a ubiased estimator of a real parameter λ = λ(θ). Use sufficiecy ad properties of coditioal expectatio ad coditioal variace to show that c. d. θ (V U) is a valid statistic. That is, it does ot deped o θ, i spite of the formal depedece o θ i the expected value. (V U) is a fuctio of U. (V U) is a ubiased estimator of λ. var θ ( (V U)) var θ (V) for ay θ Θ so (V U) is uiformly better tha V. Complete Statistics Suppose that U = h(x) is a statistic takig values i a set T. The U is a complete statistic for θ if for ay real-valued fuctio g o T

5 Sufficiet, Complete ad Acillary Statistics 5 of 7 7/16/2009 6:13 AM θ (g(u)) = 0 for all θ Θ P θ (g(u) = 0) = 0 for all θ Θ To uderstad this rather strage lookig coditio, suppose that g(u) is a statistic costructued from U that is beig used as a estimator of 0 (thought of as a fuctio of θ). The completeess coditio meas that the oly such ubiased estimator is the statistic that is 0 with probability Show that if U ad V are equivalet statistics ad U is complete for θ the V is complete for θ. S pecial Distributios 21. Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Beroulli distributio with success parameter p ( 0, 1). Show that the umber or successes Y = i =1 X i is complete for p. Hit: Note that p( g(y)) ca be writte as a polyomial i t = p If this polyomial is 0 for all t i a ope 1 p iterval, the the coefficiets must be Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Poisso distributio with parameter a ( 0, ). Show that the sum of the sample values Y = i =1 X i is complete for Hit: Note that a (g(y)) ca be writte as a power series i If this series is 0 for all a i a ope iterval, the the coefficiets must be Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the expoetial distributio with scale parameter b ( 0, ). Show that the sum of the sample values Y = i =1 X i is complete for Hit: Note that b (g(y)) is the Laplace trasform of a certai fuctio. If this trasform is 0 for all b i a ope iterval, the the fuctio must be 0. The results i the previous exercises geeralize to expoetial families, although the proof is complicated. Specifically, if the distributio of X is a k-parameter expoetial family with the atural sufficiet statistic U = h(x) the U is complete for θ (as well as miimally sufficiet for θ). This applies to radom samples from the Beroulli, Poisso, ormal, gamma, ad beta distributios discussed above. The otio of completeess depeds very much o the parameter space. 24. Suppose that X = (X 1, X 2, X 3 ) is a radom sample of size 3 from the Beroulli distributio with success parameter p { 1 3, 1 2}. Show that Y = X 1 + X 2 + X 3 is ot complete for p. The Lehma-S cheffé Theorem The ext exercise shows the importace of complete sufficiet statistics; it is kow as the Lehma- Scheffé theorem, amed for Erich Lehma ad Hery Scheffé. 25. Suppose that U is sufficiet ad complete for θ ad that T = r(u) is a ubiased estimator of a real parameter λ = λ(θ). Show that T is a uiformly miimum variace ubiased estimator of λ. The proof is

6 Sufficiet, Complete ad Acillary Statistics 6 of 7 7/16/2009 6:13 AM based o the followig steps: Suppose that V is a ubiased estimator of λ. By the Rao-Blackwell theorem, (V U) is also a ubiased estimator of λ ad is uiformly better tha V. Sice (V U) is a fuctio of U, use completeess to coclude that T = (V U) with probability Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Beroulli distributio with success parameter p [ 0, 1]. As usual, let Y = i =1 X i deote the umber of successes. Show that a UM VUE for p (1 p), the variace of the sample distributio, is Y 1 ( 1 Y ) 27. Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the Poisso distributio with parameter μ. Let Y = i =1 X i. Show that a UMVUE for P(X = 0) = e μ is Y 1 ( ) Hit: Use the probability geeratig fuctio of Y. Acillary Statistics Suppose that V = r(x) is a statistic takig values i a set T. If the distributio of V does ot deped o θ, the V is called a acillary statistic for θ. Thus, the otio of a acillary statistic is complemetary to the otio of a sufficiet statistics (which cotais all iformatio about the parameter that is cotaied i the sample). Thus, the result i the followig exercise, kow as Basu's Theorem ad amed for Debabrata Basu, makes this poit more precisely. 28. Suppose that U is complete ad sufficiet for a parameter θ ad that V is a acillary statistic. Show that U ad V are idepedet. The followig steps sketch the proof: c. Let g deote the probability desity fuctio of V ad let v g(v U) deote the coditioal probability desity fuctio of V give U. Use properties of coditioal expected value to show that (g(v U)) = g(v) for v T. Use completeess to coclude that g(v U) = g(v) with probability Show that if U ad V are equivalet statistics ad U is acillary for θ the V is acillary for θ. 30. Suppose that X = (X 1, X 2,..., X ) is a radom sample from a scale family with scale parameter b ( 0, ). Show that V is a acillary statistic for b if V is a fuctio of X 1, X 2,..., X 1 ( X X X )

7 Sufficiet, Complete ad Acillary Statistics 7 of 7 7/16/2009 6:13 AM 31. Suppose that X = (X 1, X 2,..., X ) is a radom sample of size from the gamma distributio with shape parameter k ( 0, ) ad scale parameter b ( 0, ). Let M deote the sample arithmetic mea of X ad let U deote the sample geometric mea of X. Show that M U is acillary for b, ad thus coclude that M ad M U are idepedet. Hit: Use the previous exercise. Virtual Laboratories > 7. Poit Estimatio > Cotets Applets Data Sets Biographies Exteral Resources Key words Feedback

Chapter 6 Principles of Data Reduction

Chapter 6 Principles of Data Reduction Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a