Minimal sufficient statistics emerge from the observed likelihood functions

Transcription

1 International Journal of Statistical Sciences ISSN Vol. 5(Special Issue), 2006, pp *-* c 2006 Dept. of Statistics, Univ. of Rajshahi, Bangladesh Minimal sufficient statistics emerge from the observed likelihood functions D.A.S Fraser Dept. of Statistics University of Toronto Toronto, Canada Amir Naderi Dept. of Mathematical Sciences Isfahan University of Technalogy Isfahan, Iran [Received ** ; Revised ** ; Accepted ** ] Abstract The likelihood statistic guides statistical analysis in almost all areas of application. A precise definition of likelihood statistic is given, and simple and easy to use criteria are proposed to establish under weak conditions that minimal sufficiency in statistics emerges from observed likelihood functions. Some examples are presented. Keywords and Phrases: Probability measures, derivatives of probability measures, likelihood, likelihood statistic, minimal sufficiency. AMS Classification: *** 1 Introduction With model f θ (x), the observed likelihood function from x 0 is cf θ (x 0 )wherec is an arbitrary positive constant; this can be formalized in terms of an equivalence class R(f θ (x 0 )) = {cf θ (x 0 ),c R + } of functions on the parameter space, say Ω. The likelihood map L(θ x) is defined as the mapping that carries a point x in the sample space X to R(f θ (x)), see Naderi [14] or Fraser et al [8]. Fisher [4]-[7] noted that the general dependence of L(θ x) onx is that of minimal sufficiency but did this without detailed formulation. Neyman [15], Halmos and Savage [10], Lehmann and Scheffé

2 2 International Journal of Statistical Sciences, Vol.5s, 2006 [12], Dynkin [3] and Bahadur [1] extended Fisher s results. The theory of minimal sufficiency was initiated by Lehmann and Scheffé [12] and Dynkin [3]. Pitcher [16] constructed a family of probability measures for which a minimal sufficient statistic did not exist. Landers and Rogge [11] showed that even for a dominated family of probability measures a minimal sufficient statistic did not exist. Barndorff-Nielsen et al [2] and Fraser et al [8] however showed that the minimal sufficient statistic exists under some regularity conditions. This paper establishes that minimal sufficiency in statistics emerges from the observed likelihood functions under weak conditions. 2 Likelihood Partition For each θ in Ω, let (X, B,P θ ) be a probability space, P θ be absolutely continuous with respect to a σ-finite measure µ, andf θ (x) be the probability density of P θ with respect to µ. Suppose P = {P θ : θ Ω} and F = {f θ (x) :θ Ω}. Foreachθ, f θ (x) is unique uptoasetofµ-measure zero. The general uniqueness of f θ (x) however, is required for the definition of our mappings. This may be achived by defining the probability P density f θ (x) as the lim θ (E n) n µ(e n) for all sequences {E n } converging regularly to x and possessing some covering properties. For a detailed treatment see for example Naderi [14] or Fraser et al [8]. The question of the existence, in general, of a minimal sufficient partition or the associated statistic involves measure theoretic difficulties and some regularity conditions need to be imposed on F. Two such regularity conditions are: Separability Condition. F is separable if there exists a fixed countable subset F 0 = {f θ (x) :θ Ω 0 } of F such that for each f θ (x) inf there is a sequence {f θn (x)} in F 0 for which f θ (x) f θn (x) dµ(x) 0 X as n. In this case F 0 is said to be dense in F. When X is a Euclidean space, the class of probability densities is a separable metric space with distance defined by d(g, h) = X g(x) h(x) dµ(x). Continuity Condition. F is continuous on Ω if Ω is a separable metric space and if, for each x in X, f θ (x) is continuous in θ. The Continuity Condition is stronger than the Separability Condition. Let R+ Ω be the class of all functions from Ω to R + =[0, ) andletr Ω = {R(g) :g R+} Ω where R(g) ={cg : c>0} is the same class reduced modulo the scale group. An element R(f θ (x)) of R Ω is called the likelihood function from x. The likelihood map L is the function from X to R Ω that maps x in X to R(f θ (x)). For countable Ω the likelihood map is always minimal sufficient. For a general Ω, however, the likelihood map is minimal sufficient under the Continuity Condition, and the restricted likelihood map to Ω 0 say L Ω 0, is minimal sufficient under the Separability Condition, see Naderi [14] or Fraser et al [8]. The partition of X induced by both L and L Ω 0 coincides with

3 Fraser and Naderi: Minimal sufficient statistics emerge 3 the minimal sufficient partition proposed by Lehmann and Scheffé [12] a.e.[p ], see McDunnough et al [13]. The standardized likelihood map is defined as follows. There is a countable subfamily {P θ n,n 1} of P which is equivalent to P (Halmos and Savage [10]), and a Probability measure λ on B defined by λ(e) = α n P θ n (E), α n =1, α n > 0, which is equivalent to P.Letfbethe probability density of λ with respect to µ. Then f(x) = α n f θ n (x) except on a set N 1 of µ measure zero. Let N 2 = {x : f(x) =0}, thenλ(n 2 ) = 0. Since λ is equivalent to P, P θ (N 2 ) = 0 for every θ in Ω. If N = N 1 N 2,sinceµ dominates P and N 2 is a null set for P,thenN is a null set for P. The standardized likelihood map is a mapping r from X to R Ω that carries x in X N to q θ = dp θ /dλ = f θ (x)/f(x). The values of r on N are immaterial and may be defined arbitrarily. Barndorff-Nielsen et al [2] have shown that r is minimal sufficient under the Continuity Condition in the Bahadur sense. Their result may also be obtained via the following theorem and the fact that the likelihood map is minimal sufficient under the Continuity Condition; or more directly by showing that the partition of X induced by r coincides a.e.[p ] with the minimal sufficient partition proposed by Lehmann and Scheffé [12], see McDunnough et al [13]. As L Ω 0 is a minimal sufficient statistic under the Separability Condition this theorem also implies that r Ω 0, the restriction of r to Ω 0, is minimal sufficient statistic under the Separability Condition. Theorem 1. L and r induce the same partition of X, a.e.[p ]. Proof. For each x and y in X N let r(x) = r(y). Then q θ (x) = q θ (y) or f θ (x)/f(x) =f θ (y)/f(y). This may be written as f θ (x) = f(x) f(y) f θ(y) or f θ (y) = f(y) f(x) f θ(x) Hence R(f θ (x)) = R(f θ (y)) and thus L(x) =L(y). Now for each x and y in X N let L(x) =L(y). Then by definition R(f θ (x)) = R(f θ (y)).this implies that there is a nonnegative function h(x, y) such that for each θ, f θ (x) =h(x, y)f θ (y). But f(x) = α n f θ n (x) = α n h(x, y)f θ n (y) = h(x, y)f(y) Hence h(x, y) = f(x)/f(y) and thus f θ (x) = (f(x)/f(y))f θ (y). This implies that f θ (x)/f(x) =f θ (y)/f(y) orr(x) =r(y). Hence for each x and y in X N, L(x) =L(y) if and only if r(x) =r(y) and thus L and r induce the same partition of X, a.e.[p ].

4 4 International Journal of Statistical Sciences, Vol.5s, Likelihood Statistic We shall call a statistic T a likelihood statistic if the partition of X induced by T coincides with the partition of X induced by the likelihood map L, a.e.[p ]. The likelihood statistic is essentially unique up to a one to one equivalence. Clearly L is a likelihood statistic and by Theorem 1, r is also a likelihood statistic. The definition of likelihood statistic implies that a likelihood statistic is minimal sufficient if and only if the likelihood map is minimal sufficient. Hence when Ω is countable, or more generally when F is continuous on Ω, a likelihood statistic is minimal sufficient. The Likelihood statistic guides statistical analysis in almost all areas of application. Some important likelihood statistics, for example, ( x, s 2 x), arise from the likelihood function with a normal random sample. The next theorm gives a simple criteria to establish the minimal sufficiency of such statistics. Theorem 2. Suppose θ 0 is an arbitrary but fixed point in Ω such that for each x in X N, f θ0 (x) > 0. Suppose t is a statistic such that for each θ in Ω, g θ θ0 (t(x)) = f θ (x)/f θ0 (x) is a function of t(x) forx in X N. If g θ θ0 (t(x)) = g θ θ0 (t(y)), θ in Ω implies that t(x) = t(y), then t is a likelihood statistic. Proof. For each x and y in X N, t(x) =t(y) implies that g θ θ0 (t(x)) = g θ θ0 (t(y)) and hence that f θ (x)/f θ0 (x) = f θ (y)/f θ0 (y). This in turn implies that f θ (x) = (f θ0 (x)/f θ0 (y))f θ (y)andthatf θ (y) =(f θ0 (y)/f θ0 (x))f θ (x). Hence R(f θ (x)) = R(f θ (y)) and thus L(x) =L(y). Now for each x and y in X N, letl(x) =L(y). Then by definition R(f θ (x)) = R(f θ (y)). Thus there is a nonnegative function h(x, y), free of θ, such that for each θ in Ω,f θ (x) =h(x, y)f θ (y). But, as in the proof of Thearem 1, h(x, y) =f(x)/f(y). Hence for each θ in Ω, f θ (x) =(f(x)/f(y))f θ (y). The last equation implies that for an arbitrary but fixed point θ 0 in Ω,f θ0 (x) =(f(x)/f(y))f θ0 (y). Hence for each θ in Ω,f θ (x)/f θ0 (x) =f θ (y)/f θ0 (y) org θ θ0 (t(x)) = g θ θ0 (t(y)) and thus by the assumption t(x) =t(y). Since for each x and y in X N, t(x) =t(y) ifand only if L(x) =L(y), the partition of X induced by t coincides with the partition of X induced by L, a.e[p ] and hence t is a likelihood statistic. When Ω is Countable or more generally when F is continuous on Ω the likelihood statistic t in Theorem 2 is minimal sufficient. When the Continuity Condition is not fulfilled the criteria given in the next theorem may be used to establish the minimal sufficiency of the statistic t. Theorem 3. Suppose F is separable and F 0 = {f θ (x) :θ Ω 0 } is dense in F. Define g θ θ0 (t(x)) as in Theorem 2. If g θ θ0 (t(x)) = g θ θ0 (t(y)),θ in Ω 0,resultsint(x) =t(y) then t is a minimal sufficient statistic for P. Proof. Consider L Ω 0, the restriction of the likelihood map L to Ω 0. A similar argument as in the proof of Theorem 2 implies that L Ω 0 (x) =L Ω 0 (y) if and only if t(x) =t(y),a.e.[p ]. Thus both L Ω 0 and t induce the same partition of X, a.e.[p ]. But L Ω 0 is minimal sufficient under the Separability Condition for P, hence t is minimal sufficient for P. Note that the Separability Condition holds when X is a Euclidean space. Hence the criteria in Theorem 3 are widely applicable. In practice the conditions of Theorems

5 Fraser and Naderi: Minimal sufficient statistics emerge 5 2 and 3 are verified by showing that t(x) =t(y) isaθ-free solution to the equation g θ θ0 (t(x)) = g θ θ0 (t(y)), for θ in Ω for the continuity case and for θ in Ω 0 for the separability case. 4 Examples 1. Consider a random sample from the Normal (θ, σ0 2) distribution with σ2 0 known and (θ, σ0 2)inR R+ (location normal model). If θ 0 =0andt(x) = x then g θ θ0 (t(x)) = exp( n (θ 2 2θ x)). For each θ in R, g 2σ0 2 θ θ0 (t(x)) = g θ θ0 (t(y)) if and only if for each θ in R, exp( nθ ( x ȳ)) = 1. A θ-free solution to the equation σ0 2 is x = ȳ. Hence t(x) = t(y) and thus t(x) = x is a minimal sufficient statistic. 2. Consider a random sample from Normal (µ, σ 2 ) distribution with µ, σ 2 both unknown and θ =(µ, σ 2 )inr R + (location-scale normal model). If θ 0 =(0, 1) and t(x) =( x, s 2 x )then g θ θ0 (t(x)) = σ n exp( nµ2 2σ 2 n 2 ( 1 σ 2 1) x2 + nµ (n 1) x ( 1 σ2 2 σ 2 1)s2 x) For each θ in R R +, g θ θ0 (t(x)) = g θ θ0 (t(y)) if and only if for each µ in R and each σ 2 in R 2, exp( n 2 ( 1 σ 2 1)( x2 ȳ 2 )+ n (n 1) µ( x ȳ) ( 1 σ2 2 σ 2 1)(s2 x s2 y )) = 1 A θ-free solution to the equation is x =ȳ and s 2 x = s 2 y. Hence t(x) =( x, s 2 x)= (ȳ,s 2 y )=t(y) and thus t(x) =( x, s2 x ) is a minimal sufficient statistic. 3. Consider a random sample from an exponential model with probability density f θ (x) =γ(θ)exp{t 1 (x)ϕ 1 (θ)+ + t r (x)ϕ r (θ)}h(x) where 1,ϕ 1 (θ),,ϕ r (θ) are linearly independent on Ω (exponential family). If t(x) =(t 1 (x),,t r (x) then g θ θ0 (t(x)) = γ(θ) r γ(θ 0 ) exp{ t i (x)(ϕ i (θ) ϕ i (θ 0 ))} For each θ, g θ θ0 (t(x)) = g θ θ0 (t(y)) if and only if for each θ, exp{ i=1 r (t i (x) t i (y))(ϕ i (θ) ϕ i (θ 0 ))} =1 i=1 A θ-free solution to the equation is t i (x) =t i (y) fori =1,,r. Hence t(x) = (t 1 (x),,t r (x)) = (t 1 (y),,t r (y)) = t(y) and thus t(x) =(t 1 (x),,t r (x)) is a minimal sufficient statistic.

6 6 International Journal of Statistical Sciences, Vol.5s, Consider a random sample from Uniform (0,θ) distribution, θ>0. If θ 0 =1and t(x) =x (n) =max(x 1,,x n )theng θ θ0 (t(x)) = θ n I [x(n), )(θ). For each θ>0, g θ θ0 (t(x)) = g θ θ0 (t(y)) if and only if for each θ>0, I [x(n), )(θ) =I [y(n), )(θ). A θ-free solution to the equation is x (n) = y (n). Hence t(x) =t(y) and thus t(x) =x (n) is a minimal sufficient statistic. Acknowledgments This work was financially supported by Isfahan University of Technology, Isfahan, Iran and the Natural Sciences and Engineering Research Council of Canada.

7 Fraser and Naderi: Minimal sufficient statistics emerge 7 References 1. Bahadur, R.R. (1954). Sufficiency and statistical decision functions, Ann. Math. Statist., 25, Barndorff-Nielsen, O., Hoffmann-Jorgensen, J., and Pedersen, L. (1976). On the minimal sufficiency of the likelihood function, Scand. J. Statist., 3, Dynkin, E.B. (1951). Necessary and sufficient statistics for a family of probability distributions, English translation in Select. Transl. Math. Statist. Prob., 1, 1961, Fisher, R.A. (1921). On the probable error of a coefficient of correlation deduced from a small sample, Metron I, Fisher, R.A. (1922). On the mathematical foundations of theoretical statistics, Philos. Trans. Roy. Sor. London Ser., A 222, Fisher, R.A.(1934). Two new properties of mathematical likelihood, Proc. Roy. Soc. Ser., A 144, Fisher, R.A. (1956). Statistical Methods and Scientific Inference, Oliver and Boyd, London. 8. Fraser, D.A.S., McDunnough, P., Naderi, A. and Plante, A. (1995). On the definition of probability densities and sufficiency of the likelihood map, Prob. Math. Statist., 15, Fraser, D.A.S., McDunnough, P., Naderi, A., and Plante, A. (1997). From the likelihood map to Euclidean minimal sufficiency, Prob. Math. Statist., 17, Halmos, P.R. and Savage, L.J. (1949). Application of the Radon-Nikodym theorem to the theory of sufficient statistics, Ann. Math. Statist., 20, Landers, D. and Rogge, L. (1972). Minimal sufficient σ-fields and minimal sufficient statistics. Two counterexamples, Ann. Math. Stat., 43, Lehmann, E.L. and Scheffé, H.(1950). Completeness, similar regions, and unbiased estimation, Sankhya, 10, McDonnough, P. and Naderi, A. (2001). Likelihood map and minimal sufficiency: The intertwined concepts, Data Analysis from Statistical Foundations. A Festschrift in honor of D.A.S Fraser, Editor: A.K.MD.E. Saleh, Nova Science Publishers Inc., New York, 3-8.

8 8 International Journal of Statistical Sciences, Vol.5s, Naderi, A. (1988). Measurability and other properties of likelihood map, Ph.D. Thesis, University of Toronto, Neyman, J. (1935). Sur un teorema concernente le cosidette statistiche sufficienti, Giorn. Ist. Ital. Att., 6, Pitcher, T.S. (1957). Sets of measures not admitting necessary and sufficient statistics or subfields., Ann. Math. Stat., 28,