On Information and Sufficiency

Size: px

Start display at page:

Download "On Information and Sufficiency"

Christian Stevenson
5 years ago
Views:

On Information and Sufficienc Huaiu hu SFI WORKING PAPER: 997-02-04 SFI Working Papers contain accounts of scientific work of

We accept papers intended for publication in peer-reviewed journals or proceedings volumes, but not papers that have alread

Ecept for papers b our eternal facult, papers must be based on work done at SFI, inspired b an invited visit to or

NOTICE: This working paper is included b permission of the contributing author(s) as a means to ensure timel distribution of

1 On Information and Sufficienc Huaiu hu SFI WORKING PAPER: SFI Working Papers contain accounts of scientific work of the author(s) and do not necessaril represent the views of the Santa Fe Institute. We accept papers intended for publication in peer-reviewed journals or proceedings volumes, but not papers that have alread appeared in print. Ecept for papers b our eternal facult, papers must be based on work done at SFI, inspired b an invited visit to or collaboration at SFI, or funded b an SFI grant. NOTICE: This working paper is included b permission of the contributing author(s) as a means to ensure timel distribution of the scholarl and technical work on a non-commercial basis. Copright and all rights therein are maintained b the author(s). It is understood that all persons coping this information will adhere to the terms and constraints invoked b each author's copright. These works ma be reposted onl with the eplicit permission of the copright holder. SANTA FE INSTITUTE

2 SFI working paper ftp://ftp.santafe.edu/pub/zhuh/suff-dev.ps. Submitted to Annals of Statistics on Jan 7, 997. Revised. On Information and Sucienc Huaiu hu March 3, 997 Abstract The information deviation between an two nite measures cannot be increased b an statistical operations (Markov morphisms). It is invariant if and onl if the morphism is sucient for these two measures. Abbreviated Title: Information and Sucienc Aliation: Santa Fe Institute, 399 Hde Park Road, Santa Fe, NM 8750, USA. zhuh@santafe.edu, Tel: , Fa: AMS 99 Classication: Primar 62B0 (stat info) Secondar 62B20 (meas th), 62B05 (su stat), 94A7 (meas of info). Kewords: information, sucienc. Acknowledgment: I thank Kim Sau Chung for pointing out several inaccuracies in a previous version. M work in SFI is sponsored b TXN, Inc. Introduction The -deviations, 2 [0 ], generalized to the space of nite measures, plas a uniquel important role in statistical inference (hu and Rohwer, 995). One basic requirement for a measure of information is that it should not be increased b an statistical operation, and should be invariant if and onl if the operation is sucient. The most general form of statistical operation is Markov morphism ( Cencov, 982), which includes deterministic mapping as a special case. It was proved (Kullback andleibler, 95) that the -deviation (known as the Kullback-Leibler divergence) for probabilit measures satises this criterion for deterministic mappings. Here we etend this result to all -deviations on an nite measures and for an Markov morphisms. A b-product is a concise proof of sucienc of MLE, slightl dierent from Fisher's original, which is also reproduced here.

3 2 Background We generall follow (Halmos and Savage, 949) for notation and terminolog concerning measures, with the eception that, following (hu, 996), the Radon-Nikodm densit will be denoted as quotient instead of derivative. If A is a proposition then A() [p] means A is satised b all ecept a set of measure zero in p. Let e P the space of nite measures. The -deviations between p q 2 e P is dened as D (p q) := 8 >< >: p +(; )q ; p q ; 2 (0 ) ( ; ) D (p q) =0 lim!0 lim! D (p q) =: We shall make use of the following simple properties (hu and Rohwer, 995). (2.) D (p q) 0: D (p q) =0 () p = q: (2.2) Let [X X ] and [Y Y] betwo measurable spaces. Denote b P e X and P e Y the corresponding spaces of nite measures, and b P X and P Y the spaces of probabilit measures. A Markov morphism ( Cencov, 982), also called a Markov kernel or a transitional probabilit distribution, is a mapping P j : X Y! R + (2.3) such that for each it is a probabilit measure on Y. It is also a mapping from e P X to e P Y which preserves total mass p = P j p p = p : (2.4) Let M = fp gp e X and N = fp gp e Y is sucient for N if there eists a Markov morphism P j such that N P j M (Cencov, 982). If, furthermore, P j M is also sucient for M, we sa that P j is sucient for M. This means that we have a Markov morphism P j which maps P j M back to M. For an given p and P j, there alwas eists an inverse kernel P j given b the Baes theorem, mapping p back top (Kolmogorov, 956). It ma be dierent for dierent p even if P j is ed. The condition of sucienc is that one ed inverse kernel P j works for the whole model M. Its intuitive meaning is as follows: After sampling from a particular p and transforming it to according to P j, it is alwas possible, using the inverse kernel P j, to generate from a \mock sample" of with eactl the same distribution p. Therefore keeping and discarding will not lose an information concerning the original distribution, as long as we do not need knowledge of p for the inverse kernel. See also (Fisher, 925, 4) and (Halmos and Savage, 949, 0). One important special case of Markov morphism is a deterministic mapping T : X! Y, as considered in (Fisher, 925 Halmos and Savage, 949 Kullback and Leibler, 95). The transformed variable = T () is called a statistic Its measure is p = p T ;. be two families of measures. We sa that M 2

4 3 Sucienc According to Information Deviations Consider a Markov morphism P j which transforms p q 2 e P X into p q 2 e P Y : p = P j p q = Our main result in this note is that P j q : (3.) D (p q ) D (p q ) (3.2) and that the equalit holdsif and onl if P j is sucient for fp q g. Denote the joint distributions as p and q, and the Baes inverse as P j and Q j, Then p := P j p = P j p q := P j q = Q j q : (3.3) p q; = P j p q; = p q ; = p q; P j Q; j : (3.4) For 2 (0 ), denoting r := p q;, we have D (p q ) ; D (p q )= ( ; ) = r ; P ( ; j ) Q; j = p q; ; p q; rd (P j Q j ): (3.5) B taking limits of we see this is true also for 2f0 g. Therefore, D (p q ) D (p q ), for 2 [0 ]. The equalit holds if and onl if P j = Q j [r]: (3.6) The eceptional set E with r(e) = 0 can be decomposed into E = E p [ E q such that p(e p )=q(e q )=0. B changing P j to Q j on E p,we have p = P j p q = P j q : (3.7) That is, P j is sucient for fp q g. The main thread of this proof is essentiall the same as that of (Kullback and Leibler, 95). This theorem establishes that the -deviations as measures of information are selfconsistent and keeps track of all the information in a statistical problem. 3

5 4 Sucienc According to the Information Metric For smooth nite dimensional models as commonl emploed in statistical problems, we can recover Fisher's proof (Fisher, 925, 7,0) of the sucienc and ecienc of maimum likelihood estimates, b dierentiating the formulas term b term in the above proof. Despite man later epositions, Fisher's original proof remains enigmatic to most statisticians. Therefore we consider it appropriate here to reproduce his proof in modern notations, which aords more generalit and rigor while being more concise and intuitive. Let M = fp g be a famil of measures parameterized b a vector = [ ::: m ]. The partial derivatives are i i. Let P j be a ed Markov morphism. Denote b l l l L j L j the log-likelihoods corresponding to p p p P j P j. We use hi and h i to denote the mean and variance, respectivel, with possible subscripts to denote conditioning. Then, b consecutivel taking logarithm, dierentiating, and taking average b P j, we obtain l = L j + l = L j + l : i l i l i L j i l : (4.2) h@ i li = h@ i l i i l : (4.3) Therefore, using the notation of (Amari, 985, Ch. 7) for the information metric, g ij () ; g ij () = i j l ; i j l D E = i l@ j l ;h@ i li h@ j li = h@i j li =: gij (j) 0: (4.4) The inequalit is in the sense of positive deniteness. The equalit holds if and onl i l i l i L j = 0 [p ]. That is, there eists one unique P j for the whole model M. In other words, P j is sucient. If, furthermore, asmptotic normalit is assumed, i l ;g ij ()( j ; b j ) (4.5) where b is the MLE. So g ij (j) =0if and onl if b is constant for each. In other words, b is a function of. Hence if a sucient estimate eists, then MLE is sucient. In an case it is alwas among the most ecient estimates. Fisher's original proof was in fact based on ;@ j l j p p 2 i@ j p p = h@ i li h@ j li ;h@ i l@ j l j li = g ij () ;h@ i j li (4.6) using g ij = ; R p@ j l instead of g ij = R p@ i l@ j l. The advantage of considerations based on information deviations instead of the information metric is, as noted in (Halmos and Savage, 949), that neither the proof nor its interpretation requires assumptions about smoothness, dimensionalit, dominatedness, asmptotic normalit, or an other similarl \etraneous" assumptions. 4

6 References Amari, S. (985). Dierential-Geometrical Methods in Statistics, volume 28 of Springer Lecture Notes in Statistics. Springer-Verlag, New York. Cencov, N. N. (982). Optimal Decision Rules and Optimal Inference. Amer. Math. Soc., Rhode Island. Translation from Russian, 972, Nauka, Moscow. Fisher, R. A. (925). Theor of statistical estimation. Proc. Camb. Phi. Soc., 22:700{725. Halmos, P. R. and Savage, L. J. (949). Application of the Radon-Nikodm theorem to the theor of sucient statistics. Ann. Math. Statist., pages 225{24. Kolmogorov, A. N. (956). Foundations of the Theor of Probabilit. Chelsea., New York. Translation of Grundbegrie der Wahrscheinlichkeitsrechnung, 933. Kullback, S. and Leibler, R. A. (95). On information and sucienc. Ann. Math. Statist., 22:79{86. hu, H. (996). On topologies and geometries of information deviations of nite measures. Submitted. hu, H. and Rohwer, R. (995). Baesian geometric theor of statistical inference. Submitted to Ann. Stat. 5

On the Speed of Quantum Computers with Finite Size Clocks

On the Speed of Quantum Computers with Finite Size Clocks Tino Gramss SFI WORKING PAPER: 1995-1-8 SFI Working Papers contain accounts of scientific work of the author(s) and do not necessarily represent