Entropy power inequality for a family of discrete random variables

Transcription

1 20 IEEE International Symposium on Information Theory Proceedings Entropy power inequality for a family of discrete random variables Naresh Sharma, Smarajit Das and Siddharth Muthurishnan School of Technology and Computer Science Tata Institute of Fundamental Research Mumbai , India {nsharma,smarajit}@tifr.res.in, smrish@tcs.tifr.res.in. Abstract It is nown that the Entropy Power Inequality (EPI) always holds if the random variables have density. Not much wor has been done to identify discrete distributions for which the inequality holds with the differential entropy replaced by the discrete entropy. Harremoës and Vignat showed that it holds for the pair (B(m,p),B(n, p)), m,n N, (where B(n, p) is a Binomial distribution with n trials each with success probability p) forp =0.5. Inthispaper, we considerably expand the set of Binomial distributions for which the inequality holds and, in particular, identify n 0(p) such that for all m, n n 0(p), theepi holds for (B(m,p),B(n, p)). Wefurthershowthatthe EPI holds for the discrete random variables that can be expressed as the sum of n independent and identically distributed (IID) discrete random variables for large n. Index Terms Entropy power inequality, Taylor s theorem, asymptotic series, binomial distribution. I. INTRODUCTION The Entropy Power Inequality e 2h(X+Y ) e 2h(X) +e 2h(Y ) () holds for independent random variables X and Y with densities, where h( ) is the differential entropy. It was first stated by Shannon in Ref. [], and the proof was given by Stam and Blachman [2]. See also Refs. [3], [4], [5], [6], [7], [8], [9]. This inequality is, in general, not true for discrete distributions where the differential entropy is replaced by the discrete entropy. For some special cases (binary random variables with modulo 2 addition), results have been provided by Shamai and Wyner in Ref. [0]. More recently, Harremoës and Vignat have shown that this inequality will hold if X and Y are B(n, /2) and B(m, /2) respectively for all m, n []. Significantly, the convolution operation to get the distribution of X +Y is performed over the usual addition over reals and not over finite fields. Recently, another approach has been expounded by Harremoës et. al. [2] and by Johnson and Yu [3], wherein they interpret Rényi s thinning operation on a discrete random variable as a discrete analog of the scaling operation for continuous random variables. They provide inequalities for the convolutions of thinned discrete random variables that can be interpreted as the discrete analogs of the ones for the continuous case. In this paper, we tae a re-loo at the result by Harremoës and Vignat [] for the Binomial family and extend it for all p (0, ). Weshowthatthere always exists an n 0 (p) that is a function of p, such that for all m, n n 0 (p), e 2H[B(m+n,p)] e 2H[B(m,p)] +e 2H[B(n,p)], (2) whereh( ) is the discrete entropy. The result in Ref. [] is a special case of our result since we obtain n 0 (0.5) = 7 and it can be checed numerically by using a sufficient condition that the inequality holds for m, n 6. We then extend our results for the family of discrete random variables that can be written as the sum of n IID random variables and show that for large n, EPI holds. We also loo at the semi-asymptotic case for the distributions B(m, p) with m small and B(n, p) with n large. However, when n is large, there may exist some m such that EPI does not hold. The proofs for all the lemmas and theorems except that of Theorem 2 and 3 are omitted due to lac of space, but they are available in Ref. [4]. II. EPI FOR THE BINOMIAL DISTRIBUTION Our aim is to have an estimate on the threshold n 0 (p) such that e 2H[B(m+n,p)] e 2H[B(m,p)] +e 2H[B(n,p)], (3) //$ IEEE 945

2 holds for all m, n n 0 (p). It is observed that n 0 (p) depends on the sewness of the associated Bernoulli distribution. Sewness of a probability distribution is defined as κ 3 / κ 3 2 where κ 2 and κ 3 are respectively the second and third cumulants of the Bernoulli distribution B(,p), anditturnsouttobe(2p )/ p( p). Let (2p )2 ω(p) = p( p). (4) We find an expression for n 0 (p) that depends on ω(p). The well nown Taylor s theorem will be useful for this purpose (see for example p. 0 in Ref. [5]) and is stated as follows. Suppose f is a real function on [a, b], n N, the (n )th derivative of f denoted by f (n ) is continuous on [a, b], andf (n) (t) exists for all t (a, b). Letα, β be distinct points of [a, b], thenthere exists a point y between α and β such that n f(β) = f(α)+ f () (α) (β α)! + f (n) (y) (β α) n. (5) n! For 0 p, leth(p) denote the discrete entropy of a Bernoulli distribution with probability of success p, thatis,h(p) p log(p) ( p)log( p). Weshallusethenaturallogarithm throughout this paper. Note that we earlier defined H( ) to be the discrete entropy of a discrete random variable. The definition to be used would be amply clear from the context in what follows. Let Ĥ(x) H(p) H(x), x (0, ), (6) F () (x) Ĥ() (x). (7)! Note that Ĥ(x) satisfies the assumptions in the Taylor s theorem in x (0, ). Therefore,wecan write n Ĥ(x) = Ĥ(p)+ F () (x)(x p) + F (n) (x )(x p) n, (8) for some x (x, p). NotethatĤ(p) =0. For even, F () (x) 0 for all x (0, ), and hence, Ĥ(x) 2l+ F () (p)(x p) (9) for all x (0, ) and any non-negative integer l. The following useful identity would be employed at times 2 2ν ( log(2) H(p) = p 2ν. (0) 2ν(2ν ) 2) ν= Let P {p i } and Q {q i } be two probability measures over a finite alphabet A. LetC (p) (P, Q) and ν (p) (P, Q) be measures of discrimination defined as C (p) (P, Q) pd(p M)+qD(Q M), () ν (p) pp i qq i 2ν, (pp i + qq i ) 2ν i A (2) where M = pp + qq,q = p and D( ) is the Kullbac-Leibler divergence. These quantities are generalized capacitory discrimination and triangular discrimination of order ν respectively that were introduced by Topsøe [6]. The following theorem relates C (p) (P, Q) with (P, Q) and would be used later to derive an expression for n 0 (p). ItgeneralizesTheoremin Ref. [6]. (p) ν Theorem. Let P and Q be two distributions over the alphabet A and 0 <p<. Then C (p) (P, Q) = ν= ν (p) (P, Q) [log(2) H(p)]. 2ν(2ν ) Let X (n) be a discrete random variable that can be written as X (n) = Z +Z 2 + +Z n,wherez i s are IID random variables. We note that when X (n) is defined as above, we have X (n) +X (m) = X (n+m). Let Y n e 2[H(X(n) )].Wefirstusealemmadueto Harremoës and Vignat []. Lemma (Harremoës and Vignat []). If Y n /n is increasing, then Y n is super-additive, i.e., Y m+n Y m + Y n. It is not difficult to show that this is a sufficient condition for the EPI to hold []. By the above lemma, the inequality H(X (n+) ) H(X (n) ) 2 log ( n + n ) (3) is sufficient for EPI to hold. Let X (n) = B(n, p). We have P X (n+)( +)=pp X (n)()+qp X (n)( +). (4) 946

3 f(2,, p) p Fig.. Plot of e 2H[B(3,p)] e 2H[B(2,p)] +e 2H[B(,p)] versus p. Define a random variable X (n) + as P X (n) + ( + ) = P X (n)() for all {0,,,n}. Hence, using H(X (n) +)=H(X (n) ), P X (n+) = pp X (n) + +qp X (n) and (), we generalize Eq. (3.7) in Ref. [] as H(X (n+) )=H(X (n) )+C (p) (P X (n) +,P X (n)). (5) We now derive the lower bound for C (p) (P X (n) +,P X (n)). Lemma 2. For l N, 2l+ C (p) (P X (n),p X (n) + ) X F () (p)(n +) µ (n+), where µ (n) is the -th central moment of B(n, p), i.e., n µ (n) = (i np) P X (n)(i). (6) i=0 We note that there exist m, n for p 0.5 for which EPI does not hold, that is, e 2H[B(m+n,p)] < e 2H[B(m,p)] +e 2H[B(n,p)]. In fact, one can easily see that e 2H[B(2,p)] e 2H[B(,p)] +e 2H[B(,p)] p, with equality if and only if p =0.5. For the case m =and n =2,itisclearfromth Fig. that EPI holds when p is close to 0.5, while EPI does not hold if p is close to 0 or. Thisleads us to the question that for a given p, whatshould m, n be such that the EPI would hold. The main theorem of this section answers this question. Theorem 2. H[B(n +,p)] H[B(n, p)] ( ) n + 2 log n n n 0 (p). Several candidates of n 0 (p) are possible such as n 0 (p) = 4.44 ω(p) +7 and n 0 (p) = ω(p) ω(p)+7. Proof: See the Appendix. Using Lemma 2, we can obtain a non-asymptotic lower bound to the entropy of the binomial distribution unlie the asymptotic expansion of H[B(n, p)] given in Ref. [7]. For details, see Ref. [4]. III. EPI FOR THE SUM OF IID We showed in the previous section that EPI holds for the pair (B(n, p),b(m, p)) for all m, n n 0 (p). The question naturally arises whether EPI holds for all such discrete random variables that can be expressed as sum of IID random variables. Let X (n) be a discrete random variable such that X (n) X + X X n, (7) where X i s are IID random variables and σ 2 is the variance of X.Theseries l= a lφ l (n) is said to be an asymptotic expansion for f(n) as n if N f(n) = a l φ l (n)+o[φ N (n)] N, (8) l= and is written as f(n) l a lφ l (n), where if γ(n) o[φ(n)], thenlim n γ(n)/φ(n) =0.We shall use the asymptotic expansion due to Knessl [7]. 947

4 Lemma 3 (Knessl [7]). For a random variable X (n),asdefinedabove,havingfinitemoments,we have as n, g(n) H(X (n) ) 2 log(2πenσ2 ) κ2 3 2σ 6 n + β l, (9) nl+ l= where κ j is the jth cumulant of of X.Ifκ 3 = κ 4 = = κ N =0but κ N+ 0,then g(n) 2(N +)!σ 2N+2 n N + l=n β l n l+. Note that the leading term in the asymptotic expansion is always negative. We also note using Lemma 3 that as n, H(X (n) ) < 2 log(2πenσ2 ). (20) To see this, we invoe the definition of the asymptotic series to get g(n) = 2(N +)!σ 2N+2 n N + β ( ) N n N +o n N. From the definition of the little-oh notation, we now that given any ɛ>0, thereexistsal(ɛ) > 0 such that for all n>l(ɛ), g(n) = 2(N +)!σ 2N+2 n N + β N + ɛ n N. Choosing n large enough, we get the desired result. We consider the case of the pair (X (n),x (m) ) when both m, n are large and have the following result. Theorem 3. There exists a n 0 N such that e 2H(X(m) +X (n)) e 2H(X(m)) +e 2H(X(n) ) for all m, n n 0. (2) Proof: We shall prove the sufficient condition for the EPI to hold (as per Lemma ) and show that H(X (n+) ) H(X (n) ) ( ) n + 2 log (22) n for n n 0 for some n 0 N. Let us tae the first three terms in the above asymptotic series as g(n) C n + C 2 n 2 + C 3 n 3 (23) where 0 < < 2 < 3 and C is some non-zero positive constant, and hence, g(n)+ C n C 2 n C ( ) 3 2 n = o. (24) 3 n 3 and given any ɛ>0, thereexistsal(ɛ) > 0 such that for all n>l(ɛ), g(n)+ C n C 2 n C 3 2 n ɛ 3 n. (25) 3 From the above inequality, we have, by using the lower and upper bounds respectively for g(n +) and g(n), that [ ] g(n +) g(n) C + [ C 2 (n +) 2 n 2 n (n +) [ C3 ɛ (n +) C 3 + ɛ 3 n 3 ] + ]. From the above expression, we can clearly see that the first term is strictly positive and is Θ ( /n +). The second and third terms (their signs are irrelevant) are of the order Θ(/n 2+ ) and Θ(/n 3 ) respectively. The Θ( ) means both asymptotic lower and asymptotic upper bounding. It is clear that there exists some positive integer n 0 such that for all n n 0,first(positive)termwilldominateandthe other two terms will be negligible compared to the first and hence g(n +) g(n) 0. It must be noted that if one considers the pair (X (n),x (m) ) where n and m is fixed, then the EPI need not hold. As can can be seen in our paper [4], it can be shown that e 2H(X(m) +X (n)) e 2H(X(m)) +e 2H(X(n) ) if n and H(X (m) ) < 2 log[2πemσ2 ]. IV. CONCLUSIONS We have expanded the set of pairs of Binomial distributions for which the EPI holds. We identified athresholdthatisafunctionoftheprobabilityof success beyond which the EPI holds. We further show that EPI would hold for discrete random variables that can be written as sum of IID random variables. We also obtain an improvement of a bound given by Artstein et. al. [6] (see Ref. [4]). It would be interesting to now if C (p) (P X (n) +,P X (n)) for X(n) = B(n, p) is a concave function in p. It would also be of interest to now that for a given p (0, 0.5) if H[B(n +,p)] H[B(n, p)] 0.5log( + /n) would have a single zero crossing as a function of n when n increases from to. 948

5 REFERENCES [] C. E. Shannon, A mathematical theory of communication, Bell Syst. Tech. J., vol.27,pp and ,July and Oct [2] N. M. Blachman, The convolution inequality for entropy powers, IEEE Trans. Inf. Theory, vol., pp , Apr [3] M. H. M. Costa, A new entropy power inequality, IEEE Trans. Inf. Theory, vol.3,pp ,Nov.985. [4] A. Dembo, T. M. Cover, and J. A. Thomas, Information theoretic inequalities, IEEE Trans. Inf. Theory, vol. 37, pp , Nov. 99. [5] O. T. Johnson, Log-concavity and the maximum entropy property of the Poisson distribution, Stoch. Proc. Appl., vol. 7, pp , June [6] S. Artstein, K. M. Ball, F. Barthe, and A. Naor, Solution of Shannon s problem on the monotonicity of entropy, J. Amer. Math. Soc., vol.7,pp ,May2004. [7] A. M. Tulino and S. Verdú, Monotonic decrease of the non-gaussianness of the sum of independent random variables: A simple proof, IEEE Trans. Inf. Theory, vol. 52, pp , Sep [8] S. Verdú and D. Guo, A simple proof of the entropy-power inequality, IEEE Trans. Inf. Theory, vol. 52, pp , May [9] M. Madiman and A. Barron, Generalized entropy power inequalities and monotonicity properties of information, IEEE Trans. Inf. Theory, vol.53,pp ,July2007. [0] S. Shamai (Shitz) and A. D. Wyner, A binary analog to the entropy-power inequality, IEEE Trans. Inf. Theory, vol.36, pp , Nov [] P. Harremoës and C. Vignat, An entropy power inequality for the binomial family, J. Inequal. Pure Appl. Math., vol. 4, no. 5, Oct [2] P. Harremoës, O. Johnshon, and I. Kontoyiannis, Thinning, entropy, and the law of thin numbers, IEEE Trans. Inf. Theory, vol.56,pp ,Sep.200. [3] O. Johnshon and Y. Yu, Monotonicity, thinning, and discrete versions of the entropy power inequality, IEEE Trans. Inf. Theory, vol.56,pp ,Nov.200. [4] N. Sharma, S. Das, and S. Muthurishnan, Entropy power inequality for a family of discrete random variables, http: //arxiv.org/abs/02.042, Dec [5] W. Rudin, Principles of Mathematical Analysis, 3rd ed. McGraw-Hill, 976. [6] F. Topsøe, Some inequalities for information divergence and related measures of discrimination, IEEE Trans. Inf. Theory, vol.46,pp ,July2000. [7] C. Knessl, Integral representations and asymptotic expansions for Shannon and Renyi entropies, Appl. Math. Lett., vol., pp , Mar APPENDIX PROOF OF THEOREM 2 We prove that for all n n 0 (p), H[B(n +,p)] H[B(n, p)] n + 2 log n Using Lemma 2, we have H[B(n +,p)] H[B(n, p)] 2l+ «(26) F () (p)(n +) µ (n+). (27) Let r = p /2 and t = ω(p) =6r 2 /( 4r 2 ). We have r 2 = t/[4(t +4)]. Note that r 2 [0, /4) and t [0, ). Wewill use the first seven central moments of B(n, p) and they contain only even powers of r and hence, can be written as a function of t. We upper bound the right hand side of (26) as ( ) n + log n n 2n 2 + 3n 3. (28) Define f(n, t) 7X»r t F () 4(t +4) + (n +) µ (n+) 2 2 n + 2n «. 2 3n 3 Proving (26) is equivalent to showing that f(n, t) 0 n>n 0 (p). Simplifying h f(n, t) = 35n 7 t +(70+35t +35t 2 )n 6 420(n +) 6 n 3 ( t +3339t 2 +72t 3 )n 5 (826 72t 37t t t 4 )n 4 ( t +35t 2 +57t 3 +0t 5 +66t 4 )n 3 i 630n 2 35n 70. (29) Define g(n, t) 420(n +) 6 n 3 f(n, t). A simple but elaborate calculation yields g(4.44t +7+m, t) 35tm 7 +(22.8t t +70)m 6 +( t t t +2625)m 5 +(.0t t t t+0.40)0 5 m 4 +(3.85t t t t t+3.02)0 5 m 3 +(7.76t t t t t t +.80)0 5 m 2 +(6.47t t t t t t t )0 5 m +(0.5t t t t t t t t +64.5)0 4. Note that all the coefficients are positive and hence, f(4.44t +7+m, t) 0 for all m 0 or f(n, t) 0 for all n 4.44t +7. A more careful choice would yield all coefficients to be positive for n 4.438t +7.Yetanotherchoice that would yield all coefficients as positive would be n t t +7.Notethatthischoicewould be better for 0 <t<2. and, in particular, for t =,thefirstchoiceyields(afterconstrainingn to be a natural number) n 2 while the second one yields n.furtherrefinementsarealsopossible. 949