VARIANTS OF ENTROPY POWER INEQUALITY

Transcription

1 VARIANTS OF ENTROPY POWER INEQUALITY Sergey G. Bobkov and Arnaud Marsiglietti Abstract An extension of the entroy ower inequality to the form N α r (X + Y ) N α r (X) + N α r (Y ) with arbitrary indeendent summands X and Y in R n is obtained for the Rényi entroy and owers α (r + )/. Introduction Given a continuous random vector X in R n with density f, define the (Shannon) entroy and the associated entroy ower h(x) = f(x) log f(x) dx, N(f) = N(X) = e n h(x). R n Serving as measures of chaos or randomness hidden in the distribution of X, these functionals ossess a number of remarkable roerties, esecially when they are considered on convolutions. For examle, we have the famous entroy ower inequality (EPI), fundamental in Information Theory, which states that N(X + Y ) N(X) + N(Y ), (.) for arbitrary indeendent summands X and Y in R n (cf. [Sh], [St]). Several roofs of the EPI exist (see e.g. [L], [D-T-C], [S-V], [V-G], [R], [W-M]), as well as refinements (see e.g. [A-B-B-N], [M-B], [Co]). We refer to the survey [M-M-X] for further details. Moreover, when a Gaussian noise is added to X, i.e., if Y = t Z with Z standard normal, the random vector X + t Z has density f t whose entroy ower is a concave function in t, so that d dt N(f t) 0 (t > 0). (.) This observation due to Costa [C], which strengthens (.) in the secial case where Y is Gaussian, is known as the concavity of entroy ower theorem (cf. also [D], [V]). Key words: Entroy ower inequality, Rényi entroy. MSC 94A 60F School of Mathematics, University of Minnesota, 7 Vincent Hall, 06 Church St. S.E., Minneaolis, MN USA. bobkov@math.umn.edu Partially suorted by the Alexander von Humboldt Foundation and NSF grant DMS-696 California Institute of Technology, 00 East California Boulevard, MC 305-6, Pasadena, CA 95 USA. amarsigl@caltech.edu Suorted by the Walter S. Baer and Jeri Weiss CMI Postdoctoral Fellowshi.

2 Of a large interest there has been the question on ossible extensions of such roerties to more general informational characteristics given, in articular, by the Rényi entroy ower N r (X) = f(x) r dx R n ) n of a fixed order r > 0, or by some natural functionals of N r. As one interesting examle, for the densities u(x, t) = f t (x) solving the nonlinear heat equation t u = ur with r > n, Savaré and Toscani [S-T] have extended roerty (.) to the functional Nr α in lace of N, where α = + n (r ). Therefore, in this PDE context, it is natural to work with Ñ r (X) = f(x) r dx R n ) n r r, called the r-th Rényi ower in [S-T]. Although the solutions f t lose the convolution structure, one may wonder whether or not the Savaré-Toscani entroy ower Ñr shares the EPI (.) as well. Here we give an affirmative answer to this question, including sharer owers of N r. Theorem.. Given indeendent random vectors X and Y in R n with densities, we have N α r (X + Y ) N α r (X) + N α r (Y ) (.3) whenever α r+ (r > ). Letting r, inequality (.3) returns us to the classical EPI. This inequality is getting sharer when r is fixed and α decreases, but it is not clear to us, if the ower α = r+ is critical (which aears in the definition of Ñr for dimension one). Anyhow, (.3) is no longer true for α = like in (.). For the range r > 3, this fact was mentioned in [B-C] in case where both X and Y are uniformly distributed. As we will see, (.3) may be violated with α = for any r >, even when one of the summands is normally distributed (that is, for the densities f t in the heat semigrou model). In the roof of (.3) we follow an aroach of Lieb [L], emloying Young s inequality with best constants. Although the basic argument is rather standard, we remind it in the next section. In our situation it leads to some routine calculus comutations, so we move the involved analysis to searate sections (starting with the case of equal entroy owers). In Section 5, we analyze (.3) with α = and show that this inequality cannot be true in general. Finally, in Section 6 we rovide a simle lower bound on the otimal exonent α = α(r) in (.3). Information-theoretic formulation of Young s inequality The Young inequality with otimal constants (due to Beckner [B] and Brascam and Lieb [B-L]) indicates that, for any two indeendent random vectors X and Y in R n with densities f and g, resectively, and for all arameters, q, r such that + q = r, (.)

3 3 we have with f g r C n f g q (.) C = C(, q, r) = c c q c r, where c α = α/α (α ) /α. (.3) As usual, f g denotes the convolution, = f = f(x) dx R n is the conjugate ower, and stands for the L -norm of a non-negative function f on R n with resect to the Lebesgue measure. In general, we have C, with equality in (.) attainable for Gaussian densities (The traditional Young inequality is formulated without this constant, so, in a weaker form). Since f r = N r (X) n r, the inequality (.) may be stated as a dimension-free relation between the corresonding entroy owers, namely ) / N r (X + Y ) r C N (X) N q (Y ) q. (.4) This is an equivalent information-theoretic formulation of Beckner s result, secialized to the class of robability densities, which aears, for examle, in the book by Cover and Thomas [C-T] (in a slightly different form, cf. Theorem 7.8.3,. 677). It is natural to have an analog of (.4) for one functional N r only (rather than for three arameters). This can be done on the basis of (.) by noting that, due to Jensen s (or Hölder s) inequality, and since, q r in (.) and f is a robability density function, f r f f r r = f r r r (r > ). As an alternative aroach, one can just use the monotonicity of the function r N r, which follows, for examle, from the reresentation N n r (X) = [ E f(x) r ] r. Hence N N r, N q N r in (.4), and with these bounds it immediately yields: Proosition.. Given indeendent random vectors X and Y in R n with densities, we have N r (X + Y ) r C N r(x) N r (Y ) q, (.5) which holds true for all, q, r subject to (.) with constant C = C(, q, r) as in (.3). A weak oint of this inequality is however the loss of equality on Gaussian densities. Nevertheless, there is still freedom to otimize the right-hand side over all admissible coules (, q), or to choose secific values, even if they are not otimal.

4 4 3 The case of equal entroy owers Let us illustrate this aroach in the simler situation of equal Rényi entroies. N r (X) = N r (Y ) = N, inequality (.5) is simlified to When N r (X + Y ) C r N, (3.) and our task reduces to the minimization of C as a function of (, q) for a fixed r >. Putting x =, y = q, so that = = x and q = q = y, from (.3), C = c r c c q = c r ( )/ ( ) / ( q )/q Hence, we need to maximize the quantity ( x) x ( = c r q ) /q x x ( y) y y y. (3.) ψ(x) = log C = log c r ( x log x ( x) log( x) ) ( y log y ( y) log( y) ) subject to the constraint (.), that is, for x, y 0, x+y = r, or equivalently, on the interval 0 x r. At the endoints, we have ψ(0) = ψ(/r ) = 0, while inside the interval ψ (x) = log y( y) x( x) = 0 if and only if y( y) = x( x). This equation is solved either as y = x = But the latter contradicts x + y = r <. Moreover, ψ (x) = x x( x) + y y( y) r or as y = x. is negative at x 0 = r, which imlies that x 0 is the oint of maximum of the function ψ. Thus, the coefficient C in (.5) is maximized, when = q = r = r r. For these values, = q = r r+, so c = c q = (r) r (r ) r r (r + ) r+ r and C = r r (r + ) r+ r. It remains to raise C to the ower r = r r, and then we obtain an exlicit exression for the otimal constant in (3.) derived on the basis of (.5). Proosition 3.. If the indeendent random vectors X and Y satisfy N r (X) = N r (Y ) = N for some r, then N r (X + Y ) A r N, A r = 4 r (r + ) r+ r r r r. (3.3) Proosition 3. is not new and a more general version where the distributions have different Rényi entroies was obtained in [R-S, Theorem ]. Moreover, in the case of different Rényi entroies, tighter bounds are rovided in [R-S, Corollary 3]. Now, it is easy to see that < A r <. Moreover, (3.3) rovides the desired linear bound (.3) in case of equal entroy owers, N α r (X + Y ) N α (X), as long as A r /α, or equivalently, when α > α(r) = (log )/(log A r ). A simle analysis shows that α(r) (r+)/.

5 5 4 The general case Here we derive the extension (.3) of the EPI for the ower α = r+ in the case of arbitrary values N r (X) and N r (Y ). As a reliminary ste, let us return to the inequality (.5) and raise it to the ower α, so as to rewrite it as Nr α (X + Y ) C αr N r (X) αr N r (Y ) αr q. Putting x = N α r (X), y = N α r (Y ) and assuming without loss of generality that x + y = r (using homogeneity of these functionals), we need to show that C αr x r y r q r for some admissible, q, i.e., satisfying the condition + q = r. The best values of and q can be described imlicitly as solutions to a certain equation, and we refer to take some secific values. As a natural choice, consider (, q) such that = x and q = y and try to check the desired inequality x xr y yr C αr r, i.e., x x y y C α (r ) /r ( x, y > 0, x + y = r ). Equivalently, so that to eliminate the arameter r, we need to check whether or not As in Section 3, cf. (3.), and (4.) takes the form x x y y C α (x + y) x+y (x, y > 0, x + y < ). (4.) C = c c q c r = x x y y ( x y) x y ( x) x ( y) y (x + y) x+y, ( ) (x + y) x+y α x x y y Now, in case α = r+, we have ( ( x y) x y ( x) x ( y) y ) α. (4.) and (4.) becomes α = r ( r ) = x y ( x y), α = x + y ( x y), ( ) (x + y) x+y x+y x x y y ( ( x y) x y ( x) x ( y) y ) x y, (4.3) which is to be verified in the triangle x, y > 0, x + y <. This inequality is obviously true for boundary values x = 0, y = 0, and when x + y =, with equality of both sides. To roceed, we choose an argument based on the following calculus lemma.

6 6 Lemma 4.. Given 0 < c < and β c, the function ψ(x) = ( x)β( x) ( y) β( y) x x y y (0 x c, y = c x) attains minimum either at the endoints x = 0, x = c, or at the center x = c. Moreover, in case β = c, this function attains minimum at the endoints. Proof. Inside the interval (0, c) the function v(x) = log ψ(x) has the first two derivatives = β(( x) log( x) + ( y) log( y)) (x log x + y log y) v (x) = v (x) = β ( log( x) log( y) ) (log x log y), ( β x + β ) ( y x + ) y = β( c) ( x)( y) c xy. Note that v(0) = v(c). Also, v (0+) =, v (c ) =, so v is increasing near zero and is decreasing near the oint c. In addition, v (x) is vanishing, if and only if w(x) β( c) xy c ( x)( y) = 0, (4.4) which is a quadratic equation (recall that y = c x). In general it has at most two roots. Case : Equation (4.4) has at most one root in (0, c). Since w(0) < 0, it means that w(x) 0 in (0, c). Therefore, v is concave, and thus attains its minimum at the endoints of this interval. Case : Equation (4.4) has exactly two roots in (0, c), say 0 < x < x < c. Since w(0) < 0 and w(c) < 0, it means that w(x) < 0 in (0, x ) and (x, c), while w(x) > 0 in (x, x ). That is, v is strictly concave on (0, x ) and (x, c), and is strictly convex on the intermediate interval. Hence, in this case there is at most one oint of local minimum. If there is no oint of local minimum, then v attains its minimum at the endoints. It there is one oint x 0 of local minimum of v, then it must belong to (x, x ), and there are two oints of local maximum, say z and z belonging to the other subintervals. In articular, v < 0 on (z, x 0 ) and v > 0 on (x 0, z ). Note that v (c/) = 0, so this oint is a candidate for local extremum. Moreover, by the assumtion on β, v 4 (βc ( c)) (c/) = 0. c( c) If β > c, then v (c/) > 0 which means that x 0 = c/ is a local minimum for v and therefore for ψ, and the first assertion follows. If β = c, then v (c/) = 0 which means that either c/ = x or c/ = x. But at these oints the derivative of v may not vanish. In other words, the equality β = c is only ossible under Case.

7 7 Proof of Theorem.. Let us return to the inequality (4.3) and rewrite it as (x + y) x+y x x y y ( ( x y) x y ( x) x ( y) y ) β, β = α α, or equivalently ( x) β( x) ( y) β( y) x x y y ( x y)β( x y) (x + y) x+y. (4.5) Here the right-hand side deends only on c = x + y (since α may only deend on r which is a function of x + y). Hence, to rove (4.5), it is sufficient to minimize the left-hand side under the constraint x, y 0, x + y = c, and then to comare the minimum with the right-hand side. In case α = r+, we have α α = x y = c = x + y c c, which is exactly the extreme value for β in Lemma 4.. Therefore, by its conclusion, the left-hand side of (4.5) is minimized either at x = 0 or x = c. But for such boundary values there is equality in (4.5), as was already stressed before Lemma 4.. As a result, we obtain the desired inequality (4.3) for all x, y 0 such that x + y. 5 Rényi entroy owers for the heat semi-grou Let us now look at the ossible behavior of the Rényi entroy owers in the class of densities f t of X t = X + tz, assuming that X has a sufficiently regular ositive density f (on the line), and Z is a standard normal random variable indeendent of X. Since for small t > 0 f t (x) = f(x) + f (x) t + o(t), we find, by Taylor exansion and integrating by arts, and thus, for r >, f t (x) r dx = f(x) r dx t r(r ) f(x) r f (x) dx + o(t), ) +r N r (X t ) = N r (X) + tr f(x) r r dx f(x) r f (x) dx + o(t). Using this reresentation, we are going to test the inequality (.3) for α =, when it becomes N r (X t ) N r (X) + tn r (Z). Comaring the linear terms in front of t and using N r (Z) = πr r, we would be led to a Nash-tye inequality ) +r r f(x) r r dx f(x) r f (x) dx π r r, (5.)

8 8 holding already without too restrictive conditions (e.g., for all C -smooth f > 0). Now, let us take f(x) = Be x with, where B is a normalizing constant, i.e., B = Γ( ). In this case, so that Similarly, f(x) r dx = B r f (x) f(x) r dx = B r e r x dx = B r r /, ) +r f(x) r r dx = B (r+) r r+ and thus the left-hand side in (5.) is equal to r ( B r r+ (r ) ) ( r Γ ) Hence, inequality (5.) says that x ( ) e r x π 4 r r( ) (r ) Γ (r ). ( dx = B r ) ( r Γ ), ( ( = 4 Γ Γ ) ) r r (r ). ( ) ( Γ ). (5.) We claim that it cannot be true for all > sufficiently close to (i.e., when X itself is almost standard normal). To see this, denote by G() the right-hand side of (5.) and note that there is equality at =. So, let us look at the derivative and show that G () < 0, i.e., H (/) > 0 for H(x) = log G(/x). Indeed, H (x) = r r log r + Γ (x) Γ(x) Γ ( x) Γ( x). From the fundamental relation Γ(x + ) = xγ(x), it follows that Γ (x + ) = Γ(x) + xγ (x), so Γ (3/) = Γ(/) + Γ (/), while Γ(3/) = Γ(/). Hence, H (/) = r r log r > 0. We may conclude that the entroy ower inequality for N r of any order r > does not hold in general, even when one of the variable is Gaussian. For another, less direct argument, one may return to (5.) and rewrite it as a homogeneous inequality ) +r ) r r f(x) r r dx f(x) r f (x) dx π r r r f(x) dx. R R R After the change f = u r, it takes the form of the Nash-tye inequality u(x) dx R ) r+ r Kr R ) r u (x) dx u(x) r r dx R (5.3)

9 9 with K r = πr r r. In fact, the Nash inequality in Rn asserts that R n u(x) dx with shar constant given by ) + n C n = Cn u(x) dx u(x) dx R n R n ( + ) ( n ) Γ n + n πjn/ (cf. [C-L], [B]). Here j n/ denotes the smallest ositive zero of the Bessel function J n/ of order n/. In dimension n =, one has J / (x) = πx sin(x) (cf. [W],. 54, eq. (3)]), thus j / = π. Hence the shar Nash inequality in dimension reads ) 3 u(x) dx 7 6π u (x) dx u(x) dx) 4, which is the same as (5.3) for r =, however, with a larger constant. Hence, as we have already seen, inequality (.3) cannot be true for α = and r =. Let us notice that the Nash inequality with the asymtotically shar constant /(πen) can be deduced from the classical EPI (.) (cf. [T]). For the arameter r =, routine comutations also rovide a counter-examle in the case where both X and Y have the beta distribution with density f(x) = 3 4 ( x ), x <. ) 4 n 6 Lower bound on the otimal exonent One may also rovide a simle lower bound on the otimal exonent α = α ot that satisfies the inequality N r (X + Y ) α N r (X) α + N r (Y ) α for all indeendent random vectors X and Y. Together with the uer bound of Theorem., we have: Proosition 6. One has α ot [ log r log( r+ ] r+ ),. Proof. For the lower bound, let X and Y be indeendent and uniformly distributed on [0, ], in which case N r (X) = N r (Y ) =. The sum X + Y has the triangle density (f g)(x) = x on [0, ] and (f g)(x) = x on [, ]. Hence, Thus (f g)(x) r dx = 0 x r dx + ( ) r + r N r (X + Y ) =. ( x) r dx = r +.

10 0 Since N r (X + Y ) αot N r (X) αot + N r (Y ) αot, we deduce that ( r+ required statement. ) α ot r, which is the Acknowledgement. The authors would like to thank Eric Carlen, Eshed Ram and Igal Sason for reading the manuscrit and for their valuable comments. References [A-B-B-N] S. Artstein, K. M. Ball, F. Barthe, and A. Naor. Solution of Shannon s roblem on the monotonicity of entroy. J. Amer. Math. Soc., 7(4): (electronic), 004. [B] W. Beckner. Inequalities in Fourier analysis. Ann. of Math. () 0 (975), no., [B] W. Beckner. Geometric roof of Nash s inequality. Internat. Math. Res. Notices (998), [B-C] [B-L] S. G. Bobkov; G. P. Chistyakov. Entroy ower inequality for the Rényi entroy. IEEE Trans. Inform. Theory 6 (05), no., H. J. Brascam; E. H. Lieb. Best constants in Young s inequality, its converse, and its generalization to more than three functions. Advances in Math. 0 (976), no., [C] M. Costa. A new entroy ower inequality. IEEE Trans. Inf. Theory 3 (985), no. 6, [Co] T. A. Courtade. Strengthening the entroy ower inequality. Prerint, arxiv: [C-L] E. A. Carlen; M. Loss. Shar constant in Nash s inequality. Internat. Math. Res. Notices 7 (993), 3 5. [C-T] [D] T. M. Cover; J. A. Thomas. Elements of information theory. Wiley Series in Telecommunications. A Wiley-Interscience Publication. John Wiley & Sons, Inc., New York, 99. xxiv+54. A. Dembo. A simle roof of the concavity of the entroy ower with resect to the variance of additive normal noise. IEEE Trans. Inform. Theory 35 (989), [D-C-T] A. Dembo; T. M. Cover; J. A. Thomas. Information-theoretic inequalities. IEEE Trans. Inform. Theory, 37(6), 50 58, 99. [L] E. H. Lieb. Proof of an entroy conjecture of Wehrl. Comm. Math. Phys. 6 (978), no., [M-B] M. Madiman; A.R. Barron. Generalized entroy ower inequalities and monotonicity roerties of information. IEEE Trans. Inform. Theory 53 (007), no. 7, [M-M-X] M. Madiman; J. Melbourne; P. Xu. Forward and Reverse Entroy Power Inequalities in Convex Geometry. Prerint, arxiv: [R-S] [R] [S-T] E. Ram and I. Sason. On Rényi Entroy Power Inequalities. Prerint, arxiv: O. Rioul. Information theoretic roofs of entroy ower inequalities. IEEE Trans. Inform. Theory 57 (0), no., G. Savaré; G. Toscani. The concavity of Rényi entroy ower. IEEE Trans. Inform. Theory 60 (04), no. 5,

11 [Sh] C. E. Shannon. A mathematical theory of communication. Bell System Tech. J. 7, (948) , [St] [S-V] [T] [V-G] A. J. Stam. Some inequalities satisfied by the quantities of information of Fisher and Shannon. Information and Control, :0, 959. S. J. Szarek and D. Voiculescu. Shannon s entroy ower inequality via restricted Minkowski sums. Geometric asects of functional analysis, 745 (000), Lecture Notes in Math Sringer, Berlin. G. Toscani. An information-theoretic roof of Nash s inequality. (English summary) Atti Accad. Naz. Lincei Cl. Sci. Fis. Mat. Natur. Rend. Lincei (9) Mat. Al. 4 (03), no., S. Verdù; D. Guo. A simle roof of the entroy-ower inequality. IEEE Trans. Inform. Theory 5 (006), no. 5, [V] C. Villani. A short roof of the concavity of entroy ower. IEEE Trans. Inform. Theory 46 (000), no. 4, [W-M] L. Wang; M. Madiman. Beyond the entroy ower inequality, via rearrangements. IEEE Trans. Inform. Theory 60 (04), no. 9, [W] G. N. Watson. A Treatise on the Theory of Bessel Functions. Cambridge University Press, Cambridge, 944.