Fisher information and Stam inequality on a finite group

Transcription

1 Fisher information and Stam inequality on a finite group Paolo Gibilisco and Tommaso Isola February, 2008 Abstract We prove a discrete version of Stam inequality for random variables taking values on a finite group Mathematics Subject Classification: 94A7, 62B0 Key words and phrases. Fisher score, Fisher information, Stam inequality. Introduction The Fisher information I X of a real random variable (with strictly positive differentiable density function f) is defined as f (x) 2 I X := dx. (.) f(x) If X, Y are independent random variables such that I X, I Y <, Stam was able to prove the inequality +, (.2) where equality holds iff X, Y are Gaussian (see [7, ]). This result has been very useful in a manifold of different areas: analysis, probability, statistics, information theory, statistical mechanics and so on (see [2, 4, 0, 5, 8]). Therefore it is not surprising that different proofs and generalizations appear in the recent literature on the subject (see for example [20, 4]). A free analogue of Fisher information has been introduced in free probability. Also in this case one can prove a Stam-like inequality. The equality case characterizes the Wigner distribution that, in many respects, is the free analogue of the Gaussian distribution (see [9]). Discrete versions of Fisher information and the Stam inequality are well-known. On the integers Z, equality characterizes the Poisson distribution, while on the cyclic group Z n, equality occurs for the uniform distribution (see [9, 6,, 2, 3, 5, 6]). It has been observed that there are group-theoretical features in the proof of Stam inequality (see []). Nevertheless, up to now, inequality (.2) has been proved only on specific groups like R, Z and Z n. In this paper we consider the family of all finite groups and we show that, mutatis mutandis one can introduce Fisher information and prove Stam inequality on an arbitrary finite group. 2 Preliminaries We recall the formulation of Stam inequality in the known cases, where it has already been proved. Dipartimento SEFEMEQ, Facoltà di Economia, Università di Roma Tor Vergata, Via Columbia 2, 0033 Rome, Italy. gibilisco@volterra.uniroma2.it URL: Dipartimento di Matematica, Università di Roma Tor Vergata, Via della Ricerca Scientifica, 0033 Rome, Italy. isola@mat.uniroma2.it URL: isola

2 2 2. Stam inequality on R and R n Let f : R R be a differentiable, strictly positive density. One may define the Fisher f-score J f : R R by J f := f f. Let (Ω, F, p) be a probability space. In general, if X : (Ω, F, p) R is a random variable with density f we write J X = J f and define the Fisher information as namely, I X := Var f (J f ) = E f (J 2 f ); I X = (f (x)/f(x)) 2 f(x)dx. (2.) R Theorem 2.. [7, ] If X, Y : (Ω, F, p) R are independent random variables such that I X, I Y <, then +, (2.2) with equality if and only if X, Y are Gaussian (with the same covariance). The same result holds for random vectors. Let f : R n R be a differentiable, strictly positive density. We use the notation f xi = f x i. One may define the Fisher f-score (in the direction x i ) J xi f : R n R by J xi f := f x i f. Let (Ω, F, p) be a probability space. In general, if X : (Ω, F, p) R n is a random vector with density f we write J xi X = J xi f and define the Fisher information as (see p.20 in [2] and p. 838 in [3]) I X := n i= E f [(J xi f )2 ]. Note that in this case I X is the trace of the Fisher information matrix [ ] log(f) log(f) E f i, j =,..., n x i x j Theorem 2.2. If X, Y : (Ω, F, p) R n are independent random vectors such that I X, I Y <, then +, (2.3) with equality if and only if X, Y are Gaussian (with the same covariance matrix). 2.2 Stam inequality on Z Let f : Z R be a (discrete) density. We say that f belongs to the class RSP (right side positivity) if f(k) > 0 = f(k + ) > 0. If f RSP, then we may define the Fisher f-score by { f(k) f(k ) f(k) f(k) > 0, J f (k) = 0 f(k) = 0. If X : (Ω, F, p) Z is a random variable with (discrete) density f RSP we write J X = J f and define the Fisher information as I X := Var f (J f ) = E f (J 2 f ).

3 3 Theorem 2.3. [6, ] If X, Y : (Ω, F, p) Z are independent random variables with densities in RSP and such that I X, I Y <, then +, (2.4) with equality if and only if X, Y have (possibly shifted) Poisson distribution. 2.3 Stam inequality on Z n Let f : Z n R be a (discrete) density that is strictly positive. We define the Fisher f-score by J f (k) = f(k) f(k ) f(k) If X : (Ω, F, p) Z n is a random variable with positive (discrete) density f we write J X = J f and define the Fisher information as I X := Var f (J f ) = E f (J 2 f ). Theorem 2.4. [6] If X, Y : (Ω, F, p) Z are independent random variables with positive densities then with equality if and only if X and Y have uniform distribution. +, (2.5) In the following section we generalize this result to an arbitrary finite group. 3 Stam inequality on a finite group Let G be a finite group. Introduce the class P of strictly positive densities, that is { } P := f : G R f(j) =, f(k) > 0 k G. j G We assume, from now on, that all densities belong to P. Let f P, g G. In analogy with the previous definitions, we may introduce J g f f-score in the direction g, by J g f (k) := f(k) f(g k). f(k) Then, J g f is an f-centered random variable : G R, the E f [J g f ] := J g f (k)f(k) = 0. (3.) k G If X : (Ω, F, p) G is a random variable with density f X (k) := p(x = k), and if f X P, define the score J g X := J g f. Lemma 3.. Let g G, X, Y : (Ω, F, p) G be two independent random variables with densities f X, f Y P and let Z := XY. Then, Proof. Let f Z be the density of Z; namely, J g Z (Z) = E p[j g X (X) Z] = E p[j g Y (Y ) Z]. f Z (k) = j G f X (j)f Y (j k), k G,

4 4 so that f Z P. Then, f Z (k) f Z (g k) = j G f X (j)f Y (j k) j G f X (j)f Y (j g k) = j G( fx (j) f X (g j) ) f Y (j k). Therefore, for k G, J g Z (k) = f Z(k) f Z (g k) f Z (k) = j G f X (j)f Y (j k) f X (j) f X (g j) f Z (k) f X (j) = J g X (j)p(x = j Z = k) j G = E fx [J g X Z = k] = E p [J g X (X) Z = k]. Similarly, by symmetry of the convolution formula one obtains proving the claim. J g Z (k) = E p[j g Y (Y ) Z = k], k G, Lemma 3.2. Let g G, X, Y : (Ω, F, p) G be two independent random variables with densities f X, f Y P and let Z := XY. Then, for any a, b R, we have (a + b) 2 E p [J g Z (Z)2 ] a 2 E p [J g X (X)2 ] + b 2 E p [J g Y (Y )2 ]. (3.2) Moreover, if equality holds in (3.2) then J g X, J g Y Proof. By Lemma 3. are constant on G. E p [aj g X (X) + bj g Y (Y ) Z] = ae p[j g X (X) Z] + be p[j g Y (Y ) Z] = (a + b)j g Z (Z). (3.3) Hence, by applying Jensen s inequality it holds and thus E p [(aj g X (X) + bj g Y (Y ))2 ] = E p [E p [(aj g X (X) + bj g Y (Y ))2 Z]] E p [E p [aj g X (X) + bj g Y (Y ) Z]2 ] (3.4) = E p [(a + b) 2 J g Z (Z)2 ] = (a + b) 2 E p [J g Z (Z)2 ], (a + b) 2 E p [J g Z (Z)2 ] E p [(aj g X (X) + bj g Y (Y ))2 ] = a 2 E p [J g X (X)2 ] + 2abE p [J g X (X)J g Y (Y )] + b2 E p [J g Y (Y )2 ] = a 2 E p [J g X (X)2 ] + b 2 E p [J g Y (Y )2 ], where the last equality follows from independence and due to (3.). We now consider the case of equality in (3.2). Set c = a + b, and let us prove that equality in (3.2) holds iff aj g X (X) + bj g Y (Y ) = cj g Z (XY ). (3.5) Indeed, let H := aj g X (X) + bj g Y (Y ); then equality in (3.4) occurs if and only if E p [H 2 Z] = (E p [H Z]) 2,

5 5 i.e. Therefore, H = E p [H Z], so that E p [(H E p [H Z]) 2 Z] = 0. E p [aj g X (X) + bj g Y (Y ) Z] = aj g X (X) + bj g Y (Y ) = cj g Z (Z), due to (3.3). Conversely, if (3.5) holds, then by applying the squared power and the expectation operator we obtain equality in (3.2). Using (3.5), we now prove the last statement of the Lemma. Let us choose a set of generators of the group G, i.e. Γ := {γ,..., γ n } G such that the subgroup generated by them is the whole G. Let us also denote by Γ := {γ,..., γ n }. Let now x, y G; because of independence, for any γ Γ Γ, { p(x = xγ, Y = y) = p(x = xγ) p(y = y) > 0 p(x = x, Y = γy) = p(x = x) p(y = γy) > 0. Thus, it makes sense to write equality (3.5) on A := {X = xγ} {Y = y} and on B := {X = x} {Y = γy}, so that { aj g X (xγ) + bj g Y (y) = cj g Z (xγy) aj g X (x) + bj g Y (γy) = cj g Z (xγy). Subtracting these relations one has a[j g X (xγ) J g X (x)] = b[j g Y (γy) J g Y (y)], x, y G. Therefore, for any γ Γ Γ, there is a(γ) R such that J g X (xγ) J g X (x) = a(γ), for any x G. Thus, if n is the order of γ in G, i.e. γ n = e, the identity of G, then J g X (x) = J g X (xγn ) = J g X (x) + na(γ), for any x G, which implies a(γ) = 0. Therefore J g X (xγ) = J g X (x), x G, γ Γ Γ. (3.6) Since any k G, k e, can be written as a product of elements in Γ Γ, i.e. k = γ i γ i2 γ il, for γ ij Γ Γ, we can use (3.6) iteratively, and obtain J g X (xk) = J g X (xγ i γ i2 γ il ) = J g X (xγ i γ i2 γ il ) =... = J g X (x), x G. In particular, for x = e we obtain J g X (k) = J g X (e), for any k G, i.e. J g X for J g Y is analogous. is constant on G. The proof Let us now fix a set of generators of the group G, i.e. Γ := {γ,..., γ n } G such that the subgroup generated by them is the whole G. If X : (Ω, F, p) G is a random variable with density f X P, define the Fisher information I X := E f [(J γ f )2 ] = ( f(k) f(γ k) ) 2f(k). f(k) γ Γ γ Γ k G Note that, due to the finiteness of G, the condition I X < always holds. However, we cannot ensure in general that I X 0. In fact, it is easy to characterize this degenerate case. Lemma 3.3. The following conditions are equivalent () X has uniform distribution; (2) J γ X = 0, for any γ Γ; (3) I X = 0; (4) J γ X is constant, for any γ Γ.

6 6 Proof. The implications () = (2) = (4) are immediately proved. The equivalence of (2) and (3) is also easy to show. Therefore, it is enough to prove that (4) implies (). So, for any γ Γ there is a(γ) R such that J γ X (x) = a(γ), for any x G, i.e., with f f X, f(x) f(γ x) f(x) = a(γ), for any x G, which is equivalent to ( a(γ))f(x) = f(γ x), x G. (3.7) Thus, if n is the order of γ in G, i.e.γ n = e, the identity of G, then ( a(γ)) n f(x) = ( a(γ)) n f(γ x) =... = f(γ n x) = f(x), for any x G, which implies a(γ) = 0. Therefore, From this it also follows f(γ x) = f(x), x G, γ Γ. (3.8) f(γx) = f(x), x G, γ Γ. (3.9) Since any k G, k e, can be written as a product of elements in Γ Γ, i.e. k = γ i γ i2 γ il, for γ ij Γ Γ, we can use (3.8) and (3.9) iteratively, and obtain f(kx) = f(γ i γ i2 γ il x) = f(γ i2 γ il x) =... = f(x), x G. In particular, for x = e we obtain f(k) = f(e), for any k G, i.e. f is constant on G, that is, X is uniform. This concludes the proof. Let us recall also the following result that is immediate by using the convolution formula. Proposition 3.4. If X, Y : (Ω, F, p) G are independent random variables and X is uniform then also Z = XY is uniform. Proposition 3.5. Let X, Y : (Ω, F, p) be independent random variables such that their densities belong to P. If X or Y has uniform distribution, then = +, I XY I X I Y in the sense that both sides of equality are equal to infinity. Proof. Let Z = XY. If X is uniform, then Z is uniform by Proposition 3.4 and we are done by Lemma 3.3. Because of the above proposition, it remains to consider random variables with strictly positive Fisher information. We are ready to prove the main result. Theorem 3.6. Let X, Y : (Ω, F, p) G be two independent random variables such that I X, I Y > 0. Then > +. (3.0) I XY I X I Y Proof. Define Z := XY, and let a, b R. Then, for any γ Γ we have from (3.2) Summing up over γ Γ, we obtain (a + b) 2 I Z = (a + b) 2 γ Γ (a + b) 2 E p [J γ Z (Z)2 ] a 2 E p [J γ X (X)2 ] + b 2 E p [J γ Y (Y )2 ]. (3.) E p [J γ Z (Z)2 ] a 2 γ Γ E p [J γ X (X)2 ] + b 2 γ Γ E p [J γ Y (Y )2 ] = a 2 I X + b 2 I Y.

7 7 Now, take a := /I X and b := /I Y ; then we obtain ( + ) 2 I Z +. I X I Y I X I Y It remains to be proved that the equality sign cannot hold in (3.0). To this purpose, define c := a+b, where, again, a = /I X and b = /I Y ; then equality holds if and only if c 2 I Z = a 2 I X + b 2 I Y. This implies equality in (3.), for any γ Γ. From Lemma 3.2 it follows that J γ X, J γ Y are constant on G, for any γ Γ. But then, from Lemma 3.3, I X = I Y = 0, which is absurd. So equality cannot hold in (3.0). References [] Blachman N. M., The convolution inequality for entropy powers. IEEE Trans. Inform. theory, :267 27, 965. [2] Carlen E., Superadditivity of Fisher s information and logarithmic Sobolev inequalities. J. Funct. Anal., 0(): 94-2, 99. [3] Costa M.H.M. and Cover T., On the similarity of the entropy power inequality and the Brunn- Minkowski inequality IEEE Trans. Inform. Theory, vol. IT-30: , 984. [4] Dembo A., Cover T. and Thomas J., Information theoretic inequalities, IEEE Trans. Inform. Theory, 37(6): 50 58, 99. [5] Gardner, R.J., The Brunn-Minkowski inequality, Bull. Amer. Math. Soc., 39(3): , [6] Gibilisco P., Imparato D., Isola T., Stam inequality on Z n, Statis. Probab. Lett. (to appear), [7] Harremoes P., Binomial and Poisson distribution as maximum entropy distributions. IEEE Trans. Inform. Theory, 47(5): , 200. [8] Johnson O. T., Log-concavity and the maximum entropy property of the Poisson distribution. Stoch. Proc. Appl., 7(6): , [9] Johnstone I.M. and MacGibbon B., Une mesure d information caractérisant la loi de Poisson. In Séminaire de Probabilités, XXI, vol. 247 of Lecture Notes in Math., , Springer, Berlin, 987. [0] Kagan A., Landsman Z., Statistical meaning of Carlen s superadditivity of the Fisher information. Statis. Probab. Lett., 32:75 79, 997. [] Kagan A., A discrete version of Stam inequality and a characterization of the Poisson distribution. J. Statist. Plann. Inference, 92(-2):7 2, 200. [2] Kagan A., Letter to the editor: A discrete version of Stam inequality and a characterization of the Poisson distribution. [J. Statist. Plann. Inference, 92(-2):7 2, 200]. J. Statist. Plann. Inference, 99():, 200. [3] Kontoyiannis I., Harremoës P. and Johnson O., Entropy and the law of small numbers. IEEE Trans. Inform. Theory, 5(2): , [4] Madiman M., and Barron R. A., Generalized entropy power inequalities and monotonicity properties of information. IEEE Trans. Inform. Theory, 53(7): , [5] Madiman M., Johnson O. and Kontoyiannis I., Fisher information, compound Poisson approximation and the Poisson channel. Proc. IEEE Intl. Symp. Inform. Theory, Nice, France, 2007.

8 8 [6] Papathanasiou V., Some characteristic properties of the Fisher information matrix via Cacoullotype inequalities. J. Multivariate Anal., 44(2): , 993. [7] Stam A.J., Some inequalities satisfied by the quantities of information of Fisher and Shannon. Information and Control, 2:0-2, 959. [8] Villani C., Cercignani s conjecture is sometimes true and always almost true. Comm. Math. Phys., 234(3): , [9] Voiculescu D., The analogues of entropy and of Fisher s information measure in free probability theory. V. Noncommutative Hilbert transforms. Invent. Math., 32(): , 998. [20] Zamir R., A proof of the Fisher information inequality via a data processing argument. IEEE Trans. Inform. Theory, 44(3): , 998.