which element of ξ contains the outcome conveys all the available information.

Transcription

1 CHAPTER III MEASURE THEORETIC ENTROPY In this chapter we define the measure theoretic entropy or metric entropy of a measure preserving transformation. Equalities are to be understood modulo null sets; thus two σ algebras A, B or partitions ξ, η will be treated as equal if, for each A A A an atom of ξ there is a B B B an atom of η with the property that A B is a null set. Partitions and algebras A partition ξ of the probability space X, B, µ is a disjoint collection ξ = {B i } i I of measurable sets whose union is all of X. The partition is finite if I is finite. If X is the sample space of some experiment a throw of a dice for example, then a partition of X has associated to it an amount of information: when the experiment is performed, finding out which element B i of the partition the outcome lies in tells you something about the outcome. Consider some examples: 1 ξ = {X} here, knowing which element of ξ the outcome landed in conveys no information. 2 ξ = {{1}, {2}, {3}, {4}, {5}, {6}} has maximal information, since knowing which element of ξ contains the outcome conveys all the available information. 3 An intermediate case is given by the partition that tells us if the number thrown is odd or even, ξ = {{1, 3, 5}, {2, 4, 6}}. Associated to a finite partition ξ there is a finite σ algebra Aξ, comprising all unions of elements of ξ. Conversely, if A is a finite sub σ algebra of B, a finite partition ξa may be associate to it as follows. Let A = {A j } j J ; then the collection of non empty sets formed by intersections of the A j and their complements forms a partition of X. Notice that AξC = C for any finite algebra C, and ξaη = η for any finite partition η. Given partitions ξ = {B i } i I and η = {C j } j J of X, let ξ η = {B i C j i I, j J} denote their join. Analogously, A C denotes the intersection of all the σ algebras containing both A and C. A partition ξ is said to refine or to be finer than a partition η if every element of η is a union of elements of ξ; this is written η ξ. Now let T be a measure preserving transformation of X, B, µ, and let ξ = {B i } i I be a finite partition of X. For each n 1 notice that T n ξ = {T n ξ} i I is another partition of X. See Exercise Typeset by AMS-TEX

2 20 CHAPTER 3. MEASURE THEORETIC ENTROPY Entropy of partitions Recall that a partition ξ of the probability space X, B, µ may be viewed as an enumeration of the possible outcomes of an experiment. The information gained by performing the experiment and learning which element of ξ the outcome landed in will be denoted Hξ. In order that H be well defined invariant under isomorphism of measure spaces we require that H{A i } i I = HµA i i I. That is, Hξ is a function of the measures of the sets in the partition ξ but not of the sets themselves. Now consider two partitions, ξ = {A 1,..., A k } and η = {B 1,..., B l }, and let Hξ η denote the information gained in learning the outcome of ξ given the outcome of η. First, if we know that the outcome B j occurs, then outcome A i occurs with probability µa i B j /µb j. It follows that the additional information gained from ξ if we already know that B j occurs is HµA 1 B j /µb j,..., µa k B j /µb j. Since the outcome B j occurs with probability µb j, we then have Hξ η = l µb j H µa 1 B j /µb j,..., µa k B j /µb j. j=1 For brevity, write Hξ for HµA i i I, and let k = {p 1,..., p k R k p i 0, k p i = 1}, so that the domain of H is k N k. The following properties are then reasonable for the function H. Zero information is only gained from an experiment whose outcome is almost surely known, so 3.1 Hp 1,..., p k 0; Hp 1,..., p k = 0 if and only if some p i = 1. If the elements of the partition ξ are perturbed slightly, the information should not change too much, so for any k, 3.2 H k is a continuous function. The information depends on the numbers {p 1,..., p k } only, so for any k 3.3 H k is symmetric. The maximum possible amount of information carried by a member of k corresponds to having all outcomes equally likely, so for each k 3.4 max{h k } = H 1 k,..., 1 k. The information gained from two experiments, Hξ η is not of course the sum of the informations in particular, Hξ ξ = Hξ. Instead we have 3.5 Hξ η = Hξ + Hη ξ, where Hη ξ is defined above in terms of H. Finally, H k+1 is linked to H k by 3.6 Hp 1,..., p k, 0 = Hp 1,..., p k. These properties are sufficient to determine the function H up to a normalization.

3 CONDITIONAL ENTROPY 21 Theorem 3.1. If H : k k R has properties , then there is a λ > 0 for which k Hp 1,..., p k = λ p i log p i, where the function x x log x is extended to x = 0 by setting 0 log 0 = 0. See Exercises 3.4 and 3.5. We therefore define the entropy of the partition ξ or the algebra Aξ to be the quantity Hξ, where λ = 1. Conditional Entropy Recall that if ξ = {A 1,..., A k } and η = {B 1,..., B l } are finite partitions, then the conditional entropy of ξ given η is 3.6 Hξ η = HAξ Aη = l µb j j=1 k µa i B j µb j log µa i B j. µb j See Exercise 3.7, 3.8 and 3.9. Notice that 3.6 also defines the conditional entropy of a finite σ algebra A given another finite σ algebra B. Statements about partitions often only make sense for finite partitions sometimes this can be extended to countable partitions, but we will see below that conditional entropy can be defined on all σ algebras. Given two finite partitions ξ, η, let ρξ, η = Hξ η + Hη ξ. Theorem 3.2. The function ρ defines a metric on the space of all finite partitions of X. See Exercise Now let A be a finite sub σ algebra of B, and let F be any sub σ algebra of B. The following argument shows how to define the conditional entropy HA F. Recall that the conditional expectation operator E F : L 1 X, B, m L 1 X, F, m is defined by the property that Ef F is an F measurable function on X with the property that C Ef Fdm = C fdm. Define then the conditional entropy of the finite σ algebra A with respect to the σ algebra F to be 3.7. HA F = k Eχ Ai F log Eχ Ai Fdm If {A n } is a family of sub σ algebras, then denote by n=1 A n the intersection of all sub σ algebras of B that contain all the A n s. Theorem 3.3. Let X, B, µ be a probability space, and let {A n } be an increasing sequence of sub σ algebras of B. Then for a finite sub σ algebra F of B, HF A n = lim HF A n. n n=1 See Exercise 3.12, 3.13 and 3.14.

4 22 CHAPTER 3. MEASURE THEORETIC ENTROPY Theorem 3.4. Let A be a finite sub σ algebra of B, and let F be any sub σalgebra. Then HA F = 0 if and only if A F, and HA F = HA if and only if A and F are independent. Entropy of a measure preserving transformation Let A be a finite sub σ algebra of B, and let T be a measure preserving transformation of X, B, µ. Lemma 3.5. The quantity 1 n 1 n H decreases in n. Proof. We claim that Pn H n 1 P1 is clear, so assume Pp. Then n 1 = HA + H A j=1 p p H = H A j. p = H + H A = H p 1 = HA + + H p H A j=1 A p p j by the assumption Pp. So Pn holds for all n by induction. This shows that the following definition makes sense. Definition 3.6. If A is a finite sub σ algebra of B, then the entropy of T with respect to A is the quantity n 1 1 ht, A = ht, ξa = lim n n H. The entropy of T is then defined to be ht = sup ht, A, A where the supremum is taken over all finite sub σ algebras A of X. Notice that Lemma 3.5 shows that the limit exists and shows that 0 ht, A HA <. The entropy itself cannot be bounded: ht = is possible.

5 ENTROPY OF A MEASURE PRESERVING TRANSFORMATION 23 Theorem 3.7. If S is a factor of T then hs ht. The proof is clear. Corollary 3.8. Entropy is an invariant of weak isomorphism. Notes. The exposition in this chapter essentially follows Chapter 4 of Walters book, [35]. The entropy of a measure preserving transformation was introduced by Kolmogorv [15] and Sinai [33]. Exercises 3.1 Prove that ξa C = ξa ξc and Aξ η = Aξ Aη. 3.2 Prove that η ξ if and only if Aη Aξ, and B C if and only if ξb ξc. 3.3 Let ξ be a finite partition, and let A be a finite σ algebra. Prove that ξt n A = T n ξa; AT n ξ = T n Aξ; T n ξ η = T n ξ T n η; ξ η = T n ξ T n η; A C = T n A T n C. 3.4 Let φx = x log x extended to the domain [0, by setting φ0 = 0. k Prove that φ a ix i k a iφx i when the a i > 0, a i = 1, and that equality holds only when all the x i are equal. That is, φ is strictly convex. 3.5 Show that any function of the form stated in Theorem 3.1 does satisfy properties Prove Theorem 3.1 the details are in the book A. Khinchine, Mathematical Foundations of Information Theory, Dover Let ξ, η and ζ be finite partitions of X. Prove the following: a. Hξ η ζ = Hξ ζ + Hη ξ ζ; b. Hξ η = Hξ + Hη ξ; c. ξ η = Hξ ζ Hη ζ; d. η ζ = Hξ η Hξ ζ; e. Hξ Hξ η; f. Hξ η ζ Hξ ζ + Hη ζ; g. Hξ η Hξ + Hη; h. HT 1 ξ T 1 η = Hξ η. 3.8 Prove that Hξ η = 0 if and only if ξ η. 3.9 Prove that Hξ η = Hξ if and only if ξ and η are independent partitions that is, for any sets A ξ and B η, µa B = µaµb Prove Theorem 3.2 use 3.8 and Show that 3.7 coincides with 3.6 when F is a finite σ algebra Let A be a finite σ algebra. Prove that HA F is finite for any F In the notation of Theorem 3.3, prove, for any f L 2, that Ef A n converges to Ef n=1 A n in L 2. Deduce Theorem Assume that X, B, µ has a countable basis, and let D be a sub σ algebra of B. By choosing a sequence of finite σ algebras increasing to D, prove the following A and C are finite sub σ algebras of B. a. HA C D = HA D + HC A D. b. HA C = HA + HC A. c. A C = HA D HC D. d. C D = HA C HA D.

6 24 CHAPTER 3. MEASURE THEORETIC ENTROPY e. HA C D HA D + HC D. f. HT 1 A T 1 D = HA D Prove Theorem 3.4.