Entropy and Ergodic Theory Notes 22: The Kolmogorov Sinai entropy of a measure-preserving system

Size: px

Start display at page:

Download "Entropy and Ergodic Theory Notes 22: The Kolmogorov Sinai entropy of a measure-preserving system"

Andrea Parrish
6 years ago
Views:

1 Entropy and Ergodic Theory Notes 22: The Kolmogorov Sinai entropy of a measure-preserving system 1 Joinings and channels 1.1 Joinings Definition 1. If px, µ, T q and py, ν, Sq are MPSs, then a joining of them is a coupling λ P Probpµ, νq which is invariant under T ˆ S. The product measure µˆν is always a joining, but some pairs of systems have many others. Example. If π : px, µ, T q ÝÑ py, ν, Sq is a factor map, it has an associated graphical joining: grpµ, πqpuq : µtx : px, πpxqq P Uu for measurable U Ď X ˆ Y. It is the unique joining supported on the pt ˆ Sq-invariant subset graphpπq tpx, πpxqq : x P Xu. Example. A little more generally, given a diagram pz, θ, Rq ϕ π px, µ, T q py, ν, Sq, the associated joint distribution is the image measure: pϕ, πq θpuq θtz : pϕpzq, πpzqq P Uu. A graphical joining grpµ, πq is the joint distribution of id X and π. Definition 1 and the second example above have obvious generalizations to collections of more than two systems. 1

2 1.2 Channels Joinings show up all over ergodic theory, but in this course we mostly use them for pairs of sources. In this setting, we start with a simple but powerful description of general joinings provided by the disintegration theorem from Lecture 20. Definition 2. Let A and B be finite alphabets and let S A and S B be the respective shifts on A Z and B Z. A stationary channel from A Z to B Z is a kernel θ from A Z to B Z such that θ SA pxqps B V q θ x pv q for all x P A Z and measurable V Ď B Z. The next definition is also a generalization of an idea from the information theory part of the course. Definition 3. If px, B, µq is a probability space and θ is a stochastic kernel from px, Bq to py, C q, then the input-output measure or hookup of µ and θ is the measure on px ˆ Y, B b C q defined by pµ θqpuq : θ x ty P Y : px, yq P Uu µpdxq. Example. 1. A constant probability kernel is one for which θ x ν for some fixed probability measure ν on py, C q. In this case µ θ µ ˆ ν. 2. Any stationary code π : A Z ÝÑ B Z defines a deterministic channel according to θ x δ πpxq. If µ is a shift-invariant measure on A Z, then its hookup to this channel equals the graphical joining grpµ, πq. 3. If ra, θ 0, Bs is a DMC, then its infinite extension is the stationary channel from A Z to B Z defined by θ x : ą npz θ 0 p x n q. A stationary channel of this form is sometimes also called a DMC. 2

3 Stationary channels are the basic objects in various generalizations of the channel coding theorem beyond the memoryless setting. Here we use them as a way of describing joinings between sources. Theorem 4. Let ra Z, µs and rb Z, νs be sources and let λ be a joining of them. Then there is a stationary channel θ from A Z to B Z such that λ µ θ, and θ is unique up to agreement µ-a.e. Proof. On the probability space pa Z ˆ B Z, λq, let F be the σ-subalgebra of measurable sets which are lifted from A Z : F tu ˆ B Z : U Ď A Z measurableu. Applying the disintegration theorem to λ and F, we obtain a kernel px, yq ÞÑ λ px,yq from A Z ˆ B Z to itself such that i) for each measurable U Ď A Z ˆ B Z, the map px, yq ÞÑ λ px,yq puq is F - measurable, and ii) for each bounded measurable f : A Z ˆ B Z ÝÑ R, we have E λ pf F qpx, yq f dλ px,yq for λ-a.e. px, yq. Any function on A Z ˆ B Z which is F -measurable must depend only on the coordinate in A Z. So property (i) implies that λ px,yq is really just a function of x. We henceforth write it as λ x instead. Moreover, each of the maps x ÞÑ λ x puq is F -measurable as a function of px, yq if and only if it is measurable as a function of x alone, by the definition of F. Therefore we have identified λ with a kernel from A Z to A Z ˆ B Z. On the other hand, consider a bounded measurable function f on A Z, and define fpx, p yq : fpxq. Then f p is F -measurable, so property (ii) implies that fpxq fpx, p yq pfpx 1, y 1 q λ x pdx 1, dy 1 q fpx 1 q λ x pdx 1, dy 1 q for µ-a.e. x. By applying this to all functions f from a countable dense subset of CpA Z q, it follows that µ-a.e. x has the property that fpx 1 q λ x pdx 1, dy 1 q P CpA Z q. 3

4 This is possible only if λ x is supported on txu ˆ A Z. So for each x P X 0 we can write λ x δ x ˆ θ x for some θ x P ProbpB Z q. The map x ÞÑ θ x is now a kernel from A Z to B Z such that λ µ θ. This kernel is essentially unique because of the corresponding property of λ. Finally, observe from the shift-invariance of µ and λ that δ x ˆ S 1 B pθ S A pxqq µpdxq ps 1 A ˆ S 1 B q pδ x ˆ θ x q µpdxq ps 1 A ˆ S 1 B q λ λ Therefore, by essential uniqueness, we must have δ x ˆ θ x µpdxq. S 1 B pθ S A pxqq θ x µ-a.s. 2 The Kolmogorov Sinai theorem Theorem 5 (Monotonicity under factor maps). If there exists a factor map ra Z, µs ÝÑ rb Z, νs then hpνq ď hpµq. The proof first treats the case of a sliding block code, and then deduces the general case by an approximation. The first step is quite easy, but we need some preparations for the second step. The next lemma provides the approximations we need. Lemma 6 (Approximation of stationary codes by sliding-block). If ra Z, µs is a source, π : A Z ÝÑ B Z is a stationary code, and ε ą 0, then there is a sliding block code ϕ : A Z ÝÑ B Z such that µtx : π 0 pxq ϕ 0 pxqu ă ε. Proof. This is just simple measure theory: any finite valued measurable function on A Z can be approximated in measure by local functions. In order to use Lemma 6, we need the ability to control entropy rates under the kind of approximation that it provides. This is done using a version of Fano s inequality. 4

5 Lemma 7 (Fano s inequality for general sources). Let ra Z, µs and ra Z, νs be two sources with the same alphabet, and let λ be a joining. Let p : λtpx, yq : x 0 y 0 u. Then hpµq hpνq ď Hpp, 1 pq ` p logp A 1q. Proof. For each m P Z, let pα m, β m q be the m th coordinate projection from A Z ˆ A Z to A ˆ A. According to the measure λ on A Z ˆ A Z, the process pα n q n has joint distribution µ, and the process pβ n q n has joint distribution ν. So for each n ě 1 we have Hpµ n q Hpα 1,..., α n q ď Hpβ 1,..., β n q ` Hpα 1,..., α n β 1,..., β n q, by the monotonicity of entropy and the chain rule. By subadditivity of conditional entropy, the right-hand side above is at most Hpβ 1,..., β n q ` nÿ Hpα i β i q. By the original Fano inequality, each term in the sum here is at most i 1 Hpp, 1 pq ` p logp A 1q. Dividing by n and letting n ÝÑ 8, we obtain hpµq ď hpνq ` Hpp, 1 pq ` p logp A 1q. The corresponding inequality with µ and ν switched follows by symmetry. Proof of Theorem 5. Let π pπ n q n : ra Z, µs ÝÑ rb Z, νs be the factor map, and let α pα n q n and β pβ n q n be stationary stochastic processes with distributions µ and ν respectively. Our assertion about π implies that π α law β. Step 1. Suppose first that π 0 is local. This means that π 0 pxq π 1 0px m, x m`1,..., x m q for some m P N and some π 1 0 : A 2m`1 ÝÑ B. By stationarity, this turns into π n pxq π 1 0px n m, x n m`1,..., x n`m P Z, 5

6 and therefore pβ 1,..., β n q law pπ 1 pαq, π 2 pαq,..., π n pαqq `π 1 0pα 1 m,..., α 1`m q, π 1 0pα 2 m,..., α 2`m q,..., π 1 0pα n m,..., α n`m q. The right-hand side here is determined by pα 1 m,..., α n`m q, so stationarity and the monotonicity of entropy imply that Hpν n q Hpβ 1,..., β n q Hpπ 1 pαq, π 2 pαq,..., π n pαqq ď Hpα 1 m,..., α n`m q Hpµ n`2m q. Dividing by n and letting n ÝÑ 8, this gives hpνq ď hpµq, since m is fixed. Step 2. Now consider a general factor map π. Let ε ą 0. By Lemma 6 there is a sliding block code ϕ pϕ n q n such that By Step 1, we know that On the other hand, the joint distribution is a joining of ν and ϕ µ which satisfies Therefore Lemma 7 gives µtπ 0 ϕ 0 u ă ε. hpϕ µq ď hpµq. λ pπ, ϕq µ P ProbpA Z ˆ A Z q λtpy, y 1 q : y 0 y 1 0u µtπ 0 ϕ 0 u ă ε. hpνq ď hpϕ µq ` Hpε, 1 εq ` ε logp A 1q ď hpµq ` Hpε, 1 εq ` ε logp A 1q. The second and third terms on the right can be made arbitrarily small by choosing ε sufficiently small, so this completes the proof. Corollary 8 (Kolmogorov Sinai theorem). Isomorphic shift-systems have the same entropy rate. Finally, let us extend the definition of entropy rate to arbitrary MPSs. First, recall that if px, µ, T q is a MPS, A is a finite alphabet, and ϕ 0 : X ÝÑ A is measurable, then we obtain from these a stationary A-valued process ϕ pϕ 0 T n q npz. 6

7 Definition 9. The entropy rate of px, µ, T q and ϕ is hpµ, T, ϕq : hpϕ µq. The Kolmogorov Sinai or KS entropy of px, µ, T q is! ) hpµ, T q : sup hpµ, T, ϕq : A ă 8 and ϕ 0 : X ÝÑ A measurable. We could have made this definition much sooner. It is easily checked to be invariant under isomorphism. But the power of this definition is much clearer now that we have Theorem 5. If ϕ and ψ are two finite-valued processes on the same underlying MPS px, µ, T q, and if ϕ is an isomorphism of MPSs, then that theorem gives hpµ, T, ψq ď hpµ, T, ϕq. Therefore we may compute hpµ, T q as hpϕ µq whenever we can find a process ϕ on px, µ, T q which is an isomorphism to a shift-system. In that case the time-zero observable ϕ 0 is said to be generating. Because of this, our previous calculations of entropy rates for processes all give the KS entropies of the underlying systems: A Bernoulli shift ra Z, pˆz s has KS entropy equal to Hppq. If µ P ProbpA Z q is the joint distribution of a stationary Markov chain with transition kernel θ and equilibrium distribution π, then its KS entropy is ÿ πpaqh`pθp aq. apa A circle rotation has entropy zero. Indeed, for an irrational circle rotation, we previously saw that any non-trivial arc of T defines a generating observable of entropy rate zero. The case of a rational rotation is even simpler, since then any process it generates is periodic. The KS entropy is one of the most important invariants of a MPS. Its first major application, and Kolmogorov and Sinai s original motivation, was the following. Corollary 10. Two Bernoulli shifts ra Z, pˆz s and rb Z, qˆz s can be isomorphic only if Hppq Hpqq. Before the introduction of entropy, it was open whether in fact all Bernoulli shifts are isomorphic. It turns out that the condition Hppq Hpqq is also sufficient for isomorphism. This much harder result is Ornstein s theorem. We return to it later in the course. 7

8 3 Notes and remarks Joinings were introduced into ergodic theory in Furstenberg s classic paper [Fur67]. It remains highly influential, and is well worth reading. The survey [Šuj83] is a good place to start learning more about the study of general sources in information theory. References [Fur67] Harry Furstenberg. Disjointness in ergodic theory, minimal sets, and a problem in Diophantine approximation. Math. Systems Theory, 1:1 49, [Šuj83] Štefan Šujan. Ergodic theory, entropy, and coding problems of information theory. Kybernetika (Prague) Suppl., 19(1-4):67, TIM AUSTIN tim@math.ucla.edu URL: math.ucla.edu/ tim 8

Entropy and Ergodic Theory Lecture 28: Sinai s and Ornstein s theorems, II

Entropy and Ergodic Theory Lecture 28: Sinai s and Ornstein s theorems, II 1 Proof of the update proposition Consider again an ergodic and atomless source ra Z, µs and a Bernoulli source rb Z, pˆz s such