Entropy Rates and Asymptotic Equipartition

Transcription

1 Chapter 29 Etropy Rates ad Asymptotic Equipartitio Sectio 29. itroduces the etropy rate the asymptotic etropy per time-step of a stochastic process ad shows that it is well-defied; ad similarly for iformatio, divergece, etc. rates. Sectio 29.2 proves the Shao-MacMilla-Breima theorem, a.k.a. the asymptotic equipartitio property, a.k.a. the etropy ergodic theorem: asymptotically, almost all sample paths of a statioary ergodic process have the same log-probability per time-step, amely the etropy rate. This leads to the idea of typical sequeces, i Sectio Sectio 29.3 discusses some aspects of asymptotic likelihood, usig the asymptotic equipartitio property, ad allied results for the divergece rate. 29. Iformatio-Theoretic Rates Defiitio 376 (Etropy Rate) The etropy rate of a radom sequece X is h(x) lim H ρ [X ] (29.) whe the limit exists. Defiitio 377 (Limitig Coditioal Etropy) The limitig coditioal etropy of a radom sequece X is h (X) lim H ρ [X X ] (29.2) whe the limit exists. 97

2 CHAPTER 29. RATES AND EQUIPARTITION 98 Lemma 378 For a statioary sequece, H ρ [X X ] is o-icreasig i. Moreover, its limit exists if X takes values i a discrete space. Proof: Because coditioig reduces etropy, H ρ [X + X ] H[X + X2 ]. By statioarity, H ρ [X + X2 ] = H ρ [X X ]. If X takes discrete values, the coditioal etropy is o-egative, ad a o-icreasig sequece of oegative real umbers always has a limit. Remark: Discrete values are a sufficiet coditio for the existece of the limit, ot a ecessary oe. We ow eed a atural-lookig, but slightly techical, result from real aalysis. Theorem 379 (Cesàro) For ay sequece of real umbers a a, the sequece b = i= a also coverges to a. Proof: For every ɛ > 0, there is a N(ɛ) such that a a < ɛ wheever > N(ɛ). Now take b ad break it up ito two parts, oe summig the terms below N(ɛ), ad the other the terms above. lim b a = lim a i a (29.3) i= lim a i a (29.4) i= N(ɛ) lim a i a + ( N(ɛ))ɛ (29.5) i= N(ɛ) lim a i a + ɛ (29.6) i= N(ɛ) = ɛ + lim Sice ɛ was arbitrary, lim b = a. i= a i a (29.7) = ɛ (29.8) Theorem 380 (Etropy Rate) For a statioary sequece, if the limitig coditioal etropy exists, the it is equal to the etropy rate, h(x) = h (X). Proof: Start with the chai rule to break the joit etropy ito a sum of coditioal etropies, use Lemma 378 to idetify their limit as h ]prime (X), ad

3 CHAPTER 29. RATES AND EQUIPARTITION 99 the use Cesàro s theorem: h(x) = lim H ρ[x ] (29.9) = lim H ρ [X i X i ] (29.0) i= = h (X) (29.) as required. Because h(x) = h (X) for statioary processes (whe both limits exist), it is ot ucommo to fid what I ve called the limitig coditioal etropy referred to as the etropy rate. Lemma 38 For a statioary sequece h(x) H[X ], with equality iff the sequece is IID. Proof: Coditioig reduces etropy, uless the variables are idepedet, so H[X X ] < H[X ], uless X X. For this to be true of all, which is what s eeded for h(x) = H[X ], all the values of the sequece must be idepedet of each other; sice the sequece is statioary, this would imply that it s IID. = Example 382 (Markov Sequeces) If X is a statioary Markov sequece, the h(x) = H ρ [X 2 X ], because, by the chai rule, H ρ [X ] = H ρ [X ] + t=2 H ρ[x t X t ]. By the Markov property, however, H ρ [X t X t ] = H ρ [X t X t ], which by statioarity is H ρ [X 2 X ]. Thus, H ρ [X ] = H ρ [X ]+( )H ρ [X 2 X ]. Dividig by ad takig the limit, we get H ρ [X ] = H ρ [X 2 X ]. Example 383 (Higher-Order Markov Sequeces) If X is a k th order Markov sequece, the the same reasoig as before shows that h(x) = H ρ [X k+ X k ] whe X is statioary. Defiitio 384 (Divergece Rate) The divergece rate or relative etropy rate of the ifiite-dimesioal distributio Q from the ifiite-dimesioal distributio P, d(p Q), is ( )] dp d(p Q) = lim E P [log (29.2) dq σ(x 0 ) if all the fiite-dimesioal distributios of Q domiate all the fiite-dimesioal distributios of P. If P ad Q have desities, respectively p ad q, with respect to a commo referece measure, the [ d(p Q) = lim E P log p(x ] 0 X ) q(x 0 X (29.3) )

4 CHAPTER 29. RATES AND EQUIPARTITION The Shao-McMilla-Breima Theorem or Asymptotic Equipartitio Property This is a cetral result i iformatio theory, actig as a kid of ergodic theorem for the etropy. That is, we wat to say that, for almost all ω, log P (X (ω)) lim E [ log P (X )] = h(x) At first, it looks like we should be able to make a ice time-averagig argumet. We ca always factor the joit probability, log P (X ) = t= log P ( X t X t ) with the uderstadig that P ( ) X X 0 = P (X ). This looks rather like the sort of Cesàro average that we became familiar with i ergodic theory. The problem is, there we were averagig f(t t ω) for a fixed fuctio f. This is ot the case here, because we are coditioig o log ad loger stretches of the past. There s o problem if the sequece is Markovia, because the the remote past is irrelevat, by the Markov property, ad we ca just coditio o a fixedlegth stretch of the past, so we re averagig a fixed fuctio shifted i time. (This is why Shao s origial argumet was for Markov chais.) The result oetheless more broadly, but requires more subtlety tha might otherwise be thought. Breima s origial proof of the geeral case was fairly ivolved, requirig both martigale theory, ad a sort of domiated covergece theorem for ergodic time averages. (You ca fid a simplified versio of his argumet i Kalleberg, at the ed of chapter.) We will go over the sadwichig argumet of Algoet ad Cover (988), which is, to my mid, more trasparet. The idea of the sadwich argumet is to show that, for large, log P (X ) must lie betwee a upper boud, h k, obtaied by approximatig the sequece by a Markov process of order k, ad a lower boud, which will be show to be h. Oce we establish that h k h, we will be doe. Defiitio 385 (Markov Approximatio) For each k, defie the order k Markov approximatio to X by µ k (X ) = P ( X k ) t=k+ P ( X t X t ) t k (29.4) µ k is the distributio of a statioary Markov process of order k, where the distributio of X k+ matches that of the origial process. Notoriously, the proof i his origial paper was actually ivalid, forcig him to publish a correctio.

5 CHAPTER 29. RATES AND EQUIPARTITION 20 Lemma 386 For each k, the etropy rate of the order k Markov approximatio is is equal to H[X k+ X k ]. Proof: Uder the approximatio (but ot uder the origial distributio of X), H[X ] = H[X k ]+( k)h[x k+ X k ], by the Markov property ad statioarity (as i Examples 382 ad 383). Dividig by ad takig the limit as gives the result. Lemma 387 If X is a statioary two-sided sequece, the Y t = f(x t ) defies a statioary sequece, for ay measurable f. If X is also ergodic, the Y is ergodic too. Proof: Because X is statioary, it ca be represeted as a measure-preservig shift o sequece space. Because it is measure-preservig, θx t d = X, t so Y (t) = d Y (t + ), ad similarly for all fiite-legth blocks of Y. Thus, all of the fiite-dimesioal distributios of Y are shift-ivariat, ad these determie the ifiite-dimesioal distributio, so Y itself must be statioary. To see that Y must be ergodic if X is ergodic, recall that a radom sequece is ergodic iff its correspodig shift dyamical system is ergodic. A dyamical system is ergodic iff all ivariat fuctios are a.e. costat (Theorem 304). Because the Y sequece is obtaied by applyig a measurable fuctio to the X sequece, a shift-ivariat fuctio of the Y sequece is a shift-ivariat fuctio of the X sequece. Sice the latter are all costat a.e., the former are too, ad Y is ergodic. Lemma 388 If X is statioary ad ergodic, the, for every k, P (lim ) log µ k(x (ω)) = h k = (29.5) i.e., log µ k(x (ω)) coverges a.s. to h k. Proof: Start by factorig the approximatig Markov measure i the way suggested by its defiitio: log µ k(x ) = log P ( X k ) t=k+ log P ( X t X t ) t k (29.6) As grows, log P ( ) ( X k 0, for every fixed k. O the other had, log P Xt X t ) t k is a measurable fuctio of the past of the process, ad sice X is statioary ad ergodic, it, too, is statioary ad ergodic (Lemma 387). So by Theorem 32. log µ k(x ) log P ( X t X t ) t k (29.7) t=k+ a.s. E [ log P ( X k+ X k )] (29.8) = h k (29.9)

6 CHAPTER 29. RATES AND EQUIPARTITION 202 Defiitio 389 The ifiite-order approximatio to the etropy rate of a discretevalued statioary process X is h (X) E [ log P ( X 0 X )] (29.20) Lemma 390 If X is statioary ad ergodic, the almost surely. lim log P ( X X 0 ) = h (29.2) Proof: Via Theorem 32 agai, as i Lemma 388. Lemma 39 For a statioary, ergodic, fiite-valued radom sequece, h k (X) h (X). Proof: By the martigale covergece theorem, for every x 0 Ξ, P ( X 0 = x 0 X ) a.s. P ( X 0 = x 0 X ) (29.22) Sice Ξ is fiite, the probability of ay poit i Ξ is betwee 0 ad iclusive, ad p log p is bouded ad cotiuous. So we ca apply bouded covergece to get that h k = E [ x 0 P ( X 0 = x 0 X ) ( k)] k log P X0 = x 0 X (29.23) [ E P ( X 0 = x 0 X ) ( log P X0 = x 0 X ) x 0 (29.24) = h (29.25) Lemma 392 h (X) is the etropy rate of X, i.e. h (X) = h(x). Proof: Clear from Theorem 380 ad the defiitio of coditioal etropy. We are almost ready for the proof, but eed oe techical lemma first. Lemma 393 If R 0, E [R ] for all, the almost surely. lim sup Proof: Pick ay ɛ > 0. ( ) P log R ɛ log R 0 (29.26) = P (R e ɛ ) (29.27) E [R ] e ɛ (29.28) e ɛ (29.29) by Markov s iequality. Sice e ɛ, by the Borel-Catelli lemma, lim sup log R ɛ. Sice ɛ was arbitrary, this cocludes the proof.

7 CHAPTER 29. RATES AND EQUIPARTITION 203 Theorem 394 (Asymptotic Equipartitio Property) For a statioary, ergodic, fiite-valued radom sequece X, log P (X ) h(x) a.s. (29.30) Proof: For every k, µ k (X )/P (X ) 0, ad E [µ k (X )/P (X )]. Hece, by Lemma 393, lim sup log µ k(x ) P (X ) 0 (29.3) a.s. Maipulatig the logarithm, lim sup log µ k(x ) lim sup log P (X ) (29.32) From Lemma 388, lim sup log µ k(x ) = lim log µ k(x ) = h k (X), a.s. Hece, for each k, h k (X) lim sup log P (X ) (29.33) almost surely. A similar maipulatio of P (X ) /P ( X X 0 ) gives h (X) lim if log P (X ) (29.34) a.s. As h k h, it follows that the limif ad the limsup of the ormalized log likelihood must be equal almost surely, ad so equal to h, which is to say to h(x). Why is this called the AEP? Because, to withi a o() term, all sequeces of legth have the same log-likelihood (to withi factors of o(), if they have positive probability at all. I this sese, the likelihood is equally partitioed over those sequeces Typical Sequeces Let s tur the result of the AEP aroud. For large, the probability of a give sequece is either approximately 2 h or approximately zero 2. To get the total probability to sum up to oe, there eed to be about 2 h sequeces with positive probability. If the size of the alphabet is s, the the fractio of sequeces which are actually exhibited is 2 (h log s), a icreasigly small fractio (as h log s). Roughly speakig, these are the typical sequeces, ay oe of which, via ergodicity, ca act as a represetative of the complete process. 2 Of course that assumes usig base-2 logarithms i the defiitio of etropy.

8 CHAPTER 29. RATES AND EQUIPARTITION Asymptotic Likelihood Asymptotic Equipartitio for Divergece Usig methods aalogous to those we employed o the AEP for etropy, it is possible to prove the followig. Theorem 395 Let P be a asymptotically mea-statioary distributio, with statioary mea P, with ergodic compoet fuctio φ. Let M be a homogeeous fiite-order Markov process, whose fiite-dimesioal distributios domiate those of P ad P ; deote the desities with respect to M by p ad p, respectively. If lim log p(x ) is a ivariat fuctio P -a.e., the log p(x (ω)) a.s. d(p φ(ω) M) (29.35) where P φ(ω) is the statioary, ergodic distributio of the ergodic compoet. Proof: See Algoet ad Cover (988, theorem 4), Gray (990, corollary 8.4.). Remark. The usual AEP is i fact a cosequece of this result, with the appropriate referece measure. (Which?) Likelihood Results It is left as a exercise for you to obtai the followig result, from the AEP for relative etropy, Lemma 367 ad the chai rules. Theorem 396 Let P be a statioary ad ergodic data-geeratig process, whose etropy rate, with respect to some referece measure ρ, is h. Further let M be a fiite-order Markov process which domiates P, whose desity, with respect to the referece measure, is m. The P -almost surely. log m(x ) h + d(p M) (29.36) 29.4 Exercises Exercise 29. Markov approximatios are maximum-etropy approximatios. (You may assume that the process X takes values i a fiite set.) a Prove that µ k, as defied i Defiitio 385, gets the distributio of sequeces of legth k + correct, i.e., for ay set A X k+, ν(a) = P ( X k+ A ). b Prove that µ k, for ay ay k > k, also gets the distributio of legth k + sequeces right.

9 CHAPTER 29. RATES AND EQUIPARTITION 205 c I a slight abuse of otatio, let H[ν(X )] stad for the etropy of a sequece of legth whe distributed accordig to ν. Show that H[µ k (X )] H[µ k (X )] if k > k. (Note that the k case is easy!) d Is it true that that if ν is ay other measure which gets the distributio of sequeces of legth k + right, the H[µ k (X )] H[ν(X )]? If yes, prove it; if ot, fid a couter-example. Exercise 29.2 Prove Theorem 396.