8. Compressing stationary ergodic sources

Transcription

1 8. Compressig statioary ergodic sources We have examied the compressio of i.i.d. sequece {S i }, for which l (f (S )) H( S) i prob. (8.) lim ɛ ( S R > H(S), R) = { (8.2) R < H( S) I this lecture, we shall examie similar results for ergodic processes ad we first state the mai theory as follows: Theorem 8. (Shao-McMilla). Let {S, S 2,... } be a statioary ad ergodic discrete process, the where H = lim H( S ) is the etropy rate. log P P S (S H, also a.s. ad i L (8.3) ) Corollary 8.. For ay statioary ad ergodic discrete process { S, S 2,... }, (8.) (8.2) hold with H( S) replaced by H. Proof. Shao-McMilla (we oly eed covergece i probability) + Theorem Theorem 7. which tie together the respective CDF of the radom variable l(f ( S )) ad log P S (s ). I Lecture 7 we leared the asymptotic equipartitio property (AEP) for iid sources. Here we geeralize it to statioary ergodic sources thaks to Shao-McMilla. Corollary 8.2 (AEP for statioary ergodic sources). Let { S, S 2,... } be a statioary ad ergodic discrete process. For ay δ >, defie the set The. P [S T δ ] as (H δ) ( + o()) T δ 2 (H+δ) ( + o()). Note: T δ = {s log H δ}. (S ) Covergece i probability for statioary ergodic Markov chais [Shao 948] Covergece i L for statioary ergodic processes [McMilla 953] P S 9

2 Covergece almost surely for statioary ergodic processes [Breima 956] (Either of the last two results implies the covergece Theorem 8. i probability.) For a Markov chai, existece of typical sequeces ca be uderstood by thikig of Markov process as sequece of idepedet decisios regardig which trasitios to take. It is the clear that Markov process s trajectory is simply a trasformatio of trajectories of a i.i.d. process, hece must similarly cocetrate similarly o some typical set. 8. Bits of ergodic theory Let s start with a dyamic system view ad itroduce a few defiitios: Defiitio 8. (Measure preservig trasformatio). τ Ω Ω is measure preservig (more precisely, probability preservig) if E F, P (E) = P ( τ E ). The set E is called τ-ivariat if E = τ E. The set of all τ-ivariat sets forms a σ-algrebra (check!) deoted F iv. Defiitio 8.2 (statioary process). A process { S, =,...} is statioary if there exists a measure preservig trasformatio τ Ω Ω such that: S = S τ = S τ j j j Therefore a statioary process ca be described by the tuple (Ω, F, P, τ, S ) ad S k = S τ k. Notes:. Alteratively, a radom process (S, S, S 2,... ) is statioary if its joit distributio is ivariat with respect to shifts i time, i.e., P S m = P S m+t,, m, t. Ideed, give such a process we ca + t defie a m.p.t. as follows: (s, s,... ) τ (s, s 2,... ) (8.4) So τ is a shift to the right. 2. A evet E F is shift-ivariat if (s, s 2,... ) E s (s, s, s 2,... ) E or equivaletly E = τ E (check!). Thus τ-ivariat evets are also called shift-ivariat, whe τ is iterpreted as (8.4). 3. Some examples of shift-ivariat evets are { x i = i }, {lim sup x i < } etc. A o shift-ivariat evet is A = { x = x = = }, sice τ(,,,...) A but (,,...) / A. 4. Also recall that the tail σ-algebra is defied as Ftail σ{ S, S +,...}. It is easy to check that all shift-ivariat evets belog to F tail. The iclusio is strict, as for example the evet { x i =, odd i } is i F tail but ot shift-ivariat. 9

3 Propositio 8. (Poicare recurrece). Let τ be measure-preservig for ( Ω, F, P ). The for ay measurable A with P[ A] > we have P[ τ k k A A] = P[ τ k ( ω) A ifiitely ofte A] =. Proof. Let B = k τ k A. It is sufficiet to show that P[ A B] = P[ A] or equivaletly To that ed otice that τ A τ B = B ad thus P[ A B] = P[ B ]. (8.5) P[τ (A B)] = P[B], but the left-had side equals P[ A B] by the measure-preservatio of τ, provig (8.5). Note: Cosider τ mappig iitial state of the coservative (Hamiltoia) mechaical system to its state after passage of a give uit of time. It is kow that τ preserves Lebesgue measure i phase space (Liouville s theorem). Thus Poicare recurrece leads to rather couter-ituitive coclusios. For example, opeig the barrier separatig two gases i a cylider allows them to mix. Poicare recurrece says that evetually they will retur back to the origial separated state (with each gas occupyig roughly its half of the cylider). Defiitio 8.3 (Ergodicity). A trasformatio τ is ergodic if E F iv we have P[ E] = or. A process { S } [ i is ergodic if all shift ivariat evets are determiistic, i.e., for ay shift ivariat evet E, P S E] = or. Example: { S k = k 2 }: ergodic but ot statioary {S k = S }: statioary but ot ergodic ( uless S = {( )} [ is a costat). Note that the sigleto set E s, s,... is shift ivariat ad P S E] = P [ S = s] (, ) ot determiistic. {S k } i.i.d. is statioary ad ergodic (by Kolmogorov s - law, tail evets have o radomess) (Slidig-widow costructio of ergodic processes) If {S i } is ergodic, the {X i = f(s i, S i+,... )} is also ergodic. It is called a B-process if S i is i.i.d. Example, S i Ber( ) i.i.d., X k = 2 2 = S k + = 2X k mod. The margial distributio of X i is uiform o [, ]. Note that X k s behavior is completely determiistic: give X, all the future Xk s are determied exactly. This example shows that certai determiistic maps exhibit ergodic/chaotic behavior uder iterative applicatio: although the trajectory is completely determiistic, its time-averages coverge to expectatios ad i geeral look radom. There are also stroger coditios tha ergodicity. Namely, we say that τ is mixig (or strog mixig) if P[A τ B] P[A]P[B]. We say that τ is weakly mixig if P [A τ B] P[A]P[B]. k= Strog mixig implies weak mixig, which implies ergodicity (check!). 92

4 {S i }: fiite irreducible Markov chai with recurret states is ergodic (i fact strog mixig), regardless of iitial distributio. Toy example: kerel P ( ) = P ( ) = with iitial dist. P (S = ) =.5. This process oly has two sample paths: P [S = (...)] = P [S = (...)] = 2. It is easy to verify this process is ergodic (i the sese defied above!). Note however, that i Markov-chai literature a chai is called ergodic if it is irreducible, aperiodic ad recurret. This example does ot satisfy this defiitio (this clash of termiology is a frequet source of cofusio). (optioal) { S i }: statioary zero-mea Gaussia process with autocovariace fuctio R( ) = E[ S S ]. lim t R[ t] = { Si} ergodic { S i } weakly mixig + = lim R[] = { } mixig S i Ituitively speakig, a ergodic process ca have ifiite memory i geeral, but the memory is weak. Ideed, we see that for a statioary Gaussia process ergodicity meas the correlatio dies (i the Cesaro-mea sese). The spectral measure is defied as the (discrete time) Fourier trasform of the autocovariace sequece { R( )}, i the sese that there exists a uique probability measure µ o [ 2, 2] such that R( ) = E exp(i2πx) where X µ. The spectral criteria ca be formulated as follows: {S i } ergodic spectral measure has o atoms (CDF is cotiuous) { S i } B-process spectral measure has desity Detailed expositio o statioary Gaussia processes ca be foud i [Doo53, Theorem 9.3.2, pp. 474, Theorem 9.7., pp ]. 8.2 Proof of Shao-McMilla We shall show the covergece i L, which implies covergece i probability automatically. I order to prove Shao-McMilla, let s first itroduce the Birkhoff-Khitchie s covergece theorem for ergodic processes, the proof of which is preseted i the ext subsectio. Theorem 8.2 (Birkhoff-Khitchie s Ergodic Theorem). If { S i } statioary ad ergodic, fuctio f L, i.e., E f(s,... ) <, lim f(s k,... ) = E f(s,... ). a.s. ad i L k= I the special case where f depeds o fiitely may coordiates, say, f = f(s,..., S m ), we have lim f(s k,..., S k+m ) = E f(s,..., S m ). a.s. ad i L k = Iterpretatio: time average Example: Cosider f = f(s ) Thaks Prof. Bruce Hajek for the poiter. coverges to esemble average. 93

5 {S i } is iid. The Theorem 8.2 is SLLN (strog LLN). {S i } is such that S i = S for all i o-ergodic. The Theorem 8.2 fails uless S is a costat. Defiitio 8.4. {S i i N} is a m th order Markov chai if P St+ S t = P S t t S for all t m. It + t m+ is called time homogeeous if P S = P m t+ S t S + m+ S. t m Remark 8.. Showig (8.3) for a m th order time homogeeous Markov chai {S i } is a direct applicatio of Birkhoff-Khitchie. log P S (S ) = t= log P St S t (S t S t ) = log P S m(s m ) + = log P S (S m) + where we applied Theorem 8.2 with f(s, s 2,...) = log log t=m+ t=m+ log P St S t t m (S l S l l m ) P Sm+ S m(s t St m t ) H(S m+ S m ) by Birkhoff-Khitchie P + s + s. S Sm ( m m m ), (8.6) Now let s prove (8.3) for a geeral statioary ergodic process {S i } which might have ifiite memory. The idea is to approximate the distributio of that ergodic process by a m-th order MC (fiite memory) ad make use of (8.6); the let m to make the the approximatio accurate (Markov approximatio). Proof of Theorem 8. i L. To show that (8.3) coverges i L, we wat to show that E log P S (S H,. ) To this ed, fix a m N. Defie the followig auxiliary distributio for the process: Q (m) (S ) = PS m (S ) stat. m P St S t (S t S t t m t m t= m+ t t= m+ = P S m(s m ) P S (S t m+ S m t S m) Note that uder Q (m), {S i} is a m th -order time-homogeeous Markov chai. By triagle iequality, E log P S (S ) H E log P S (S ) log (S ) m where Hm H( S m + S ). Now ) Q (m) S A + E log H Q (m ) S ( ) m + Hm S 94 B C H

6 C = H m H as m by Theorem 5.4 (Recall that for statioary processes: H( Sm S m ) H from above). + As show i Remark 8., for ay fixed m, B i L as, as a cosequece of Birkhoff-Khitchie. Hece for ay fixed m, EB as. For term A, where E[A] = E P log dp S dq (m) S D(P S Q(m) S ) + 2 log e e D(P S Q(m) S ) = E P S log (S ) P S m(s m ) t=m+ P Sm+ Sm(S t S t H ( (S ) + H(S m ) + ( m)h m ) H as stat. = H m t m ) ad the ext Lemma 8.. Combiig all three terms ad sedig, we obtai for ay m, lim sup E log P S (S ) H 2(H m H). Sedig m completes the proof of L -covergece. Lemma 8.. E P [ log dp D 2 log e ] (P Q) +. dq e Proof. x log x x log x 2 log e e, x >, sice LHS is zero if x, ad otherwise upper bouded by 2 sup x x log x = 2 log e e. 8.3* Proof of Birkhoff-Khitchie Proof of Theorem 8.2. fuctio f L, ɛ, there exists a decompositio f = f + h such that f is bouded, ad h L, h ɛ. Let us first focus o the bouded fuctio f. Note that i the bouded domai L L 2, thus f L2. Furthermore, L2 is a Hilbert space with ier product ( f, g) = E[f(S )g( S )]. For the measure preservig trasformatio τ that geerates the statioary process { S i }, defie the operator T ( f) = f τ. Sice τ is measure preservig, we kow that T f 2 2 = f 2 2, thus T is a uitary ad bouded operator. Defie the operator A (f) = f τ k k= Ituitively: k = A = k T = (I T )(I T ) 95

7 The, if f ker(i T ) we should have A f, sice oly compoets i the kerel ca blow up. This ituitio is formalized i the proof below. Let s further decompose f ito two parts f = f + f 2, where f ker(i T ) ad f 2 ker(i T ). Observatios: if g ker(i T ), g must be a costat fuctio. This is due to the ergodicity. Cosider idicator fuctio A, if A = A τ = τ A, the P[ A] = or. For a geeral case, suppose g = T g ad g is ot costat, the at least some set {g ( a, b)} will be shift-ivariat ad have o-trivial measure, violatig ergodicity. ker(i T ) = ker(i T ). This is due to the fact that T is uitary: g = T g g 2 = (T g, g) = (g, T g) ( T g, g) = g T g T g = g where i the last step we used the fact that Cauchy-Schwarz ( f, g) f g oly holds with equality for g = cf for some costat c. ker(i T ) = ker( I T ) = [ Im( I T )], where [Im(I T )] is a L 2 closure. g ker( I T ) E[ g] =. Ideed, oly zero-mea fuctios costats. are orthogoal to With these observatios, we kow that = m is a cost. Also, f 2 [ Im( I T )] so we further approximate it by f 2 = f + h, where f f Im( I T ), amely f = g g τ for some fuctio g L2, ad h h 2 < ɛ. Therefore we have A f = f = E[f] A f = (g g τ ) a.s. ad L (sice E[ ( g τ )2 ] = E[g 2 ] 2 < g The proof completes by showig Ideed, the by takig ɛ we will have show as required. τ a.s. ) 2 P[ lim sup A (h + h ) δ] ɛ. (8.7) δ P[ lim sup A ( f) E[ f] + δ] = Proof of (8.7) makes use of the Maximal Ergodic Lemma stated as follows: Theorem 8.3 (Maximal Ergodic Lemma). Let (P, τ) be a probability measure ad a measurepreservig trasformatio. The for ay f L ( P) we have where A f = k= f τ k. P [ > ] E[f sup Af sup A f a a 96 >a] f a

8 Note: This is a so-called weak L estimate for a subliear operator sup A ( ). I fact, this theorem is exactly equivalet to the followig result: Lemma 8.2 (Estimate for the maximum of averages). Let {Z, =,...} be a statioary process with E[ Z ] < the Z Z P [ sup > a] E[ Z ] a > a Proof. The argumet for this Lemma has origially bee quite ivolved, util a dramatically simple proof (below) was foud by A. Garcia. Defie It is sufficiet to show that S = Z k (8.8) k= L = max{, Z,..., Z + + Z } (8.9) = max{, Z 2, Z2 + Z 3,..., Z Z } (8.) S Z = sup (8.) M E[Z {Z >}]. (8.2) Ideed, applyig (8.2) to Z = Z a ad oticig that Z = Z a we obtai E[Z {Z >a}] ap[z > a], from which Lemma follows by upper-boudig the left-had side with E[ Z ]. I order to show (8.2) we first otice that {L > } { Z > }. Next we otice that ad furthermore Thus, we have Z + M = max{ S,..., S} Z + M = L o {L > } Z {L >} = L M {L> where we do ot eed idicator i the first term sice L = o {L > } c. Takig expectatio we get E[Z {L>}] = E[L ] E[ M ] } {L >} (8.3) E[ L ] E[ M ] (8.4) = E[L ] E[L ] = E[L L ], (8.5) where we used M, the fact that M has the same distributio as L, ad L respectively. Takig limit as i (8.5) we obtai (8.2). L, 97

9 8.4* Siai s geerator theorem It turs out there is a way to associate to every probability-preservig trasformatio τ a umber, called Kolmogorov-Siai etropy. This umber is ivariat to isomorphisms of p.p.t. s (appropriately defied). Defiitio 8.5. Fix a probability-preservig trasformatio τ actig o probability space ( Ω, F, P ). Kolmogorov-Siai etropy of τ is defied as H( τ) sup lim H ( X X, X τ,..., X where supremum is take over all radom variables with respect to F. X τ ), Ω X with fiite rage X ad measurable Note that every radom variable X geerates a statioary process adapted to τ, that is X k X τ k. I this way, Kolmogorov-Siai etropy of τ equals the maximal etropy rate amog all statioary processes adapted to τ. This quatity may be extremely hard to evaluate, however. Oe help comes i the form of the famous criterio of Y. Siai. We eed to elaborate o some more cocepts before: σ-algebra G F is P-dese i F, or sometimes we also say G = F mod P or eve G = F mod, if for every E F there exists E G s.t. P[ E E ] =. Partitio A = { A i, i =, 2,...} measurable with respect to F is called geeratig if = σ{τ A} = F mod P. Radom variable Y Ω Y with a coutable alphabet Y is called a geerator of (Ω, F, P, τ) if σ{y, Y τ,..., Y τ,...} = F mod P Theorem 8.4 (Siai s geerator theorem). Let Y be the geerator of a p.p.t. ( Ω, F, P, τ ). Let H( Y) be the etropy rate of the process Y = { Yk = Y τ k, k =,...}. If H( Y) is fiite, the H( τ) = H( Y). Proof. Notice that sice H(Y) is fiite, we must have H( Y ) < ad thus H( Y ) <. First, we argue that H( τ) H( Y ). If Y has fiite alphabet, the it is simply from the defiitio. Otherwise let Y be Z -valued. Defie a trucated versio Y + m = mi( Y, m), the sice Y m Y as m we have from lower semicotiuity of mutual iformatio, cf. (3.9), that lim I( Y ; Y m ) H(Y ), ad cosequetly for arbitrarily small ɛ ad sufficietly large m H(Y Ỹ ) ɛ, m 98

10 The, cosider the chai H(Y ) = H(Ỹ, Y H ) = ( Y ) + H( Y Y ) = H( Y ) + H(Y i Ỹ, Y i i= H( Y ) + H( Y i Ỹi) i= = H( Y ) + H( Y Y Thus, etropy rate of Y (which has fiite-alphabet) ca be made arbitrarily close to the etropy rate of Y, cocludig that H( τ) H( Y ). The mai part is showig that for ay statioary process X adapted to τ the etropy rate is upper bouded by H( Y). To that ed, cosider X Ω X with fiite X ad defie as usual the process X = { X τ k, k =,,...}. By geeratig property of Y we have that X (perhaps after modificatio o a set of measure zero) is a fuctio of Y. So are all X k. Thus ) ) H(Ỹ ) + ɛ H(X ) = I(X ; Y ) = lim I( X ; Y ), where we used the cotiuity-i-σ-algebra property of mutual iformatio, cf. (3.). Rewritig the latter limit differetly, we have lim H ( X Y ) =. Fix ɛ > ad choose m so that H(X Y m ) ɛ. The cosider the followig chai: H(X ) H(X, Y ) = H(Y ) + H H(Y ) + H( Xi Yi ) i= = H(Y ) + H( X Y i= i (X Y ) H( Y ) + m log X + ( m) ɛ, where we used statioarity of (X k, Y k ) ad the fact that H(X Y by ad passig to the limit our argumet implies Takig here ɛ completes the proof. H(X) H(Y) + ɛ. ) i ) < ɛ for i m. After dividig Alterative proof: Suppose X is takig values o a fiite alphabet X ad X = f( Y ). The (this is a measure-theoretic fact) for every ɛ > there exists m = m(ɛ) ad a fuctio fɛ Y m + X s.t. P[ f( Y ) f Y m ɛ ( )] ɛ. (This is just aother way to say that X as Notice that sice X is a fuctio of Y + σ{y } is P-dese i σ( Y ).) Defie a statioary process m X j f ɛ ( Y j we have m+ j ). H( X ) H(Y +m ). 99

11 Dividig by m ad passig to the limit we obtai that for etropy rates H (X ) H(Y). Fially, to relate X to X otice that by costructio P[ X j X j ] ɛ. Sice both processes take values o a fixed fiite alphabet, from Corollary 5.2 we ifer that Altogether, we have show that Takig ɛ we coclude the proof. Examples: H(X) H(X ) ɛ log X + h(ɛ). H( X) H( Y) + ɛ log X + h( ɛ ). Let Ω = [, ], F Borel σ-algebra, P = Leb ad ω τ(ω) = 2ω mod = 2, ω < /2 2ω, ω / 2 It is easy to show that Y (ω) = {ω < /2} is a geerator ad that Y is a i.i.d. Beroulli( / 2) process. Thus, we get that Kolmogorov-Siai etropy is H( τ) = log 2. Let Ω be the uit circle S, F Borel σ-algebra, P be the ormalized legth ad τ(ω) = ω + γ γ 2π i.e. τ is a rotatio by the agle γ. (Whe is irratioal, this is kow to be a ergodic p.p.t.). Here Y = { ω < 2πɛ} is a geerator for arbitrarily small ɛ ad hece This is a example of a zero-etropy p.p.t. H(τ) H(X) H(Y ) = h(ɛ) as ɛ. Remark 8.2. Two p.p.t. s ( Ω, τ, P ) ad ( Ω, τ, P ) are called isomorphic if there exists f i Ω i Ω i defied P i -almost everywhere ad such that ) τ i f i = f i τ i ; 2) f i f i is idetity o Ωi (a.e.); 3) P i [f i E] = P i [E]. It is easy to see that Kolmogorov-Siai etropies of isomorphic p.p.t.s are equal. This observatio was made by Kolmogorov i 958. It was revoluatioary, sice it allowed to show that p.p.t.s correspodig shifts of iid Ber( / 2) ad iid Ber( / 3) procceses are ot isomorphic. Before, the oly ivariats kow were those obtaied from studyig the spectrum of a uitary operator U τ L 2 (Ω, P) L 2 (Ω, P) (8.6) φ( x) φ( τ( x )). (8.7) However, the spectrum of τ correspodig to ay o-costat i.i.d. process cosists of the etire uit circle, ad thus is uable to distiguish Ber( / 2) from Ber(/3). 2 2 To see the statemet about the spectrum, let X i be iid with zero mea ad uit variace. The cosider φ(x ) defied as m iω k= e k iω x k. This φ has uit eergy ad as m we have Uτ φ e φ m L2. Hece every e iω belogs to the spectrum of U τ.

12 MIT OpeCourseWare Iformatio Theory Sprig 26 For iformatio about citig these materials or our Terms of Use, visit: