Memory in Classical Information Theory: A Brief History

Transcription

1 Beyond iid in information theory 8- January, 203 Memory in Classical Information Theory: A Brief History sergio verdu princeton university

2 entropy rate

3 Shannon, 948

4 entropy rate: memoryless process H(X) =H(P X )= X a2a P X (a) log P X (a)

5 n 3 ition.8. When the limit exists, the entropy rate of the random process X also a non-negative supermartingale, which according to the martingale co X2,...) is lim inf H(X,..., Xn ) = (.8 n! n nce theorem converges almost surely to the right 3side of (.87) with a = n! Ve pplying the boundedh(x) convergence theorem, we can claim that for each a2 = lim H(X,..., Xn ) (..2 Limiting conditional n! information h i n h and entropyhttp://www i E f (ıxk X k (a X k n ))! E f (ıxk X k (a X k )) ( n ple.2. If {X } are independent, distributed according i for the existence of the entropy rate X, then sufficient condition is that to thepcondition py rate is = equal to has tropy the past limit. here fgiven (x) x exp( x) isa bounded and continuous in the positive real line. ing both sides of (.95) over a 2 A we obtain (.88) if A is finite (See.3. If the on the right exists, then H(X) =side H(X) (. reorem the case when A islimit countably infinite)..3 H(X) = lim H(Xn X,..., Xn n! ) (.8 Stationary processes oof. Recall the chain rule (Theorem.9): * ationarity is sufficient to ensure that the conditional entropy in (.85) conv n X every stationary source, and integer n, the following ho heorem.5. For H(X,... Xn ) = H(Xi X,..., Xi ) (.8 n n i= a), if H(Xi X,..., Xi )!, then its arithmetic average (also known as Ces`a H(Xn X,..., Xn ) H(Xn X,..., Xn 2 ) ( m) must also converge to.

6 entropy rate: Markov process H(X) =E H P X X 0 ( X 0 ) = H(X X 0 )

7 entropy rate: stationary process H(Y) =E h H i P Y Y 0 ( Y 0 )

8 entropy rate: hidden Markov process X i X i X i+ Y i Y i Y i+ P Y Y 0 = CPP X 0 Y 0

9 Blackwell, 957. P Xn Y n 0 is a measure-valued Markov process 2. Blackwell measure: stationary distribution P X0 Y 0 3. Conjecture: {X i } irreducible =) {P Xn Y n 0 } ergodic : Chigansky-Van Handel condition () {P Xn Y n 0 } ergodic

10 ergodic theory. Kolmogorov, 958: Entropy rate of stationary dynamical system 2. Amended by Sinai, Kolmogorov-Sinai entropy also applies to analog processes 4. Kolmogorov, 959: isomorphism theorem if a stationary process can be generated by a deterministic sliding transformation of another stationary process and viceversa, then they must have the same entropy 5. Ornstein, 970, 973: deterministic sliding transformations of memoryless processes with identical entropy are isomorphic. 6. Identical entropy rate is not sufficient.

11 asymptotic equipartition property

12 Shannon, 948 stationary ergodic Markov

13 information 2 A ı X (a) = log P X (a)

14 ergodic process X:., Xi Shannon-MacMilla!McMillan, E[f (X,n..., Xk953 )] (8.28) X n g(..., X, X, X )! E[f (X X i 2 i i n g(..., X, X i=, X )! E[f (X,..., X )] 2, Xi, X i ) i give ka key re In this section we n i= Millan-Breiman Theorem arity/ergodicity is sufficient fo Shannon-MacMillan-Breiman Theorem asymptotically for (8 ey result in information theory that implies that (namely, stationi 2 i hannon-macmillan-breiman Theorem nt for the typical exhaust the probability In this sequences section we to give a key result in information theor Theorem 8.6. If (X, X,... or (8.2) to hold for > 0.) arity/ergodicity is sufficient for the typical sequences 2 tion we give a key result in information theory that implies tha asymptotically (namely, for (8.2) to hold for > 0.) alphabet A, with finite entropy dicity sufficient for the taking typicalvalues sequences exhaust the pr,...) isis stationary ergodic on a to discrete cally forthen (8.2) Theorem 8.6.toIfhold (X,for X2,..>.) 0.) is stationary ergodic ta ropy (namely, rate H(X), alphabet A, with finite entropy rate H(X), then 8.6. If (X, X2,...) is stationary ergodic taking values onıxa n ıx n (X,..., Xn )! H(X) (8.29),n with finite entropy rate H(X), then ıx n (X,..., Xn )! H(X n and almost surely. both in L ely...., Xnsurely. )! H(X) Xn (X both in ıl and,almost n ution it is possible to writeby the telescoping left side of (8.29) as a the distributio

15 ergodic process X:., Xi 2, Xi, X i )! E[f (XX,n. 957,.., Xk )] 960 (8.28) Breiman, n g(..., X, X, X )! E[f (X X i 2 i i n g(..., Xi 2, Xi i=, Xi )! E[f (X,..., Xk )] n i= Millan-Breiman Theorem Shannon-MacMillan-Breiman Theorem ey result in information theory thattheorem implies that stationhannon-macmillan-breiman nt for the typical exhaust the probability In this sequences section we to give a key result in information theor or (8.2) to hold for > 0.) arity/ergodicity is sufficient for the typical sequences tion we give a key result in information theory that implies tha asymptotically (namely, for (8.2) to hold for > 0.) dicity sufficient for the taking typicalvalues sequences exhaust the pr,...) isis stationary ergodic on a to discrete cally forthen (8.2) Theorem 8.6.toIfhold (X,for X2,..>.) 0.) is stationary ergodic ta ropy (namely, rate H(X), alphabet A, with finite entropy rate H(X), then 8.6. If (X, X2,...) is stationary ergodic taking values on a ıx n (X,..., Xn )! H(X) (8.29),n with finite entropy rate H(X), then ıx n (X,..., Xn )! H(X n a.s. ely...., Xnsurely. )! H(X) Xn (X both in ıl and,almost n ution it is possible to write the left side of (8.29) as a

16 Han-Verdú, 993 information spectrum probability mass function of n ı X n (X n ) H(X) R H(X)

17 sup-entropy rate H(X) = sup-entropy rate = limsup in probability of n ı X n (X n )

18 Verdú-Han, 993 new observation. Theorem 3: A source all satisfies the AEP if and only if for (4)

19

20 Verdú-Han, 993 Definition 7: Denote the infinite-order Rényi entropy by Note that flat-top source if for any is satisfied. Theorem 5: A flat-top source satisfies the AEP. Theorem 6: The Poisson source (Example 4) with mean satisfies Theorem 7: The geometric distribution (Example 5) with mean is flat-top and (8)

21 losless data compression COMPRESSOR DECOMPRESSOR

22 Han-Verdú, 993

23 lossless data compression: single-shot achievability Theorem 2: For any a 0, P [`(f (X)) a] apple P [ı X (X) a]. Since converse the labeling of the values Theorem 3: For any nonnegative integer k, max >0 P [ı X (X) k + ] 2 apple P [`(f (X)) k]. 0

24 Verdú-Kontoyiannis, 202 exact fundamental limit Theorem 4.3. For all, R ( ) =dlog 2 ( + M X ( ))e = P [ı X (X) log 2 ] M X ( )= P [ı X (X) < log 2 ] Z P [ı X (X) apple log 2 t] dt (38)

25 Steinberg-Verdú, 994 channel simulation

26 data transmission S ENCODER DECODER

27 Shannon, 949 waterfilling

28 Dobrushin, 959 information stability If P Xn maximizes I(X n ; Y n ), then ı Xn ;Ȳ n ( X n ; Ȳ n ) I( X n ; Ȳ n ) i.p.! C = lim n! + + n I( X n ; Ȳ n )

29 Wolfowitz, 963 finite memory n + m I( X n ; Ȳ n ) apple C apple n I( X n ; Ȳ n )+ k n

30 Verdú-Han, 994 general formula channel capacity

31 open problems deletion channel trapdoor channel...

32 successes Gaussian channel with intersymbol interference Erasure channel Additive-noise discrete channel Queue Gaussian channel with feedback (partial) Trapdoor channel with feedback

33 Vembu-Verdú-Steinberg, 995 source-channel separation principle

34 Vembu-Verdú-Steinberg, 995 source-channel separation principle

35 Polyanskiy-Poor-Verdú, 200 Finite blocklength Decouple non-asymptotics (information theory) from asymptotics (limit theorems) Single-shot (Converse) Upper bounds to maximal achievable rate (Achievability) Lower bounds Back to asymptotics (via CLT) Channel Dispersion

36 Verdú, 202 multiuser information theory: single-shot achievability bounds multiaccess channels interference channels broadcast channels Wyner-Ziv Slepian-Wolf Gelfand-Pinsker Ahlswede-Korner

37 lossy compression b X COMPRESSOR DECOMPRESSOR X^ {,..., M }

38

39 Steinberg-Verdú, 996 channel simulation with finite precision

40