Source Coding with Lists and Rényi Entropy or The Honey-Do Problem

Transcription

1 Source Coding with Lists and Rényi Entropy or The Honey-Do Problem Amos Lapidoth ETH Zurich October 8, 2013 Joint work with Christoph Bunte.

2 A Task from your Spouse Using a fixed number of bits, your spouse reminds you of one of the following tasks: Honey, don t forget to feed the cat. Honey, don t forget to go to the dry-cleaner. Honey, don t forget to pick-up my parents at the airport. Honey, don t forget the kids violin concert.

3 A Task from your Spouse Using a fixed number of bits, your spouse reminds you of one of the following tasks: Honey, don t forget to feed the cat. Honey, don t forget to go to the dry-cleaner. Honey, don t forget to pick-up my parents at the airport. Honey, don t forget the kids violin concert. The combinatorical approach requires # of bits = log 2 # of tasks. It guarantees that you ll know what to do...

4 The Information-Theoretic Approach Model the tasks as elements of X n generated IID P. Ignore the atypical sequences. Index the typical sequences using n H(X) bits. Send the index. Typical tasks will be communicated error-free.

5 The Information-Theoretic Approach Model the tasks as elements of X n generated IID P. Ignore the atypical sequences. Index the typical sequences using n H(X) bits. Send the index. Typical tasks will be communicated error-free. Any married person knows how ludicrous this is: What if the task is atypical? Yes, this is unlikely, but: You won t even know it! Are you ok with the consequences?

6 Improved Information-Theoretic Approach First bit indicates whether task is typical. You ll know when the task is lost in transmission.

7 Improved Information-Theoretic Approach First bit indicates whether task is typical. You ll know when the task is lost in transmission. What are you going to do about it?

8 Improved Information-Theoretic Approach First bit indicates whether task is typical. You ll know when the task is lost in transmission. What are you going to do about it? If I were you, I would perform them all. Yes, I know there are exponentially many of them. Are you beginning to worry about the expected number of tasks?

9 Improved Information-Theoretic Approach First bit indicates whether task is typical. You ll know when the task is lost in transmission. What are you going to do about it? If I were you, I would perform them all. Yes, I know there are exponentially many of them. Are you beginning to worry about the expected number of tasks? You could perform a subset of the tasks. You ll get extra points for effort. But what if the required task is not in the subset? Are you ok with the consequences?

10 Our Problem A source generates X n in X n IID P. The sequence is described using nr bits. Based on the description, a list is generated that is guaranteed to contain X n. For which rates R can we find descriptions and corresponding lists with expected listsize arbitrarily close to 1? More generally, we ll look at the ρ-th moment of the listsize.

11 What if you are not in a Relationship? Should you tune out?

12 Rényi Entropy H α (X) = α [ ] 1/α 1 α log P(x) α x X Alfréd Rényi ( )

13 A Homework Problem Show that 1. lim α 1 H α (X) = H(X). 2. lim α 0 H α (X) = log suppp. 3. lim α H α (X) = log max x X P(x).

14 Do not Tune Out Our problem gives an operational meaning to H 1, ρ > 0 (i.e., 0 < α < 1). It reveals many of its properties. And it motivates the conditional Rényi entropy.

15 Lossless List Source Codes Rate-R blocklength-n source code with list decoder: f n : X n {1,..., 2 nr }, λ n : {1,..., 2 nr } 2 X n The code is lossless if x n λ n (f n (x n )), x n X n ρ-th listsize moment (ρ > 0): E[ λ n (f n (X n )) ρ ] = P n (x n ) λ n (f n (x n )) ρ x n X n

16 The Main Result on Lossless List Source Codes Theorem 1. If R > H 1 (X), then there exists (f n, λ n ) n 1 such that 2. If R < H 1 (X), then lim E[ λ n(f n (X n )) ρ ] = 1. n lim E[ λ n(f n (X n )) ρ ] =. n

17 Some Properties of H 1 (X) 1. Nondecreasing in ρ

18 Some Properties of H 1 (X) 1. Nondecreasing in ρ (Monotonicity of ρ a ρ when a 1.)

19 Some Properties of H 1 (X) 1. Nondecreasing in ρ 2. H(X) H 1 (X) log X

20 Some Properties of H 1 (X) 1. Nondecreasing in ρ 2. H(X) H 1 (X) log X (R < H(X) = listsize 2 w.p. tending to one. And R = log X can guarantee listsize = 1.)

21 Some Properties of H 1 (X) 1. Nondecreasing in ρ 2. H(X) H 1 (X) log X 3. lim ρ 0 H 1 (X) = H(X)

22 Some Properties of H 1 (X) 1. Nondecreasing in ρ 2. H(X) H 1 (X) log X 3. lim ρ 0 H 1 (X) = H(X) (R > H(X) = prob(listsize 2) decays exponentially. For small ρ beats λ n (f n (X n )) ρ, which cannot exceed e nρ log X.)

23 Some Properties of H 1 (X) 1. Nondecreasing in ρ 2. H(X) H 1 (X) log X 3. lim ρ 0 H 1 (X) = H(X) 4. lim ρ H 1 (X) = log supp(p)

24 Some Properties of H 1 (X) 1. Nondecreasing in ρ 2. H(X) H 1 (X) log X 3. lim ρ 0 H 1 (X) = H(X) 4. lim ρ H 1 (X) = log supp(p) (R < log supp(p) = x 0 supp(p) n for which ϕ n (f n (x 0 )) e n(log supp(p) R). Since P n (x 0 ) pmin n, where p min = min{p(x) : x supp(p)} P n (x) ϕ n (f n (x)) ρ e nρ(log supp(p) R 1 ρ log 1 ) p min. x Hence R is not achievable if ρ is large.)

25 Some Properties of H 1 (X) 1. Nondecreasing in ρ 2. H(X) H 1 (X) log X 3. lim ρ 0 H 1 (X) = H(X) 4. lim ρ H 1 (X) = log supp(p)

26 Sketch of Direct Part 1. Partition each type-class T Q into 2 nr lists of lengths 2 nr T Q 2 n(h(q) R). 2. Describe the type of x n using o(n) bits. 3. Describe the list containing x n using nr bits. 4. Pr(X n T Q ) 2 nd(q P) and small number of types, so Pr(X n T Q ) 2 n(h(q) R) ρ Q where δ n By Arıkan 96, nρ(r max Q{H(Q) ρ 1 D(Q P)} δ n) { max H(Q) ρ 1 D(Q P) } = H 1 (X). Q

27 The Key to the Converse Lemma If 1. P is a PMF on a finite nonempty set X, 2. L 1,..., L M is a partition of X, 3. L(x) L j if x L j. Then x X P(x)L ρ (x) M ρ[ x X P(x) 1 ].

28 A Simple Identity for the Proof of the Lemma x X 1 L(x) = M.

29 A Simple Identity for the Proof of the Lemma Proof: x X 1 L(x) = M. 1 L(x) = M 1 L(x) x X j=1 x L j M 1 = L j=1 x L j j M = 1 j=1 = M.

30 Proof of the Lemma 1. Recall Hölder s Inequality: If p, q > 1 and 1/p + 1/q = 1, then [ ] 1 [ a(x)b(x) a(x) p p ] 1 b(x) q q, a( ), b( ) 0. x x x 2. Rearranging gives [ ] p [ a(x) p b(x) q q p a(x)b(x)]. x x x 3. Choose p = 1 + ρ, q = (1 + ρ)/ρ, a(x) = P(x) 1 L(x) ρ and b(x) = L(x) ρ, and note that x X 1 L(x) = M.

31 Converse 1. WLOG assume λ n (m) = { x n X n : f n (x n ) = m }. 2. The lists λ n (1),..., λ n (2 nr ) partition X n. 3. λ n (f n (x n )) is the list containing x n. 4. By the lemma: [ PX n (x n ) λ n (f n (x n )) ρ 2 nρr PX n (x n ) 1 x n X n x n X ( n ) = 2 nρ H 1 (X) R. ] Recall the lemma: x X P(x)L ρ (x) M ρ[ x X P(x) 1 ].

32 How to Define Conditional Rényi Entropy? Should it be defined as P Y (y) H α (X Y = y)? y Y

33 How to Define Conditional Rényi Entropy? Should it be defined as P Y (y) H α (X Y = y)? y Y Consider Y as side information to both encoder and decoder, (X i, Y i ) IID P XY. You and your spouse hopefully have something in common...

34 Lossless List Source Codes with Side-Information (X 1, Y 1 ), (X 2, Y 2 ),... IID P X,Y Y n is side-information. Rate-R blocklength-n source code with list decoder: f n : X n Y n {1,..., 2 nr }, λ n : {1,..., 2 nr } Y n 2 X n Lossless property: x n λ n (f n (x n, y n ), y n ), (x n, y n ) X n Y n ρ-th listsize moment: E[ λ n (f n (X n, Y n ), Y n ) ρ ]

35 Result for Lossless List Source Codes with Side-Information Theorem 1. If R > H 1 (X Y ), then there exists (f n, λ n ) n 1 such that 2. If R < H 1 (X Y ), then lim E[ λ n(f n (X n, Y n ), Y n ) ρ ] = 1. n lim E[ λ n(f n (X n, Y n ), Y n ) ρ ] =. n Here H 1 (X Y ) is defined to make this correct...

36 So H 1 (X Y ) is: H α (X Y ) = α 1 α log [ ] 1/α P X,Y (x, y) α y Y x X

37 Some Properties of H 1 (X Y ) 1. Nondecreasing in ρ > 0 2. lim ρ 0 H 1 (X Y ) = H(X Y ) 3. lim ρ H 1 (X Y ) = max y log supp(p X Y =y ) 4. H 1 (X Y ) H 1 (X)

38 Direct Part 1. Fix a side-information sequence y n of type Q. 2. Partition each V -shell of y n into 2 nr lists of lengths at most 2 nr T V (y n ) 2 n(h(v Q) R). 3. Describe V and the list containing x n using nr + o(n) bits. 4. The ρ-th moment of the listsize can be upper-bounded by Pr ( (X n, Y n ) ) T Q V 2 n(h(v Q) R) ρ Q,V where δ n nρ(r max Q,V {H(V Q) ρ 1 D(Q V P X,Y )} δ n), 5. Complete the proof by showing that H 1 { (X Y ) = max H(V Q) ρ 1 D(Q V P X,Y ) }. Q,V

39 Conditional Rényi Entropy H α (X Y ) = α 1 α log [ ] 1/α P X,Y (x, y) α y Y x X Suguru Arimoto

40 Arimoto s Motivation Define capacity of order α as Arimoto showed that C α = max P X { Hα (X) H α (X Y ) } C 1 = 1 ρ max E 0(ρ, P), P where E 0 (ρ, P) is Gallager s exponent function: E 0 (ρ, P) = log y [ x P(x)W (y x) 1 ], Gallager s random coding bound thus becomes ( P e exp nρ ( C 1 R )), 0 ρ 1.

41 List Source Coding with a Fidelity Criterion 1. Rate-R blocklength-n source code with list decoder: f n : X n {1,..., 2 nr }, λ n : {1,..., 2 nr } 2 ˆX n 2. Fidelity criterion: d(f n, λ n ) max min d(x n, ˆx n ) D x n X n ˆx n λ n(f n(x n )) 3. ρ-th listsize moment: E[ λ n (f n (X n )) ρ ]

42 A Rate-Distortion Theorem for List Source Codes Theorem 1. If R > R ρ (D), then there exists (f n, λ n ) n 1 such that sup d(f n, λ n ) D & lim E[ λ n(f n (X n )) ρ ] = 1. n n 2. If R < R ρ (D) and lim sup n d(f n, λ n ) D, then But what is R ρ (D)? lim E[ λ n(f n (X n )) ρ ] =. n

43 A Rényi Rate-Distortion Function { R ρ (D) max R(Q, D) ρ 1 D(Q P) }, Q where R(Q, D) is the rate-distortion function of the source Q.

44 Direct Part 1. Type Covering Lemma: If n n 0 (δ), then for every type Q we can find B Q ˆX n such that B Q 2 n(r(q,d)+δ) and max min d(x n, ˆx n ) D. x n T Q ˆx n B Q 2. Partition each B Q into 2 nr lists of lengths at most 2 n(r(q,d) R+δ). 3. Use nr + o(n) bits to describe the type Q of x n and a list in the partition of B Q that contains some ˆx n with d(x n, ˆx n ) D. 4. The ρ-th moment of the listsize can be upper-bounded by Pr(X n T Q ) 2 n(r(q,d) R+δ) ρ Q nρ(r max Q{R(Q,D) ρ 1 D(Q P)} δ δ n).

45 Converse 1. WLOG assume λ n (m) λ n (m ) = if m m. 2. For each ˆx n 2 nr m=1 λ n(m) let m(ˆx n ) be the unique index s.t. ˆx n λ n (m(ˆx n )). 3. Define g n : X n ˆX n such that g n (x n ) λ n (f n (x n )) and d(x n, g n (x n )) D, x. 4. Observe that x n P n X (x n ) λ n (f n (x n )) ρ = ˆx n P n X ( g 1 n ({ˆx n }) ) λ n (m(ˆx n )) ρ = ˆx n P n (ˆx n ) λ n (m(ˆx n )) ρ, where P n (ˆx n ) = P n X ( g 1 n ({ˆx n }) ).

46 5. Applying the lemma yields Converse contd. ˆx n P n (ˆx n ) λ n (m(ˆx n )) ρ 2 nρr 2 ρh 1 6. It now suffices to show that 7. The PMF P n can be written as H 1 ( P n ) nr ρ (D). P n = P n X W n, ( P n) where W n (ˆx n x n ) = 1 {ˆx n = g n (x n ) }.

47 Converse contd. 8. Let Q achieve R ρ (D), i.e., 9. For every PMF Q on ˆX n 10. Choosing Q = Q n W n gives R ρ (D) = R(Q, D) ρ 1 D(Q P X ). H 1 ( P n ) H(Q) ρ 1 D(Q P n ). H 1 ( P n ) H(Q n W n ) ρ 1 D(Q n W n P n W X n ) H(Q n W n ) ρ 1 D(Q n P n X ) = H(Q n W n ) nρ 1 D(Q P X ). (Data processing)

48 Converse contd. 11. Let X n be IID Q and let ˆX n = g n ( X n ). Then H(Q n W n ) = H( ˆX n ) = I( X n ; ˆX n ). 12. By construction of g n ( ) E[d( X n, ˆX n )] D. 13. From the converse to the Rate-Distortion Theorem it follows I( X n ; ˆX n ) nr(q, D).

49 Example: Binary Source with Hamming Distortion X = ˆX = {0, 1} Pr(X i = 1) = p d(x, ˆx) = 1{x ˆx} R(D) = h(p) h(d) + R ρ (D) = H 1 (p) h(d) + where ξ + = max{0, ξ} and h(p) = p log 1 p + (1 p) log 1 1 p.

50 Example: Binary Source with Hamming Distortion contd. Rρ(D) ρ 0 ρ = 1/2 ρ = 1 ρ = 2 ρ D R ρ (D) plotted for binary source (p = 1/4) and Hamming distortion

51 This function Is also not New! { R ρ (D) max R(Q, D) ρ 1 D(Q P) }, Q where R(Q, D) is the rate-distortion function of the source Q. Erdal Arıkan Neri Merhav

52 Arıkan & Merhav s Motivation Let G n = {ˆx n 1, ˆx n 2,...} be an ordering of ˆX n. Define G n (x n ) = min { j : d(x n, ˆx n j ) D }. If X 1, X 2,... are IID P, then 1 lim n n min log E[G n (X 1,..., X n ) ρ ] 1/ρ = R ρ (D). G n

53 To Recap Replacing messages with tasks leads to new operational characterizations of for all ρ > 0. H 1 (X) = 1 ρ log [ H 1 (X Y ) = 1 ρ log y R ρ (D) = max Q ] P(x) 1 x [ x ] P X,Y (x, y) 1 { R(Q, D) ρ 1 D(Q P) }

54 To Recap Replacing messages with tasks leads to new operational characterizations of for all ρ > 0. H 1 (X) = 1 ρ log [ H 1 (X Y ) = 1 ρ log y R ρ (D) = max Q ] P(x) 1 x [ x ] P X,Y (x, y) 1 { R(Q, D) ρ 1 D(Q P) } Thank You!