The Gallager Converse

Transcription

1 The Gallager Converse Abbas El Gamal Director, Information Systems Laboratory Department of Electrical Engineering Stanford University Gallager s 75th Birthday 1

2 Information Theoretic Limits Establishing information theoretic limits, e.g., channel capacity, requires: Finding a single-letter expression for the limit Proving achievability Proving a converse While achievability tells us about how to improve system design, converse is necessary to prove optimality Proving a converse is typically harder and there are very few tools available, e.g., Fano s inequality, data processing inequality, convexity Gallager s 75th Birthday 2

3 Information Theoretic Limits Establishing information theoretic limits, e.g., channel capacity, requires: Finding a single-letter expression for the limit Proving achievability Proving a converse While achievability tells us about how to improve system design, converse is necessary to prove optimality Proving a converse is typically harder and there are very few tools available, e.g., Fano s inequality, data processing inequality, convexity Gallager s identification of the auxiliary random variable: Crucial step in proof of the degraded broadcast channel converse Has been used in the proof of almost all subsequent converses in multi-user information theory I used it many times in my papers Gallager s 75th Birthday 3

4 Broadcast Channel T. M. Cover, Broadcast Channels, IEEE Trans. Info. Theory, vol. IT-18, pp. 2-14, Jan Discrete memoryless (DM) broadcast channel (X,p(y 1,y 2 x), Y 1, Y 2 ): (W 1,W 2 ) Encoder X n p(y 1,y 2 x) Y n 1 Y n 2 Decoder 1 Decoder 2 Ŵ 1 Ŵ 2 Send independent message W j [1, 2 nr j] to receiver j = 1, 2 Average probability of error: P (n) e = P{Ŵ1 W 1 or Ŵ2 W 2 } (R 1, R 2 ) achievable if there exists a sequence of codes with P (n) e 0 The capacity region is the closure of the set of achievable rates Gallager s 75th Birthday 4

5 Example 1: Binary Symmetric BC Assume that p 1 p 2 < 1/2, H(a), a [0, 1] binary entropy function Z 1 Bern(p 1 ) R 2 X Y 1 1 H(p 2 ) time-sharing Y 2 Z 2 Bern(p 2 ) 1 H(p 1 ) R 1 Gallager s 75th Birthday 5

6 Example 1: Binary Symmetric BC Assume that p 1 p 2 < 1/2 Z 1 Bern(p 1 ) cloud (radius nβ) {0, 1} n satellite codeword X n X Y 1 cloud center U n Y 2 Z 2 Bern(p 2 ) Superposition coding: Use random coding. Let 0 β 1/2; U Bern(1/2) and X Bern(β) independent, X = U + X Generate 2 nr 2 i.i.d. U n (w 2 ), w 2 [1, 2 nr 2] (cloud centers) Generate 2 nr 1 i.i.d. X n (w 1 ), w 1 [1, 2 nr 1] Send X n (w 1, w 2 ) = U n (w 2 ) + X n (w 1 ) (satellite codeword) Gallager s 75th Birthday 6

7 Example 1: Binary Symmetric BC Assume that p 1 p 2 < 1/2 Z 1 Bern(p 1 ) R 2 X Y 1 C 2 time-sharing Y 2 Decoding: Z 2 Bern(p 2 ) Decoder 2 decodes U n (W 2 ) R 2 < 1 H(β p 2 ) Decoder 1 first decodes U n (W 2 ), subtracts it off, then decodes X n (W 1 ) R 1 < H(β p 1 ) H(p 1 ) C 1 R 1 Gallager s 75th Birthday 7

8 Example 2: AWGN BC Assume N 1 N 2, average power constraint P on X Z 1 N(0, N 1 ) Y 1 X Y 2 Z 2 N(0, N 2 ) Superposition coding: Let α [0, 1], U N(0, (1 α)p), X N(0, αp) independent, X = U + X Decoder 2 decodes cloud center R 2 C ((1 α)p/(αp + N 2 )) Decoder 1 decodes clound center, subtracts it off, then decode the sattelite codeword R 1 C (αp/n 1 ) Gallager s 75th Birthday 8

9 Degraded Broadcast Channel Stochastically degraded BC [Cover 1972]: There exists p(y 2 y 1 ) such that p(y 2 x) = y 1 p(y 1 x)p(y 2 y 1 ) Special case of channel inclusion in: C. E. Shannon, A note on a partial ordering for communication channels, Information and Control, vol. 1, pp , 1958 Gallager s 75th Birthday 9

10 Degraded Broadcast Channel Stochastically degraded BC [Cover 1972]: There exists p(y 2 y 1 ) such that p(y 2 x) = y 1 p(y 1 x)p(y 2 y 1 ) Special case of channel inclusion in: C. E. Shannon, A Note on a Partial Ordering for Communication Channels, Information and Control, vol. 1, pp , 1958 Physically degraded version [Bergmans 73]: X p(y 1 x) Y 1 p(y 2 y 1 ) Y 2 Since the capacity region of any BC depends only on marginals p(y 1 x), p(y 2 x) The capacity region of the degraded broadcast channel is the same as that of the physically degraded version Gallager s 75th Birthday 10

11 Degraded Broadcast Channel Cover conjectured that the capacity region is set of all (R 1,R 2 ) such that for some p(u,x) R 2 I(U;Y 2 ), R 1 I(X;Y 1 U), First time an auxiliary random variable is used in characterizing an information theoretic limit Gallager s 75th Birthday 11

12 Achievability P. P. Bergmans, Random Coding Theorem for Broadcast Channels with Degraded Components, IEEE Trans. Info. Theory, vol. IT-19, pp , Mar Use superposition coding: Fix p(u)p(x u) U n U n X n X n Decoder 2 decodes the cloud center U n R 2 I(U; Y 2 ) Decoder 1 decodes first decodes the cloud center, then the sattelite codeword X n R 1 I(X;Y 1 U) Gallager s 75th Birthday 12

13 The Converse A. Wyner, A Theorem on the Entropy of Certain Binary Sequences and Applications: Part II IEEE Trans. Info. Theory, vol. IT-10, pp , Nov Proved weak converse for binary symmetric broadcast channel (used Mrs. Gerber s Lemma) Gallager s 75th Birthday 13

14 The Converse A. Wyner, A Theorem on the Entropy of Certain Binary Sequences and Applications: Part II IEEE Trans. Info. Theory, vol. IT-10, pp , Nov Proved weak converse for binary symmetric broadcast channel (used Mrs. Gerber s Lemma) P.P. Bergmans, A Simple Converse for Broadcast Channels with Additive White Gaussian Noise (Corresp.), IEEE Trans. Info. Theory, vol. IT-20, pp , Mar Proved the converse for the AWGN BC (used Entropy Power Inequality very similar to Wyner s proof for binary symmetric case) Gallager s 75th Birthday 14

15 The Converse A. Wyner, A Theorem on the Entropy of Certain Binary Sequences and Applications: Part II IEEE Trans. Info. Theory, vol. IT-10, pp , Nov Proved weak converse for binary symmetric broadcast channel (used Mrs. Gerber s Lemma) P.P. Bergmans, A Simple Converse for Broadcast Channels with Additive White Gaussian Noise (Corresp.), IEEE Trans. Info. Theory, vol. IT-20, pp , Mar Proved converse for AWGN BC (used Entropy Power Inequality) R. Gallager, Capacity and Coding for Degraded Broadcast Channels, Problemy Peredaci Informaccii, vol. 10, no. 3, pp. 3-14, July-Sept 1974 Proved the weak converse for the discrete-memoryless degraded BC identification of auxiliary random variable Established bound on cardinality of the auxiliary random variable Gallager s 75th Birthday 15

16 Weak Converse for Shannon Channel Capacity Shannon capacity: C = max p(x) I(X;Y ) (only uses channel variables) Weak converse: Show that for any sequence of (n,r) codes with 0, R C P (n) e For each code form the empirical joint pmf: (W, X n, Y n ) p(w)p(x n w) n p(y i x i ) Fano s inequality: H(W Y n ) nrp (n) e Now consider nr = I(W; Y n ) + H(W Y n ) + H(P e (n) ) = nǫ n I(W; Y n ) + nǫ n I(X n ; Y n ) + nǫ n data processing inequality = H(Y n ) H(Y n X n ) + nǫ n I(X i ; Y i ) + nǫ n convexity, memorylessness n(c + ǫ n ) Gallager s 75th Birthday 16

17 Gallager Converse Show that given a sequence of (n,r 1,R 2 ) codes with P (n) e 0, R 1 I(X;Y 1 U), R 2 I(U;Y 2 ) for some p(u,x) such that U X (Y 1, Y 2 ) Key is identifying U The converses for binary symmetric and AWGN BCs are direct and do not explicitly identify U As before, by Fano s inequality So, we have H(W 1 Y1 n ) nr 1 P e (n) H(W 2 Y2 n ) nr 2 P e (n) + 1 = nǫ 1n + 1 = nǫ 2n nr 1 I(W 1 ;Y n 1 ) + nǫ 1n, nr 2 I(W 2 ;Y n 2 ) + nǫ 2n Gallager s 75th Birthday 17

18 A First attempt: U = W 2 (satisfies U X i (Y i1,y 2i )) So far so good. I(W 1 ;Y1 n ) I(W 1 ; Y1 n W 2 ) = I(W 1 ; Y1 n U) = = I(W 1 ; Y 1i U,Y i 1 1 ) chain rule ( H(Y1i U) H(Y 1i U,Y i 1 1, W 1 ) ) ( H(Y1i U) H(Y 1i U,Y i 1 1, W 1,X i ) ) I(X i ;Y 1i U) Gallager s 75th Birthday 18

19 A First attempt: U = W 2 (satisfies U X i (Y i1,y 2i )) I(W 1 ;Y1 n ) I(W 1 ; Y1 n W 2 ) = I(W 1 ; Y1 n U) = = I(W 1 ; Y 1i U,Y i 1 1 ) ( H(Y1i U) H(Y 1i U,Y i 1 1, W 1 ) ) ( H(Y1i U) H(Y 1i U,Y i 1 1, W 1,X i ) ) I(X i ;Y 1i U) So far so good. Now let s try the second inequality I(W 2 ; Y2 n ) = I(W 2 ; Y 2i Y2 i 1 ) = I(U;Y 2i Y2 i 1 ) But I(U;Y 2i Y i 1 2 ) is not necessarily I(U;Y 2i ), so U = W 2 does not work Gallager s 75th Birthday 19

20 Gallager: Ok, let s try U i = W 2, Y1 i 1 (U i X i (Y 1i,Y 2i )), so I(W 1 ; Y1 n W 2 ) I(X i ; Y 1i U i ) Now, let s consider the other term I(W 2 ; Y2 n ) I(W 2,Y2 i 1 ; Y 2i ) = I(W 2,Y i 1 2, Y i 1 1 ;Y 2i ) ( H(Y2i ) H(Y 2i W 2,Y2 i 1, Y1 i 1 ) ) But H(Y 2i W 2, Y i 1 2,Y i 1 1 ) is not in general equal to H(Y 2i W 2, Y i 1 1 ) Gallager s 75th Birthday 20

21 Gallager: Ok, let s try U i = W 2, Y1 i 1 (U i X i (Y 1i,Y 2i )), so I(W 1 ; Y1 n W 2 ) I(X i ; Y 1i U i ) Now, let s consider the other term I(W 2 ; Y2 n ) I(W 2,Y2 i 1 ; Y 2i ) = I(W 2,Y i 1 2, Y i 1 1 ;Y 2i ) ( H(Y2i ) H(Y 2i W 2,Y2 i 1, Y1 i 1 ) ) But H(Y 2i W 2, Y i 1 2,Y i 1 1 ) is not in general equal to H(Y 2i W 2, Y i 1 1 ) Key insight: Since capacity is the same as physically degraded, can assume X Y 1 Y 2, thus Y2 i 1 (W 2,Y1 i 1 ) Y 2i, and I(W 2 ; Y2 n ) I(U i ; Y 2i ) Eureka! Gallager s 75th Birthday 21

22 Note: Proof also works with U i = W 2, Y2 i 1 or U i = W 2,Y1 i 1, Y2 i 1 (both satisfy U i X i (Y 1i, Y 2i )). Let s try U i = W 2,Y2 i 1 First consider Now consider I(W 1 ;Y n 1 W 2 ) = I(W 2 ; Y n 2 ) = = I(W 2,Y i 1 2 ; Y 2i ) = I(W 1 ; Y 1i W 2,Y i 1 1 ) I(X i ; Y 1i W 2, Y i 1 1 ) I(U i ;Y 2i ) ( H(Y1i W 2, Y i 1 1 ) H(Y 1i X i ) ) ( H(Y1i W 2, Y i 1 1,Y i 1 2 ) H(Y 1i X i,u i ) ) ( H(Y1i W 2, Y i 1 2 ) H(Y 1i X i, U i ) ) = I(X i ; Y 1i U i ) Gallager s 75th Birthday 22

23 Thus we finally have nr 1 nr 2 I(X i ; Y 1i U i ) + nǫ 1n I(U i ;Y 2i ) + nǫ 2n By convexity and letting n, it follows that there exists U X Y 1 Y 2 such that R 1 I(X;Y 1 U), R 2 I(U;Y 2 ) Gallager s 75th Birthday 23

24 Thus we finally have nr 1 nr 2 I(X i ; Y 1i U i ) + nǫ 1n I(U i ;Y 2i ) + nǫ 1n By convexity and letting n, it follows that there exists U X Y 1 Y 2 such that R 1 I(X;Y 1 U) + ǫ 1n, R 2 I(U;Y 2 ) + ǫ 2n But there is still a problem here; U can have unbounded cardinality, so this is not a computable expression Gallager then argues that U min{ X, Y 1, Y 2 } suffices Remark: Proof extends to more than 2 receivers Gallager s 75th Birthday 24

25 Subsequent Broadcast Channel Results Direct application of Gallager converse: Physically degraded BC with feedback [El Gamal 1978] Product and sum of degraded and reversely degraded BC [Poltyrev 1977, El Gamal 1980] Gallager s 75th Birthday 25

26 Subsequent Broadcast Channel Results Direct application of Gallager converse: Physically degraded BC with feedback [El Gamal 1978] Product and sum of degraded and reversely degraded BC [Poltyrev 1977, El Gamal 1980] Gallager converse+csizsar Lemma: Less noisy [Korner, Marton 1975]: I(U;Y 1 ) I(U;Y 2 ) for all U X (Y,Z) BC with degraded message sets [Korner, Marton 1977] More capable [El Gamal 1979] (I(X;Y 1 ) I(X;Y 2 ) for all p(x)) Semi-deterministic BC [Gelfand-Pinsker 1980] Nair-El Gamal outer bound for BC [2006] All BC channel converses have used Gallager converse Gallager s 75th Birthday 26

27 Degraded Broadcast Channel With Feedback Assume at time i: X i = f i (W 1, W 2,Y i 1 1, Y i 1 2 ) Capacity region can now depend on joint pmf p(y 1,y 2 x) capacity of stochastically degraded BC with feedback not necessarily equal to that of physically degraded counterpart My first result was showing that feedback doesn t increase capacity of physically degraded BC [El Gamal 78] Converse: nr 2 I(W 2 ;Y2 n ) + nǫ n I(W 2, Y2 i 1 ;Y 2i ) + nǫ n I(W 2, Y i 1 1,Y i 1 2 ; Y 2i ) + nǫ n But because of feedback, Y i 1 2, (W 2, Y i 1 1 ),Y 2i do not necessarily form a Markov chain, thus cannot in general get rid of Y i 1 2 Gallager s 75th Birthday 27

28 Let U i = W 2, Y i 1 1,Y i 1 2 (U i X i Y 1i Y 2i ) Now conisder nr 1 I(W 1 ;Y1 n W 2 ) + nǫ n I(W 1 ;Y1 n,y2 n W 2 ) + nǫ n = = I(W 1 ; Y 1i, Y 2i W 2,Y i 1 1, Y i 1 2 ) + nǫ n I(W 1 ; Y 1i, Y 2i U i ) + nǫ n I(X i ; Y 1i, Y 2i U i ) + nǫ n = = I(X i ; Y 1i U i ) + I(X i ;Y 2i Y 1i,U i ) + nǫ n I(X i ; Y 1i U i ) + nǫ n, (U i X i Y 1i Y 2i ) In general, feedback can enlarge the capacity region of degraded BC Capacity region of degraded BC with feedback is not known Gallager s 75th Birthday 28

29 Broadcast Channel with Degraded Message Sets Consider a general DM broadcast channel (X,p(y 1,y 2 x), Y 1, Y 2 ) Common message W 0 is to be sent to both Y 1 and Y 2, while message W 1 is to be sent only to Y 1 Korner-Marton 77: The capacity region is given by the set C 1 of all (R 0, R 1 ) such that: R 0 I(U;Y 2 ) R 1 I(X;Y 1 U) Achievability: Superposition coding R 0 + R 1 I(X;Y 1 ), for some p(u,x) Converse: Gallager-type converse doesn t seem to work: The first inequality holds with either U i = W 0, Y2 i 1 or U i = W 0, Y1 i 1, Y2 i 1, but neither works for the second inequality The second inequality works with U i = W 0, Y1 i 1, but the first doesn t Gallager s 75th Birthday 29

30 BC with Degraded Message Sets A Gallager type converse works, but needs a new ingredient [El Gamal] Define the set C 2 of all (R 0, R 1 ) such that: R 0 min {I(U;Y 1 ),I(U;Y 2 )} R 0 + R 1 min {I(X;Y 1 ),I(X;Y 1 U) + I(U;Y 2 )}, for some p(u,x) It is easy to show that C 2 C 1 Weak converse for C 2 : Define U i = W 0,Y i 1 1, Y n 2(i+1)!! It is not difficult to show that { } R 0 1 n min I(U i ; Y 1i ), I(U i ;Y 2i ) +ǫ n, R 0 +R 1 1 n The difficult step is to show that R 0 + R 1 1 (I(X i ; Y 1i U i ) + I(U i ;Y 2i )) + ǫ n n I(X i ;Y 1i )+ǫ n Gallager s 75th Birthday 30

31 To show this, consider n(r 0 + R 1 ) I(W 1 ; Y1 n W 0 ) + I(W 0 ; Y2 n ) + nǫ n ( ) = I(W 1 ;Y 1i W 0, Y1 i 1 ) + I(W 0 ; Y 2i Y2(i+1) n ) + nǫ n + ( ) I(W 1,Y2(i+1) n ; Y 1i W 0, Y1 i 1 ) + I(W 0,Y2(i+1) n ; Y 2i) ( ) I(X i ;Y 1i U i ) + I(Y2(i+1) n ;Y 1i W 0,Y1 i 1 ) ( I(U i ;Y 2i ) I(Y i 1 1 ; Y 2i W 0,Y n 2(i+1) ) ) + nǫ n. + nǫ n Gallager s 75th Birthday 31

32 To show this, consider n(r 0 + R 1 ) I(W 1 ; Y1 n W 0 ) + I(W 0 ; Y2 n ) + nǫ n ( ) = I(W 1 ;Y 1i W 0, Y1 i 1 ) + I(W 0 ; Y 2i Y2(i+1) n ) + nǫ n + ( ) I(W 1,Y2(i+1) n ; Y 1i W 0, Y1 i 1 ) + I(W 0,Y2(i+1) n ; Y 2i) ( ) I(X i ;Y 1i U i ) + I(Y2(i+1) n ;Y 1i W 0,Y1 i 1 ) ( I(U i ;Y 2i ) I(Y i 1 1 ; Y 2i W 0,Y n 2(i+1) ) ) + nǫ n. Now using the Csiszar lemma [Csiszar-Korner 1978] I(Y1 i 1 ; Y 2i W 0, Y2(i+1) n ) = I(Y2(i+1) n ;Y 1i W 0,Y1 i 1 ), the second and fourth terms cancel and the converse is proved Note: This problem is open for more than two users + nǫ n Gallager s 75th Birthday 32

33 Tightest Bounds on Capacity of Broadcast Channel Inner bound [Marton 79]: Set of rate triples (R 0, R 1, R 2 ) such that R 0 min {I(W; Y 1 ),I(W; Y 2 )}, R 0 + R 1 I(W,U; Y 1 ), R 0 + R 2 I(W,V ; Y 2 ), R 0 + R 1 + R 2 min{i(w, U; Y 1 ) + I(V ; Y 2 W), I(U;Y 1 W) + I(W, V ; Y 2 )} I(U;V W), for some p(w, u, v, x) Outer bound [Nair-El Gamal 2006]: Set of rate triples (R 1, R 2 ) such that R 0 min {I(W; Y 1 ),I(W; Y 2 )} R 0 + R 1 I(W,U; Y 1 ) R 0 + R 2 I(W,V ; Y 2 ) R 0 + R 1 + R 2 min{i(w, U; Y 1 ) + I(V ; Y 2 W, U),I(W, V ; Y 2 ) + I(U;Y 1 W,V )}, for some p(u, v, w, x) = p(u)p(v)p(w u, v)p(x u, v, w) Gallager s 75th Birthday 33

34 Tightest Bounds for Independent Messages Inner bound: Set of all rate pairs (R 1,R 2 ) such that R 1 I(W, U;Y 1 ) R 2 I(W, V ; Y 2 ) R 1 + R 2 min{i(w, U;Y 1 ) + I(V ; Y 2 W), I(W, V ; Y 2 ) + I(U; Y 1 V )} I(U; V W), for some p(u,v,w,x) = p(w)p(u,v w)p(x u,v,w) Outer bound: Set of all rate pairs (R 1,R 2 ) such that R 1 I(W, U;Y 1 ) R 2 I(W, V ;Y 2 ) R 1 + R 2 min{i(w, U;Y 1 ) + I(V ;Y 2 W, U),I(W, V ; Y 2 ) + I(U;Y 1 W,V )}, for some joint distribution p(u, v, w, x) = p(u)p(v)p(w u, v)p(x u, v, w) Gallager s 75th Birthday 34

35 Broadcast Channel: Summary Marton region is tight for all classes of BC channels where capacity is known and contains all other achievable rate regions Nair-El Gamal region is tight for all classes of BC channels where capacity is known Uses Gallager converse+csizsar Lemma Capacity of BC is still not known in general Is there a single-letter characterization? Does it use auxiliary random variables? If so, then the Gallager converse is again likely to play a key role Gallager s 75th Birthday 35

36 Gallager Converse: Other Multi-User Channels/Sources Lossless source coding with side information [Ahlswede-Korner 1975] Lossy source coding with side information [Wyner-Ziv 1976] Channel with state known at encoder [Gelfand-Pinsker 1980] uses Gallager converse+csiszar Lemma MAC with partially cooperating encoders [Willems 1983] MAC with cribbing encoders [Willems-van der Meulen, 1985] Rate distortion when side information may be absent [Heegard-Berger 1985] Multi-terminal source coding with one distortion criterion [Berger-Yeung 1989] Relay without delay [El Gamal-Hassanpour 2005] and, many others... Gallager s 75th Birthday 36

37 Conclusion Gallager converse has had a huge impact on the development of multi-user information theory Gallager converse has had a huge impact on my work Gallager s 75th Birthday 37

38 Conclusion Gallager converse has had a huge impact on the development of multi-user information theory Gallager converse has had a huge impact on my work But, Bob, what does the identification U i = W 2, Y i 1 1 really mean? Gallager s 75th Birthday 38