UCSD ECE 255C Handout #12 Prof. Young-Han Kim Tuesday, February 28, Solutions to Take-Home Midterm (Prepared by Pinar Sen)

Transcription

1 UCSD ECE 255C Handout #12 Prof. Young-Han Kim Tuesday, February 28, 2017 Solutions to Take-Home Midterm (Prepared by Pinar Sen) 1. (30 points) Erasure broadcast channel. Let p(y 1,y 2 x) be a discrete memoryless broadcast channel with capacity region C. Suppose that this channel is followed immediately by an erasure channel p(y j y j) at each receiver j 1,2 that erases the symbols with probability p, that is, Y j { Y j which results in a new erasure broadcast channel with probability 1 p, e with probability p, p(y 1,y 2 x) p(y 1,y 2 x)p(y 1 y 1)p(y 2 y 2). Note that the erasures at the receivers are independent of each other. Suppose that the underlying broadcast channel p(y 1,y 2 x) is degraded. (a) Is the erasure broadcast channel degraded, less noisy, or more capable? (b) Find the capacity region of the erasure broadcast channel. (c) Now suppose that the underlying broadcast channel is not necessarily degraded and has the capacity region C. Comment on the capacity region of the erasure broadcast channel. Solution. (a) The erasure broadcast channel is (stochastically) degraded (therefore it is also less noisy and more capable). To see this, define Ỹ 1 as Ỹ 1 { Y 1 if Y 2 Y 2, e if Y 2 e. Then, Ỹ 1 {X x} p Y 1 X (ỹ 1 x), i.e., Ỹ 1 has the same conditional pmf as Y 1 given X and X Ỹ 1 Y 2 form a Markov chain. Therefore, by definition, the erasure broadcast channel p(y 1,y 2 x) is degraded (and hence less noisy as well as more capable). (b) Since it is degraded, by Theorem 5.2 in the text, the capacity region of the erasure broadcast channel p(y 1,y 2 x) is the set of rate pairs (R 1,R 2 ) such that R 1 I(X;Y 1 U), R 2 I(U;Y 2 ) for some pmf p(u,x). To express this region in terms of C, define binary random variables E j for j 1,2 as { 1 if Y j E j e, 0 if Y j Y j. 1

2 For any p(u,x), we have I(U;Y 2 )(a) I(U;Y 2,E 2), I(U;E 2 )+I(U;Y 2 E 2 ), (b) I(U;Y 2 E 2 0)P(E 2 0), (c) (1 p)i(u;y 2 ), where (a) follows since E 2 is a function of Y 2, (b) follows by the independence of E 2 and U, and (c) follows since E 2 is independent from (U,Y 2 ). Following similar steps, it is easy to verify that I(X;Y 1 U) (1 p)i(x;y 1 U). Thus, the capacity region of the erasure broadcast channel p(y 1,y 2 x) is the set of rate pairs (R 1,R 2 ) such that for some pmf p(u,x), which is equivalent to R 1 (1 p)i(x;y 1 U), R 2 (1 p)i(u;y 2 ) (1 p)c {(1 p)(r 1,R 2 ) : (R 1,R 2 ) C}. (c) We can assume without loss of generality that erasures occur simultaneously at both receivers (why?). Therefore, if a genie provides the locations of erasures to the sender, then the capacity region (with side information) is (1 p)c and hence the capacity region of the erasure broadcast channel (without genie), denoted by C, satisfies C (1 p)c. A more rigorous proof is too large to fit in the margin. 2. (40 points) Common-message broadcasting with state information. Consider the DM-BC with DM state p(y 1,y 2 x,s)p(s). Establish the common-message capacity C 0 (that is, the maximum achievable common rate R 0 when R 1 R 2 0) for the following settings. You are expected to prove achievability and the converse. (a) The state information is available only at decoder 1. (b) The state information is causally available at both the encoder and decoder 1. (c) The state information is noncausally available at the encoder and decoder 1. (d) The state information is causally available only at the encoder. Solution. (a) The common-message capacity is C 0 max p(x) min{i(x;y 1 S),I(X;Y 2 )}. This follows immediately by considering the augmented output Ỹ1 (Y 1,S) at receiver 1 and using Problem 5.9(c). Alternatively, this is proved step-by-step as follows. Achievability: Fix a pmf p(x) that attains C 0 and randomly and independently generate 2 nr0 codewords x n (m), m [1 : 2 nr0 ], each according to n p X(x i ). To send m, the encoder transmits x n (m). Decoder 1 finds the unique message ˆm 1 such that (x n (ˆm 1 ),s n,y1 n) T ǫ (n) (X,S,Y 1 ) and decoder 2 finds the unique message ˆm 2 such that (x n (ˆm 2 ),y2 n) T ǫ (n) (X,Y 2 ). By the LLN 2

3 and the packing lemma, the average probability of error for decoder 1 tends to 0 as n if R 0 < I(X;Y 1 S) δ(ǫ), and the average probability of error for decoder 2 tends to 0 as n if R 0 < I(X;Y 2 ) δ(ǫ). The proof of achievability follows by letting ǫ 0. Proof of the converse: Consider nr 0 H(M) I(M;Y n 1,S n )+H(M Y n 1,S n ) (a) I(M;Y1 n,s n )+nǫ n I(M;Y 1i,S i Y1 i 1,S i 1 )+nǫ n (b) I(M,Y i 1 1,S i 1,X i ;Y 1i,S i )+nǫ n I(X i ;Y 1i,S i )+nǫ n, where (a) follows by Fano s inequality, and (b) follows since (M,Y i 1 1,S i 1 ) (X i,s i ) Y 1i form a Markov chain and S i is independent of (M,Y i 1 1,S i 1,X i ). Similarly, nr 0 H(M) I(M;Y2 n )+H(M Y2 n ) I(M;Y2 n )+nǫ n I(M;Y 2i Y i 1 2 )+nǫ n I(M,Y i 1 2,S i 1,X i ;Y 2i )+nǫ n I(X i ;Y 2i )+nǫ n. The rest of the proof follows by introducing a time sharing random variable Q Unif[1 : n] independent of (M,X n,y1 n,y2 n,s n ), defining X X Q,Y 1 Y 1Q,Y 2 Y 2Q, and S S Q, and letting n. (b) The common-message capacity is C 0 max p(u),x(u,s) max p(u),x(u,s) min{i(u;y 1 S),I(U;Y 2 )} (1) min{i(x;y 1 S),I(U;Y 2 )}, where I(U;Y 1 S) I(X;Y 1 S) since X is a function of (U,S) and U (X,S) Y 1 form a Markov chain. To see this, first consider the case with causal state information available only at the encoder (see part (d)). Then, by the Shannon strategy, the common message capacity is max p(u),x(u,s) min{i(u;y 1 ),I(U;Y 2 )}. (The converse proof follows by adapting the case with a single receiver.) With state information available at receiver 1 as well, we use the output augmentation trick in part (a) to establish (1). Alternatively, this is proved step-by-step as follows. 3

4 Achievability: Fix p(u) and x(u,s) that attain C 0 and randomly and independently generate 2 nr0 sequences u n (m), m [1 : 2 nr0 ], each according to n p U(u i ). To send m given the state s i, the encoder transmits x(u i (m),s i ) at time i [1 : n]. Decoder 1 finds the unique message ˆm 1 such that (u n (ˆm 1 ),s n,y1 n) T ǫ (n) (U,S,Y 1 ) and decoder 2 finds the unique message ˆm 2 such that (u n (ˆm 2 ),y2) n T ǫ (n) (U,Y 2 ). By the LLN and the packing lemma, the average probability of error for decoder 1 tends to 0 as n if R 0 < I(U;Y 1 S) δ(ǫ), and the average probability of error for decoder 2 tends to 0 as n if R 0 < I(U;Y 2 ) δ(ǫ). The result follows by letting ǫ 0. Proof of the converse: Define U i (M,S i 1 ). Then, we have nr 0 H(M) I(M;Y1 n,sn )+H(M Y1 n,sn ) I(M;Y1 n,sn )+nǫ n (a) I(M;Y 1i,S i Y i 1 1,S i 1 )+nǫ n I(M,S i 1,Y i 1 1,X i 1 ;Y 1i,S i )+nǫ n I(M,S i 1,X i 1 ;Y 1i,S i )+nǫ n I(M,S i 1 ;Y 1i,S i )+nǫ n I(U i ;Y 1i,S i )+nǫ n, where (a) follows since (Y 1i,S i ) (X i 1,S i 1 ) Y i 1 1 form a Markov chain. Similarly, nr 0 H(M) I(M;Y2 n )+H(M Y 2 n ) I(M;Y2 n )+nǫ n I(M;Y 2i Y i 1 2 )+nǫ n I(M,S i 1,Y i 1 2,X i 1 ;Y 2i )+nǫ n I(M,S i 1,X i 1 ;Y 2i )+nǫ n I(M,S i 1 ;Y 2i )+nǫ n I(U i ;Y 2i )+nǫ n. The rest of the proof follows by introducing a time sharing random variable Q Unif[1 : n] independent of (M,X n,y1 n,y2 n,s n ), defining U (U Q,Q),Y 1 Y 1Q,Y 2 Y 2Q, and S S Q (note that U and S are independent), and letting n. 4

5 (c) The common-message capacity is C 0 max p(u s),x(u,s) max p(u s),x(u,s) min{i(u;y 1 S),I(U;Y 2 ) I(U;S)} min{i(x;y 1 S),I(U;Y 2 ) I(U;S)}. Achievability: Apply Gelfand Pinsker coding at the encoder. Fix a conditional pmf p(u s) and function x(u,s) that attain C 0, and let R > R 0. For each message m [1 : 2 nr0 ], generate a subcodebook C(m) consisting of 2 n( R R 0) randomly and independently generated sequences u n (l), l [(m 1)2 n( R R 0) +1 : m2 n( R R 0) ], each according to n p U(u i ). To send m given s n, the encoder chooses a sequence u n (l) C(m) such that (u n (l),s n ) T (n) ǫ (U,S). If no such sequence exists, it chooses l 1. Then, the encoder transmits x(u i (l),s i ) at time i [1 : n]. Let ǫ > ǫ. Decoder 1 finds the unique message ˆm 1 such that (u n (l),s n,y1 n) T ǫ (n) (U,S,Y 1 ) for some u n (l) C(ˆm 1 ), and decoder 2 finds the unique message ˆm 2 such that (u n (l),y2) n T ǫ (n) (U,Y 2 ) for some u n (l) C(ˆm 2 ). By the covering lemma, the conditional typicality lemma, and the packing lemma, the average probability of error tends to 0 as n if R R 0 > I(U;S)+δ(ǫ ), R < I(U;Y 1,S) δ(ǫ), and R < I(U;Y 2 ) δ(ǫ). By Fourier Motzkin elimination method, the average probability of error tends to 0 as n if R 0 < min{i(u;y 1 S) δ(ǫ) δ(ǫ ),I(U;Y 2 ) I(U;S) δ(ǫ) δ(ǫ )}. The proof of achievability follows by letting ǫ 0. Proof of the converse: Define U i (M,Si+1 n i 1,Y2 ). Then, we have nr 0 H(M) I(M;Y2 n )+H(M Y 2) I(M;Y2 n )+nǫ n (a) (b) I(M;Y 2i Y i 1 2 )+nǫ n I(M,Si+1 n i 1,Y2 ;Y 2i ) I(M,S n i+1,y i 1 2 ;Y 2i ) I(M,Si+1 n i 1,Y2 ;Y 2i ) I(U i ;Y 2i ) I(U i ;S i )+nǫ n, I(S n i+1 ;Y 2i M,Y i 1 2 )+nǫ n I(Y i 1 2 ;S i M,S n i+1)+nǫ n I(M,Si+1 n i 1,Y2 ;S i )+nǫ n where (a) follows by the Csiszár sum identity, and (b) follows by the independence of S i and (M,Si+1 n ). Similarly, nr 0 H(M) I(M;Y1 n S n )+H(M Y1 n,s n ) I(M;Y1 n Sn )+nǫ n I(M;Y 1i Y i 1 1,S n )+nǫ n I(M,Y i 1 1,S i 1,S n i+1,x i;y 1i S i )+nǫ n 5

6 I(X i ;Y 1i S i )+nǫ n. ByintroducingatimesharingrandomvariableQ Unif[1 : n]independentof(m,u n,y n 1,Y n 2,Sn ) and defining U (U Q,Q),Y 1 Y 1Q,Y 2 Y 2Q,S S Q, we have C 0 max min{i(x;y 1 S),I(U;Y 2 ) I(U;S)}, p(u s),p(x u,s) as n since X i is not a function of (U i,s i ). Now, we show that max p(u s), p(x u,s) min{i(x;y 1 S),I(U;Y 2 ) I(U;S)} max p(u s), x(u,s) min{i(x;y 1 S),I(U;Y 2 ) I(U;S)}. One direction of inequality ( ) is trivial. For the other direction, note that by the functional representation lemma, any p(x u,s) can be expressed as x(u,s,v), where V is a random variable independent of (U,S) with finite cardinality. Thus, by defining Ũ (U,V), we have max p(u s), p(x u,s) min{i(x;y 1 S),I(U;Y 2 ) I(U;S)} max p(u s)p(v), x(u,s,v) max p(u,v s), x(u,s,v) max p(ũ s), x(ũ,s) min{i(x;y 1 S),I(U;Y 2 ) I(U;S)} min{i(x;y 1 S),I(U,V;Y 2 ) I(U,V;S)} min{i(x;y 1 S),I(Ũ;Y2) I(Ũ;S)}. (d) The common-message capacity is C 0 max p(u),x(u,s) min{i(u;y 1 ),I(U;Y 2 )}, as argued before in part (b). We still provide a formal proof. Achievability: Fix apmf p(u) and x(u,s) that attain C 0 and randomlyand independently generate 2 nr0 sequences u n (m), m [1 : 2 nr0 ], each according to n p U(u i ). To send m given the state s i, the encoder transmits x(u i (m),s i ) at time i [1 : n]. Decoder 1 finds the unique message ˆm 1 such that (u n (ˆm 1 ),y1 n) T ǫ (n) (U,Y 1 ) and decoder 2 finds the unique message ˆm 2 such that (u n (ˆm 2 ),y2 n) T ǫ (n) (U,Y 2 ). By the LLN and the packing lemma, the average probability of error for decoder j 1,2 tends to 0 as n if R 0 < I(U;Y j ) δ(ǫ). The proof of achievability follows by letting ǫ 0. Proof of the converse: Define U i (M,S i 1 ). Then, we have nr 0 H(M) I(M;Yj n )+H(M Yj n ) I(M;Yj n )+nǫ n I(M;Y ji Y i 1 j )+nǫ n I(M,S i 1,Yj i 1,X i 1 ;Y ji )+nǫ n I(M,S i 1,X i 1 ;Y ji )+nǫ n I(M,S i 1 ;Y ji )+nǫ n 6

7 I(U i ;Y ji )+nǫ n for j 1,2. The rest of the proof follows by introducing the standard time sharing random variable and letting n. 3. (30 points) Broadcasting with side information. Consider the 3-receiver deterministic BC p(y 1,y 2,y 3 x) such that Y j y j (X), j 1,2,3, with message triple (M 1,M 2,M 3 ) uniformly distributed over [1 : 2 nr1 ] [1 : 2 nr2 ] [1 : 2 nr3 ] as depicted below. The sender wishes to communicate M j to receiver j {1,2,3}. We assume that each receiver has side information of some other messages, denoted as M(A j ) for receiver j. Find the capacity region for the following cases. You are expected to prove achievability and the converse. (a) A 1, A 2, A 3, that is, the receivers have no side information. (b) A 1 {2,3}, A 2 {1,3}, A 3 {1,2}, that is, the receivers already know all the messages other than they are receiving. (c) A 1, A 2 {1}, A 3 {1,2}. M(A 1 ) Y n 1 Decoder 1 ˆM 1 M(A 2 ) M 1,M 2,M 3 X n Encoder (y 1,y 2,y 3 )(x) Y n 2 Decoder 2 ˆM 2 M(A 3 ) Y n 3 Decoder 3 ˆM 3 Solution. (a) The capacity region consists of the rate triples (R 1,R 2,R 3 ) such that R 1 < H(Y 1 ), R 2 < H(Y 2 ), R 3 < H(Y 3 ), R 1 +R 2 < H(Y 1,Y 2 ), R 1 +R 3 < H(Y 1,Y 3 ), R 2 +R 3 < H(Y 2,Y 3 ), R 1 +R 2 +R 3 < H(Y 1,Y 2,Y 3 ) for some pmf p(x). Achievability follows by Marton s inner bound with U j Y j y j (X). For the converse, consider nr j H(M j ) 7

8 I(M;Yj n )+H(M Y j n ) I(M;Yj n )+nǫ n H(Yj n )+nǫ n H(Y ji )+nǫ n for j 1,2,3. Similarly, for j k and j,k {1,2,3}, we have n(r j +R k ) H(M j )+H(M k M j ) I(M j ;Y n j )+H(M j Y n j )+I(M k ;Y n k M j )+H(M k Y n k,m j ) I(M j ;Yj n )+H(M j Yj n )+I(M k ;Yk n M j )+H(M k Yk n ) I(M j ;Yj n )+I(M k;yk n M j)+2nǫ n I(M j,m k ;Yj n,y k n )+2nǫ n H(Y ji,y ki )+nǫ n. For the last inequality in the capacity region, consider n(r 1 +R 2 +R 3 ) H(M 1 )+H(M 2 M 1 )+H(M 3 M 1,M 2 ) I(M 1 ;Y1 n )+H(M 1 Y1 n )+I(M 2 ;Y2 n M 1 )+H(M 2 Y2 n )+ +I(M 3 ;Y3 n M 1,M 2 )+H(M 3 Y3 n ) I(M 1,M 2,M 3 ;Y1 n,y2 n,y3 n )+3nǫ n H(Y 1i,Y 2i,Y 3i )+nǫ n. The rest of the proof follows by introducing the standard time sharing random variable and letting n. (b) The capacity region consists of the rate triples (R 1,R 2,R 3 ) such that R 1 < H(Y 1 ), R 2 < H(Y 2 ), R 3 < H(Y 3 ) for some pmf p(x). Achievability: Fix a pmf p(x). Randomly and independently generate 2 n(r1+r2+r3) sequences x n (m 1,m 2,m 3 ), m j [1 : 2 nrj ], for j 1,2,3, each according to n p X(x i ). To send the message triple (m 1,m 2,m 3 ), the encoder transmits x n (m 1,m 2,m 3 ). Decoder 1 finds the unique message ˆm 1 such that (x n (ˆm 1,m 2,m 3 ),y1 n) T ǫ (n) (X,Y 1 ), decoder 2 finds the unique message ˆm 2 such that (x n (m 1, ˆm 2,m 3 ),y2 n) T ǫ (n) (X,Y 2 ), and decoder 3 finds the unique message ˆm 3 such that (x n (m 1,m 2, ˆm 3 ),y3) n T ǫ (n) (X,Y 3 ). By the LLN and the packing lemma, the average probability of errortends to 0 as n if R j < I(X;Y j ) δ(ǫ). The proofof achievability follows by letting ǫ 0. Proof of converse: Consider nr j H(M j M(A j )) I(M j ;Y n j M(A j))+h(m j Y n j,m(a j)) 8

9 I(M j ;Yj n M(A j))+nǫ n H(Yj n )+nǫ n H(Y ji )+nǫ n for j 1,2,3. The rest of the proof follows by standard arguments. (c) The capacity region consists of the rate triples (R 1,R 2,R 3 ) such that R 1 < H(Y 1 ), R 2 < H(Y 2 ), R 3 < H(Y 3 ), R 1 +R 2 < H(Y 1,Y 2 ), R 1 +R 3 < H(Y 1,Y 3 ), R 2 +R 3 < H(Y 2,Y 3 ), R 1 +R 2 +R 3 < H(Y 1,Y 2,Y 3 ) for some pmf p(x). Notice that the capacity is the same as part a) and does not increase with this set of receiver side information. Achievability follows from part (a). For the converse, consider for j 1,2,3. Now, consider Similarly, we have and nr j H(M j M(A j )) I(M;Yj n M(A j ))+H(M Yj n,m(a j )) I(M;Yj n M(A j))+nǫ n H(Yj n )+nǫ n H(Y ji )+nǫ n n(r 1 +R 2 ) H(M 1 )+H(M 2 M 1 ) I(M 1 ;Y1 n )+H(M 1 Y1 n )+I(M 2 ;Y2 n M 1 )+H(M 2 Y2 n,m 1 ) I(M 1 ;Y1 n )+I(M 2;Y2 n M 1)+2nǫ n I(M 1,M 2 ;Y1 n,y2 n )+2nǫ n H(Y 1i,Y 2i )+2nǫ n. n(r 2 +R 3 ) H(M 2 M 1 )+H(M 3 M 1,M 2 ) I(M 2 ;Y2 n M 1)+H(M 2 Y2 n,m 1)+I(M 3 ;Y3 n M 1,M 2 )+H(M 3 Y3 n,m 1,M 2 ) I(M 2,M 3 ;Y2 n,y3 n M 1 )+2nǫ n H(Y 2i,Y 3i )+2nǫ n n(r 1 +R 3 ) H(M 1 )+H(M 3 M 1,M 2 ) 9

10 I(M 1 ;Y1 n )+H(M 1 Y1 n )+I(M 3;Y3 n M 1,M 2 )+H(M 3 Y3 n,m 1,M 2 ) I(M 1,M 2,M 3 ;Y1 n,y3 n )+2nǫ n H(Y 1i,Y 3i )+2nǫ n. Finally, for the last term, consider n(r 1 +R 2 +R 3 ) H(M 1 )+H(M 2 M 1 )+H(M 3 M 1,M 2 ) I(M 1 ;Y1 n )+H(M 1 Y1 n )+I(M 2;Y2 n M 1)+H(M 2 Y2 n,m 2)+ +I(M 3 ;Y3 n M 1,M 2 )+H(M 3 Y3 n,m 1,M 2 ), I(M 1,M 2,M 3 ;Y1 n,y 2 n,y 3 n )+3nǫ n H(Y 1i,Y 2i,Y 3i )+3nǫ n. The rest of the proof follows by standard arguments. 10