National University of Singapore Department of Electrical & Computer Engineering Examination for EE5139R Information Theory for Communication Systems (Semester I, 2014/15) November/December 2014 Time Allowed: 3.0 hours INSTRUCTIONS FOR CANDIDATES: ˆ This paper contains FOUR (4) questions, printed on FIVE (5) pages. ˆ Answer all questions. ˆ Programmable calculators are NOT allowed. ˆ Electronic communicating devices MUST be turned off and inaccessible throughout the examination. They CANNOT be used as calculators, timers or clocks. ˆ You are allowed to bring SEVEN (7) HANDWRITTEN A4 size sheets. You may write on both sides of the sheet. No other material is allowed.
EE5139R Communication Systems Page 2 of 5 Problem 1 1(a) (7 points) State whether the following statement is TRUE or FALSE: There exists a discrete memoryless channel (DMC) with a binary (i.e., X = 2 symbols) input alphabet and a ternary (i.e., Y = 3 symbols) output alphabet such that its capacity is equal to C = 1.5 bits/channel use. Please justify your answer carefully. 1(b) (8 points) Prove a list decoding version of Fano s inequality. In particular, show that if (W, Y ) W Y are jointly distributed as P W,Y, and L(Y ) is a subset of W of size l 1, the probability of error P e := Pr(W / L(Y )) can be lower bounded as P e H(W L(Y )) 1 log l log W l. l Hint: Define an error random variable E which takes the value of 1 when W / L(Y ) and 0 otherwise. Then expand H(W, E L(Y )) in two different ways using the chain rule for joint entropy. 1(c) (3 points) For a DMC P Y X (y x) over n channel uses, find an upper bound for 1 n I(Xn ; Y n ) in terms of the channel capacity C = max PX I(X; Y ). State where you used the assumption of memorylessness in your derivations. 1(d) (7 points) A (2 nr, 2 nl, n)-list code for a DMC P Y X (y x) with capacity C consists of an encoder that assigns a codeword x n (w) to each message w W := 1,..., 2 nr } and a decoder that upon receiving y n tries to finds the list of messages L(y n ) W = 1,..., 2 nr } of size L(y n ) 2 nl that contains the transmitted message. An error occurs if the list does not contain the transmitted message W, i.e., P (n) e := Pr(W / L(Y n )). A rate-list exponent pair (R, L) is said to be achievable if there exists a sequence of (2 nr, 2 nl, n)-list codes with P (n) e 0 as n. Show using parts (b) and (c) that there exists an upper bound on R called R + such that every sequence of (2 nr, 2 nl, n) list codes with P e (n) 0 (i.e., every (R, L) pair that is achievable) must satisfy R R +. Find this upper bound R + in terms of C and L.
EE5139R Communication Systems Page 3 of 5 Problem 2 2(a) (10 points) Consider the Gaussian channel shown above, in which the transmitted signal X with E[X 2 ] = P is received by two antennas with Y 1 = X + Z 1, Y 2 = X + Z 2 where Z 1 and Z 2 are independent, and E[Zi 2] = σ2 i with σ2 1 < σ2 2. Moreover, the signals at the two antennas are combined as Y = αy 1 + (1 α)y 2 before decoding where 0 α 1. Find the capacity of the channel for a given α. Please provide the units of capacity. 2(b) (10 points) The following parts are separate from part (a). Now assume that you have two parallel Gaussian channels with inputs X 1 and X 2 and noises Z 1 and Z 2, respectively. Assume that the noise powers are E[Z1 2] = 0.3 and E[Z2 2 ] = 0.6, while the total available power is P = E[X1 2] + E[X2 2 ] = 0.1. Find the optimal power allocation (waterfilling) and corresponding capacity. Please provide the units of capacity. Hint: You may assume the waterfilling conditions derived in class (without proving them again). Separately consider the cases P 1, P 2 > 0 and one of the powers is 0. 2(c) (5 points) Now let S n be a binary memoryless source (BMS) with entropy 0.15 bits per source symbol. Is it possible to transmit S n over the channel in part (b)? If it is possible, please explain how to achieve sending the BMS over the Gaussian channel. If it is not possible, explain clearly why not. Hint: Even if you didn t get part (b), you can still do (c) by assuming the capacity of the channel in part (c) is some value C > 0.
EE5139R Communication Systems Page 4 of 5 Problem 3 In this problem we use minimum distance decoding to establish achievability of the capacity for a Binary Symmetric Channel (BSC) with crossover probability p < 1/2. Define the Hamming distance d(x n, y n ) between two binary sequences x n, y n 0, 1} n as the number of positions where they differ, i.e., n n d(x n, y n ) = i 1, 2,..., n} : x i y i } = 1x i y i } = x i y i Here x y = 0 if x = y and x y = 1 if x y. We fix a rate R and generate 2 nr codewords X n (w) each from the uniform distribution on 0, 1} n so each X i (w) is uniform on 0, 1}, i.e., Pr(X i (w) = 0) = Pr(X i (w) = 1) = 1/2. The random codebook is C = X n (1), X n (2)..., X n (2 nr )}. For any codebook, the decoder chooses message ŵ such that the Hamming distance of X n (ŵ) to the channel output Y n is minimized, i.e., d(x n (ŵ), Y n ) < d(x n ( w), Y n ), for all w ŵ. Advice: Each part below is designed so that it may be done independently of earlier parts. So if you get stuck, move on. 3(a) (2 points) Assume that message W = 1 was sent. Argue that the probability of error averaged over the random code is P e (n) := Pr ( A ) 2nR W = 1, where A := w=2 i=1 i=1 d(x n (w), Y n ) d(x n (1), Y n ) } Hint: How does an error occur under the minimum distance decoding rule above? 3(b) (4 points) Recall the law of total probability Pr(A) = Pr(A, E) + Pr(A, E c ). Let Now show the upper bound E := d(x n (1), Y n ) > n(p + ɛ)}, for some 0 < ɛ < 1 2 p P (n) e Pr ( d(x n (1), Y n ) > n(p + ɛ) W = 1 ) + 2 nr Pr ( d(x n (2), Y n ) n(p + ɛ) W = 1 ) (1) Hint: Use Bayes rule and the fact that probabilities are no larger than one. 3(c) (2 points) Let Z i = Y i X i (1). What is the distribution of Z i? 3(d) (4 points) Write the first term on the right-hand-side of (1), namely Pr(d(X n (1), Y n ) > n(p + ɛ) W = 1) in terms of the Z i random variables of part (c) and argue using Chebyshev s inequality that this first term tends to zero as n tends to infinity. 3(e) (2 points) Let U i = Y i X i (2). What is the distribution of U i? 3(f) (8 points) Write the second probability in (1), namely Pr ( d(x n (2), Y n ) n(p + ɛ) W = 1 ) in terms of the U i random variables of part (e) and hence show using Sanov s theorem that Pr ( d(x n (2), Y n ) n(p + ɛ) W = 1 ) (n + 1) 2 2 nd(p+ɛ 1 2 ) where D(p q) is the relative entropy between a Bernoulli-p and Bernoulli-q random variable. 3(g) (3 points) Using the above, show that the capacity of the BSC with crossover probability p is at least 1 H b (p) where H b (p) = p log p (1 p) log(1 p) is the entropy of a Bernoulli-p random variable. Hint: Use parts (d) and (f) to show that P (n) e goes to zero under some condition on R.
EE5139R Communication Systems Page 5 of 5 Problem 4 Let X n := (X 1, X 2,..., X n ) be independent (but not identically distributed) discrete random variables on a finite set X, drawn according to the following rule: P1 (x) i odd Pr(X i = x) = P 2 (x) i even (2) Let Q be any other distribution on X (not the same as P 1 or P 2 ). Let r(n) be the remainder when you divide n by 2, i.e., r(n) = 0 if n is even and 1 otherwise. 4(a) (7 points) Fix ɛ > 0. Define the relative entropy typical set A (n) ɛ (P 1, P 2 Q) := x n : D ɛ < 1 n log P } 1(x 1, x 3,..., x n r(n)+1 )P 2 (x 2, x 4,..., x n r(n) ) < D + ɛ Q(x 1, x 2,..., x n ) for some D > 0. Note that 1, 3,..., n r(n) + 1 is simply the collection of odd numbers up to and including n. Similarly, 2, 4,..., n r(n) is the collection of even numbers up to and including n. Under hypothesis H 0, let X n be distributed as in (2). Find D in terms of P 1, P 2, Q such that ( Pr X n A (n) ɛ (P 1, P 2 Q) ) H 0 1 Hint: By independence, P 1 (X 1, X 3,..., X n r(n)+1 ) = P 1 (X 1 )P 1 (X 3 )... P 1 (X n r(n)+1 ) and similarly for the other probabilities where X n follows (2). Apply the weak law of large numbers or Chebyshev s inequality. 4(b) (10 points) Under hypothesis H 1, let X n be distributed i.i.d. Q. Find the largest E such that E satisfies ( Pr X n A (n) ɛ (P 1, P 2 Q) ) H 1 2 ne for every integer n 1. The number E should be stated in terms of D and ɛ. 4(c) (8 points) Now consider the binary hypothesis test H 0 : X n according to (2) H 1 : X n i.i.d. Q Let A n X n be an acceptance region for hypothesis H 1 and let the probabilities of error be α n (A n ) := Pr(A c n H 0 ), β n (A n ) := Pr(A n H 1 ). Define Find an upper bound for in terms of P 1, P 2, Q and ɛ. β ɛ n := min A n X n :α n(a n)<ɛ β n 1 lim n n log βɛ n END OF PAPER