Note that the new channel is noisier than the original two : and H(A I +A2-2A1A2) > H(A2) (why?). min(c,, C2 ) = min(1 - H(a t ), 1 - H(A 2 )).

Size: px

Start display at page:

Download "Note that the new channel is noisier than the original two : and H(A I +A2-2A1A2) > H(A2) (why?). min(c,, C2 ) = min(1 - H(a t ), 1 - H(A 2 ))."

Stephanie Henry
5 years ago
Views:

1 l I ~-16 / (a) (5 points) What is the capacity Cr of the channel X -> Y? What is C of the channel Y - Z? (b) (5 points) What is the capacity C 3 of the cascaded channel X -3 Z? (c) (5 points) A ow let. its actively intervene between channels 1 and, rather than passively transmitting yr`. What is the capacity of channel 1 followed by channel if you are allowed to decode the output y" of channel 1 and then re-encode it as y" for transmission over channel? Cascaded BSCs (a) C, is just a capacity of a BSC(A 1 ). Thus, C, = I -H(A 1 ). Similarly. C = 1-H(A ). (b) The cascaded channel is also a BSC. Since the new BSC has a crossover probability of a, (1 - X ) + (1 - A! )A = a, + A - A 1 A. C3 = 1 - H(a, + A - A 1 A ). Note that the new channel is noisier than the original two : and H(A I +A-A1A) > H(A) (why?). H(A,+A-A1A) > TI(AI) (c) Since we are allowed to decode the intermediate outputs and re-encode theta prior to the second transmission, any rate less than both C t and C can be achievable and at the same time any rate greater than either C, or C will cause P(O 1 exponentially_. Hence. the overall capacity is the minimum of two capacities, min(c,, C ) = min(1 - H(a t ), 1 - H(A )). 3. Gaussian Mixture Channel Consider a memoryless Gaussian mixture channel Y = X + Z, where X is the transmitted signal with a fixed power P. Y is the received signal, and Z is an additive noise defined as follows : f B= 1 with prob z where B= if 0 = with prob z Z i N(0, (T?), Z - A (0., a?) ; Zt, Z. 0 are independent ; and o- < az. (a) (5 points) Suppose that both the transmitter and the receiver know the value of 0 in each time instant prior to transmission. What is the capacity of the channel? (b) (5 points) How would you achieve the capacity found in (c) (5 _points) Now. suppose that neither the transmitter nor the receiver knows the value of 0. Suppose further that a; = 0 and a' = 1. What is the capacity of the channel? (d) (5 points) How would you achieve the capacity found in (a)? (c)?

2 Gaussian Multiple Access Channel with Multiple Antennas Consider a Gaussian multiple access channel with vector input and vector output : 0 Y=H,X r +H7 X +Z, where X1, X, Y and Z are n x 1 vectors, H 1. H are a x n matrices, and Z -N(0, KC). Let X r and X be Gaussian independent vectors with fixed covariance matrices K L and K respectively, i.e. X r - N(0, K I ) and X - N(0, K ). (a) (5 points) What is the maximum rate at which X r can communicate with Y? (b) (5 points) Assuming that Xi is communicating with Y at the rate found in (a), what is the maximum additional rate at, which X can communicate with Y? (c) ;'5 points) Sketch the capacity region of the multiple access channel. Label all corner points. Gaussian Multiple Access Channel with Multiple Antennas (a) With a fixed K1 : H1K,H7+K ; C ' = 1 log K I (b) With a fixed K and with H i K H1 as noise : C 1 IH i K1 H,' +H K H'+K I = log H K, HT + Kz ', (c) The capacity region is : R, R < 1 l log 1 HK1 H,±KI log Kzl HKH' + K, K i R, + R; < < 1, og H j K i H; +H KH; +K- I K,' 6. Cooperative receivers in a broadcast channel Consider a broadcast channel with the following transition probability p(y n, y, x) : If X = 0. then 1 Y ifx=1.then, L 5

3 Rate Distortion Theory 1-a 0 1-a 1 Figure 10.1 : Joint distribution for erasure rate distortion of a binary source error distortion, show that 1 1 h(x) - log(ared) < R(D) < log (io.7tr} For the upper bound, consider the joint distribution shown in Figure 10.. Are Gaussian random variables harder or easier to describe than other random variables with the same variance? Bounds on the rate distortion function for squared error distortion. We assume that X has zero mean and variance a. To prove the lower bound, we use the same techniques as used for the Guassian rate distortion function. Let (X, k) be random variables such that E(X -X) < D. Then I(X ; X) h(x) - h(x}x) (10.78) = h(x) - h(x - XjX) (10.79) > h(x) - h(x - X) (10.80) > h(x) - h(n(0, E(X - x))) (10.81) = h(x) - a log(vre)e(x - X) (10.8) > h(x) - log(ire)d. (10.83) To prove the upper bound, we consider the joint distribution as shown in Figure 10.3,

4 49 50 Rate Distortion Theory P D a o -D, (x + Z) Figure 10. : Joint distribution for upper bound on rate distortion function. (10.77) aussian he same (10.78) (10.79) (10.80) (10.81 (10.8) (10.83) Figure 10.3 : Joint distribution for upper bound on rate distortion function

5 Rate Distortion Theory and calculate the distortion and the mutual information between X and a -D X = a (X + Z), we have _ \ E(X X) ECDX aa D Z 1 (D )EX'+ ( aad) / I1 EZ - (D) (a-d a a ) ada - D D, since X and Z a e independent and zero mean. Also the mutual information is Now I(X ;X) = h(x)-h(xjx) (la. _ h(x)-h( a D Z). (19. a EX a = -D )/ E(X + Z) Or /a-da +/\ (EX" + EZ a ) (aad/ = a -D. ( a+aod ) (10.94) Hence, we have I(X ;x) h(x) -h( a- D Z) (10.95) h(x) - h(z) - log a ad (10.96) < h(ai(0, a - D)) - log(ue) ada D - log a ad (10.97) z -D) log(~re)(a -D)- ~log(7re) a- log ~a a (10.98) - 01 log D, (10.99)

6 Network Information Theory 4. Multiplicative multiple access channel. Find and sketch the cap the multiplicative multiple access channel with X 1 E {0,1}. X e {1..3}, and Y=X 1 X. Multiplicative multiple access channel. Since Y = X1X, if X1 = 0, Y = 0 and we receive no information about < When X1 = 1, Y = X, and we can decode X perfectly, thus we can achieve a R1 = 0, R = log 3. Let a be the probability that X 1 = 1. By symmetry, X should have an unifo distribution on { 1,, 31. The capacity region of the multiple access channel I(X1 ;YIX) = H(X11X) - H(X1IY,X)=H(Xi)=H(a) I(X ;YIX 1 ) = H(YIXI )=ah(x)=alog3 I(X1,X:Y) = H(Y)=11(1-0,3,3,3)=H(a)+alog3 ( (15.198) (15.199) - Thus the rate region is characterized by the equations R1 < H(a) (15.00) R < alog3 (15.01) where a varies from 0 to 1 The maximum value for R1 occurs for a =. The maximum value for the sum of the rates occurs (by calculus) at a = a 5. Distributed data compression. Let Z 1i Z,Z3 be independent Bernoulli (p). Find the Slepian-Wolf rate region for the description of (XI, X,X3 ) where X1 - Z1 X = Z 1 + Z X 3 = Z1 + Z + Z3.

7 30 Gaussian channel 14. Additive noise channel. Consider the channel Y = X + Z, where X is the transmitted signal with power constraint P, Z is independent additive noise, and Y is the received signal. Let J 0 with prob. Z Z', with prob. t o ' where Z" - N(0, N). Thus Z has a mixture distribution which is the mixture of a Gaussian distribution and a degenerate distribution with mass 1 at 0. (a) What is the capacity of this channel? This should be a pleasant surprise. (b) How would you signal in order to achieve capacity? Additive Noise channel The capacity of this channel is infinite, since at the times the noise is 0 the output is exactly equal to the input, and we can send an infinite number of bits. To send information through this channel, just repeat the same real number at the input. When we have three or four outputs that agree, that should correspond to the points where the noise is 0, and we can decode an infinite number of bits. 15. Discrete input continuous output channel. Let Pr{X = 1} = p, Pr{X = 0} _ 1 - p, and let Y = X + Z, where Z is uniform over the interval [0, a], a > 1, and Z is independent of X. (a) Calculate I(X;Y) = H(X) - H(XIY). (b) Now calculate I(X ; Y) the other way by I(X ;Y) = h(y) - h(y{x). (c) Calculate the capacity of this channel by maximizing over p (a) Since Discrete input Continuous Output channel 1 0<y<a f (YIX = 0) _ m 0 otherwise 9.9 and Therefore, (1 - p)a 0<y<1 f(yix= 1)= a l 1 < y < a (9.93) pa a < y < 1 + a (1 f(y) = 1<y<a (9.94) a l a<y<1+a Pa - P)a 0<y<1

8 Gaussian channel 3 (b) H(X) = H(p). H(XIY = y) is nonzero only for 1 < y < a, and by Bayes rule, conditioned on Y, the probabilty that X = 1 is P(X = 1) f (Y1 X = 1) P(X = 1IY = y) = P(X = 1)f (yix =1) + P(X = 0)f (yi X = 0) = p (9.95) and hence H(XIY) = P(1 < Y < a)h(p) = a'h(p). Therefore I(X ;Y) H(X) - H(X IY) = ah(p). (c) f (YI X = 0) - U(0, a), and hence h(yix = 0) = log a, and similarly for X = so that h(yix) = log a. The differential entropy h(y) can be calculated from (9.94) as h(y) = -Z1(1-p) a log 1 a 1+a a pdy- alog ady- a log ad,(9.96) J J a (-p log p - (1 - p) log(1 - p)) +logo (9.97) a H(p) + log a (9.98) and again I(X ; Y) = h(y) - h(yix) = ah(p). (d) The mutual information is maximized for p = 0.5, and the corresponding capacity of the channel is o. 16. Gaussian mutual information Suppose that (X, Y, Z) are jointly Gaussian and that X - Y -. Z forms a Markov chain. Let X and Y have correlation coefficient pi and let Y and Z have correlation coefficient P. Find I(X ; Z). Solution: Gaussian Mutual Information (Repeat of problem 8.9) First note that we may without any loss of generality assume that the means of X, Y and Z are zero. If in fact the means are not zero one can subtract the vector of means without affecting the mutual information or the conditional independence of X, Z given Y. Let A = ox QxQ QxQZPxz az be the covariance matrix of X and Z. We can now use Eq. (9.93) and Eq. (9.94) to compute I(X ; Z) = h(x) + h(z) - h(x, Z) log (1rea ) + log (ireai) - log(1 - Psz) log (7reIA~)

9 366 Network Information Theory Converse for deterministic broadcast channel. We can derive this bound from single user arguments. sender can send information to receiver 1 is less than The maximum rate that the R1 :5 I(X ;Yl) = H(Yi) - H(Yi IX) = H(Yi) (15.6) since the channel is deterministic and therefore H(Yi IX) = H(YIX) = 0. Similarly, R :5 H(Y). Also, if the receivers cooperated with each other, the capacity Ri + R < I(X;Y1,Y ) = H(Y1,Y) (15.7) since the sum of rates to the two receivers without cooperation cannot be greater than the single user capacity of a channel from X to (Y1, Y). 34. Multiple access channel Consider the multiple access channel Y = X 1+X (mod 4), where X1 E 10, 1,, 3}, X E {0 ' 11. (a) Find the capacity region (R1, R). (b) What is the maximum throughput R1 + R? Multiple access channel (a) The MAC capacity region is given by the standard set of equations which reduce as follows since there is no noise : R1 < I(Xi ;YI X) = H(YIX) - H(YIX1,X) = H(YIX) = H(X1) R < I(X ;YI X1) = H(YI Xi) - H(YIX1, X) = H(YI Xi) = H(X) Ri + R < I(Xi, X ;Y) = H(Y) - H(YIX1, X) = H(Y) Since entropy is maximized under a uniform distribution over the finite alphebet, R1 < H(Xl ) <, R < H(X ) < 1, and Rl + R < H(Y) <. Further, if X1 - uni f (0, 1,, 3), and X - uni f (0,1) then Y - uni f (0, 1,, 3), so the upper bounds are achieved. This gives the capacity region in Figure (b) The throughput of R1 + R < by the third constraint above, and is achieved at many points including when R1 = and R = 0. So the maximum throughput is R1+R=. 35. Distributed source compression Let Z1 1, p 0, q, p 4,

ECE Information theory Final

ECE Information theory Final ECE 776 - Information theory Final Q1 (1 point) We would like to compress a Gaussian source with zero mean and variance 1 We consider two strategies In the first, we quantize with a step size so that the