Chapter 9. Gaussian Channel

Size: px

Start display at page:

Download "Chapter 9. Gaussian Channel"

Amber Lamb
5 years ago
Views:

1 Chapter 9 Gaussian Channel Peng-Hua Wang Graduate Inst. of Comm. Engineering National Taipei University

2 Chapter Outline Chap. 9 Gaussian Channel 9.1 Gaussian Channel: Definitions 9.2 Converse to the Coding Theorem for Gaussian Channels 9.3 Bandlimited Channels 9.4 Parallel Gaussian Channels 9.5 Channels with Colored Gaussian Noise 9.6 Gaussian Channels with Feedback Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 2/31

3 9.1 Gaussian Channel: Definitions Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 3/31

4 Introduction Y i = X i +Z i, Z i N(0,N) X i : input,y i :output,z i : noise. Z i is independent ofx i. Without further constraint, the capacity of this channel may be infinite. If the noise variancen is zero, the channel can transmit an arbitrary real number with no error. If the noise variancen is nonzero, we can choose an infinite subset of inputs arbitrary far apart, so that they are distinguishable at the output with arbitrarily small probability of error. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 4/31

5 Introduction The most common limitation on the input is an energy or power constraint. We assume an average power constraint. For any codeword (x 1,x 2,...,x n ) transmitted over the channel, we require that 1 n x 2 i P n i=1 Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 5/31

6 Information Capacity Definition 1 (Capacity) The information capacity of the Gaussian channel with powerp is C = max f(x):e[x 2 ] P I(X;Y) We can calculate the information capacity as follows. I(X;Y) = h(y) h(y X) = h(y) h(x +Z X) = h(y) h(z X) = h(y) h(z) 1 2 log2πe(p +N) 1 2 log2πen = 1 ( 2 log 1+ P ) N Note thate[y 2 ] = E[(X +Z) 2 ] = P +N and the entropy of gaussian with varianceσ 2 is 1 2 log2πeσ2. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 6/31

7 Information Capacity Therefore, the information capacity of the Gaussian channel is C = max E[X 2 ] P I(X;Y) = 1 2 log ( 1+ P N and the equality holds whenx N(0,P). Next, we will show that this capacity is achievable. ) Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 7/31

8 Code for Gaussian Channel Definition 2 ((M,n) code for Gaussian Channel) An(M,n) code for the Gaussian channel with power constraintp consists the following: 1. An index set{1,2,...,m}. 2. An encoding functionx : {1,2,...,M} X n, yielding codewordsx n (1),x n (2),...,x n (M), satisfying the power constraintp 1 n n x 2 i(w) P, w = 1,2,...,M. i=1 3. A decoding functiong : Y n {1,2,...,M}. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 8/31

9 Definitions Definition 3 (Conditional probability of error) λ i = Pr(g(Y n ) i X n = x n (i)) = g(y n ) i p(y n x n (i)) = y n p(y n x n (i))i(g(y n ) i) I( ) is the indicator function. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 9/31

10 Definitions Definition 4 (Maximal probability of error) λ (n) = max λ i i {1,2,...,M} Definition 5 (Average probability of error) P (n) e = 1 M M i=1 λ i The decoding error is Pr(g(Y n ) W) = M i=1 Pr(W = i)pr(g(y n ) i W = i) If the indexw is chosen uniformly from{1,2,...,m}, then P (n) e = Pr(g(Y n ) W). Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 10/31

11 Definitions Definition 6 (Rate) The raterof an(m,n) code is R = logm n bits per transmission Definition 7 (Achievable rate) A rateris said to be achievable for a Gaussian channel with a power constraintp if there exists a ( 2 nr,n) code with codewords satisfying the power constraint such that the maximal probability of errorλ (n) tends to0asn. Definition 8 (Channel capacity) The capacity of a channel is the supremum of all achievable rates. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 11/31

12 Capacity of a Gaussian Channel Theorem 1 (Capacity of a Gaussian Channel) The capacity of a Gaussian channel with power constraintp and noise variancen is 1 2 log ( 1+ P N ) bits per transmission. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 12/31

13 Sphere Packing Argument Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 13/31

14 Sphere Packing Argument For each sent codeword, the received codeword is contained in a sphere of radius nn. The received vectors have energy no grater thann(p +N), so they lie in a sphere of radius n(p +N). How many codeword can we use without intersection in the decoding sphere? M = A n ( n(p +N) ) n A n ( nn) n = ( 1+ P N ) n/2 whereathe constant for calculating the volume ofn-dimensional sphere. For example, A 2 = π,a 3 = 4 3 π. Therefore, the capacity is 1 n logm = 1 2 log ( 1+ P ). N Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 14/31

15 R < C Achievable Codebook. LetX i (w),i = 1,2,...,n,w = 1,2,...,2 nr be i.i.d. N(0,P ǫ). For largen, 1 n X 2 i P ǫ. Encoding. The codebook is revealed to both the sender and the receiver. To send the message indexw, the transmitter sends the wth codewordx n (w) in the codebook. Decoding. The receiver searches for the one that is jointly typical with the received vector. If there is one and only one such codeword X n (w), the receiver declaresŵ = w. Otherwise, the receiver declares an error. If the power constraint is not satisfied, the receiver also declare an error. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 15/31

16 R < C Achievable Probability of error. Assume that codeword 1 was sent. Y n = X n (1)+Z n. Define the events { } 1 n E 0 = X 2 n j(1) > P and Then an error occurs if j=1 E i = { ( ) X n (i),y n (i) is ina (n) ǫ }. The power constraint is violate. E 0 occurs. The transmitted codeword and the received sequence are not jointly typical. E c 1 occurs. Wrong codeword is jointly typical with the received sequence. E 2 E 3 E 2 nr occurs. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 16/31

17 R < C Achievable LetW be uniformly distributed. We have P (n) e = 1 2 nr λi = P(E) = Pr(E W = 1) = P(E 0 E c a E 2 E 3 E 2 nr) P(E 0 )+P ( E c 1)+ ǫ+ǫ+ 2 nr i=2 2 nr i=2 2 n(i(x;y) 3ǫ) P(E i ) 2ǫ+2 n(i(x;y) R 3ǫ) 3ǫ fornsufficient large andr < I(X;Y) 3ǫ. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 17/31

18 R < C Achievable, final part Since the average probability of error over codebooks is less then3ǫ, there exists at least one codebookc such thatpr(e C ) < 3ǫ. C can be found by an exhaustive search over all codes. Deleting the worst half of the codewords inc, we obtain a code with low maximal probability of error. The codewords that violates the power constraint is definitely deleted. (why?) Hence, we have construct a code that achieves a rate arbitrarily close toc. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 18/31

19 9.2 Converse to the Coding Theorem for Gaussian Channels Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 19/31

20 Achievable R < C 0 thenr C = 1 log(1+ P 2 N be distributed uniformly. We havew X n Y n We will prove that ifp (n) e inequality, ). LetW Ŵ. By Fano s H(W Ŵ) 1+nRP(n) e = nǫ n, whereǫ n = 1 n +RP(n) e 0 asp (n) e 0. Now, nr = H(W) = I(W;Ŵ)+H(W Ŵ) I(W;Ŵ)+nǫ n I(X n ;Y n )+nǫ n (data processing ineq.) = h(y n ) h(y n X n )+nǫ n = h(y n ) h(z n )+nǫ n n n n h(y i ) h(z n )+nǫ n h(y i ) h(z i )+nǫ n i=1 i=1 i=1 Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 20/31

21 Achievable R < C nr n (h(y i ) h(z i ))+nǫ n i=1 ( 1 2 log(2πe(p i +N)) 1 ) 2 log2πen = ( 1 2 log 1+ P ) i +nǫ n N n ( 2 log 1+ P ) +nǫ n N +nǫ n since every codeword satisfies the power constraint. Thus, R 1 2 log ( 1+ P N ) +ǫ n. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 21/31

22 9.3 Bandlimited Channels Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 22/31

23 Capacity of Bandlimited Channels Suppose the output of a band-limited channel can be represented by Y(t) = (X(t)+N(t)) h(t) wherex(t) is the input signal,z(t) is the white Gaussian noise, andh(t) is the impulse response of the channel with bandwidthw. The sampling frequency is2w. If the channel be used over the time interval[0,t], then there are2wt samples transmitted. Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 23/31

24 Capacity of Bandlimited Channels If the noise has power spectral densityn 0 /2 watts/hz, the noise power is(n 0 /2)(2W) = N 0 W. The noise energy per sample is N 0 W T/2WT = N 0 /2. If the signal power isp. The signal energy per sample is P T/2W T = P/2W. The capacity is (1+ 1 log P/2W 2 N 0 /2 ( C = W log 1+ P N 0 W ) bits/sample or ) bits/second Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 24/31

25 9.4 Parallel Gaussian Channels Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 25/31

26 Capacity of Bandlimited Channels In this section we considerk independent Gaussian channels in parallel with a common power constraint. The objective is to distribute the total power among the channels so as to maximize the capacity. The channels are modeled as Y j = X j +Z j,j = 1,2,...,k. withz j N(0,N j ). There is a common power constraint [ k ] E P. j=1 X 2 j Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 26/31

27 Capacity of Bandlimited Channels The information capacity is C = max I(X 1,X 2...,X k ;Y 1,Y 2,...,Y k ) f(x 1,...,x n ):EXi 2<P SinceZ 1,Z 2,...,Z k are independent, I(X 1,X 2...,X k ;Y 1,Y 2,...,Y k ) =h(y 1,Y 2,...,Y k ) h(y 1,Y 2,...,Y k X 1,X 2...,X k ) =h(y 1,Y 2,...,Y k ) h(z 1,Z 2,...,Z k X 1,X 2...,X k ) =h(y 1,Y 2,...,Y k ) h(z 1,Z 2,...,Z k ) =h(y 1,Y 2,...,Y k ) i i h(y i ) i h(z i ) h(z i ) i wherep i = EX 2 i and P i = P ( 1 2 log 1+ P ) i N i Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 27/31

28 Capacity of Bandlimited Channels Therefore, we have a constrained optimization problem max i 1 2 log ( 1+ P i N i ) subject to P i P,P i 0. This can be solved by Lagrange multiplier together with the Kuhn-Tucker condition /N i 1+P i /N i µ i +λ = 0 P i 0, i i P i P 0 µ i P i = 0,λ( i P i P) = 0 µ i 0,λ 0 Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 28/31

29 Capacity of Bandlimited Channels Case I.λ = 0. We have P i +N i = 1 2µ i, P i = 1 2µ i N i This violates the condition P i 0 sincen i > 0 andµ i 0. Case II.λ 0. We have P i +N i = 1 2(λ µ i ) = 1 2λ = constant, P i > 0( implyµ i = 0) 1 2(λ µ i ), P i = 0. We can solveλby i P i = i ( 1 2λ N i) + = P Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 29/31

30 Capacity of Bandlimited Channels Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 30/31

31 Nonlinear Optimization For the problem minf(x 1,x 2,...,x n ) subject to g j (x 1,x 2,...,x n ) 0,j = 1,2,...m The necessary conditions for optimization are f x i + j µ j g j x i = 0, i = 1,2,...,n g j (x 1,x 2,...,x n ) 0, j = 1,2,...,m µ j g j (x 1,x 2,...,x n ) = 0, j = 1,2,...,m µ j 0, j = 1,2,...,m Peng-Hua Wang, May 14, 2012 Information Theory, Chap. 9 - p. 31/31

Lecture 3: Channel Capacity

Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.