meghdadi@ensil.unilim.fr Reference : Elements of Information Theory by Cover and Thomas
Continuous random variable Gaussian multivariate random variable AWGN Band limited channel Parallel channels Flat fading channel Shannon Capacity of fading Channel CSI known at TX
Continuous random variable Continuous random variable Gaussian multivariate random variable In the case where X is a continuous RV, how the entropy is defined? For discrete RV we used the mass probability function, here it is replaced by probability distribution function (PDF). Definition The random variable X is said to be continuous if its cumulative distribution function F (x) = Pr(X x) is continuous.
Outline Continuous random variable Gaussian multivariate random variable Definition The differential entropy h(x ) of a continuous random variable X with a PDF P X (x) is defined as 1 h(x ) = P X (x) log = E S [ log 1 P X (x) P X (x) dx ] where S is the support set of the random variable. (1)
Example: Uniform distribution Continuous random variable Gaussian multivariate random variable Show that for X U(0, a) the differential entropy is log a. 1/a P X (x) Note Unlike discrete entropy, the differential entropy can be negative. However, 2 h(x ) = 2 log a = a is the volume of the support set, which is always non-negative. Note A horizontal shift does not change the entropy. a x
Continuous random variable Gaussian multivariate random variable Example: Normal and exponential distribution Show that for X N (0, σ 2 ) the differential entropy is h(x) = 1 2 log(2πeσ2 ) bits Show that for P X (x) = λe λx for X 0 the differential entropy is h(x) = log e λ bits What is the entropy if P X (x) = λ 2 e λ x?
Exercise Outline Continuous random variable Gaussian multivariate random variable Suppose an additive Gaussian channel defined by Y = X + N with: X N (0, P X ) and N N (0, P N ). Because of the independence of X and N, Y N (0, P X + P N ). Defining I (X ; Y ) = h(y ) h(y X ), show that I (X ; Y ) = 1 ( 2 log 2 1 + P ) X P N Hint: You can use the fact that h(y X ) = h(n) (why?). Actually this is the capacity of a noisy continuous channel.
Gaussian random vector Continuous random variable Gaussian multivariate random variable Suppose that the vector X is defined as [ ] X1 X = where X 1 and X 2 are i.i.d. N (0, 1). What is the entropy of X? h(x) = h(x 1, X 2 ) = h(x 1 ) + h(x 2 X 1 ) = h(x 1 ) + h(x 2 ) Therefore h(x) = 1 2 log(2πe)2 And for a vector of dimension n: h(x) = 1 2 log(2πe)n X 2
Continuous random variable Gaussian multivariate random variable Some properties 1. Chain rule: h(x, Y ) = h(x Y ) + h(y ) 2. h(x + cte) = h(x ) 3. h(cx ) = h(x ) + log c (note that in discrete case, H(cX ) = H(X )) 4. Let X be a random vector and Y = AX where A is a square non singular matrix. Then h(y) = h(x) + log A. 5. Suppose X is a random vector with E(X) = 0 and E(XX T ) = K, then h(x) 1 2 log(2πe)n K. The equality is achieved only if X is Gaussian N (0, K)
Shannon capacity Outline AWGN Band limited channel Parallel channels In the early 1940s it was thought to be impossible to send information at a positive rate with negligible probability of error. Shannon showed that (1948): For every channel there exists a maximum information transmission rate, below which, BER can be made nearly zero. If the entropy of source is less than channel capacity, asymptotically error free communication can be achieved. To obtain an error free communication, a coding scheme should be used. Shannon did not show the optimal coding. Today, the predicted capacity by Shannon can be achieved within only a few tenth of db. For every channel there exists a maximum information transmission rate, below which, the error probability can be made nearly zero.
Additive white Gaussian channel AWGN Band limited channel Parallel channels As we have seen before with an additive Gaussian noise channel, the mutual input-output information can be calculated as I (X ; Y ) = h(y ) h(y X ) = h(y) h(z) = h(y ) 1 log 2πeN 2 To maximize the mutual information, one should maximize h(y ) with the power constraint of P Y = P + N. The distribution maximizing the entropy for a continuous random variable is Gaussian. This can be obtain if X is Gaussian. C = max I (X ; Y ) = 1 ( p(x):e X 2 P 2 log 1 + P ) N
Band limited channels AWGN Band limited channel Parallel channels Suppose we have a continuous channel with bandwidth B and the power spectral density of noise is N 0 /2. So the analog noise power is N 0 B. On the other hand, supposing that the channel is used over the time interval [0, T ]. So the power of analog signal times T gives the total energy of the signal in this period. Using Shannon sampling theorem, there are 2B samples per second. So the power of discrete signal per sample will be PT /2BT = P/2B. The same argument can be used for the noise, so the power of samples of noise is N 0 2 2B T 2BT = N 0/2. So the capacity of the Gaussian channel per sample is: C = 1 ( 2 log 1 + P ) bits per sample N 0 B
Band limited channel capacity AWGN Band limited channel Parallel channels Since there are maximum 2B independent samples per second the capacity can be written as: ( C = B log 1 + P ) bits per second N 0 B Sometimes this equation is divided by B to obtain: ( C B = log 1 + P ) bits per second per Hz N 0 B It is the maximum achievable spectral efficiency through the AWGN channel.
AWGN Band limited channel Parallel channels Parallel independent Gaussian channel Here we consider k independent Gaussian channels in parallel with a common power constraint. The objective is to maximize the capacity by optimal distribution of the power among the channels: C = max p X1,...,X k (x 1,...,x k ): I (X 1,..., X k ; Y 1,..., Y k ) EXi 2 P
AWGN Band limited channel Parallel channels Parallel independent Gaussian channel Using the independence of Z 1,..., Z k : C = I (X 1,..., X k ; Y 1,..., Y k ) = h(y 1,..., Y k ) h(y 1,..., Y k X 1,..., X k ) = h(y 1,..., Y k ) h(z 1,..., Z k ) i i h(y i ) h(z i ) ( 1 2 log 1 + P ) i N i If there is no common power constraint, it is clear that the total capacity is the sum of the capacities of each channel.
Common power constraint AWGN Band limited channel Parallel channels The question is: how to distribute the power among the transmitter to maximize the capacity? The capacity for the equivalent channel is: C = max P 1 +P 2 P x [ B 1 log(1 + P 1h2 1 N 0 B 1 ) + B 2 log(1 + P 2h 2 2 N 0 B 2 ) ]
Common power constraint AWGN Band limited channel Parallel channels So we should maximize C subjected to P 1 + P 2 P x. Using Lagrangian, one can define: L(P 1, P 2, λ) = B 1 log(1+ P 1h 2 1 N 0 B 1 )+B 2 log(1+ P 2h 2 2 N 0 B 2 ) λ(p 1 +P 2 P x ) Let d(.)/dp 1 = 0 and d(.)/dp 2 = 0 and using ln instead of log 2 : B 1 1 + P 1h 2 1 N 0 B 1 h 2 1 N 0 B 1 = λ P 1 = 1 1 B 1 N 0 λn 0 h1 2
AWGN Band limited channel Parallel channels With the same operations we obtain: P 1 B 1 N 0 = Cst 1 h1 2 P 2 B 2 N 0 = Cst 1 h2 2 Where the Cte can be found by setting P 1 + P 2 = P x. Since the two powers are found, the capacity of the channel is calculated easily. The only constraint that to be considered is that P 1 and P 2 cannot be negative. If one of these is negative, the corresponding power is zero and all the power are assigned to the other one. This principle is called water filling.
Exercise Outline AWGN Band limited channel Parallel channels Exercise Use the same principle (water filling) and give the power allocation for a channel with three frequency bands defined as follows: h 1 = 1/2, h 2 = 1/3 and h 3 = 1; B 1 = B, B 2 = 2B and B 3 = B; N 0 B = 1; P x = P 1 + P 2 + P 3 = 10. Solution: P 1 = 3.5, P 2 = 0 and P 3 = 6.5.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Flat fading channel (frequency non-selective) A non LOS urban transmission results in general in many of multi paths : the received signal is the sum of many replicas of transmitted signal. Using I and Q components of received signal: r(t) = cos(2πf c t) I a i cos(φ i ) sin(2πf c t) i=0 I a i sin(φ i ) + n(t) i=0 With the central limit theorem, A = I i=0 a i cos(φ i ) and B = I i=0 a i sin(φ i ) are i.i.d. Gaussian random variables.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX The envelope of the received signal h = A 2 + B 2 will be Rayleigh random variable with: f h (h) = h ( ) h 2 σ 2 exp 2σ 2 r 0 with σ 2 the variance of A and B. The received power will be an exponential RV with the pdf: f (p) = 1 ( ) p 2σ 2 exp 2σ 2 r 0 Therefore, the received signal can be modeled as: Y = hx + N
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Shannon (ergodic) capacity when Rx knows CSI The Channel coefficient h is an i.i.d. random variable independent of signal and noise. We assume that the receiver knows the channel coefficient but the transmitter does not. The capacity is: C = max px :E[X ] P I (X ; Y, h) Using chain rule: I (X ; Y, h) = I (X ; h) + I (X ; Y h) = I (X ; Y h)
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Conditioned on the fading coefficient h, the channel is transformed into a simple AWGN with equivalent P equal to h 2 P X. So we can write: I (X ; Y h = h) = 1 ( ) 2 log 1 + h 2 P X P N The ergodic capacity of the flat fading channel will be : [ ( 1 C = E h 2 log 1 + P X h 2 )] Note: Normally all the signals are complex and they are the base band equivalent of reel signals. In this case, the capacity is multiplied by two since the real and imaginary parts of signals are decorrelated. P N
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Example(Wireless transmission by Andreas Goldsmith) Consider a wireless channel where power falloff with distance follows the formula P r (d) = P t (d 0 /d) 3 for d 0 = 10m. Assume the channel band width of B = 30 khz and AWGN with noise PSD N 0 /2, where N 0 = 10 9 W/Hz. For a transmit power of 1 W flind the capacity of the channel for a distance of 100m and 1km. Solution: The received signal to noise ratio SNR is γ = P r (d)/p N = p t (d 0 /d) 3 /(N 0 B). That is γ = 15 db for d = 100m, and 15 db for d = 1km. The capacity of complex transmission is C = B log(1 + SNR) and is 156.6 kbps for d = 100m and 1.4 kbps for d = 1000 m.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Example(Wireless transmission by Andreas Goldsmith) Consider a flat fading channel with i.i.d. channel gain h, which can take on three possible values: 0.05 with the probability of 0.1, 0.5 with 0.5, and 1 with 0.4. The transmitted power is 10 mw, N 0 = 10 9 W/Hz, and the channel band width is 30 khz. Assume the receiver has the knowledge of instantaneous value of h but the transmitter does not. Find the Shannon capacity of this channel. Solution: The channel has three possible received SNRs: γ 1 = P t h 1 /N 0 B = 0.83, γ 2 = P t h 2 /N 0 B = 83.33, and γ 3 = P t h 3 /N 0 B = 333.33. So the Shannon capacity is given by: C = i B log 2 (1 + γ i )p(γ i ) = 199.26Kbps Note: The average SNR is 175 and the corresponding capacity would be 223.8 Kbps.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Shannon capacity defines the maximum data rate that can be sent over the channel with asymptotically small error probability. Since the TX does not know the channel, the transmitted rate is constant. When channel is in deep fade, the BER is not zero because the TX cannot adapt its rate relative to CSI. So the capacity with outage is defined and is the maximum rate that can be achieved with some outage probability (the probability of deep fading). By allowing some losses in deep fading, higher data rate can be achieved.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Fixing the required rate, C, a corresponding minimum SNR can be calculated (assuming complex transmission): C = log 2 (1 + γ min ) If TX sends the date at this rate, the outage (non zero BER) occurs when γ < γ min. Therefore the probability of outage is p out = p(γ < γ min ). The average rate of data that correctly received at RX is C O = (1 p out )B log 2 (1 + γ min ). The value of γ min is a design parameter based on the acceptable outage probability. Normally one draws the normalized capacity C/B = log 2 (1 + γ min ) as a function of p out = p(γ < γ min ).
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Example(Wireless transmission by Andreas Goldsmith) Consider the same channel as in the last example with BW=30kHz and p(γ = 0.83) = 0.1, p(γ = 83.33) = 0.5, and p(γ = 333.33) = 0.4. Find the capacity versus outage and the average rate correctly received for outage probabilities p out < 0.1, p out = 0.1 and p out=0.6. Solution: For p out < 0.1, we must decode in all the channel states. Therefore the rate must be less than the worst case: γ min = γ 1 = 0.83. The corresponding capacity is 26.23 Kbps. For 0.1 P out < 0.6, we can decode incorrectly only if the channel is in the weakest state: γ = 0.83. So γ min = γ 2 with corresponding capacity of 191.94 Kbps. For 0.6 P out < 1, we can decode incorrectly if received γ is γ 1 or γ 2. Thus, γ min = γ 3 with corresponding capacity of 251.55 Kbps.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Example (cont.) For p out < 0.1 data rates close to 26.23 Kbps are always correctly received. For p out = 0.1 we transmit at the rate 191.94 but can only corecte when γ = γ 2 or γ 3. So the rate correctly received is (1-0.1)191.94=172.75 Kbps. For p out = 0.6 the rate correctly received is (1-0.6)251.55=125.78 Kbps.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Since the channel is known at the TX, the outage cannot be produced. That is because the TX can adapt its power to avoid the outage. The capacity is (which is the same as Shannon capacity as before): C = 0 B log 2 (1 + γ)p(γ)dγ Now we add also the power adaptation with a power constraint: 0 P(γ)p(γ)dγ P So the problem is how to distribute the available power as a function of SNR to maximize the rate while the average power dose not exceed a predefined value.
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Water-filling The capacity is then C = max P(γ): P(γ)p(γ)dγ= P 0 0 ( B log 2 1 + P(γ)γ ) p(γ)dγ P Note that γ = P h 2 N 0 B. It means that for each channel level realization, a coding is employed to adjust the rate To find the optimal power allocation P(γ) we form the Lagrangian. ( J(P(γ)) = B log 2 1 + P(γ)γ ) p(γ)dγ λ P(γ)p(γ)dγ P 0
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Water-filling Setting the derivative with respect to P(γ) equal to zero and solving for P(γ) with the constraint P(γ) 0: { P(γ) P = 1/γ 0 1/γ γ γ 0 0 γ < γ 0 It means that if γ is under a threshold γ 0, the channel will not be used. The capacity formula is then: ( ) γ C = B log 2 p(γ)dγ γ 0 γ 0
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Water-filling Therefore, the capacity can be achieved by adapting the rate as a function of SNR. Another strategy would be fixing the rate and adapting only the power. Note that γ 0 must be found numerically. Replacing the optimal power allocation calculated in the constraint power, we obtain the following expression that should be satisfied to calculate γ 0. ( 1 1 ) p(γ)dγ = 1 γ 0 γ γ 0
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Water-filling γ 0 1/γ 0 P(γ) P 1/γ γ 03 C6 03 C1 03 B5 03 C1 03 C6 03 C6 03 C1 Figure above shows why this principle is called water-filling. 1F 62 03 B2 03 B2
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Example With the same example as before: p(γ 1 = 0.83) = 0.1, p(γ 1 = 83.33) = 0.5, and p(γ 3 = 333.33) = 0.4. Find the ergodic capacity of the channel with CSI at TX and RX. Solution: Since water-filling will be used, we must first calculate γ 0 satisfying: ( 1 1 ) p(γ i ) = 1 γ 0 γ i γ i γ 0
Flat fading channel Shannon Capacity of fading Channel CSI known at TX Example (cont.) First we assume that all channel states will be used. In the above equation everything is known except for γ 0 which is calculated to be 0.884. Since this value exceeds γ 1 = 0.83, the first channel state should not be used. At the first iteration, the above equation will be calculated only for the second and third channel giving γ 0 = 0.893. This value is acceptable because the weakest channel is better than this minimum threshold. Using this values the channel capacity can be calculated and is 200.82 Kbps.