Exercises with solutions (Set D)

Transcription

1 Exercises with solutions Set D. A fair die is rolled at the same time as a fair coin is tossed. Let A be the number on the upper surface of the die and let B describe the outcome of the coin toss, where B is equal to if the result is head and it is equal to if the result if tail. The random variables X and Y are given by X = A + B and Y = A B, respectively. Calculate the marginal entropies HX and HY, the conditional entropies HX Y and HY X, the joint entropy HX, Y and the mutual information IX; Y. Solution: Let a, b, x, and y denote possible values of the random variables A, B, X, and Y, respectively. Each event {a, b} is associated with exactly one event {x, y} and the probability for such an event is given by p AB a, b = p XY x, y = 6 = Consequently, we obtain for the joint entropy HX, Y = p XY x, y log p XY x, y = x,y = log = + log 3 log The following tables list the possible values of the random variables X and Y, the associated events {a, b}, and the probability masses p X x and p Y y. x events {a, b} p X x {, } / {, }, {, } /6 3 {3, }, {, } /6 4 {4, }, {3, } /6 5 {5, }, {4, } /6 6 {6, }, {5, } /6 7 {6, } / y events {a, b} p Y y {, } / {, }, {, } /6 {, }, {3, } /6 3 {3, }, {4, } /6 4 {4, }, {5, } /6 5 {5, }, {6, } /6 6 {6, } / The random variable X = A + B can take the values to 7. The probability masses p X x for the values and 7 are equal to /, since they correspond to exactly one event. The probability masses for the values to 6 are equal to /6, since each of these values corresponds to two events {a, b}. Similarly, the random variable Y = A B can take the values to 6, where the probability masses for the values and 6 are equal to /, while the probability masses for the values to 5 are equal to /6.

2 Hence, the marginal entropies are given by HX = x p X x log p X x = log = 6 log 4 + log log + log 3 = log log 6 and HY = y p Y y log p Y y = log 5 6 log 6 = log 3 The conditional entropies can now be determined using the chain rule HX Y = HX, Y HY = + log log 3 = 5 6 HY X = HX, Y HX = + log log 3 = 5 6 Alternatively, we can also calculate the conditional entropies based on the probability mass functions. The conditional probability mass function p X Y x y is given by { : x = y + p X Y x y = = { : x = y p X Y x y = 6 = / : x = y p X Y x < y < 6 = / : x = y + Hence, we obtain HX Y = y p Y y x p X Y x y log p X Y x y = = 5 6 log 5 6 log Similarly, the conditional probability mass function p Y X y x is given by { : y = x p Y X y x = = { : y = x p Y X y x = 7 =

3 Hence, we obtain / : y = x p Y X y < x < 7 = / : y = x HY X = x p X x y p Y X y x log p Y X y x = = 5 6 log 5 6 log The mutual information IX; Y can be determined according to IX; Y = HX HX Y = log = 3 + log 3 or IX; Y = HY HY X = log = 3 + log 3 Alternatively, it can also be determined based on the probability mass functions, IX; Y = p XY x, y p XY x, y log p x,y X x p Y y = 8 log log = 3 log log 6 = 3 log = 3 + log log 3 3

4 . Consider a stationary Gauss-Markov process X = {X n } with mean µ, variance σ, and the correlation coefficient ρ correlation coefficient between two successive random variables. Determine the mutual information IX k ; X k+n between two random variables X k and X k+n, where the distance between the random variables is N times the sampling interval. Interpret the results for the special cases ρ =, ρ =, and ρ =. Hint: In the lecture, we showed E { X µ N T C N X µ N } = N, which can be useful for the problem. Solution: The mutual information IX k ; X k+n between the random variables X k and X k+n can be expressed using differential entropies IX k ; X k+n = hx k hx k X k+n = hx k + hx k+n hx k, X k+n = hx k hx k, X k+n The marginal pdf of the source is given by f x = x µ e σ πσ For the marginal differential entropy hx k, we obtain hx k = E { log f x} { = E log πσ + } x µ ln σ = log πσ + σ ln E { x µ } = log πσ + ln = log πσ + log e = log πe σ The joint pdf of two samples of a Gaussian process is a Gaussian pdf. With x = [x k x k+n ] T being a vector of potential outcomes of the random variables, the joint pdf is given by f,n x k, x k+n = f,n x = π C,N e x µ T C,N x µ 4

5 where µ = [µ µ] T is the vector of mean values and C,N is the covariance matrix. With X = [X k X k+n ] T being a vector of two random variables X k and X k+n the covariance matrix is given by C,N = E { X µ T X µ } [ ] E{Xk µ = } E{X k µx k+n µ} E{X k µx k+n µ} E{X k µ } Hence, we obtain for the joint differential entropy hx k, X k+n = E { log f,n X k, X k+n } C,N = E {log π = log + } ln X µ T C,N X µ π C,N + ln E { X µ T C,N X µ Inserting the expression given as hint yields hx k, X k+n = log π C,N + { } ln E X µ T C,N X µ = log π C,N + ln = log π C,N + ln = log π C,N + ln e ln = log π C,N + log e = log πe C,N } Note that a continuous stationary Markov process with correlation factor ρ can be represented by X k+n µ = ρ X k+n µ + Z k+n = ρ X k+n µ + ρ Z k+n + Z k+n N = ρ N X k µ + ρ i Z k+n i where Z = {Z k } is a zero-mean iid process; for Gauss-Markov processes, it is an zero-mean Gaussian iid process. i= 5

6 The covariance E{X k µx k+n µ} is given by E{X k µx k+n µ} { = E X k µ ρ N X k µ + = ρ N E { X k µ } + N i= = ρ N σ N i= ρ i Z k+n i } ρ i E {X k Z k+n i } + µ E {Z k+n i } Consequently, the covariance matrix C,N is given by [ ] σ ρ C,N = N σ ρ N σ σ For the determinant, we obtain C,N = σ σ ρ N σ ρ N σ = σ 4 ρ N Inserting this expression into the formula for the joint differential entropy, which we have derived above, yields hx k, X k+n = log = log πe C,N πe σ 4 N ρ For the mutual information, we finally obtain IX k ; X k+n = hx k hx k, X k+n = log πe σ log = log πe σ 4 πe σ 4 ρ N = log ρ N πe σ 4 N ρ The mutually information between two random variables of a Gauss- Markov process depends only on the correlation factor ρ and the distance N between the two considered random variables. For ρ =, the process is a Gaussian iid process, and the mutual information is equal to. A random variable X k does not contian any information about any other random variable X k+n with N. For ρ = ±, the process is deterministic, and the mutual information is infinity. By knowing any random variable X k, we know all other random variables X k+n. 6

7 3. Show that for discrete random processes the fundamental bound for lossless coding is a special case of the fundamental bound for lossy coding. Solution: The fundamental bound for lossy coding is the information rate-distortion function given by R I D = lim inf N g N :δg N D = lim inf N g N :δg N D H N S N = lim lim N N I N S N ; S N N HN S N H N S N ; S N sup N N g N :δg N D HN S N S N For lossless coding, the distortion D is equal to and the vector of reconstructed samples S N is equal to the vector of source samples S N. Hence, we have R I H N S N D = = lim lim N N sup N g N :δg N = N HN S N S N The conditional entropy H N S N S N is equal to, and thus R I H N S N D = = lim lim N N H N S N = lim N N = HS sup N g N :δg N = N N For zero distortion, the information rate-distortion function is equal to the entropy rate. 7

8 4. Determine the Shannon lower bound with MSE distortion, as distortionrate function, for iid processes with the following pdfs: The exponential pdf f E x = λ e λ x, with x The zero-mean Laplace pdf f L x = λ e λ x Express the distortion-rate function for the Shannon lower bound as a function of the variance σ. Which of the given pdfs is easier to code if the variance is the same? Solution: The mean of the exponential pdf is given by µ E = x f E x dx = λ x e λx dx Using the substitution t = λx and applying the integration rule uv = uv u v with u = t and v = e t yields µ E = λ x e λx dx = t = [ t e t] λ λ = [ ] [e t] = [ ] = λ λ λ For the variance of the exponential pdf, we obtain σ E = x f E x dx µ E = λ x e λ x dx λ Using the substitution t = λx and applying the integration rule uv = uv u v with u = t n and v = e t yields σ E = λ x e λx dx λ = λ = [ λ t e t] = [ ] λ = [ λ te t] t λ t λ = ] [ λ λ = λ λ = λ t λ λ = [ ] λ [e t] λ 8

9 Similarly, for the variance of the Laplace pdf, we obtain σ L = x f L x dx = λ = λ x e λx dx = λ x e λ x dx x e λx dt = λ The Shannon lower bound for iid processes and MSE distortion is given by D L R = πe hx R For the differential entropy of the exponential pdf, we obtain h E X = = λ = λ ln f E x log f E x dx e λ x log λ x e λx dx λ log λ λ ln x dx e λx dx By setting t = λx and using the integration rule uv = uv u v with u = x and v = e t, we obtain h E X = ln = ln t + log λ [ t e t] = λ [ ] + log ln λ ln + log λ = log λ log e [e t] λ [ ] = log e e = log λ = e log λ = log e σ 9

10 Similarly, the differential entropy for the Laplace pdf is given by h L X = = λ = λ = λ ln f L x log f L x dx e λ x λ log λ ln x dx e λ x log λ λ ln x x e λx dx λ log λ dx e λx dx By setting t = λx and using the integration rule uv = uv u v with u = x and v = e t, we obtain h L X = ln = ln t + log λ [ t e t] λ + log = λ [ ] λ + log ln ln λ [ = log log e e t] λ = log e = log e λ = log [ ] 4e = log e σ Inserting the expressions for the differential entropy into the formula for the Shannon lower bound yields, for the exponential pdf, D E L R = πe h EX R = e σ πe R = e π σ R and, for the Laplace pdf, D L L R = πe h LX R = e σ R πe = e π σ R Hence, at high rates R, the distortion for the Laplace pdf is twice as large as the distortion for the exponential pdf with the same variance. The exponential pdf is easier to code. λ