Communication Theory and Engineering

Size: px

Start display at page:

Download "Communication Theory and Engineering"

Clarissa Bennett
5 years ago
Views:

1 Communication Theory and Engineering Master's Degree in Electronic Engineering Sapienza University of Rome A.A

2 Information theory Practice work 3

3 Review For any probability distribution, we define a quantity called the entropy Entropy is a measure of the uncertainty of a random variable H ( X ) = p( x) log p( x) x Conditional entropy H(X Y), is the entropy of a random variable conditional on the knowledge of another random variable H(Y X ) = p(x) H(Y X = x) = p(x, y)log p( y x) x Ω X x Ω X y Ω Y

4 Review The relative entropy is a measure of the distance between two distributions p(x) D( p q) = p(x)log q(x) x Ω X 0 0log 0 = 0; 0log 0 q = 0; p log Mutual information is a measure of the amount of information one random variable contains about another. p(x, y) p(x,y ) I (X;Y ) = D( p(x, y) p(x) p( y)) = p(x, y)log = E p(x) p( y) p(x, y) log p(x ) p(y ) x Ω X y Ω Y p 0 = The mutual information I(X;Y) is the relative entropy between the joint distribution p(x,y) and the product of the distributions p(x)p(y)

5 Example 1 Show that: a) I(X;Y) = H(X) H(X Y) = H(Y) H(Y X) b) I(X;Y) = H(X) + H(Y) H(Y,X) c) I(X;X) =?

6 Example 1 Solution a) I (X;Y ) = p(x, y) p(x, y)log x, y p(x) p( y) = p(x, y)log p(x y) x, y p(x) = = p(x)log p(x) + p(x, y)log p(x y) = H(X ) H(X Y ) = H(Y ) H(Y X ) x x, y b) We know that H(X,Y) = H(X) + H(Y X) = H(Y) + H(X Y) so I(X;Y) = H(X) - H(X Y) = H(X) + H(Y) H(X,Y) c) I(X;X) = H(X) H(X X) = H(X)

7 Example Show that: H(X,Y Z ) = H(X Z ) + H(Y X,Z ) Solution: H(X,Y Z ) = p(x, y,z) log p(x, y z) = x, y,z = p(x, y,z) log p(x z) p(x, y,z) log p( y x,z) = H(X Z ) + H(Y X,Z ) x, y,z Note: x, y,z p(x, y z) = p(x, y,z) p(z) * p(x,z) p(x,z) = p( y x,z) p(x z)

8 Example 3 solution What is the entropy and coding of an equiprobable binary source? If the outcomes of X are equally likely (q=1/), H(X) is maximum and equal to 1 bit, i.e. we need, on average, one bit to code each possible outcome of X In this case, the possible outcomes are two (the source is binary), so exactly 1 bit is used for each outcome

9 Example 4 What is the entropy and coding of a non equiprobable binary source? a) Evaluate the average number of bits necessary to encode a non-equiprobable binary source, characterized by q=0.1. Solution: The entropy is H(X)= 0.1*log *log 0.9 = 0.47 bits We need, on average, 0.47 bits to code each outcome. b) Implement a possible binary code that efficiently represents this source Solution: A code, in which the outcomes are coded in pairs, e.g.: outcomes Associated Code not optimized n bit = [p(0,0)*1+p(0,1)*+p(1,0)*3+p(1,1)*3] / = = [0.9* *0.1* + 0.1*0.9* *0.1*3]/ = bits > 0.47 bits

10 Example 5 What is the entropy and coding of a non binary and non equiprobable source? a) Evaluate entropy and a possible binary representation code. If X is a r.v. such that Ω X ={a 1, a, a 3, a 4 } and: Solution: the entropy is: H(X ) = 1 log log log log 8 =1.75 bits 8 A possible code that represents the outcomes of X with an average number of bits n bit equal to 1.75 is: outcomes Associated Code a 1 0 p X (a 1 ) = 1, p X (a ) = 1 4, p X (a 3 ) = 1 8, p X (a 4 ) = 1 8 a 10 a n bit = [p(a 1 )*1+p(a )*+p(a 3 )*3+p(a 4 )*3] / 1 = 1.75 bits a 4 111

11 Example 6 Let! (X,Y ) p(x, y) Compute the following: a)h(x), H(Y) b)h(x Y), H(Y X) c)h(x,y) d)h(y) H(Y X) e)i(x;y) Y y 1 y y 3 X x 1 0 1/4 1/16 x 1/8 0 1/8 x 3 1/16 1/8 1/4 Joint probability of p(x,y)

12 p X (x) = p(x = x) = p(x) = p(x, y) p y Y p(x = x 1 ) = = 5 16 Example 6 solution In the first step, we compute the marginal distribution of X and Y with respect to the given jiont probabilities in the table. x ( y) = p(y = y) = p( y) = p(x, y) p(y = y 1 ) = = 3 16 p(x = x ) = = 1 4 p(y = y ) = = 3 8 p(x = x 3 ) = = 7 16 H(X ) = p(x)log x H(Y ) = p( y)log y 1 p(x) = 5 16 log log log p( y) = 5 16 log log p(y = y 3 ) = = 7 16 a) Now we can compute the entropy and conditional entropy regarding to the jiont probabilities and marginal distribution log 16 7 =1.543 bits =1.505 bits

13 Example 6 solution b) The conditional entropies H(X Y) and H(Y X) are: H(X Y ) = p(y = y)h(x Y = y) = p(y = y) p(x = x y)log p(x y) = p(x, y) log p( y x) = y y = p(x 1, y 1 )log p(x 1 y 1 ) p(x 1, y )log p(x 1 y ) p(x 1, y 3 )log p(x 1 y 3 ) p(x, y 1 )log p(x y 1 ) p(x, y )log p(x y ) p(x, y 3 )log p(x y 3 ) p(x 3, y 1 )log p(x 3 y 1 ) p(x 3, y )log p(x 3 y ) p(x 3, y 3 )log p(x 3 y 3 ) = log log log log log log log 4 7 x x, y =1.119 bits H(Y X ) = p(x = x)h(y X = x) = 5 x 16 H(Y X = x ) H(Y X = x ) H(Y X = x ) = 3 = 5 16 H(0, 4 5,1 5 ) H(1,0,1 ) H(1 7, 7, 4 ) = =1.077 bits 7

14 1 H(X,Y ) = p(x, y)log = p(x, y) x y Example 6 solution c) The joint entropy with respect to the definition is: = log log log log log log log 4 =.65 bits = = H(X ) + H(Y X ) = H(Y ) + H(X Y ) d, e) The mutual information is: I (X;Y ) = H(Y ) H(Y X ) = = 0.48 bits = H(X ) H(X Y )

15 Consider the distributions in the table the ternary source X. Calculate: a) H(p), H(q) b) D(p q), D(q p) H( p) = 1 16 log log log log 4 = H(q) = 4 4 log 4 = bits D( p q) = 1 16 log D(q p) = 1 4 log log log log log Example 7 solution log 1= log 1= Symbol p(x) q(x) a 1/16 1/4 b 1/ 1/4 c 3/16 1/4 d 1/4 1/4 =1.703 bits

16 Example 7 solution b) Shows an example of two distributions p and q for which D(p q) = D(q p) Symbol p(x) q(x) a z 1-z b 1-z z z D( p q) = z log 1 z + (1 z)log 1 z z D(q p) = (1 z)log 1 z z + z log z 1 z

17 Excercise MATLAB 1: Entropy of a binary source Write a MATLAB script that produces a graph representing the entropy H(X) of a binary X source, as the pmf of the source changes. the algorithm can be written as: 0- begin 1- set counter i=1, - get the step size Δ 3- p(1)=0 4- i= i+1 5- p(i)=p(i-1)+δ 6- H(i)=-(1-p(i))log (1-p(i))-p(i)log p(i) 7- if p(i)<1 then go to 4 8- H(1)=H(end)=0 9- plot H on the Y axis and p on the X axis 10- end Then we test the algorithm Set step size = 0.1 i p(i) H(i)

18 Excercise MATLAB 1: Entropy of a binary source After testing the algorithm, now we can draw the flowchart as: begin i=1, p(1)=0, get the step size Δ i=i+1 p(i)=p(i-1)+δ, H(i)=-(1-p(i))log (1-p(i))- p(i)log p(i) H(1)=H(end)=0 Plot(p,H) end Yes p(i)<1 No

19 MATLAB excercise : Entropy of a non equiprobable discrete source Write a MATLAB script that produces the entropy calculation H(X) of a discrete source X. a) The script must provide for the possibility of choice, by a generic user: i. of the cardinality of the source (number of outcomes) ii. of the source pmf b) In addition, the script must predict the user's possible error in entering data (eg, on the total sum of the probability), and react appropriately. c) Note: Use the MATLAB help command to get information about the input() and fprintf() functions.

20 MATLAB excercise 3: Relative entropy Consider two distributions p(x) and q(x) of q(x):pr(x q =0)=[0:0.1:1]; a binary X source. Whereas a) Write a proper algorithm and draw its flow chart to plot the trend of D(p q) to the variation of Pr(x q = 0), in the following cases : 1) p(x) uniform ) p(x) non uniform and such that Pr(x p =0)=1/3; 3) p(x) non uniform and such that Pr(x p =0)=/3; b) Produce a script to plot, in the same figure, the trend of D(p q) to the variation of Pr(x=0), in the previous cases c) For the three cases of the previous point, show in the same figure, but in a suitable subplot number, the curves obtained by calculating D(p q) and D(q p), showing the anti-symmetry property of the relative entropy (in general D(p q) D(q p).

Chapter 2: Entropy and Mutual Information. University of Illinois at Chicago ECE 534, Natasha Devroye

Chapter 2: Entropy and Mutual Information Chapter 2 outline Definitions Entropy Joint entropy, conditional entropy Relative entropy, mutual information Chain rules Jensen s inequality Log-sum inequality