Communication Theory and Engineering Master's Degree in Electronic Engineering Sapienza University of Rome A.A. 018-019
Information theory Practice work 3
Review For any probability distribution, we define a quantity called the entropy Entropy is a measure of the uncertainty of a random variable H ( X ) = p( x) log p( x) x Conditional entropy H(X Y), is the entropy of a random variable conditional on the knowledge of another random variable H(Y X ) = p(x) H(Y X = x) = p(x, y)log p( y x) x Ω X x Ω X y Ω Y
Review The relative entropy is a measure of the distance between two distributions p(x) D( p q) = p(x)log q(x) x Ω X 0 0log 0 = 0; 0log 0 q = 0; p log Mutual information is a measure of the amount of information one random variable contains about another. p(x, y) p(x,y ) I (X;Y ) = D( p(x, y) p(x) p( y)) = p(x, y)log = E p(x) p( y) p(x, y) log p(x ) p(y ) x Ω X y Ω Y p 0 = The mutual information I(X;Y) is the relative entropy between the joint distribution p(x,y) and the product of the distributions p(x)p(y)
Example 1 Show that: a) I(X;Y) = H(X) H(X Y) = H(Y) H(Y X) b) I(X;Y) = H(X) + H(Y) H(Y,X) c) I(X;X) =?
Example 1 Solution a) I (X;Y ) = p(x, y) p(x, y)log x, y p(x) p( y) = p(x, y)log p(x y) x, y p(x) = = p(x)log p(x) + p(x, y)log p(x y) = H(X ) H(X Y ) = H(Y ) H(Y X ) x x, y b) We know that H(X,Y) = H(X) + H(Y X) = H(Y) + H(X Y) so I(X;Y) = H(X) - H(X Y) = H(X) + H(Y) H(X,Y) c) I(X;X) = H(X) H(X X) = H(X)
Example Show that: H(X,Y Z ) = H(X Z ) + H(Y X,Z ) Solution: H(X,Y Z ) = p(x, y,z) log p(x, y z) = x, y,z = p(x, y,z) log p(x z) p(x, y,z) log p( y x,z) = H(X Z ) + H(Y X,Z ) x, y,z Note: x, y,z p(x, y z) = p(x, y,z) p(z) * p(x,z) p(x,z) = p( y x,z) p(x z)
Example 3 solution What is the entropy and coding of an equiprobable binary source? If the outcomes of X are equally likely (q=1/), H(X) is maximum and equal to 1 bit, i.e. we need, on average, one bit to code each possible outcome of X In this case, the possible outcomes are two (the source is binary), so exactly 1 bit is used for each outcome
Example 4 What is the entropy and coding of a non equiprobable binary source? a) Evaluate the average number of bits necessary to encode a non-equiprobable binary source, characterized by q=0.1. Solution: The entropy is H(X)= 0.1*log 0.1 0.9*log 0.9 = 0.47 bits We need, on average, 0.47 bits to code each outcome. b) Implement a possible binary code that efficiently represents this source Solution: A code, in which the outcomes are coded in pairs, e.g.: outcomes Associated Code 00 0 01 10 10 110 11 111 not optimized n bit = [p(0,0)*1+p(0,1)*+p(1,0)*3+p(1,1)*3] / = = [0.9*0.9 + 0.9*0.1* + 0.1*0.9*3 + 0.1*0.1*3]/ = 0.645 bits > 0.47 bits
Example 5 What is the entropy and coding of a non binary and non equiprobable source? a) Evaluate entropy and a possible binary representation code. If X is a r.v. such that Ω X ={a 1, a, a 3, a 4 } and: Solution: the entropy is: H(X ) = 1 log + 1 4 log 4 + 1 8 log 8 + 1 log 8 =1.75 bits 8 A possible code that represents the outcomes of X with an average number of bits n bit equal to 1.75 is: outcomes Associated Code a 1 0 p X (a 1 ) = 1, p X (a ) = 1 4, p X (a 3 ) = 1 8, p X (a 4 ) = 1 8 a 10 a 3 110 n bit = [p(a 1 )*1+p(a )*+p(a 3 )*3+p(a 4 )*3] / 1 = 1.75 bits a 4 111
Example 6 Let! (X,Y ) p(x, y) Compute the following: a)h(x), H(Y) b)h(x Y), H(Y X) c)h(x,y) d)h(y) H(Y X) e)i(x;y) Y y 1 y y 3 X x 1 0 1/4 1/16 x 1/8 0 1/8 x 3 1/16 1/8 1/4 Joint probability of p(x,y)
p X (x) = p(x = x) = p(x) = p(x, y) p y Y p(x = x 1 ) = 0 + 1 4 + 1 16 = 5 16 Example 6 solution In the first step, we compute the marginal distribution of X and Y with respect to the given jiont probabilities in the table. x ( y) = p(y = y) = p( y) = p(x, y) p(y = y 1 ) = 0 + 1 8 + 1 16 = 3 16 p(x = x ) = 1 8 + 0 + 1 8 = 1 4 p(y = y ) = 1 4 + 0 + 1 8 = 3 8 p(x = x 3 ) = 1 16 + 1 8 + 1 4 = 7 16 H(X ) = p(x)log x H(Y ) = p( y)log y 1 p(x) = 5 16 log 16 5 + 1 4 log 4 + 7 16 log 16 7 1 p( y) = 5 16 log 16 5 + 3 8 log p(y = y 3 ) = 1 16 + 1 8 + 1 4 = 7 16 a) Now we can compute the entropy and conditional entropy regarding to the jiont probabilities and marginal distribution 8 3 + 7 16 log 16 7 =1.543 bits =1.505 bits
Example 6 solution b) The conditional entropies H(X Y) and H(Y X) are: H(X Y ) = p(y = y)h(x Y = y) = p(y = y) p(x = x y)log p(x y) = p(x, y) log p( y x) = y y = p(x 1, y 1 )log p(x 1 y 1 ) p(x 1, y )log p(x 1 y ) p(x 1, y 3 )log p(x 1 y 3 ) p(x, y 1 )log p(x y 1 ) p(x, y )log p(x y ) p(x, y 3 )log p(x y 3 ) p(x 3, y 1 )log p(x 3 y 1 ) p(x 3, y )log p(x 3 y ) p(x 3, y 3 )log p(x 3 y 3 ) = 0 1 4 log 4 5 1 16 log 1 7 1 8 log 5 0 + 1 8 log 7 1 16 log 1 5 1 8 log 1 3 1 4 log 4 7 x x, y =1.119 bits H(Y X ) = p(x = x)h(y X = x) = 5 x 16 H(Y X = x ) + 1 1 3 H(Y X = x ) + 7 16 H(Y X = x ) = 3 = 5 16 H(0, 4 5,1 5 ) + 1 4 H(1,0,1 ) + 7 16 H(1 7, 7, 4 ) = 0.5 + 0.5 + 0.60 =1.077 bits 7
1 H(X,Y ) = p(x, y)log = p(x, y) x y Example 6 solution c) The joint entropy with respect to the definition is: = 0 + 1 4 log 4 + 1 16 log 16 + 1 8 log 8 + 0 + 1 8 log 8 + 1 16 log 16 + 1 8 log 8 + 1 4 log 4 =.65 bits = = H(X ) + H(Y X ) = H(Y ) + H(X Y ) d, e) The mutual information is: I (X;Y ) = H(Y ) H(Y X ) = 1.505 1.077 = 0.48 bits = H(X ) H(X Y )
Consider the distributions in the table the ternary source X. Calculate: a) H(p), H(q) b) D(p q), D(q p) H( p) = 1 16 log 16 + 1 log + 3 16 log 16 3 + 1 4 log 4 = 1 4 + 1 + 0.45 + 1 H(q) = 4 4 log 4 = bits D( p q) = 1 16 log D(q p) = 1 4 log 4 + 1 4 log 1 4 + 1 log + 3 16 log 1 + 1 4 log Example 7 solution 3 4 + 1 4 log 1= 0.98 4 3 + 1 4 log 1= 0.353 Symbol p(x) q(x) a 1/16 1/4 b 1/ 1/4 c 3/16 1/4 d 1/4 1/4 =1.703 bits
Example 7 solution b) Shows an example of two distributions p and q for which D(p q) = D(q p) Symbol p(x) q(x) a z 1-z b 1-z z z D( p q) = z log 1 z + (1 z)log 1 z z D(q p) = (1 z)log 1 z z + z log z 1 z
Excercise MATLAB 1: Entropy of a binary source Write a MATLAB script that produces a graph representing the entropy H(X) of a binary X source, as the pmf of the source changes. the algorithm can be written as: 0- begin 1- set counter i=1, - get the step size Δ 3- p(1)=0 4- i= i+1 5- p(i)=p(i-1)+δ 6- H(i)=-(1-p(i))log (1-p(i))-p(i)log p(i) 7- if p(i)<1 then go to 4 8- H(1)=H(end)=0 9- plot H on the Y axis and p on the X axis 10- end Then we test the algorithm Set step size = 0.1 i p(i) H(i) 1 0 0 0.1 0.0808. 11 1 0
Excercise MATLAB 1: Entropy of a binary source After testing the algorithm, now we can draw the flowchart as: begin i=1, p(1)=0, get the step size Δ i=i+1 p(i)=p(i-1)+δ, H(i)=-(1-p(i))log (1-p(i))- p(i)log p(i) H(1)=H(end)=0 Plot(p,H) end Yes p(i)<1 No
MATLAB excercise : Entropy of a non equiprobable discrete source Write a MATLAB script that produces the entropy calculation H(X) of a discrete source X. a) The script must provide for the possibility of choice, by a generic user: i. of the cardinality of the source (number of outcomes) ii. of the source pmf b) In addition, the script must predict the user's possible error in entering data (eg, on the total sum of the probability), and react appropriately. c) Note: Use the MATLAB help command to get information about the input() and fprintf() functions.
MATLAB excercise 3: Relative entropy Consider two distributions p(x) and q(x) of q(x):pr(x q =0)=[0:0.1:1]; a binary X source. Whereas a) Write a proper algorithm and draw its flow chart to plot the trend of D(p q) to the variation of Pr(x q = 0), in the following cases : 1) p(x) uniform ) p(x) non uniform and such that Pr(x p =0)=1/3; 3) p(x) non uniform and such that Pr(x p =0)=/3; b) Produce a script to plot, in the same figure, the trend of D(p q) to the variation of Pr(x=0), in the previous cases c) For the three cases of the previous point, show in the same figure, but in a suitable subplot number, the curves obtained by calculating D(p q) and D(q p), showing the anti-symmetry property of the relative entropy (in general D(p q) D(q p).