Solutions 1. Introduction to Coding Theory - Spring 2010 Solutions 1. Exercise 1.1. See Examples 1.2 and 1.11 in the course notes.

Transcription

1 Solutions 1 Exercise 1.1. See Exaples 1.2 and 1.11 in the course notes. Exercise 1.2. Observe that the Haing distance of two vectors is the iniu nuber of bit flips required to transfor one into the other. Using this, the first three conditions are trivial to verify. As for the triange inequality d(x, z) d(x, y) + d(y, z), (1) consider each position i of the vectors x, y and z. If x i = z i, the corresponding position contributes 0 to the left-hand-side of equation (1). In this case, either y i = x i = z i, thus contributing 0 to the right-hand-side as well, or y i x i, z i, thus contributing 2 to the right-handside. If x i z i, so that the corresponding i contributes 1 to the left-hand-side of equation (1), then y i ust be different fro at least one of x i and z i, thus contributing at least 1 to the right-hand-side. Suing over all values of i we readily obtain the triangle inequality. Exercise 1.3. This is very siilar to the case of BSC(ε) considered in the courses notes. For a received vector y Σ n and any codeword z, we have p(y z) = n p(y i z i ). Fro the definition of our channel, p(y i z i ) = ε/(q 1) for y i z i (this is the case for d(y, z) coordinates) and p(y i z i ) = 1 ε for y i = z i (this is the case for n d(y, z) coordinates). Therefore p(y z) = ( ) ε d(y,z) ( ) ε/(q 1) d(y,z) (1 ε) n d(y,z) = (1 ε) n. q 1 1 ε Since ε (q 1)/q, the ratio ε/(q 1) 1 ε 1, so that the codeword z that axiizes p(y z) is the one that iniizes d(y, z). Exercise Let A(n) := H ( 1 n,..., 1 n). We first show that A(s ) = A(s). (2) To see this, note that A(s ) = H ( ) 1 1 s,..., s corresponds to a choice between s equally likely events. We can group each s of these events together using Axio 3. For exaple, grouping the first s events gives us ( ) 1 A(s ) = H s 1, 1 s,..., 1 s + 1 A(s). s 1 Siilarly grouping all the other events s by s, we obtain A(s ) = A(s 1 ) + A(s). 1

2 We can now repeat this procedure recursively to obtain A(s ) = A(s 1 ) + A(s) = A(s 2 ) + 2A(s) = = A(s). Now for s and t integers, and for n arbitrarily large, we can always find such that On one hand, this gives us s t n < s +1. (3) n log t log s n + 1 n. (4) On the other hand, fro Axio 2, we know that A is a onotonic increasing function of its arguent, so that equation (3) gives us A(s ) A(t n ) < A(s +1 ). Fro equation (2), this is equivalent to saying that which gives us A(s) na(t) < ( + 1)A(s), n A(t) A(s) < n + 1 n. (5) As we let n grow to infinity, equations (4) and (5) gives us that so that A(t) ust be of the for A(t) li n A(s) = log t log s, A(t) = K log t for a constant K, where K ust be positive to satisfy Axio Suppose the p i are coensurable probabilities, so that p i = P n i ni. Consider choosing an event fro n i equiprobable events. Fro the expression we derived above for A(n), we know that the entropy of this choice is K log n i. But using Axio 3, we can also view this choice in the following equivalent anner: we can break down a choice fro n j equiprobable events into a choice fro n events with probabilities p 1,..., p n, then if the ith event is chosen, we have a second choice between n i equiprobable events. The entropy of this event is H(p 1,..., p n ) + p i K log n i. 2

3 We thus obtain H(p 1,..., p n ) = K (log n i ) p i log n i ( = K pi log n i ) p i log n i = K p i log n i ni = K p i log p i. 3. Now suppose the p i are incoensurable. Since the rationals are dense in the reals, we can approxiate the p i with rational nubers. We can thus find rationals p 1,..., p n 1 such that p i p i < ε for any ε > 0. Define p n as 1 n 1 p i. This ensures that ( p 1,..., p n ) is indeed a probability distribution, and p n p n < (n 1)ɛ can be ade as sall as we want. By continuity of H (Axio 1), H(p 1,..., p n ) tends to H( p 1,..., p n ) = K p i log p i. But by continuity of the function f(x 1,..., x n ) = K x i log x i (defined over real probability vectors (x 1,..., x n )), K p i log p i tends to K p i log p i. Thus the expression holds in general. Exercise I(X; Y ) = x,y = x,y p(x, y) log p(x, y) p(y) = x,y p(x, y) log + x,y = x log + x,y We prove siilarly that = H(X) H(X Y ). I(X; Y ) = H(Y ) H(Y X). I(X; Y ) = x,y p(x, y) log p(x, y) p(y) = x,y p(x, y) log x,y p(x, y) log p(y) + x,y = x log y p(y) log p(y) + x,y = H(X) + H(Y ) H(X, Y ). We can clearly see that I(X; Y ) is syetric in its arguents. 3

4 I(X; X) = x p(x, x) log = log 1 x = H(X). p(x, x) We could also obtain this forula by noting that I(X; X) = H(X) H(X X) = H(X). 2. Using the chain rule for two variables, we have H(X 1, X 2 ) = H(X 1 ) + H(X 2 X 1 ) H(X 1, X 2, X 3 ) = H(X 1 ) + H(X 2, X 3 X 1 ) = H(X 1 ) + H(X 2 X 1 ) + H(X 3 X 2, X 1 ). H(X 1,..., X n ) = H(X 1 ) + H(X 2 X 1 ) + + H(X n X n 1,..., X 1 ) n = H(X i X i 1,..., X 1 ). 3. To prove the chain rule for relative entropy, note that p(x, y) p(x, y) log q(x, y) D(p(x, y) q(x, y)) = x y = p(x, y) log p(y x) q(x)q(y x) x y = p(x, y) log q(x) + x y x y = D( q(x)) + D(p(y x) q(y x)). p(x, y) log p(y x) q(y x) Exercise Let χ be the support set of the rando variable X and let A = {x : > 0} be the 4

5 support set of the probability distribution. We have D(p q) = x A = x A log x A log q(x) log q(x) = log x A q(x) q(x) log x χ q(x) = log 1 = 0, with equality if and only if q(x)/ = 1 everywhere, since log t is a strictly concave function of t. Therefore D(p q) 0 (6) with equality if and only if = q(x) for all x. For any pair X, Y of rando variables, I(X; Y ) = D(p(x, y) p(y)). Equation (6) gives us I(X; Y ) 0, (7) with equality if and only if p(x, y) = p(y) for all values x, y, that is, if and only if X and Y are independent. 2. Let X take values over χ with soe probability distribution p, and let u be the unifor distribution over χ, so that u(x) = 1 χ for all x. Consider the quantity D(p u) = χ log u(x) = χ log χ log u(x) = log χ H(X). Fro equation (6), we have that H(X) log χ, with equality if p and u are the sae distribution. 3. Fro equation (7), we have so that I(X; Y ) = H(X) H(X Y ) 0, H(X Y ) H(X), with equality if and only if I(X; Y ) = 0, i.e., if and only if X and Y are independent. Thus conditioning reduces entropy. In the previous exercise, we saw the chain rule for the entropy of n variables: H(X 1,..., X n ) = n H(X i X i 1,..., X 1 ). 5

6 Each conditional entropy ter H(X i X i 1,..., X 1 ) is such that H(X i X i 1,..., X 1 ) H(X i ), with equality if and only if X i is independent fro the (i 1)-tuple X 1,..., X i 1. We finally get n H(X 1,..., X n ) H(X i ), with equality if and only if the X i are independent. 6