Digital Communications III (ECE 154C) Introduction to Coding and Information Theory

Transcription

1 Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring / 14

2 Statement Proof Sketch Noiseless Fundamental Limits 2 / 14

3 Fundamental Limit Statement Proof Sketch [Lossless ] For any U.D.n-ary code corresponding to then th extension of the I.I.D. SourceS, for everyn = 1,2,... L N N H n(s) Furthermore for ann-ary Huffman Code corresponding to the N th extension of the I.I.D. SourceS H n (S)+ 1 N > L N N H n(s) But this implies that Huffman coding is asymptotically optimal, i.e. lim N L N N H n(s) 3 / 14

4 Sketch of the Proof Statement Proof Sketch If one can construct a U.D. code such thatl i = will have H n (S)+1 > L H n (S). log n 1 p i, we 1 But is it always possible to construct such a code? If it is, is it good enough? Can we do better? To do this, we first prove useful conditions on a set of integers if they are the length of U.D. codewords. These conditions are called Kraft and McMillan inequalities. 1 Note that x is the unique integer in the interval[x,x+1). 4 / 14

5 Proof of 5 / 14

6 Codeword Lengths: Necessary and Sufficient Condition [Kraft Inequality] A necessary and sufficient condition for the construction of an instantaneousn-ary code withm code words of lengthsl 1,l 2,..,l M where the code symbols take onndifferent values is that PROOF OF SUFFICIENCY: n l i 1 We construct an instantaneous code with these code word lengths. Let there bem j code words of lengthj for j = 1,2,...,l = maxl i. Then n l i = l j=1 m j n j 6 / 14

7 Kraft Inequality: Proof of sufficiency M In other words, if n l i l 1, then j=1 m jn l 1 n l. Or, m l +m (l 1)n+m (l 2)n m 1 n l 1 n l, and, equivalently m l n l m 1 n l 1 m 2 n l 2 t... m l 1n (1) But sincem l 0 we then have 0 n l m 1 n l 1 m 2 n l 2 t... m l 1n and can repeat the procedure to arrive at m l 1 n l 1 m 1 n l 2 m 2 n l 3... m l 2n (2) 7 / 14

8 Codeword Lengths: Necessary and Sufficient Condition But dividing bynand noting thatm l 1 0 we have equivalently 0 n l 2 m 1 n l 3 m 2 n l 4... m l 2 m l 2 n l 2 m 1 n l 3 m 2 n l 4... (3) Continuing we get 0 m 3 n 3 m 1 n 2 m 2 n (4) 0 m 2 n 2 m 1 n (5) m 1 n (6) Note that if n l i 1 then them j satisfy (1)-(6). 8 / 14

9 Kraft Inequality: Proof of sufficiency Note thatm 1 n. Ifm 1 = n then we are done (assigning each codeword a letter). Ifm 1 < n, we have(n m 1 ) unused prefixes to form code words of length2for which the code words of length1are not prefixes. This means that there are(n m 1 )n phrases of length2to select the codewords from. But this is larger than or equal to the number of codewords of length 2,m 2 according to equation (5). So we can construct our code by selectingm 2 of the(n m 1 )n prefix-free phrases of length2 Ifm 2 = (n m 1 )n we are done. Ifm 2 < n 2 m 1 n, there are (n 2 m 1 n m 2 )n code words of length3which satisfy the prefix condition. etc. 9 / 14

10 Kraft Inequality: Proof of necessity Proof of necessity follows from McMillan inequality [McMillan Inequality] A necessary and a sufficient condition for the existence of a U.D. code withm code words of lengthl 1,l 2,...,l M where the code symbols take onndifferent values is: n l i 1. Here we sketch the proof of necessity of McMillan Inequality (enough to prove the necessity of Kraft inequality). Proof by contradiction: 1. Assume ( M n l i = A > 1 and a U.D. code exists. ) [ M 2. If n l i > 1, then n l i] N A N. 10 / 14

11 Kraft Inequality: Proof of necessity 3. ( M N n i) l = N k = l j=1 m j n j i 1 +i i N =k N = Nl k=n m i1 m i2 m in N k n k where and it denotes the # of strings ofn code words that are the length of exactlyk. 4. If the code is U.D.,N k n k. But then for a U.D. code ( M N n i) l = ( Nl N k n k ) Nl 1 = Nl N +1 k=n k=n Which grows linearly withn, not exponentially withn, contradicting bullet point / 14

12 Proof Proof Complete Proof 12 / 14

13 Proof of Proof Proof Lower bound forlfor a U.D. Code : For any instantaneous code 2,L H n (S). Furthermore,L = H n (S) iffp i = n l i. Proof: Letp i = n l i. Notep i 0 M and p i = 1. From before: j=1 n l j H n (S) = p i log n 1 p i p i log n 1 p i 2 HereL = Average length of the U.D. Code andn = number of symbols in code alphabet. 13 / 14

14 Proof of Proof Proof Then: But for a U.D. code, H n (S) p i l i + p i log n n l j 1, solog n j=1 j=1 j=1 n l j n l j 0 Equality occurs if and only if H n (S) L n l j = 1 andp i = p i. j=1 But both of these conditions hold ifp i = n l i,l i is an integer. 14 / 14