Digital Communications III (ECE 154C) Introduction to Coding and Information Theory

Transcription

1 Digital Communications III (ECE 54C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 204 / 2

2 Noiseless Fundamental Limits 2 / 2

3 Shannon Entropy Let an I.I.D. sources, havem source letters that occur with probabilitiesp,p 2,...,p M, M = The entropy of the sources is denotedh(s) and is defined as H a (S) = log a = H a (S) = E [ log a log a ] The base of the logarithms is usually taken to be equal to2. In that case,h 2 (S) is simply written ash(s) and is measured in units of bits. 3 / 2

4 Shannon Entropy Other bases can be used and easily transformed to. Note: Hence, A Useful log a x = (log b x)(log a b) = (log bx) (log b a) H a (S) = H b (S) log a b = H b(s) log b a Letp,p 2,...,p M be one set of probabilities and letp,p 2,...,p M M be another (note p i =, M p i = ). Then log log p, i with equality iff = p i fori =,2,...,M 4 / 2

5 Shannon Entropy Proof. First note thatlnx x with equality iffx =. Placeholder Figure A log p i = ( M ln p i ) loge 5 / 2

6 Shannon Entropy proof continued. ( M ln p i ) loge (loge) ( ) p i ( M = (loge) p i = 0 ) In other words, log log p i 6 / 2

7 Shannon Entropy. For an equally likely i.i.d. source withm source letters H 2 (S) = log 2 M (ah a (S) = log a M,anya) 2. For any i.i.d. source withm source letters 0 H 2 (S) log 2 M, for alla This follows from previous theorem withp i = M alli. 3. Consider an iid source with source alphabets. If we consider encodingm source letters at a time, this is an iid source with alphabets M for source letters. Call this them tuple extension of the source and denote it is bys M. Then H 2 (S M ) = mh 2 (S), (and similarly for alla,h a (S M ) = mh a (S)). The proofs are omitted but are easy. 7 / 2

8 Computation of Entropy (base 2) Example : Consider a source withm = 2 and (p,p 2 ) = (0.9,0.). H 2 (S) =.9log log 2. = 0.469bits From before we gave Huffman Codes for this source and extensions of this source for which L = L2 2 L3 3 L4 4 = = = 0.49 In other words, we note thatlm m H 2(s). Furthermore, asmgets larger I m m is getting closer toh 2 (S). 8 / 2

9 Performance of Huffman Codes One can prove that, in general, for a binary Huffman Code, H 2 (S) Lm m < H 2(S)+ m Example 2: Consider a source withm = 3 and (p,p 2,p 3 ) = (.5,.35,.5). H 2 (S) =.5log log log 2.5 =.44 bits Again we have already given codes for this source such that symbol at a timel =.5 =.46 2 symbols at a timel2 2 9 / 2

10 Performance of Huffman Codes Example 3: ConsiderM = 4 and(p,p 2,p 3,p 4 ) = ( 2, 4, 8, 8 ). H 2 (S) = 2 log 2 /2 + 4 log 2 But from before we gave the code Source Symbols Codewords A 0 B 0 C 0 D for whichl = H 2 (S). /4 + 8 log 2 =.75bits /8 + 8 log 2 /8 This means that one cannot improve on the efficiency of this Huffman code by encoding several source symbols at a time. 0 / 2

11 Fundamental Limit [Lossless ] For any U.D. Binary Code corresponding to then th extension of the I.I.D. SourceS, for everyn =,2,... L N N H(S) For a binary Huffman Code corresponding to then th extension of the I.I.D. SourceS H 2 (S)+ m > L N N H 2(S) But this implies that Huffman coding is asymptotically optimal, i.e. lim N L N N H 2(S) / 2

12 NON-BINARY CODE WORDS The code symbols that make up the codewords can be from a higher order alphabet than 2. Example 4: I.I.D. Source{A,B,C,D,E} with U.D. codes (where each concatenation of code words can be decoded in only one way): SYMBOLS TERNARY QUATERNARY 5-ary A B C D E A lower bound to the average code length of any U.D.n-ary code (withn-letters) ish n (S) For example, for a ternary code, the average length (per source letter), L M M, is no less thanh 3 (S). 2 / 2