ELEC 515 Information Theory. Distortionless Source Coding

Size: px

Start display at page:

Download "ELEC 515 Information Theory. Distortionless Source Coding"

Joella Walters
6 years ago
Views:

1 ELEC 515 Information Theory Distortionless Source Coding 1

2 Source Coding Output Alphabet Y={y 1,,y J } Source Encoder Lengths 2

3 Source Coding Two coding requirements The source sequence can be recovered from the encoded sequence with no ambiguity The average number of output symbols per source symbol is as small as possible 3

4 Typical Sequences Consider the following discrete memoryless binary source: p(1) = 1/4 p(0) = 3/4 Sequences of 20 symbols 1. 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1 2. 1,0,1,0,1,0,0,0,0,0,0,0,0,0,1,1,0,0,0,1 3. 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 4

5 Tchebycheff Inequality 5

6 Weak Law of Large Numbers Sequence of N i.i.d. RVs Define a new RV 6

7 Weak Law of Large Numbers The average approaches the statistical mean 7

8 Asymptotic Equipartition Property N i.i.d. random variables X 1,, X N p(x,x,,x )=p(x )p(x ) p(x ) 1 2 N 1 2 N N 1 1 logp(x = 1,X 2,,X N) logp(x n) E[ logp(x) ] =H(X) N N n = 1 as N 8

9 Typical Sequences RV X where p(x k ) = p k Consider a sequence x of length N where x k appears Np k times 1 2 p( x) = p p p K K Np log2 p Np pk ((2 ) k= 1 k= 1 = = Np log p = 2 = 2 = 2 Np Np Np 1 2 K K k= 1 NH(X) k k k k 2 k k= 1 K K N p log p k 2 k 9

10 Summary The Tchebycheff inequality was used to prove the weak law of large numbers (WLLN) the sample average approaches the statistical mean as The WLLN was used to prove the AEP N 1 logp(x n) H(X) as N N n= 1 A typical sequence has probability p( x1, x2,, x ) 2 N N - H(X) N There are about 2 NH(X) typical sequences of length N 10

11 Typical Sequences H( X) H( X) 11

12 Typical Sequences set of typical sequences nontypical or atypical sequence set of sequences of length N 12

13 Interpretation Although there are very many results that may be produced by a random process, the one actually produced is most probably from a set of outcomes that all have approximately the same chance of being the one actually realized. Although there are individual outcomes which may have a higher probability than outcomes in this set, the vast number of outcomes in the set almost guarantees that the outcome will come from the set. ``Almost all events are almost equally surprising Cover and Thomas 13

14 Typical Sequences From the definition, the probability of occurrence of a typical sequence p(x) is 14

15 Example p(x 1 ) = p(1) = 1/4 p(x 2 ) = p(0) = 3/4 H(X) = bit N = 3 p(x 1,x 1,x 1 ) = 1/64 p(x 1,x 1,x 2 ) = p(x 1,x 2,x 1 ) = p(x 2,x 1,x 1 ) = 3/64 p(x 1,x 2,x 2 ) = p(x 2,x 2,x 1 ) = p(x 2,x 1,x 2 ) = 9/64 p(x 2,x 2,x 2 ) = 27/64 15

16 Example H(X) = bit N = 3 b = 2 2 < p( x, x, x ) < 2 3[ δ] 3[.811 δ] x 1,x 1,x 1 1/64 = 2-3[ ] x 1,x 1,x 2 3/64 = 2-3[ ] x 1,x 2,x 2 9/64 = 2-3[ ] x 2,x 2,x 2 27/64 = 2-3[ ] 16

17 Example If δ = 0.2 the typical sequences are (x 1,x 2,x 2 ), (x 2,x 1,x 2 ), (x 2,x 2,x 1 ) with probability (1,0,0), (0,1,0), (0,0,1) If δ = 0.4 the typical sequences are (x 1,x 2,x 2 ), (x 2,x 1,x 2 ), (x 2,x 2,x 1 ), (x 2,x 2,x 2 ) with probability (1,0,0), (0,1,0), (0,0,1), (0,0,0) 17

18 18

19 Typical Sequences Random variable X Alphabet size K Entropy H(X) Arbitrary number δ>0 Sequences x of blocklength N N 0 and probability p(x) c X( ) + X( ) = K N 19

20 Shannon-McMillan Theorem 20

21 The essence of source coding or data compression is that as N, atypical sequences almost never appear as the output of the source. Therefore, one can focus on representing typical sequences with codewords and ignore atypical sequences. Since there are only 2 NH(X) typical sequences of length N, and they are approximately equiprobable, it takes about NH(X) bits to represent them. On average it takes H(X) bits per source output symbol. 21

22 Variable Length Codes Output Alphabet Y={y 1,,y J } Lengths 22

23 23

24 Instantaneous Codes Definition: A uniquely decodable code is said to be instantaneous if it is possible to decode each codeword in a sequence without reference to succeeding codewords. A necessary and sufficient condition for a code to be instantaneous is that no codeword is a prefix of some other codeword. 24

25 Variable Length Codes Prefix code (also prefix-free or instantaneous) C 1 ={0,10,110,111} Uniquely decodable code (which is not prefix) C 2 ={0,01,011,0111} Non-singular code (which is not uniquely decodable) C 3 ={0,1,00,11} Singular code C 4 ={0,10,11,10} Example sequence of codewords:

26 26

27 Average Codeword Length K L(C) = p( xk) l k= 1 k 27

28 Two Binary Prefix Codes Five source symbols: x 1, x 2, x 3, x 4, x 5 K = 5, J = 2 c 1 = 0, c 2 = 10, c 3 = 110, c 4 = 1110, c 5 = 1111 codeword lengths 1,2,3,4,4 c 1 = 00, c 2 = 01, c 3 = 10, c 4 = 110, c 5 = 111 codeword lengths 2,2,2,3,3 28

29 Kraft Inequality for Prefix Codes 29

30 Code Tree 30

31 Five Binary Codes Source symbols Code A Code B Code C Code D Code E x 1 x 2 x 3 x 4 31

32 Ternary Code Example Ten source symbols: x 1, x 2,, x 9, x 10 K = 10, J = 3 l k = 1,2,2,2,2,2,3,3,3,3 l k = 1,2,2,2,2,2,3,3,3 l k = 1,2,2,2,2,2,3,3,4,4 32

33 Average Codeword Length Bound L(C) H(X) log b J 33

34 Four Symbol Source Information Source H p(x 1 ) = 1/2 p(x 3 ) = 1/4 p(x 2 ) = p(x 4 ) = 1/8 H(X) = 1.75 bits x 1 0 x x 3 10 x L(C) = 1.75 bits x 1 00 x 2 01 x 3 10 x 4 11 L(C) = 2 bits 34

35 Code Efficiency H(X) ζ= 1 L(C)log b J First code ζ = 1.75/1.75 = 100% Second code ζ = 1.75/2.0 = 87.5% 35

36 Compact Codes A code C is called compact for a source X if its average codeword length L(C) is less than or equal to the average length of all other uniquely decodable codes for the same source and alphabet J. 36

37 Upper and Lower Bounds for a Compact Code H(X) log b J H(X) L(C) < + 1 log J b 37

38 The Shannon Algorithm Order the symbols from largest to smallest probability Choose the codeword lengths according to l = log p( x ) k J k Construct the codewords according to the cumulative probability P k P k k 1 = p( x ) 1 i= i 38

39 Example K = 10, J = 2 p(x 1 ) = p(x 2 ) = 1/4 p(x 3 ) = p(x 4 ) = 1/8 p(x 5 ) = p(x 6 ) = 1/16 p(x 7 ) = p(x 8 ) = p(x 9 ) = p(x 10 ) = 1/32 39

40 P k 40

41 Shannon Algorithm p(x 1 ) =.4 p(x 2 ) =.3 p(x 3 ) =.2 p(x 4 ) =.1 H(X) = 1.85 bits Shannon Code x 1 00 x 2 01 x x L(C) = 2.4 bits ζ = 77.1% Alternate Code x 1 0 x 2 10 x x L(C) = 1.9 bits ζ = 97.4% 41

42 Shannon s Noiseless Coding Theorem 42

43 Robert M. Fano ( ) 43

44 The Fano Algorithm Arrange the symbols in order of decreasing probability Divide the symbols into J approximately equally probable groups Each group receives one of the J code symbols as the first symbol This division process is repeated within the groups as many times as possible 44

45 Shannon Algorithm vs Fano Algorithm p(x 1 ) =.4 p(x 2 ) =.3 p(x 3 ) =.2 p(x 4 ) =.1 H(X) = 1.85 bits Shannon Code x 1 00 x 2 01 x x L(C) = 2.4 bits ζ = 77.1% Fano Code x 1 0 x 2 10 x x L(C) = 1.9 bits ζ = 97.4% 45

46 David Huffman ( ) 46

47 ``It was the most singular moment in my life. There was the absolute lightning of sudden realization. David Huffman ``Is that all there is to it! Robert Fano 47

48 The Binary Huffman Algorithm 1. Arrange the symbols in order of decreasing probability. 2. Assign a 1 to the last digit of the Kth codeword c K and a 0 to the last digit of the (K-1)th codeword c K-1. Note that this assignment is arbitrary. 3. Form a new source X with x k = x k, k = 1,, K-2, and x K-1 = x K-1 U x K p(x K-1 ) = p(x K-1 ) + p(x K ) 4. Rearrange the new set of probabilities, set K = K Repeat Steps 2 to 4 until all symbols have been combined. To obtain the codewords, trace back to the original symbols. 48

49 Five Symbol Source p(x 1 )=.35 p(x 2 )=.22 p(x 3 )=.18 p(x 4 )=.15 p(x 5 )=.10 H(X) = 2.2 bits 49

50 Midterm Test Tuesday, October 24, 2017 In class test (11:30-12:20) 20 marks total 20% of final grade One 8.5" 11 sheet of notes allowed both sides Calculator allowed but no cellphones 50

51 51

52 Huffman Code for the English Alphabet 52

53 Six Symbol Source p(x 1 )=.4 p(x 2 )=.3 p(x 3 )=.1 p(x 4 )=.1 p(x 5 )=.06 p(x 6 )=.04 H(X) = bits 53

54 Second Five Symbol Source p(x 1 )=.4 p(x 2 )=.2 p(x 3 )=.2 p(x 4 )=.1 p(x 5 )=.1 H(X) = bits 54

55 Two Huffman Codes x 1 x 1 x 2 x 2 x 3 x 3 x 4 x 4 x 5 x 5 C 1 C 2 55

56 Two Huffman Codes C 1 C 2 x x x x x

57 Second Five Symbol Source p(x 1 )=.4 p(x 2 )=.2 p(x 3 )=.2 p(x 4 )=.1 p(x 5 )=.1 H(X) = bits L(C) = 2.2 bits variance of code C 1 0.4(1-2.2) (2-2.2) (3-2.2) (4-2.2) 2 = 1.36 variance of code C 2 0.8(2-2.2) (3-2.2) 2 = 0.16 which code is preferable? 57

58 Nonbinary Codes The Huffman algorithm for nonbinary codes (J>2) follows the same procedure as for binary codes except that J symbols are combined at each stage. This requires that the number of symbols in the source X is K =J+c(J-1), K K c K J = J 1 58

59 Nonbinary Example J=3 K=6 K J c = c=2 so K =7 J 1 Add an extra symbol x 7 with p(x 7 )=0 p(x 1 )=1/3 p(x 2 )=1/6 p(x 3 )=1/6 p(x 4 )=1/9 p(x 5 )=1/9 p(x 6 )=1/9 p(x 7 )=0 H(X) = 1.54 trits 59

60 Codes for Different Output Alphabets K=13 p(x 1 )=1/4 p(x 2 )=1/4 p(x 3 )=1/16 p(x 4 )=1/16 p(x 5 )=1/16 p(x 6 )=1/16 p(x 7 )=1/16 p(x 8 )=1/16 p(x 9 )=1/16 p(x 10 )=1/64 p(x 11 )=1/64 p(x 12 )=1/64 p(x 13 )=1/64 J=2 to 13 60

61 Codes for Different Output Alphabets p(x i ) x i J x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x 11 x 12 x 13 L(C) 61

62 Code Efficiency ζ J 62

63 The Binary and Quaternary Codes x x x x x x x x x x x x x

64 Huffman Codes Symbol probabilities must be known a priori The redundancy of the code L(C)-H(X) (for J=b) is typically nonzero Error propagation can occur Codewords have variable length 64

65 Variable to Fixed Length Codes M sourcewords M J L Lengths m 1,m 2,,m M Variable to fixed length encoder 65

66 Variable to Fixed Length Codes Two questions: 1. What is the best mapping from sourcewords to codewords? 2. How to ensure unique encodability? 66

67 Average Bit Rate ABR = = average codeword length average sourceword length L L(S) M L(S) = p( si) m i= 1 M - number of sourcewords s i - sourceword i m i - length of sourceword i i 67

68 Variable to Fixed Length Codes Design criteria: minimize the Average Bit Rate L ABR= L(S) ABR H(X) (vs. L(C) H(X) for fixed to variable length codes) L(S) should be as large as possible so that the ABR is close to H(X) 68

69 Variable to Fixed Length Codes Fixed to variable length codes H(X) ζ= 1 L(C) Variable to fixed length codes H(X) ζ= 1 ABR 69

70 Binary Tunstall Code K=3, L=3 Let x 1 = a, x 2 = b and x 3 = c Unused codeword is

71 Tunstall Codes Tunstall codes must satisfy the Kraft inequality M i= 1 m K i 1 M - number of sourcewords K - source alphabet size m i - length of sourceword i 71

72 Binary Tunstall Code Construction Source X with K symbols Choose a codeword length L where 2 L > K 1. Form a tree with a root and K branches labelled with the symbols 2. If the number of leaves is greater than 2 L - (K-1), go to Step 4 3. Find the leaf with the highest probability and extend it to have K branches, go to Step 2 4. Assign codewords to the leaves 72

73 K=3, L=3 p(a) =.7, p(b) =.2, p(c) =.1 73

74 H(X) ζ = H(X)/ABR = 84.7% 74

75 The Codewords aaa 000 aab 001 aac 010 ab 011 ac 100 b 101 c

76 What if a or aa is left at the end of the sequence of source symbols? There are no corresponding codewords. Solution: use the unused codeword 111. a 1110 aa

77 Tunstall Code for a Binary Source L = 3, K = 2, J = 2, p(x 1 ) = 0.7, p(x 2 ) = 0.3 J L = 8 Seven sourcewords Eight sourcewords Codewords x 1 x 1 x 1 x 1 x 1 x 1 x 1 x 1 x 1 x x 1 x 1 x 1 x 1 x 2 x 1 x 1 x 1 x 1 x x 1 x 1 x 1 x 2 x 1 x 1 x 1 x x 1 x 1 x 2 x 1 x 1 x x 1 x 2 x 1 x 2 x x 2 x 1 x 1 x 2 x x 2 x 2 x 2 x x 2 x

78 Huffman Code for a Binary Source N = 3, K = 2, p(x 1 ) = 0.7, p(x 2 ) = 0.3 Eight sourcewords A = x 1 x 1 x 1 p(a) = B = x 1 x 1 x 2 p(b) = C = x 1 x 2 x 1 p(c) = D = x 2 x 1 x 1 p(d) = E = x 2 x 2 x 1 p(e) = F = x 2 x 1 x 2 p(f) = G = x 1 x 2 x 2 p(g) = H = x 2 x 2 x 2 p(h) =

79 Code Comparison H(X) =.8813 Tunstall Code (7 codewords) ABR =.9762 ζ = 90.3% Tunstall Code (8 codewords) ABR =.9138 ζ = 96.4% Huffman Code L(C) =.9087 ζ = 97.0% 79

80 Error Propagation Received Huffman codeword sequence A B A B A B Sequence with one bit error D F F 80

81 Huffman Coding The length of Huffman codewords has to be an integer number of symbols, while the selfinformation of the source symbols is almost always a non-integer. Thus the theoretical minimum message compression cannot always be achieved. For a binary source with p(x 1 ) =.1 and p(x 2 ) =.9 H(X) =.469 so the optimal average codeword length is.469 bits. An x 1 symbol should be encoded to l 1 =-log 2 (0.1)=3.32 bits and an x 2 symbol to l 2 =-log 2 (0.9)=.152 bits 81

82 Improving Huffman Coding One way to overcome the redundancy limitation is to encode blocks of several symbols. In this way the per-symbol inefficiency is spread over an entire block. N = 1: ζ = 46.9% N = 2: ζ = 72.7% However, using blocks is difficult to implement as there is a block for every possible combination of symbols, so the number of blocks increases exponentially with their length. 82

83 Peter Elias ( ) 83

84 Arithmetic Coding Arithmetic coding bypasses the idea of replacing an input symbol (or groups of symbols) with a specific codeword. Instead, a stream of input symbols is encoded to a single number in [0,1). Useful when dealing with sources with small alphabets, such as binary sources, and alphabets with highly skewed probabilities. 84

85 Arithmetic Coding Applications JPEG, MPEG-1, MPEG-2 Huffman and arithmetic coding JPEG2000, MPEG-4 ZIP Arithmetic coding only prediction by partial matching (PPMd) algorithm H.263, H

86 Arithmetic Coding The output of an arithmetic encoder is a stream of bits However we can think that there is a prefix 0, and the stream represents a fractional binary number between 0 and To explain the algorithm, numbers will be shown as decimal, but they are always binary in practice 86

87 Example 1 Encode string bccb from the alphabet {a,b,c} p(a) = p(b) = p(c) = 1/3 The arithmetic coder maintains two numbers, low and high, which represent a subinterval [low,high) of the range [0,1) Initially low=0 and high=1 87

88 Example 1 The range between low and high is divided between the symbols of the source alphabet according to their probabilities p(c)=1/3 p(b)=1/3 high c b p(a)=1/3 low 0 a 88

89 Example 1 b high c b high = low 0 a low =

90 Example 1 high p(c)=1/ c c high = p(b)=1/3 b p(a)=1/3 low a low =

91 Example 1 high c high = p(c)=1/ c p(b)=1/3 b p(a)=1/3 a low low =

92 Example 1 high b high = p(c)=1/ c p(b)=1/3 b p(a)=1/3 a low low =

93 Symbol Ranges Determine the cumulative probabilities k Pk = p( xi) k= 1,, K i= 1 where P 0 =0 and P K =1 The range for symbol x i is low = P i-1 high = P i The ranges divide [0,1) 93

94 Arithmetic Coding Algorithm Set low to 0.0 Set high to 1.0 While there are still input symbols Do get next input symbol range = high low high = low + range symbol_high_range low = low + range symbol_low_range End While output number between high and low 94

95 Arithmetic Coding Example 2 p(x 1 ) = 0.5, p(x 2 ) = 0.3, p(x 3 ) = 0.2 Symbol ranges: 0 x 1 <.5.5 x 2 <.8.8 x 3 < 1 low = 0.0 high = 1.0 Symbol sequence x 1 x 2 x 3 x 2 Iteration 1 x 1 : range = = 1.0 high = = 0.5 low = = 0.0 Iteration 2 x 2 : range = = 0.5 high = = 0.40 low = = 0.25 Iteration 3 x 3 : range = = 0.15 high = = 0.40 low = =

96 Arithmetic Coding Example 2 Iteration 3 x 3 : range = = 0.15 high = = 0.40 low = = 0.37 Iteration 4 x 2 : range = =.03 high = = low = = x 1 x 2 x 3 x 2 < = = The first 5 bits of the codeword are If there are no additional symbols to be encoded the codeword is

97 Arithmetic Coding Example 3 Suppose that we want to encode the message BILL GATES Character Probability Range SPACE 1/ r < 0.10 A 1/ r < 0.20 B 1/ r < 0.30 E 1/ r < 0.40 G 1/ r < r < 0.60 I 1/10 L 2/10 S 1/10 T 1/ r < r < r <

98 ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T ( ) A B E G I L S T

99 Arithmetic Coding Example 3 New Symbol Low Value High Value B I L L SPACE G A T E S

100 Binary Codeword in binary is in binary is The transmitted binary codeword is then bits long 100

101 Decoding Algorithm get encoded number Do find symbol whose range contains the encoded number output the symbol subtract symbol_low_range from the encoded number divide by the probability of the output symbol Until no more symbols 101

102 Decoding BILL GATES Encoded Number Output Symbol Low High Probability B I L L SPACE G A T E S

103 Finite Precision Symbol Probability (fraction) Interval (8-bit precision) fraction Interval (8-bit precision) binary Range in binary a 1/3 [0,85/256) [ , ) b 1/3 [85/256,171/256) [ , ) c 1/3 [171/256,1) [ , )

104 Renormalization Symbol Probability (fraction) Range Digits that can be output Range after renormalization a 1/ b 1/ c 1/ none

105 Terminating Symbol Symbol Probability (fraction) Interval (8-bit precision) fraction Interval (8-bit precision) binary Range in binary a 1/3 [0,85/256) [ , ) b 1/3 [85/256,170/256) [ , ) c 1/3 [170/256,255/256) [ , ) term 1/256 [255/256,1) [ , )

106 Huffman vs Arithmetic Codes X = {a,b,c,d} p(a) =.5, p(b) =.25, p(c) =.125, p(d) =.125 Huffman code a 0 b 10 c 110 d

107 Huffman vs Arithmetic Codes X = {a,b,c,d} p(a) =.5, p(b) =.25, p(c) =.125, p(d) =.125 Arithmetic code ranges a [0,.5) b [.5,.75) c [.75,.875) d [.875, 1) 107

108 Huffman vs Arithmetic Codes H(X)=2.75 bits L(C)=2.75 bits ζ=100% Encode abcdc Huffman codewords Arithmetic codeword both 12 bits 108

109 Huffman vs Arithmetic Codes X = {a,b,c,d} p(a) =.7, p(b) =.12, p(c) =.10, p(d) =.08 Huffman code a 0 b 10 c 110 d

110 Huffman vs Arithmetic Codes X = {a,b,c,d} p(a) =.7, p(b) =.12, p(c) =.10, p(d) =.08 Arithmetic code ranges a [0,.7) b [.7,.82) c [.82,.92) d [.92, 1) 110

111 Huffman vs Arithmetic Codes H(X)=1.351 bits L(C)=1.48 bits ζ=91.3% Encode aaab Huffman bits Arithmetic 01 2 bits Encode abcdaaa Huffman bits Arithmetic bits 111

112 Gadsby by Ernest Vincent Wright If youth, throughout all history, had had a champion to stand up for it; to show a doubting world that a child can think; and, possibly, do it practically; you wouldn t constantly run across folks today who claim that a child don t know anything. A child s brain starts functioning at birth; and has, amongst its many infant convolutions, thousands of dormant atoms, into which God has put a mystic possibility for noticing an adult s act, and figuring out its purport. Up to about its primary school days a child thinks, naturally, only of play. But many a form of play contains disciplinary factors. You can t do this, or that puts you out, shows a child that it must think, practically or fail. Now, if, throughout childhood, a brain has no opposition, it is plain that it will attain a position of status quo, as with our ordinary animals. Man knows not why a cow, dog or lion was not born with a brain on a par with ours; why such animals cannot add, subtract, or obtain from books and schooling, that paramount position which Man holds today. 112

113 Lossless Compression Techniques 1 Model and code The source is modelled as a random process. The probabilities (or statistics) are given or acquired. 2 Dictionary-based There is no explicit model and no explicit statistics gathering. Instead, a codebook (or dictionary) is used to map sourcewords into codewords. 113

114 Model and Code Huffman code Tunstall code Fano code Shannon code Arithmetic code 114

115 Dictionary-based Techniques Lempel-Ziv LZ77 sliding window LZ78 explicit dictionary Adaptive Huffman coding Due to patents, LZ77 and LZ78 led to many variants: LZ77 Variants LZR LZSS DEFLATE LZH LZB LZ78 Variants LZW LZC LZT LZMW LZJ LZFG Zip methods use LZH and LZR among other techniques UNIX compress uses LZC (a variant of LZW) 115

116 Dictionary-based Techniques 116

117 Lempel-Ziv Compression Source symbol sequences are replaced by codewords that are dynamically determined. The code table is encoded into the compressed data so it can be reconstructed during decoding. 117

118 Lempel-Ziv Example 118

119 119

120 120

121 Lempel-Ziv Codeword 121

122 Compression Comparison Compression as a percentage of the original file size File Type UNIX Compact Adaptive Huffman UNIX Compress Lempel-Ziv-Welch ASCII File 66% 44% Speech File 65% 64% Image File 94% 88% 122

123 Compression Comparison 123

Source Coding. Master Universitario en Ingeniería de Telecomunicación. I. Santamaría Universidad de Cantabria

Source Coding. Master Universitario en Ingeniería de Telecomunicación. I. Santamaría Universidad de Cantabria Source Coding Master Universitario en Ingeniería de Telecomunicación I. Santamaría Universidad de Cantabria Contents Introduction Asymptotic Equipartition Property Optimal Codes (Huffman Coding) Universal