2018/5/3. YU Xiangyu

Size: px

Start display at page:

Download "2018/5/3. YU Xiangyu"

Ashlee Pitts
5 years ago
Views:

1 2018/5/3 YU Xiangyu

3 Entropy Huffman Code

4 Entropy of Discrete Source Definition of entropy: If an information source X can generate n different messages x 1, x 2,, x i,, x n, then the average information content of the source: H ( X ) will be defined as the entropy of the source; where I (x i ) = - log 2 P(x i ) 4 E[ I( x i )] i 1 (bit) I (x i ) expresses the information content of x i H(X) can be regarded as average uncertainty of the source. n P( x i )log 2 P( xi ) Fan P342

8 Entropy of binary information source Assume there are only two different messages: 0 and 1, and assume the probability of transmitting 1 is P(1) =, then the probability of transmitting 0 is P(0) = 1 =. The entropy of the information source equals H( ) log P(1) log H( )~ curve 2 (1 )log When = ½, entropy of the source is maximal. Here the occurrence of the two messages is equal probability, so the uncertainty is maximal. When ½, one message is more likely to occur than the other one, hence the uncertainty decreases. If or equals 0, then the uncertainty is P(1) P(0) log 2 (1 ) 2 P(0) Fan P342

9 Entropy of n-ary information source Assume that the source can generate n different messages, and the occurring probability of the i-th message is denoted by P i, then the entropy of the source is H n i 1 P i log 2 P i Fan P342

10 Maximum of entropy: When P k = P n, the above equation equals 0. SinceP k is the occurring probability of arbitrary message, we have P1 P2 Pn Substituting the above equation into H i 1 obtain the maximum of H: n P i log 2 P i 1 n H log 2 n Fan P343

12 A significant amount of data compression can be realized when there is a wide difference in the probabilities of the symbols. To achieve this compression, there must also be a sufficiently large number of symbols. Sometimes, in order to have a large enough set of symbols, we form a new set of symbols derived from the original set, called an extension code. Sklar P860

14 According to the number of symbols in the codeword: equal length code and unequal length code According to the principle of decoding: uniquely decodable code and non-uniquely decodable code Uniquely decodable code: Instantaneous decodable code Non-instantaneous decodable code Example: Source character Non-instantaneous decodable code Instantaneous decodable code x x x x Fan P351

15 Uniquely Decodable Property. Uniquely decodable codes are those that allow us to invert the mapping to the original symbol alphabet. Prefix-Free Property. A sufficient (but not necessary) condition to assure that a code is uniquely decodeable is that no codeword be the prefix of any other code word. Codes that satisfy this condition are called prefix-free codes. Sklar P861

18 The Huffman code is a prefix-free, variable-length code that can achieve the shortest average code length n for a given input alphabet. The shortest average code length for a particular alphabet may be significantly greater than the entropy of the source alphabet. This inability to exploit the promised data compression is related to the alphabet, not to the coding technique. Sklar P862

19 Often the alphabet can be modified to form an extension code, and the same coding technique is then reapplied to achieve better compression performance. Compression performance is measured by the compression ratio. This measure is equal to the ratio of the average number of bits per sample before compression to the average number of bits per sample after compression. Sklar P862

20 The Huffman coding procedure can be applied for transforming between any two alphabets. We will demonstrate the application of the procedure between an arbitrary input alphabet and a binary output alphabet. The process starts by listing the input alphabet symbols, along with their probabilities (or relative frequencies), in descending order of occurrence. These tabular entries correspond to the branch ends of a tree, as shown in Figure Sklar P

21 Each branch is assigned a branch weight equal to the probability of that branch. The process now forms the tree that supports these branches. The two entries with the lowest relative frequency are merged (at a branch node) to form a new branch with their composite probability. After every merging, the new branch and the remaining branches are reordered (if necessary) to assure that the reduced table preserves the descending probability of occurrence. We call this reordering bubbling. Sklar P863

22 During the rearrangement after each merging, the mew branch rises through the table until it can rise no further. Thus, if we form a branch with a weight of 0.2 and during the bubbling process find two other branches already with the 0.2 weight, the new branch is bubbled to the top of the 0.2 group, as opposed to simply joining it. The bubbling to the top of the group results in a code with reduced code length variance but otherwise a code with the same average length as that obtained by simply joining the group. This reduced code length variance lowers the chance of buffer overflow. Sklar P863

23 As an example of this part of the code process, we will apply the Huffman procedure to the input alphabet shown in Figure The tabulated alphabet and the associated probabilities are shown on the figure. After forming the tree each branch node is labeled with a binary 1/0 decision to distinguish the two branches. The labeling is arbitrary, but for consistency, at each node we will label the branch going up with a 1 and the branch going down with a 0. After labeling the branch nodes, we trace the tree path from the base of the tree (far right) to each output branch (far left). The path contains the binary sequence to reach that branch Sklar P863

24 We find that the average code length n for this alphabet is 2.4 bits per character. It does not mean that we have to find a way to transmit a noninteger number of bits. Rather, it means that, on average, 240 bits will have to be moved through the communication channel when transmitting 100 input symbols. For comparison, a fixedlength code required to span the six-character input alphabet would be of length 3 bits, and the entropy of the input alphabet, using Equation (13.2), is 2.32 bits. Thus, this code offers a compression ratio of 1.25 (3.0/2.4) and achieves 96.7% (2.32/2.40) of the possible compression ratio. Sklar P864

25 Sklar P863

26 1 Source output Result of x 7 and x 8 combination Message Probability Message Probability x x x x x x x x x x x x x x x Encoded codewords x 1 10 x 2 11 x x x x x x Fan P353

30 In many applications, lengthy runs of specific symbols characterize a sequence of symbols to be transmitted or stored. Rather than code each symbol of a lengthy run, it makes sense to describe the run with an efficient substitution code. As an example, runs of spaces (the most common symbol in text) are encoded in many communication protocols by a control character followed by the character count. The IBM 3780 BISYNC protocol has an option to replace runs of spaces with an IGS character (if EBCDIC; or GS if ASCII), followed by a count of 2 to 63. Longer runs are partitioned into successive runs of 63 characters. Sklar P866

31 The run-length substitution coding can be applied to the original symbol alphabet or the binary representation of that alphabet. Runlength coding is particularly attractive for binary alphabets derived from specific sources. The most important commercial example is facsimile coding, used for transmitting documents by instant electronic mail. Sklar P866

32 Joint Photographic Experts Group (JPEG) JPEG is the common name given to the ISO/JPEG international standard or ITU- T Recommendation T.81 standard for Digital Compression of Continuous-Tone Still Images. JPEG is primarily known as a transform-based lossy compression scheme. Motion Picture Experts Group (MPEG) MPEG is a set of standards designed to support Coding of Moving Pictures and Associated Audio for digital storage media at up to 1.5 Mbits/s. MPEG-1, ISO standard approved in November 1992, was designed to permit full motion video recordings on CD players originally designed for stereo audio playback. MPEG-2, ISO standard or ITU T-recommendation H.262, Generic coding of Moving Pictures and Associated Audio. Sklar P

Run-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE

Run-length & Entropy Coding. Redundancy Removal. Sampling. Quantization. Perform inverse operations at the receiver EEE General e Image Coder Structure Motion Video x(s 1,s 2,t) or x(s 1,s 2 ) Natural Image Sampling A form of data compression; usually lossless, but can be lossy Redundancy Removal Lossless compression: predictive