Data Compression Techniques (Spring 2012) Model Solutions for Exercise 2

Transcription

1 Data Compression Techniques (Spring 22) Model Solutions for Exercise 2 If you have any feedback or corrections, please contact nvalimak at cs.helsinki.fi.. Problem: Construct a canonical prefix code for the alphabet {a, b, c, d, e, f} with the following codeword lengths symbol a b c d e f length Specify also the intervals associated with the codewords. Solution: First, sort the alphabet by their code lengths. The resulting intervals and codewords are: symbol c b d f a e length [ [ interval, 2) 2, 8) 5 [ 5 8, 3 [ 3 4) 4, 7 ) [ 7 8 8, 5 ) [ 5 6 6, ) code 2. Problem: Give the codes γ(n), δ(n) and GR 4 (n) for n {, 2, 4}. Solution: Let n =, and k = log(n + ) = log = 6. γ() = unary(k) binary k (n 2 k + ) = unary(6) binary 6 ( ) = { { δ() = γ(k) binary k (n 2 k + ) = γ(6) binary 6 ( ) = { { Let k = 4 and q = n 2 k = 2 4 = 6. GR 4 () = unary(q) binary k (n q 2 k ) = unary(6) binary 4 ( ) = {{ Let n = 2, and k = log(n + ) = log 2 = 7. γ(2) = unary(k) binary k (n 2 k + ) = unary(7) binary 7 ( ) = { { δ(2) = γ(k) binary k (n 2 k + ) = γ(7) binary 7 ( ) = { {

2 Let k = 4 and q = n 2 k = = 2. GR 4 (2) = unary(q) binary k (n q 2 k ) = unary(2) binary 4 ( ) = { { Let n = 4, and k = log(n + ) = log 4 = 8. γ(4) = unary(k) binary k (n 2 k + ) = unary(8) binary 8 ( ) = { { δ(4) = γ(k) binary k (n 2 k + ) = γ(8) binary 8 ( ) = {{ Let k = 4 and q = n 2 k = = 25. GR 4 (4) = unary(q) binary k (n q 2 k ) = unary(25) binary 4 ( ) = {{ 3. Problem: Use the Shannon-Fano algorithm (Exercise.3) to construct a prefix code for the set N of nonnegative integers and give the codes for the integers -9 using the following probability distributions. (a) P (n) = ( p)p n, where p = 2 /4.84. (b) P (n) = (n + ) 2 /ζ(2), where ζ(2) = π 2 /6.645 (ζ is the Riemann zeta function). Solution: Below is a Python3 solution for the problem. The code words are in Table 3. Figure 3 shows the recursion tree of the algorithm for part (a). Note: the self-information of n in part (a) is log(( p)p n ) n/ , which is close to the code length of the GR 2 code, which is n/ Correspondingly, the self-information of n in part (b) is log((n+) 2 /ζ(2)) 2 log(n+)+.7, which is close to the length of the gamma code of n, which is 2 log(n+) +. import math 2 3 def P(x): 4 p = 2**(-/4) 5 return (-p) * p**x 6 7 def P2(x): 8 return (x+)**(-2) / (math.pi**2 / 6) 9 # A recursive Shannon-Fano coding function. Parameters: # P: a probability distribution for the integers 2 # [left, right]: range of integers to be coded 3 # total: total probability of integers in the range [left, right] 2

3 4 # codeword: longest prefix shared by all codewords in the range [left, right] 5 def shannon_fano(p, left, right, total, codeword): 6 7 if left > 9: 8 # Only interested in codewords for integers -9 9 return 2 2 if left == right: # Recursion bottom 22 print(str(left) + ": " + codeword) 23 return # Find the best split point 26 best_split = # Last index in the left side of the best split 27 split_sum = # Sum of probabilities in the left side of the split 28 smallest_bias = # Bias means abs(split_sum - total/2) 29 for x in range(left,right+): 3 split_sum += P(x) 3 if abs(split_sum - total/2) < smallest_bias: 32 smallest_bias = abs(split_sum - total/2) 33 best_split = x 34 else: break # Recurse into the left and right parts 37 left_total = sum(map(p, [x for x in range(left,best_split+)])) 38 right_total = total - left_total 39 shannon_fano(p, left, best_split, left_total, codeword + "") 4 shannon_fano(p, best_split+, right, right_total, codeword + "") 4 42 shannon_fano(p,,**9,,"") 43 shannon_fano(p2,,**9,,"") number (a) codes (b) codes Table : Code words generated by the Shannon-Fano algorithm 4. Problem: Let x, x 2,..., x n be a sequence of n non-negative integers with an average value x. Show that if we encode the sequence using a Golomb-Rice code GR k with a suitable k, the average code length is less than log( x + ) + 2. Compare the result to Elias δ code length. Solution: The average length of the code words for x, x 2,..., x n is: n GR k (x) = n ( x i 2 k + k + ) n ( x i 2 k + k + ) = 2 k ( n x i ) + n 3 (k + ) = x 2 k + k + ()

4 ,,2,3, /2,,2,3 4,5,6,... /4, 2,3 4,5,6,7 8,9,, / ,5 6,7 8,9,, / ,9... / Figure : Recursion of the Shannon-Fano algorithn in problem 3a. Selecting k = log 2 x gives the bound log 2 ( x) + 2. However, if x is close to zero, then k = log 2 ( x) is negative, which does not work, because parameter k can not be negative. We can fix this by choosing k = log 2 ( x + ), which gives the desired bound log 2 ( x + ) + 2. Let s now compare this result with the code lengths of delta coding. The length of the delta code of an integer x is log(x + 2) + 2 log log(x + 2), which is asymptotically larger than log 2 (x + ) + 2 because of the loglog term in the delta coding. See Figure 4 for a plot of both functions for comparison. Note: the choice k = log 2 x gives the bound that was asked for in the problem statement, but is this choice of k optimal? To find the best possible k, let us find the zero of the derivative of the average code length. The derivative is: d dk ( x 2 k + k + ) = x ln(2)e k ln(2) + Setting this to zero and solving for k gives k = log 2 ( x) + log 2 ln 2. Since log 2 ln 2 is only approximately.69, we see that k = log 2 ( x + ) was quite close to optimal. 5. Problem: Assume that the model of computation supports the following operations (in addition to standard arithmetic operations) in constant time: Read up to w consecutive bits starting from any position in a binary string, and interpret the bits as a binary representation of an integer. Determine the position of the left-most -bit in a w-bit integer. Suppose we have encoded a sequence of integers in the range [..2 w ] using the Elias -code. Show how the sequence can be decoded in constant time per integer. Solution: Suppose we want to decode a bit string B representing a concatenation of gammacoded integers. The number of leading ones in B is the parameter k in the gamma code of the first integer. The number of leading ones can be computed by computing the left-most -bit in B using second the operation given in the problem statement. The k bits after the zero are the binary part of the gamma code of the first integer, which can be interpreted using the first operation in the problem statement. In other words, the value of the first integer is: 2 k + InterpretBinary(B[(k + )..2k]) 4

5 Golomb-Rice Elias delta Figure 2: Golomb rice ( log 2 (x + ) + 2) versus Elias delta ( log(x + 2) + 2 log log(x + 2) ). After this we can forget about the first 2k + bits of B and repeat the procedure to decode the next integer, until all integers coded in B have been decoded. 6. Problem: What is (a) the entropy (b) the average Huffman code length of the following distribution symbol a b c d e f g prob. /6 /5 /2 /5 /2 /8 /4 Solution: The entropy is H(P ) 2.5 and the average Huffman code length is about See the lecture slides for details. 5