Lec 05 Arithmetic Coding

Transcription

1 ECE 5578 Multimedia Communication Lec 05 Arithmetic Coding Zhu Li Dept of CSEE, UMKC web: phone: x2346 Z. Li, Multimedia Communciation, 208 p.

2 Outline Lecture 04 ReCap Arithmetic Coding About Homework- and Lab Z. Li, Multimedia Communciation, 208 p.2

3 JPEG Coding Block (8x8 pel) based coding DCT transform to find sparse representation Quantization reflects human visual system Zig-Zag scan to convert 2D to D string Run-Level pairs to have even more compact representation Hoffman Coding on Level Category Fixed on the Level with in the category = * Quant Table: Z. Li, Multimedia Communciation, 208 p.3

4 Coding of AC Coefficients Zigzag scanning: Example Example: zigzag scanning result EOB (Run, level) representation: (0, 24), (0, -3), (, -4), (0, -2), (, 6), (0, -2), (3, -), (0, -), (3, 2), (0, -2), (5, -), EOB Z. Li, Multimedia Communciation, 208 p.4

5 Coding of AC Coefficients Run / Catg. Base codeword Run / Catg. Base Codeword Run / Cat. Base codeword EOB ZRL 00 0/ 00 / 00 5/ 00 0/2 0 /2 0 5/2 00 0/3 00 /3 00 5/3 0 0/4 0 /4 00 5/ /5 00 /5 00 5/5 00 ZRL: represent 6 zeros when number of zeros exceeds 5. Example: 20 zeros followed by -: (ZRL), (4, -). (Run, Level) sequence: (0, 24), (0, -3), (, -4), Run/Cat. Sequence: 0/5, 0/5, /3, 24 is the 24-th entry in Category 5 (0, 24): is the 3-th entry in Category 3 (, -4): 00 0 Z. Li, Multimedia Communciation, 208 p.5

6 Outline Lecture 04 ReCap Arithmetic Coding Basic Encoding and Decoding Uniqueness and Efficiency Scaling and Incremental Coding Integer Implementation About Homework- and Lab Z. Li, Multimedia Communciation, 208 p.6

7 Arithmetic Coding The SciFi Story When I was in my 5 th grade. Aliens visit earth. Z. Li, Multimedia Communciation, 208 p.7

8 Huffman Coding Replacing an input symbol with a codeword Need a probability distribution Hard to adapt to changing statistics Need to store the codeword table Minimum codeword length is bit Arithmetic Coding Replace the entire input with a single floating-point number Does not need the probability distribution Adaptive coding is very easy No need to keep and send codeword table Fractional codeword length Z. Li, Multimedia Communciation, 208 p.8

9 Introduction Recall table look-up decoding of Huffman code N: alphabet size L: Max code word length Divide [0, 2^L] into N intervals 00 One interval for one symbol Interval size is roughly 00 0 proportional to symbol prob Arithmetic coding applies this idea recursively Normalizes the range [0, 2 L ] to [0, ]. Map an input sequence (multiple symbols) to a unique tag in [0, ). abcd.. dcba.. 0 Z. Li, Multimedia Communciation, 208 p.9

10 Arithmetic Coding Symbol set and prob: a (0.8), b(0.02), c(0.8) Disjoint and complete partition of the range [0, ) [0, 0.8), [0.8, 0.82), [0.82, ) Each interval corresponds to one symbol Interval size is proportional to symbol probability The first symbol restricts the tag position to be in one of the intervals The reduced interval is partitioned recursively as more symbols are processed Observation: once the tag falls into an interval, it never gets out of it a b c Z. Li, Multimedia Communciation, 208 p.0

11 Some Questions to think about: Why compression is achieved this way? How to implement it efficiently? How to decode the sequence? Why is it better than Huffman code? Z. Li, Multimedia Communciation, 208 p.

12 Example: Symbol Prob Map to real line range [0, ) Order does not matter Decoder needs to use the same order Disjoint but complete partition: : [0, 0.8): 0, : [0.8, 0.82): 0.8, : [0.82, ): 0.82, Z. Li, Multimedia Communciation, 208 p.2

13 Concept of the encoding (not practical) Input sequence: 32 Range Range Range Range Termination: Encode the lower end or midpoint to signal the end. Difficulties:. Shrinking of interval requires very high precision for long sequence. 2. No output is generated until the entire sequence has been processed. Z. Li, Multimedia Communciation, 208 p.3

14 Encoder Pseudo Code Cumulative Density Function (CDF) For continuous distribution: x ò - F ( x) = P( X x) = p( x) dx X Probability Mass Function For discrete distribution: X Properties: Non-decreasing Piece-wise constant å F ( i) = P( X i) = P( X = k) X k =- P( X = i) = F ( i) - F ( i -). X Each segment is closed at the lower end. i X.0 CDF X Z. Li, Multimedia Communciation, 208 p.4

15 Encoder Pseudo Code Keep track of LOW, HIGH, RANGE Any two are sufficient, e.g., LOW and RANGE. LOW=0.0, HIGH=.0; while (not EOF) { n = ReadSymbol(); RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-); } output LOW; Input HIGH LOW RANGE Initial *0.8= *0 = *= *0.82= *0.82= *0.8= *0= *0.8= Z. Li, Multimedia Communciation, 208 p.5

16 Concept of the Decoding Arithmetic Decoding (conceptual) Suppose encoder encodes the lower end: Decode Zoom In by * CDF Decode 3 Decode * CDF * CDF Decode Drawback: need to recalculate all thresholds each time. Z. Li, Multimedia Communciation, 208 p.6

17 Simplified Decoding (Floating Pt Ops) Normalize RANGE to [0, ) each time No need to recalculate the thresholds. Receive Decode 2 3 x x - low range x =( ) / 0.8 = Decode x =( ) / 0.8 = Decode 2 x =( ) / 0.02 = 0 Decode. Stop Z. Li, Multimedia Communciation, 208 p.7

18 Decoder Pseudo Code Low = 0; high = ; x = GetEncodedNumber(); While (x low) { n = DecodeOneSymbol(x); output symbol n; x = (x - CDF(n-)) / (CDF(n) - CDF(n-)); }; But this method still needs high-precision floating point operations. It also needs to read all input at the beginning of the decoding. Z. Li, Multimedia Communciation, 208 p.8

19 Outline Lecture 04 ReCap Arithmetic Coding Basic Encoding and Decoding Uniqueness and Efficiency Scaling and Incremental Coding About Homework- and Lab Z. Li, Multimedia Communciation, 208 p.9

20 Uniqueness Range Termination: Encode the midpoint (or lower end) to signal the end. How to represent the final tag uniquely and efficiently? Answer: Take the binary value of the tag T(X) and truncate to (X is the sequence {x,, x m }, not individual symbol) Proof: é ù l( X ) = log + bits. ( ) ê p X ú T(X) = F(X -) + p( X ), p(x) > 2 Assuming midpoint tag: First show the truncated code is unique, that is, code is within [F(X-), F(X)). ). ë û T ( X ) F( ) T(X) X ) T(X) So ë û l( X ) < X l ( bit longer than Shannon code is below the high end of the interval. Z. Li, Multimedia Communciation, 208 p.20 0.

21 Uniqueness and Efficiency 2). By def, 2 Together with -l( X ) = 2 æ é ù ö -ç log p( X ) + è ê ú ø 2 æ ö -ç log + è p( X ) ø T(X) = F(X -) + p( X ), p(x) > 2 T(X) - F(X -) = p( X ) ³ 2 2 T(X)- êë T(X) úû l( X ) - ( ) 2 l X = l(x) p( X ) F(X) T(X) ë û ³ F( X ). T(X) l X ) ( - ³ ( 2 -l X ) ( 2 -l X ) ët(x) û l( X ) Thus ët(x) û F( ) F( X -) X ) < X l ( F(X-) So the truncated code is still in the interval. This proves the uniqueness. Z. Li, Multimedia Communciation, 208 p.2

22 Uniqueness and Efficiency Prove the code is uniquely decodable (prefix free): Any code with prefix é ö êët(x) û, ët(x) û + l( X ) l( X ) l( X ) ø ë ët(x) û l( X ) is in 2 Need to show that this is in [F(X-), F(X) ): We already show ë û ³ F( X ). T(X) l X ) ( - F ( X ) - T(X) > X 2 Only need to show ë û l( X ) l( ) p( X ) F X ) - > 2 2 ët(x) û > F( X ) -T ( X ) = l( X ) l( ) ( X ët(x) û ( X ) é ù is prefix free if l( X ) = log + bits. l ( ) ê p X ú F(X) ë T(X) + 2 û l( X ) ët(x) û l( X ) F(X-) l( X ) Z. Li, Multimedia Communciation, 208 p.22

23 Uniqueness and Efficiency Efficiency of arithmetic code: bits. ) ( log ) ( + ú ú ù ê ê é = m m X p X l { } 2 ) ( ) ( log ) ( ) ( log ) ( ) ( ) ( + = ø ö ç ç è æ + + ø ö ç ç è æ ú + ú ù ê ê é = = å å m m m m m m m X H X p X P X p X P X l X p E L l(x) is the bits to code a sequence {x,x 2,, x m }. Assume iid sequence, ) ( ) ( X mh X H m = m X H m L X H 2 ) ( ) ( + L/m H(X) for large m, stronger than prev results. : {,..., } m X x x m Z. Li, Multimedia Communciation, 208 p.23

24 Uniqueness and Efficiency Comparison with Huffman code: Expected length of Huffman code: H ( X ) * L H ( X ) + Huffman code can reduce the overhead by jointly encoding more symbols, but needs much larger alphabet size. Arithmetic coding is more efficient for longer sequences. Z. Li, Multimedia Communciation, 208 p.24

25 Binary Arithmetic Coding Arithmetic coding is slow in general: To decode a symbol, we need a series of decisions and multiplications: While (Tag > LOW + RANGE * Sum(n) / N - ) { n++; } The complexity is greatly reduced if we have only two symbols: 0 and. symbol 0 symbol 0 x Only two intervals: [0, x), [x, ) Z. Li, Multimedia Communciation, 208 p.25

26 Encoding of Binary Arithmetic Coding HIGH LOW + RANGE CDF( n) LOW LOW + RANGE CDF( n -) LOW = 0, HIGH = Prob(0)=0.6. Sequence: 00 LOW = 0, HIGH = LOW = 0.36, HIGH = LOW = 0.504, HIGH = LOW = 0.504, HIGH = Only need to update LOW or HIGH for each symbol. Z. Li, Multimedia Communciation, 208 p.26

27 Decoding of Binary Arithmetic Coding Tag General case (integer implementation): While (Tag > LOW + RANGE * Sum(n) / N - )) { n++; } Binary case: only one condition to check if (Tag > LOW + RANGE * Sum(Symbol0) / N - ) { n = ; } else { n = 0; } Z. Li, Multimedia Communciation, 208 p.27

28 Applications of Binary Arithmetic Coding Increasingly popular: JBIG, JBIG2, JPEG2000, H.264 Convert non-binary signals into binary: Golomb-Rice Code: used in H.264. Bit-plane coding: used in JPEG2000. B = [B0, B,, BK-]: binary representation of B. Chain rule: H(B) = H(B0) +H(B B0) + + H(BK- B0, BK-2). To code B0, needs P0(0): Prob(B0=0). To code B, needs P(0 i): Prob(B=0 B0 = i), i = 0,. More details: AVC Binary Adaptive Arithmetic Coding: CABAC HEVC Arithmetic Coding: Z. Li, Multimedia Communciation, 208 p.28

29 Outline Lecture 04 ReCap Arithmetic Coding Basic Encoding and Decoding Uniqueness and Efficiency Scaling and Incremental Coding About Homework- and Lab Z. Li, Multimedia Communciation, 208 p.29

30 Scaling and Incremental Coding Problems of Previous examples: Need high precision No output is generated until the entire sequence is encoded. Decoder needs to read all input before decoding. Key Observation: As the RANGE reduces, many MSB s of LOW and HIGH become identical: Example: Binary form of and : , , We can output identical MSB s and re-scale the rest: Incremental encoding Can achieve infinite precision with finite-precision integers. Three scaling scenarios: E, E2, E3 Important Rules: Apply as many scalings as possible before further encoding and decoding. Z. Li, Multimedia Communciation, 208 p.30

31 E (lower half) and E 2 (higher half) Scaling E: [LOW HIGH) in [0, 0.5) LOW: 0.0xxxxxxx (binary), HIGH: 0.0xxxxxxx Output 0, then shift left by bit [0, 0.5) [0, ): E(x) = 2 x E2: [LOW HIGH) in [0.5, ) LOW: 0.xxxxxxx, HIGH: 0.xxxxxxx Output, subtract 0.5, shift left by bit [0.5, ) [0, ): E2(x) = 2(x - 0.5) The 3 rd 0 scaling, E3(mid), will be studied later: LOW < 0.5, HIGH > 0.5, but range < 0.5. Z. Li, Multimedia Communciation, 208 p.3

32 Encoding with E and E2 Input Input Input Symbol Prob E2: Output 2(x 0.5) E2: Output E: 2x, Output 0 E: Output 0 E: Output 0 Input E2: Output Encode any value in the tag, e.g., 0.5 Output All: 000 (0.7734) Z. Li, Multimedia Communciation,

33 To verify LOW = ( in binary), HIGH = ( in binary). So we can send out 000 (0.5325) Equivalent to E2 E E E E2 After left shift by 5 bits: LOW = ( ) x 32 = HIGH = ( ) x 32 = Same as the result in the last page. (In this example, suppose 7 bits are enough for the decoding) Z. Li, Multimedia Communciation, 208 p.33

34 Comparison with Huffman Rule: Complete all possible scaling before encoding the next symbol Symbol Prob. 0.8 Input Symbol does not cause any output Input Symbol 3 generates bit Input Symbol 2 generates 5 bits Symbols with larger probabilities generates less number of bits. Sometimes no bit is generated at all Advantage over Huffman coding Large probabilities are desired in arithmetic coding Can use context-adaptive method to create larger probability and to improve compression ratio Z. Li, Multimedia Communciation, 208 p.34

35 Incremental Decoding If input is 000, the st symbol can be decoded without ambiguity after 5 bits are read. When 6 bits are read, the status is: Read 5 bits: Decode. After reading 6 bits: Tag: 000, No scaling Decode 3, E2 scaling: Shift out bit, read bit Tag: 000 ( ) Decode 2, E2 scaling Tag: 0000 ( ) E: Tag: 0000 (0.875) E: Tag: 0000 (0.375) E: Tag: 0000 (0.75) E2: Tag: (0.5) Decode Z. Li, Multimedia Communciation, 208 p.35

36 Encoding Pseudo Code with E, E2 Rule: Complete all possible scalings before further decoding. Adjust LOW, HIGH and Tag together. (For floating-point implementation) EncodeSymbol(n) { //Update variables RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-); Symbol Prob. CDF } //Apply all possible scalings before encoding the //next symbol while LOW, HIGH in [0, 0.5) or [0.5, ) { Output 0 for E and for E2 scale LOW, HIGH by E or E2 rule } Z. Li, Multimedia Communciation, 208 p.36

37 Decoding Pseudo Code with E, E2 (For floating-point implementation) DecodeSymbol(Tag) { RANGE = HIGH - LOW; n = ; While ( (tag - LOW) / RANGE >= CDF(n) ) { n++; } HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-); Symbol Prob. CDF //keep scaling before decoding next symbol while LOW, HIGH in [0, 0.5) or [0.5, ) { scale LOW, HIGH by E or E2 rule read one more bit, update Tag } return n; } Z. Li, Multimedia Communciation, 208 p.37

38 E3 Scaling: [0.25, 0.75) [0, ) If RANGE straddles /2, E and E2 cannot be applied, but the range can be quite small Example: LOW=0.4999, HIGH=0.500 Binary: LOW=0.0., HIGH=0.000 We may not have enough bits to represent the interval. E3 Scaling: [0.25, 0.75) [0, ): E3(x) = 2(x ) Z. Li, Multimedia Communciation, 208 p.38

39 Example Previous example without E3: Input Input (E2: Output ) Input E2: Output E: Output With E3: Input E3: (x-0.25)x2 E2: Output state after E2 E3 = state after E E2, but outputs are different Z. Li, Multimedia Communciation,

40 Encoding Operation with E3 Without E3: Input E2: Output E: Output With E3: Input E3 (no output) E2: Output Output 0 here! Don t send anything when E3 is used, but send a 0 after E2: Same output, same final state Equivalent operations Z. Li, Multimedia Communciation,

41 Decoding for E3 0 Without E3: Input 000 Read 6 bits: Tag: 000 ( ) Decode Decode 3, E2 scaling Tag: 000 ( ) Decode 2, E2 scaling Tag: 0000 ( ) E: Tag: 0000 (0.875) With E3: Tag: 000 ( ) E3: Tag: 000 ( ) Decode 2, E2 scaling Tag: 0000 (0.875) Apply E3 whenever it is possible, everything else is same. Z. Li, Multimedia Communciation, 208 p.4

42 Summary of Different Scalings Need E scaling Need E2 scaling Need E3 scaling No scaling is required. Continue to encode/decode the next symbol. Z. Li, Multimedia Communciation, 208 p.42

43 HW- Info Theory: Entropy, Conditional Entropy, Mutual Info, Relative Entropy (KL Divergence) Hoffman Coding Residual Image Error statistics and Golomb coding Pixel value prediction filtering Residual error distribution modeling Optimal m selection in Golomb coding Z. Li, Multimedia Communciation, 208 p.43

44 Summary VLC is the real world image coding solution Elements of Hoffman and Golomb coding schemes are incorporated JPEG: introduced DC prediction, AC zigzag scan, run-level VLC H264: introduced reverse order coding. Z. Li, Multimedia Communciation, 208 p.44