Outline CS/EE 5590 / ENG 40 Special Topics (7804, 785, 7803 Lec 05 Arithetic Coding Lecture 04 ReCap Arithetic Coding About Hoework- and Lab Zhu Li Course Web: http://l.web.ukc.edu/lizhu/teaching/06sp.video-counication/ain.htl Z. Li Multiedia Counciation, 06 Spring p. Z. Li Multiedia Counciation, 06 Spring p. JPEG Coding Coding of AC Coefficients Block (8x8 pel based coding DCT transfor to find sparse representation Quantization reflects huan visual syste Zig-Zag scan to convert D to D string Run-Level pairs to have even ore copact representation Hoffan Coding on Level Category Fixed on the Level with in the category = * Quant Table: Zigzag scanning: Exaple 8 4-0 0 0 0 0-3 -4 6-0 0 0 0 0 - - 0 0 0 0 0 0 - - 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Exaple: zigzag scanning result 4-3 0-4 - 0 6-0 0 0 - - 0 0 0-0 0 0 0 0 - EOB (Run, level representation: (0, 4, (0, -3, (, -4, (0, -, (, 6, (0, -, (3, -, (0, -, (3,, (0, -, (5, -, EOB Z. Li Multiedia Counciation, 06 Spring p.3 Z. Li Multiedia Counciation, 06 Spring p.4
Run / Catg. Base codeword Coding of AC Coefficients Run / Catg. Base Codeword Run / Cat. Base codeword EOB 00 - - ZRL 00 0/ 00 / 00 5/ 00 0/ 0 / 0 5/ 00 0/3 00 /3 00 5/3 0 0/4 0 /4 00 5/4 000 0/5 00 /5 00 5/5 00 ZRL: represent 6 zeros when nuber of zeros exceeds 5. Exaple: 0 zeros followed by -: (ZRL, (4, -. (Run, Level sequence: (0, 4, (0, -3, (, -4, Run/Cat. Sequence: 0/5, 0/5, /3, 4 is the 4-th entry in Category 5 (0, 4: 00 000-4 is the 3-th entry in Category 3 (, -4: 00 0 Lecture 04 ReCap Outline Arithetic Coding Basic Encoding and Decoding Uniqueness and Efficiency Scaling and Increental Coding Integer Ipleentation About Hoework- and Lab Thanks for Prof. Jie Liang s Slides on Arithetic Coding, very rich in details! Z. Li Multiedia Counciation, 06 Spring p.5 Z. Li Multiedia Counciation, 06 Spring p.6 Arithetic Coding The SciFi Story When I was in y 5 th grade. Aliens visit earth. Z. Li Multiedia Counciation, 06 Spring p.7 Huffan Coding Replacing an input sybol with a codeword Need a probability distribution Hard to adapt to changing statistics Need to store the codeword table Miniu codeword length is bit Arithetic Coding Replace the entire input with a single floating-point nuber Does not need the probability distribution Adaptive coding is very easy No need to keep and send codeword table Fractional codeword length Z. Li Multiedia Counciation, 06 Spring p.8
Introduction Recall table look-up decoding of Huffan code N: alphabet size L: Max code word length Divide [0, ^L] into N intervals 00 One interval for one sybol Interval size is roughly 00 0 proportional to sybol prob. 000 00 0 00 Arithetic coding applies this idea recursively Noralizes the range [0, L ] to [0, ]. Map an input sequence (ultiple sybols to a unique tag in [0,. abcd.. dcba.. 0 Z. Li Multiedia Counciation, 06 Spring p.9 Arithetic Coding Sybol set and prob: a (0.8, b(0.0, c(0.8 Disjoint and coplete partition of the range [0, [0, 0.8, [0.8, 0.8, [0.8, Each interval corresponds to one sybol Interval size is proportional to sybol probability The first sybol restricts the tag position to be in one of the intervals The reduced interval is partitioned recursively as ore sybols are processed. 0 0 0 0 Observation: once the tag falls into an interval, it never gets out of it Z. Li Multiedia Counciation, 06 Spring p.0 a b c Soe Questions to think about: Exaple: Why copression is achieved this way? How to ipleent it efficiently? How to decode the sequence? Why is it better than Huffan code? Sybol Prob. 0.8 0.0 3 0.8 3 0 0.8 0.8.0 Map to real line range [0, Order does not atter Decoder needs to use the sae order Disjoint but coplete partition: : [0, 0.8: 0, 0.799999 9 : [0.8, 0.8: 0.8, 0.89999 9 3: [0.8, : 0.8, 0.999999 9 Z. Li Multiedia Counciation, 06 Spring p. Z. Li Multiedia Counciation, 06 Spring p.
Range 0.44 Range 0.0088 Concept of the encoding (not practical Input sequence: 3 Range Range 0.8 3 0 0.8 0.8.0 3 0 0.64 0.656 0.8 3 0.656 0.77 0.77408 0.8 3 0.77 0.773504 0.773566 0.77408 Terination: Encode the lower end or idpoint to signal the end. Difficulties:. Shrinking of interval requires very high precision for long sequence.. No output is generated until the entire sequence has been processed. Z. Li Multiedia Counciation, 06 Spring p.3 Encoder Pseudo Code Cuulative Density Function (CDF For continuous distribution: x F ( x P( x p( x dx For discrete distribution: F ( i P( i P( k k P( i F ( i F ( i. Properties: Non-decreasing Piece-wise constant Each segent is closed at the lower end. i Probability Mass Function 0. 0. 3 4 CDF 0. 0.4 0.4 0.8 0. 3 4 Z. Li Multiedia Counciation, 06 Spring p.4.0 Keep track of LOW, HIGH, RANGE Any two are sufficient, e.g., LOW and RANGE. Encoder Pseudo Code LOW=0.0, HIGH=.0; while (not EOF { n = ReadSybo; RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n; LOW = LOW + RANGE * CDF(n-; output LOW; Concept of the Decoding Arithetic Decoding (conceptual Suppose encoder encodes the lower end: 0.77 3 Decode 0 0.8 0.8.0 3 Decode 3 0 0.64 0.656 0.8 Zoo In by * CDF * CDF Input HIGH LOW RANGE Initial.0 0.0.0 0.0+.0*0.8=0.8 0.0+.0*0 = 0.0 0.8 3 0.0 + 0.8*=0.8 0.0 + 0.8*0.8=0.656 0.44 0.656+0.44*0.8=0.77408 0.656+0.44*0.8=0.77 0.0088 0.77+0.0088*0=0.77 0.77+0.0088*0.8=0.773504 0.00304 Z. Li Multiedia Counciation, 06 Spring p.5 3 Decode * CDF 0.656 0.77 0.77408 0.8 3 Decode 0.77 0.773504 0.773566 0.77408 Drawback: need to recalculate all thresholds each tie.
Noralize RANGE to [0, each tie No need to recalculate the thresholds. Receive 0.77 Decode x =(0.77-0 / 0.8 = 0.964 Decode 3 3 0 0.8 0.8.0 3 0 0.8 0.8.0 x =(0.964-0.8 / 0.8 = 0.8 3 Decode x =(0.8-0.8 / 0.0 = 0 Decode. Stop. Siplified Decoding (Floating Pt Ops 0 0.8 0.8.0 3 x 0 0.8 0.8.0 x low range Low = 0; high = ; x = GetEncodedNuber(; Decoder Pseudo Code While (x low { n = DecodeOneSybox; output sybol n; x = (x - CDF(n- / (CDF(n - CDF(n-; ; 00 00 0 000 00 0 00 But this ethod still needs high-precision floating point operations. It also needs to read all input at the beginning of the decoding. Outline Uniqueness Lecture 04 ReCap Arithetic Coding Basic Encoding and Decoding Uniqueness and Efficiency Scaling and Increental Coding About Hoework- and Lab Z. Li Multiedia Counciation, 06 Spring p.9 Range 0.0088 3 0.77 0.773504 0.773566 0.77408 Terination: Encode the idpoint (or lower end to signal the end. How to represent the final tag uniquely and efficiently? Answer: Take the binary value of the tag T( and truncate to ( is the sequence {x,, x, not individual sybol log bits. ( p Proof: Assuing idpoint tag: T( F( - p(, p( 0. First show the truncated code is unique, that is, code is within [F(-, F(.. T ( F( T( l T( So ( bit longer than Shannon code is below the high end of the interval. Z. Li Multiedia Counciation, 06 Spring p.0
. By def, Together with Thus l ( Uniqueness and Efficiency log p( T( F( - T( - F( - T(- T( l ( log p( p(, p( p( ( l F(. T( l ( T( F( F( l ( p( 0. - ( l So the truncated code is still in the interval. This proves the uniqueness. F( T( ( l T( F(- Uniqueness and Efficiency Prove the code is uniquely decodable (prefix free: Any code with prefix T( is in T(, T( l ( l ( Need to show that this is in [F(-, F( : We already show F(. T( l ( F( T( Only need to show p( F( T( F( T ( T( is prefix free if log bits. ( p F( T( T( l ( F(- Z. Li Multiedia Counciation, 06 Spring p. Z. Li Multiedia Counciation, 06 Spring p. Efficiency of arithetic code: L E p( P( Uniqueness and Efficiency log p( log p( P( bits. log p( ( H is the bits to code a sequence {x,x,, x. Assue iid sequence, H ( H ( L H ( H ( x x : {,..., L/ H( for large, stronger than prev results. Z. Li Multiedia Counciation, 06 Spring p.3 Uniqueness and Efficiency Coparison with Huffan code: Expected length of Huffan code: H ( L * H ( Huffan code can reduce the overhead by jointly encoding ore sybols, but needs uch larger alphabet size. Arithetic coding is ore efficient for longer sequences. Z. Li Multiedia Counciation, 06 Spring p.4
Lecture 04 ReCap Outline Arithetic Coding Basic Encoding and Decoding Uniqueness and Efficiency Scaling and Increental Coding About Hoework- and Lab Scaling and Increental Coding Probles of Previous exaples: Need high precision No output is generated until the entire sequence is encoded. Decoder needs to read all input before decoding. Key Observation: As the RANGE reduces, any MSB s of LOW and HIGH becoe identical: Exaple: Binary for of 0.77 and 0.773504: 0.0000.., 0.000.., We can output identical MSB s and re-scale the rest: Increental encoding Can achieve infinite precision with finite-precision integers. Three scaling scenarios: E, E, E3 Iportant Rules: Apply as any scalings as possible before further encoding and decoding. Z. Li Multiedia Counciation, 06 Spring p.5 Z. Li Multiedia Counciation, 06 Spring p.6 E (lower half and E (higher half Scaling E: [LOW HIGH in [0, 0.5 LOW: 0.0xxxxxxx (binary, HIGH: 0.0xxxxxxx. Output 0, then shift left by bit [0, 0.5 [0, : E(x = x E: [LOW HIGH in [0.5, LOW: 0.xxxxxxx, HIGH: 0.xxxxxxx. 0 0.5.0 0 0.5.0 0 0.5.0 Encoding with E and E Input 0 0.8.0 Input 3 0 0.656 0.8 Input 0.3 0.544 0.5486 0.6 0.0848 0.0963 0.696 0.964 Sybol Prob. 0.8 0.0 3 0.8 E: Output (x 0.5 E: Output E: x, Output 0 E: Output 0 Output, subtract 0.5, shift left by bit [0.5, [0, : E(x = (x - 0.5 0 0.5.0 The 3 rd 0 scaling, E3(id, will be studied later: 0.5.0 LOW < 0.5, HIGH > 0.5, but range < 0.5. Z. Li Multiedia Counciation, 06 Spring p.7 Input 0.339 0.3858 E: Output 0 0.6784 0.77056 E: Output 0.3568 0.54 Encode any value in the tag, e.g., 0.5 Output 0.3568 0.50456 All: 000 (0.7734
To verify LOW = 0.544 (0.00000... in binary, HIGH = 0.5486 (0.00000... in binary. So we can send out 000 (0.535 Equivalent to EEEEE After left shift by 5 bits: LOW = (0.544 0.535 x 3 = 0.3568 HIGH = (0.5486 0.535 x 3 = 0.54 Sae as the result in the last page. (In this exaple, suppose 7 bits are enough for the decoding Z. Li, Multiedia Counciation, 06 Spring p.9 Coparison with Huffan Rule: Coplete all possible scaling before encoding the next sybol Input Sybol does not cause any output Input Sybol 3 generates bit Input Sybol generates 5 bits Sybol Prob. 0.8 0.0 3 0.8 Sybols with larger probabilities generates less nuber of bits. Soeties no bit is generated at all Advantage over Huffan coding Large probabilities are desired in arithetic coding Can use context-adaptive ethod to create larger probability and to iprove copression ratio. Z. Li, Multiedia Counciation, 06 Spring p.30 Increental Decoding If input is 000, the st sybol can be decoded without abiguity after 5 bits are read. When 6 bits are read, the status is: Read 5 bits: Decode. 0 0.8.0 After reading 6 bits: Tag: 000, 0.76565 No scaling. 0 0.656 0.8 Decode 3, E scaling: Shift out bit, read bit 0.3 0.544 0.5486 0.6 Tag: 000 (0.546875 Decode, E scaling Tag: 0000 (0.09375 0.0848 0.0963 E: Tag: 0000 (0.875 0.696 0.964 0.339 0.3858 0.6784 0.77056 0.3568 0.54 E: Tag: 0000 (0.375 E: Tag: 0000 (0.75 E: Tag: 00000 (0.5 Decode Z. Li, Multiedia Counciation, 06 Spring p.3 Encoding Pseudo Code with E, E Rule: Coplete all possible scalings before further decoding. Adjust LOW, HIGH and Tag together. (For floating-point ipleentation EncodeSybon { //Update variables RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n; LOW = LOW + RANGE * CDF(n-; Sybol Prob. CDF 0.8 0.8 0.0 0.8 3 0.8 //Apply all possible scalings before encoding the //next sybol while LOW, HIGH in [0, 0.5 or [0.5, { Output 0 for E and for E scale LOW, HIGH by E or E rule Z. Li, Multiedia Counciation, 06 Spring p.3
Decoding Pseudo Code with E, E (For floating-point ipleentation DecodeSyboTag { RANGE = HIGH - LOW; n = ; While ( (tag - LOW / RANGE >= CDF(n { n++; HIGH = LOW + RANGE * CDF(n; LOW = LOW + RANGE * CDF(n-; //keep scaling before decoding next sybol while LOW, HIGH in [0, 0.5 or [0.5, { scale LOW, HIGH by E or E rule read one ore bit, update Tag return n; Sybol Prob. CDF 0.8 0.8 0.0 0.8 3 0.8 E3 Scaling: [0.5, 0.75[0, If RANGE straddles /, E and E cannot be applied, but the range can be quite sall Exaple: LOW=0.4999, HIGH=0.500 Binary: LOW=0.0., HIGH=0.000 We ay not have enough bits to represent the interval. E3 Scaling: [0.5, 0.75 [0, : E3(x = (x - 0.5 0 0.5 0.5 0.75 0 0.5 Z. Li, Multiedia Counciation, 06 Spring p.33 Z. Li, Multiedia Counciation, 06 Spring p.34 Exaple Input 0 0.8.0 Input 3 0 0.656 0.8 Input 0.3 0.544 0.5486 0.6 0.0848 0.0963 0.696 0.964 With E3: 0.3 0.6 Input Previous exaple without E3: 0.4 0.5848 0.5963 0.7 (E: Output E: Output E: Output 0 E3: (x-0.5x E: Output 0.696 0.964 state after E E3 = state after E E, but outputs are different Encoding Operation with E3 Without E3: Input 0.3 0.544 0.5486 0.6 0.0848 0.0963 0.696 0.964 With E3: 0.3 0.6 Input E: Output E: Output 0 0.4 0.5848 0.5963 0.7 E3 (no output 0.696 0.964 E: Output Don t send anything when E3 is used, but send a 0 after E: Sae output, sae final state Equivalent operations Output 0 here!
Decoding for E3 0 Without E3: 0.8.0 Input 000 Read 6 bits: Tag: 000 (0.76565 Decode 0 0.656 0.8 Decode 3, E scaling Tag: 000 (0.546875 0.3 0.544 0.5486 0.6 0.0848 0.0963 Decode, E scaling Tag: 0000 (0.09375 0.696 0.964 E: Tag: 0000 (0.875 With E3: 0.3 0.6 Tag: 000 (0.546875 0.5848 0.5963 0.4 0.7 E3: Tag: 000 (0.59375 0.696 0.964 Decode, E scaling Tag: 0000 (0.875 Apply E3 whenever it is possible, everything else is sae. Z. Li, Multiedia Counciation, 06 Spring p.37 Suary of Different Scalings 0 0.5 0.5 0.75.0 Need E scaling Need E scaling Need E3 scaling No scaling is required. Continue to encode/decode the next sybol. Z. Li, Multiedia Counciation, 06 Spring p.38 Binary Arithetic Coding Arithetic coding is slow in general: To decode a sybol, we need a series of decisions and ultiplications: While (Tag > LOW + RANGE * Su(n / N - { n++; The coplexity is greatly reduced if we have only two sybols: 0 and. sybol 0 sybol 0 x Only two intervals: [0, x, [x, Z. Li, Multiedia Counciation, 06 Spring p.39 Encoding of Binary Arithetic Coding HIGH LOW RANGE CDF( n LOW LOW RANGE CDF( n LOW = 0, HIGH = Prob(0=0.6. Sequence: 00 LOW = 0, HIGH = 0.6 0 0.6 LOW = 0.36, HIGH = 0.6 0 0.36 0.6 LOW = 0.504, HIGH = 0.6 0.36 0.504 0.6 LOW = 0.504, HIGH = 0.566 0.504 0.566 0.6 Only need to update LOW or HIGH for each sybol. Z. Li, Multiedia Counciation, 06 Spring p.40
Decoding of Binary Arithetic Coding Tag 0 0.6 General case (integer ipleentation: While (Tag > LOW + RANGE * Su(n / N - { n++; Binary case: only one condition to check if (Tag > LOW + RANGE * Su(Sybol0 / N - { n = ; else { n = 0; Applications of Binary Arithetic Coding Increasingly popular: JBIG, JBIG, JPEG000, H.64 Convert non-binary signals into binary: Golob-Rice Code: used in H.64. Bit-plane coding: used in JPEG000. B = [B0, B,, BK-]: binary representation of B. Chain rule: H(B = H(B0 +H(B B0 + + H(BK- B0, BK-. To code B0, needs P0(0: Prob(B0=0. To code B, needs P(0 i: Prob(B=0 B0 = i, i = 0,. More details: AVC Binary Adaptive Arithetic Coding: CABAC HEVC Arithetic Coding: Z. Li, Multiedia Counciation, 06 Spring p.4 Z. Li, Multiedia Counciation, 06 Spring p.4 HW- Info Theory: Entropy, Conditional Entropy, Mutual Info, Relative Entropy (KL Divergence Hoffan Coding Residual Iage Error statistics and Golob coding Pixel value prediction filtering Residual error distribution odeling Optial selection in Golob coding JPEG VLC schees: DCT, Quantization Zigzag, RUN-LEVEL Suary VLC is the real world iage coding solution Eleents of Hoffan and Golob coding schees are incorporated JPEG: introduced DC prediction, AC zigzag scan, run-level VLC H64: introduced reverse order coding. Z. Li Multiedia Counciation, 06 Spring p.43 Z. Li Multiedia Counciation, 06 Spring p.44