Introduction to Video Compression H.6 Dirk Farin, Contact address: Dirk Farin University of Mannheim Dept. Computer Science IV L 5,6, 683 Mannheim, Germany farin@uni-mannheim.de D.F. YUV-Colorspace Computer hardware usually uses RGB colorspace for display. Video hardware uses YUV colorspace for transmission. Historical reasons: backward compatible to black/white TV. Imitation of human visual system (HVS). Allows better compression. Luminance channel Y, two chrominance channels U,V. HVS has more receptors for Y than for U,V. Spatial resolution of U,V can be reduced (e.g., 4::0) Y - luminance U - chrominance V - chrominance
YUV to RGB conversion Since U,V can be negative, an offset of 8 is added. Y U V = 0.6 0.5 0.44 0.50 0.9 0.3 0. 0.44 0.0 calculation of greyscale value according to preceived intensities R G B + 6 8 8 Grey: (Y U V) = ( x 8 8) Black: (Y U V) = (6 8 8) R G B = 0 0.34.3.3 0.0 0 Y 6 U 8 V 8 3 Transform Coding Exploit correlation between data. x Independent coding two variables with high entropy x Decorrelated data only one variable has high entropy x' x' = x x x x' x' x 4
Transform coding / Cosine transform (3) -D DCT transform cosine wave basis vectors can be used to form -D basis images 8x8 transform has 64 basis images of 8x8 samples 5 Transform coding / Cosine transform () -D Inverse Discrete Cosine Transform (idct) Orthogonal transform f ( x, y) = x + C( u) C( v) ( F( u, v) cos Separable as two times a -D transform, thus (y + ) uπ ) vπ cos u= 0 v= 0 6 6 + = + C( v) C( u) (x ) uπ ( y ) vπ f ( x, y) F ( u, v) cos cos v= 0 u= 0 6 6 Implementation as: transform rows => transpose => transform rows 6 3
Transform coding / Quantization weighting TC Weighting HVS has strong frequency dependence can be exploited for weighting of coefficients contrast sensitivity 000 00 0 0. 0 spatial freq. (cycle/degr) HVS sensitivity for sine wave gratings Weighting matrix, N=8 8 8 8 8 8 6 5 8 8 8 6 5 4 4 8 8 6 5 4 4 8 6 5 4 4 4 8 6 6 5 5 4 4 4 5 5 5 4 4 4 4 6 4 4 4 4 4 4 4 5 4 4 4 4 4 4 4 normalize with /8 Intraframe coder intraframe coder/decoder block diagram local encoding reconstruction for motion compensation mem DCT scan Q VLC video input transform scanning weighting quantize qscale IQ inv scan rate control inverse quantize inverse scanning macroblocks Buffer frame MB mem. inv DCT inverse transform 8 4
Quantization () MPEG Quantization for inter- / intraframe data DC coefficient Human eye very sensitive for DC errors, thus fixed quantizer DC = QDC*8 AC coefficients Weighting W(u,v) according to perception H.6: flat AC quantization matrix MPEG intra block weighting 08 6 9 6 9 34 6 6 4 9 34 3 9 6 9 34 34 38 6 9 34 3 40 6 9 3 35 40 48 6 9 3 35 40 48 58 6 9 34 38 46 56 69 9 35 38 46 56 69 83 inter block weighting 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 9 MPEG Video / Quantization () AC coefficients (cont.) MPEG- encoder formula QF(u,v) = 6 F(u,v) / ( q_scale W(u,v) ) MPEG-: decoder formula F(u,v) = (QF(u,v) + k) q_scale W(u,v) / 6 k = 0 for intrablocks, and k = sign(qf(u,v)) for non-intra blocks mismatch control (value closest to zero): if F(u,v) even, then F(u,v) = F(u,v) - sign( F(u,v) ) uniform quantizer q q q = q_scale T = q/ = q_scale 0 5
MPEG Video / Quantization (3) MPEG- has more precise quantization DC coefficients up to bits precision AC coefficients MPEG-: decoder formula F(u,v) = (QF(u,v) + k) q_scale W(w,u,v) / 3 q_scale is mapped onto larger range than 0...3 w is defined by intra / non-intra and colour sampling k = 0 for intrablocks, and k = sign(qf(u,v)) for non-intra blocks special additional mismatch control: F(,) = F(,) if SUM ac(f(u,v)) is odd, and F(,) = F(,) +/- if F(,) is even/odd and SUM is even. PA case study / H.6 Video decoder Implementation issues specification of DCT computation accuracy error recovery: at least intra MB every 3 inter MBs Error protect. input step size buffer 0 MUX VLC decoder inv. Q IDCT + decoded video loop filter motion compensate frame memory only P pictures basically, different E-E delay than in MPEG matching of ME search range to temporal frame rate 6
H.6 Bit-stream syntax Only two image sizes: CIF (35x88), QCIF (6x44) quarter-cif Central coding unit: macroblock 6x6 pixels luminance, two times 8x8 pixels chrominance Image is divided into groups of macroblocks (GOB). Each GOB has x3 MBs. For CIF: x6 GOBs, QCIF: x3 GOBs Purpose: resynchronization after transmission error. 3 5 9 4 6 8 0 3 5 3 H.6 Picture Header PSC 0 bits Picture Start Code = 0000 0000 0000 000 0000 TR 5 bits Temporal Reference continuous frame counter (incremented for next frame), used to code temporal distance between pictures PTYPE 6 bits Type information Bit 4: 0 QCIF, : CIF Extra data while next bit == 8 user defined bits follow 4
H.6 GOB Bit-stream Syntax GBSC 6 bits Group of blocks start code = 0000 0000 0000 000 GN 4 bits Group number The number of this GOB. Defines spatial position. Note that GN==0 is used for picture header. GQUANT 5 bits Quantizer step-size Initial quantizer setting. Extra information same as in picture header. 5 H.6 Macroblock Layer MBA vlc- Increment to get to next MB position not every macroblock has to be coded, MBA> MTYPE vlc- Coding type of MB intra / inter mquant? motion-vector? coded-block-pattern? coefficients? loop-filter? MQUANT 5 bits New quantizer setting MVD vlc Motion vector CBP vlc Coded block pattern 6 8
H.6 Block-Layer Syntax If Intra-block, then DC-coefficient is coded as fixed-length, 8 bits. All other coefficients are coded as combined Run/Value pairs. RUN: number of zeros until next non-zero coefficient. LEVEL: value of next coefficient. Special value EOB: End of Block, no more coefficients follow. Run/Value pairs are coded with a combined Huffman code. Not all combinations are in table. For other combinations, escape-code is used and run/value is coded with fixed length codes. MPEG Video / Scanning () Scanning of transform coefficients preprocessing step for variable-length coding scanning functions reorders coefficients to cluster zeros for runlength coding start with low-frequency coefficients fundamental scanning pattern is diagonal 0 3 4 5 6 0 3 4 5 6 zigzag scanning 8 9
MPEG Video / Var.-Length Coding () Variable-length coding of AC coefficients: algorithm of (runlength, amplitude) coding STEP : (load coefficient), test of coefficient is zero STEP : (update runlength), if zero coefficient, increment zero counter, go to STEP 4 STEP 3: (jointly code), if non-zero coefficient, then 3a. jointly code [runlength, amplitude] in one codeword 3b. reset runlength counter STEP 4: (do next coefficient), go to STEP. If last coefficient, then go to STEP 5. STEP 5: (EOB) Terminate block with EOB-word, ignore runlength value. Codetable is modified Huffman code. 9 MPEG Video / Var.-Length Coding () -D VLC table of codewords unlikely symbols are coded by [escape code]+[fixed suffix] also VLC coding of macroblock address, motion vectors,... zero run 0 3 4 5 6 8 9 0 amplitude 0 3 4 5 6 8 9 0 3 4 5 6 8 9 0 3 4 4 5 5 5 6 6 6 8 8 8 8 9 9 4 4 6 8 9 9 0 0 0 5 9 0 0 3 6 8 0 3 6 9 0 3 0 3 8 3 8 EOB = 4 3 9 3 9 3 Example of wordlength table MPEG- has alternative encoding table for intrablocks. For non-intra blocks, always the same table is used. 0 0
MPEG Video / Var.-Length Coding (3) -Dim. VLC table of code-words code runlength amplitude 0 EOB s (note) 0 s (note3) 0 0s 000s 0 00s 000 s 0 3 00 s 3 00 0s 4 000 0s 000 s 5 000 0s 6 000 00s 0000 0s 0 4 0000 00s 0000 s 8 0000 0s 9 0000 0 escape - code runlength amplitude 000 00s 0 5 000 000s 0 6 000 00s 3 000 000s 3 000 0s 0 000 00s 000 000s 000 0000s 3 0000 0000s 0 0000 0000s 4 0000 000s 3 0000 00s 4 0000 0000s 5 0000 000s 4 0000 000s 5 0000 00000s 6 Note: s=sign bit, 0=pos/=neg. Note : code for dct_coeff_first Note 3: code for dct_coeff_next Example Akiyo 8k: hexadecimal: 00 0 00 9F 00.80 40 00 00.E4 4B A E8 C0 binary: 00000000 0000000 00000000 00 00000000 0000000 0000000 00000000 00000000 000000 0000 0000 00000 0000 000000 00000000 0000000 0000 PSC 0000 Temporal reference = 00 Type -> CIF 00000000 Extra Information 00000000 00000000 0 0000 0000 0000 000 000 GOB-start code (GOB ) 00000 0000 00000 0000 000000
Example 00000000 0000000 00000000 00 00000000 0000000 0000000 00000000 00000000 000000 0000 0000 00000 0000 000000 0000 000000 0000 00 000000 0000 0000 0000 000 000 GOB-start code (GOB ) 00 Gquant = 4 0 no extra information MB addess increment = (vlc) 000 MB-Type = Intra, no MQUANT 0000 DC = 46 -> 368 (dequant) 0 EOB 0000 DC = 46 -> 368 0 EOB 000000 DC = 48 -> 384 3 Example 00000000 0000000 00000000 00 00000000 0000000 0000000 00000000 00000000 000000 0000 0000 00000 0000 000000 0000 000000 0000 00 000000 000000 DC = 48 -> 384 0000 / - 0 EOB 000000 DC = 48 -> 384 0000 / - 0 EOB 00 DC = -> 008 0 EOB 00000... 4