Source Coding and Compression

Size: px

Start display at page:

Download "Source Coding and Compression"

Sharyl Perry
6 years ago
Views:

1 Surce Cding and Cmpressin Heik Schwarz Cntact: Dr.-Ing. Heik Schwarz Heik Schwarz Surce Cding and Cmpressin December 7, / 539

2 PartII: Applicatin in Image and Vide Cding Heik Schwarz Surce Cding and Cmpressin December 7, / 539

3 Still Image Cding Outline Part I: Surce Cding Fundamentals Prbability, Randm Variables and Randm Prcesses Lssless Surce Cding Rate-Distrtin Thery Quantizatin Predictive Cding Transfrm Cding Part II: Applicatin in Image and Vide Cding Still Image Cding / Intra-Picture Cding Representatin f Images and Vide JPEG Intra-Picture Cding in MPEG-2 Vide Intra-Picture Cding in H.263 and MPEG-2 Visual Intra-Picture Cding in H.264/AVC Intra-Picture Cding in H.265/HEVC Hybrid Vide Cding (Frm MPEG-2 Vide t H.265/HEVC) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

4 Still Image Cding Intrductin Still Image Cding / Intra-Picture Cding Overview Still Image Cding Exchange, transmissin and strage f images Used in virtually all digital cameras and picture editing applicatins JPEG: Mst widely used image cmpressin standard (based n DCT) JPEG-2000: Wavelet-based image cmpressin (nt discussed in lecture) JPEG-XR: Several imprvements ver JPEG (nt discussed in lecture) Intra-Picture Cding fr Vide Intra-picture cding: Sme pictures f a vide sequence need t be cded withut referring t ther picture inside the vide sequence First picture f a vide sequence has t be intra-picture cded Intra pictures in regular intervals (e.g., 1s) are required fr enabling randm access Typically, regularly inserted intra pictures cnsume large amunt f bit rate H.262 MPEG-2 Vide / H.263 / MPEG-4 Visual: Cnceptually similar t JPEG H.264 MPEG-4 AVC: Additinal cding tls yielding imprved cding efficiency H.265 MPEG-H HEVC: Increased flexibility and imprved cding efficiency Heik Schwarz Surce Cding and Cmpressin December 7, / 539

5 Still Image Cding Representatin f Images and Vide Representatin f Images and Vide Heik Schwarz Surce Cding and Cmpressin December 7, / 539

6 Still Image Cding Representatin f Images and Vide Digital Images and Vide Image 2-d functin s(x, y) relating light intensity s t spatial crdinates (x, y) Digital image Representatin f a cntinuus image at discrete crdinates [x, y] Amplitudes s[x, y] have finite alphabet, typically determined by the used bit depth Digital gray-level image f size M N can be represented using a matrix ntatin s[0, 0] s[1, 0] s[m 1, 0] s[0, 1] s[1, 1] s[m 1, 1] s[0, N 1] s[1, N 1] s[m 1, N 1] Typically characterized by image size M N and bit depth Clr images are typically cmpsed f 3 sample arrays (fr different clr cmpnents) Digital vide Sequence f digital images captured at successive time instances Typically characterized by frame rate (in additin t image size and bit depth) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

7 Still Image Cding Representatin f Images and Vide Spatial Reslutin Number f samples (M N) fr discrete matrix representatin samples samples samples samples Heik Schwarz Surce Cding and Cmpressin December 7, / 539

8 Still Image Cding Representatin f Images and Vide Gray-Level Reslutin / Bit Depth Number f gray levels fr image representatin (typically determined by bit depth) 256 gray levels (8 bit) 64 gray levels (6 bit) 16 gray levels (4 bit) 4 gray levels (2 bit) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Still Image Cding Representatin f Images and Vide Representatin f Clr Images Clr cmpnents Require at least 3 clr cmpnents (trichrmatic visin) RGB typically used as reference clr space Need t specify

9 Still Image Cding Representatin f Images and Vide Representatin f Clr Images Clr cmpnents Require at least 3 clr cmpnents (trichrmatic visin) RGB typically used as reference clr space Need t specify clr f RGB primaries in CIE XYZ reference space BT.709 RGB parameters primary x y red green blue white D Heik Schwarz Surce Cding and Cmpressin December 7, / 539

10 Still Image Cding Representatin f Images and Vide YCbCr Clr Space Definitin f YCbCr clr space Mre crrect name is Y CbCr, since Y is a gamma-adjusted luminance cmpnent Nt an abslute clr space, but a different representatin fr RGB data Transfrm f gamma-adjusted and nrmalized RGB cmpnents r, g and b with a range f 0..1 Typically used transfrm fr 8-bit cmpnents Y, Cb and Cr with Y = Rund(219 y + 16) Cb = Rund(224 pb + 128) Cr = Rund(224 pr + 128) y = K R r + (1 K R K B) g + K B b pb = 0.5 (b y )/(1 K B) pr = 0.5 (r y )/(1 K R) The cefficients K R and K B are specified by applicatin standards ITU-R Rec. BT.709 specifies K R = and K B = Y is called luma cmpnent, Cb and Cr are called chrma cmpnents Heik Schwarz Surce Cding and Cmpressin December 7, / 539

cmpnent represents a red-green difference signal Cding errrs are intrduced in perceptual meaningful way Effectiveness fr cding f images

11 Still Image Cding Representatin f Images and Vide Advantages f YCbCr Clr Representatin Prperties f the YCbCr clr representatin Similar clr decrrelatin as in human visual system: Y cmpnent is related t brightness Cb cmpnent represents a yellw-blue difference signal Cr cmpnent represents a red-green difference signal Cding errrs are intrduced in perceptual meaningful way Effectiveness fr cding f images and vide experimentally verified clr image Y cmpnent Cb cmpnent Cr cmpnent Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Still Image Cding Representatin f Images and Vide Subsampling f Chrma Cmpnents Y channel Cb channel Cr channel Visual imprtance f luma and chrma

(brightness) channel than t details in chrma channels Chrma channels can be dwnsampled fr saving bit rate Chrma dwnsampling by a factr f 2 in

12 Still Image Cding Representatin f Images and Vide Subsampling f Chrma Cmpnents Y channel Cb channel Cr channel Visual imprtance f luma and chrma cmpnents YCbCr clr space rughly apprximates the clr decrrelatin in the human visual system Human visual system is mre sensitive t details in luma (brightness) channel than t details in chrma channels Chrma channels can be dwnsampled fr saving bit rate Chrma dwnsampling by a factr f 2 in hrizntal r bth spatial directins Lcatin f chrma samples relative t luma samples has t be specified Heik Schwarz Surce Cding and Cmpressin December 7, / 539

13 Still Image Cding Representatin f Images and Vide Demnstratin f Luma/Chrma Perceptin Cmpare luma and chrma perceptin Selective lw-pass filtering fr luma r chrma cmpnents Use lw-pass filter (1,4,6,4,1)/16 Order f presentatin 1 Original luma and chrma cmpnents 2 Lw-pass filtered luma cmpnent but riginal chrma cmpnents 3 Original luma and chrma cmpnents (repeated) 4 Original luma cmpnent but lw-pass filtered chrma cmpnents Heik Schwarz Surce Cding and Cmpressin December 7, / 539

14 Still Image Cding Representatin f Images and Vide Demnstratin: Original Picture Heik Schwarz Surce Cding and Cmpressin December 7, / 539

15 Still Image Cding Representatin f Images and Vide Demnstratin: Filtered Luma Cmpnent Heik Schwarz Surce Cding and Cmpressin December 7, / 539

16 Still Image Cding Representatin f Images and Vide Demnstratin: Original Picture (Repeated) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

17 Still Image Cding Representatin f Images and Vide Demnstratin: Filtered Chrma Cmpnents Heik Schwarz Surce Cding and Cmpressin December 7, / 539

18 Still Image Cding Representatin f Images and Vide Chrma Sampling Frmats fr Image and Vide Cding Heik Schwarz Surce Cding and Cmpressin December 7, / 539

19 Still Image Cding JPEG JPEG Heik Schwarz Surce Cding and Cmpressin December 7, / 539

20 Still Image Cding JPEG JPEG Overview Jint Phtgraphic Experts Grup (JPEG) Standard is named after the grup which created it Jint cmmittee between ITU-T (frmerly CCITT) and ISO/IEC JTC 1 Standard Digital Cmpressin and Cding f Cntinuus-Tne Still Images Officially ITU-T Rec. T.81 and ISO/IEC Cmmnly referred t as JPEG Specifies cmpressin fr gray-level and clr images Wrk cmmenced in 1986, standard published in 1992 Applicatins f JPEG Strage frmat used in virtually all digital cameras (except fr raw sensr data) Mst pictures in the Internet are JPEG pictures Mtin-JPEG is de fact standard fr digital vide editing Heik Schwarz Surce Cding and Cmpressin December 7, / 539

21 Still Image Cding JPEG The Scpe f Image and Vide Cding Standardizatin What is standardized? Data frmat including cnstraints fr the data Decding result t be prduced by a cnfrming decder Prvides interperability between different devices Permits ptimizatin beynd the bvius Permits cmplexity reductin fr implementability Prvides n guarantee f quality Heik Schwarz Surce Cding and Cmpressin December 7, / 539

22 Still Image Cding JPEG JPEG: Basic Cdec Structure Encder and decder structure fr each clr cmpnent Heik Schwarz Surce Cding and Cmpressin December 7, / 539

23 Still Image Cding JPEG JPEG: Partitining f Clr Cmpnents int 8 8 Blcks Clr cmpnents are cded independently f each ther Clr cmpnents are partitined int 8 8 blcks (padding at brders) The 8 8 blcks are cded using transfrm cding Heik Schwarz Surce Cding and Cmpressin December 7, / 539

24 Still Image Cding JPEG Tw-dimensinal Transfrm fr Image Cmpressin Separable and symmetric 2-d rthgnal blck transfrm 2-d linear transfrm: Each input blck is represented as a linear cmbinatin f 2-d basis functins (r basis blcks) Separable and symmetric 2-d rthgnal blck transfrm: Transfrm f an N N blck s can be written as u = A s A T (588) where A is the N N transfrm matrix and u is the N N blck f transfrm cefficients Inverse f separable and symmetric 2-d rthgnal blck transfrm is given by s = A T u A (589) Great practical imprtance: Separable transfrm requires 2 matrix multiplicatins f size N N instead f ne multiplicatin f a vectr f size 1 N 2 with a matrix f size N 2 N 2 = Cmplexity reductin frm O(N 4 ) t O(N 3 ) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Still Image Cding JPEG 2-d Transfrm used in JPEG 2-d blck transfrm in JPEG Separable DCT f type II 8 8 transfrm matrix A cnsisting f elements a ik, with i, k = 0,, 7, given by a ik = α i cs π (2k +

25 Still Image Cding JPEG 2-d Transfrm used in JPEG 2-d blck transfrm in JPEG Separable DCT f type II 8 8 transfrm matrix A cnsisting f elements a ik, with i, k = 0,, 7, given by a ik = α i cs π (2k + 1) i 16 (590) with α i = 1 4 { 1 : i = 0 2 : i > 0 (591) Transfrm can be implemented using a fast butterfly algrithm Transfrm specificatin in JPEG Ideal frward and backward transfrm are given in infrmative clause Specificatin cntains nrmative accuracy requirements Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Still Image Cding JPEG 2-d DCT Example Step 1: Vertical Transfrm Example fr a 16 16 DCT Step 1: Clumn-wise DCT n image blck yielding intermediate blck

26 Still Image Cding JPEG 2-d DCT Example Step 1: Vertical Transfrm Example fr a DCT Step 1: Clumn-wise DCT n image blck yielding intermediate blck f transfrm cefficients Ntice the energy cncentratin in the first rw (DC cefficients) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Still Image Cding JPEG 2-d DCT Example Step 2: Hrizntal Transfrm Example fr a 16 16 DCT Step 2: Rw-wise DCT n intermediate blck f transfrm cefficients

27 Still Image Cding JPEG 2-d DCT Example Step 2: Hrizntal Transfrm Example fr a DCT Step 2: Rw-wise DCT n intermediate blck f transfrm cefficients yielding the final blck f DCT cefficients Ntice the energy cncentratin in the DC cefficient (tp-left) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

28 Still Image Cding JPEG Quantizatin in JPEG JPEG specifies unifrm recnstructin quantizers fr the transfrm cefficients Inverse quantizatin (scaling) in decder is specified by t ik = ik q ik (592) with q ik : Quantizatin index fr cefficient at lcatin (i, k) inside blck i,k : Quantizatin step size fr cefficient at lcatin (i, k) t i,k : Recnstructed transfrm cefficient at lcatin (i, k) N nrmative encding prcedure, but infrmative quantizatin rule ( ) tik q ik = rund (t i,k : riginal transfrm cefficient) (593) ik Standard specifies accuracy requirements fr cmbinatin f DCT and quantizatin = Leaves freedm fr encder designers Separate quantizatin step sizes can be selected fr each cefficient lcatin (i, k) and clr cmpnent = Quantizatin tables have t be transmitted as side infrmatin (n defaults) = Additinal freedm fr encder ptimizatin Heik Schwarz Surce Cding and Cmpressin December 7, / 539

29 Still Image Cding JPEG Quantizatin Tables in JPEG Quantizatin tables Determine rate and distrtin (amng ther parameters) Need t be transmitted (n default tables in JPEG) Example tables fr YCbCr frmat are specified in Annex K f standard (empirically derived based n psychvisual threshld experiments) luma blcks chrma blcks Heik Schwarz Surce Cding and Cmpressin December 7, / 539

30 Still Image Cding JPEG Entrpy Cding in JPEG Baseline Entrpy cding f transfrm cefficient levels (quantizatin indices) Different cncepts fr DC and AC levels DC levels: Differential cding using cdewrd tables AC levels: Run-level cding f scanned cefficients Cding f DC transfrm cefficient levels DC level is predicted by previus DC level as predictr Difference t predictr is cded using VLC and FLC Categry C is cded using VLC = Specifies range f values = Specifies number f fllwing bits (fr FLC) FLC specifies actual value f DIFF inside categry C = If DIFF > 0, lw-rder bits f DIFF = If DIFF < 0, lw-rder bits f DIFF 1 VLC table fr categry needs t be transmitted = Increases side infrmatin (n default table) = Allws adaptatin t actual statistics Heik Schwarz Surce Cding and Cmpressin December 7, / 539

31 Still Image Cding JPEG Example VLC Table fr Cding DC Difference Categry Example VLC table fr cding categry C Range f DIFF values is specified in standard Cdewrd assignment has t be transmitted Example shws recmmended table fr luma DC (Annex K f JPEG) Categry C Range f DIFF value Example cdewrd , , -2, 2, , , , , , , , , , Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Still Image Cding JPEG Entrpy Cding f AC Transfrm Cefficient Levels Representatin f AC levels Cnvert int sequence using a zig-zag scan AC cefficients are likely t be quantized t zer (in particular

32 Still Image Cding JPEG Entrpy Cding f AC Transfrm Cefficient Levels Representatin f AC levels Cnvert int sequence using a zig-zag scan AC cefficients are likely t be quantized t zer (in particular thse at high-frequency lcatins) Successive runs f zers are represented using a run (number f cnsecutive levels equal t zer) Nn-zer AC levels are represented by a categry and a value inside the categry (same as fr DC levels) Cding f AC levels AC levels are cded using a cmbinatin f VLC and FLC Variable-length cde table is used fr cding events {run,categry} VLC table includes a special symbl (EOB) fr signaling the end-f-blck (all remaining AC levels are equal t zer) Fixed-length cde is used fr cding the exact value inside a categry (number f bits is given by categry) same as fr DC difference levels VLC table has t be transmitted (n default table) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

33 Still Image Cding JPEG Example VLC Table fr Run-Categry Cding f AC Levels Example fr VLC table Standard defines ranges fr categries (same as fr DC, but n categries 0 and 11) Cdewrd assignment has t be transmitted Example shws first entries f recmmended table fr luma AC (Annex K f JPEG) run/categry cdewrd run/categry cdewrd EOB / /1 00 2/ /2 01 2/ / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Still Image Cding JPEG Example: JPEG Transfrm Cefficient Level Cding Example fr an 8 8 luma blck: transfrm cefficient levels Last DC level: DC(N 1) = 178 Use recmmended luma tables Cding f DC

34 Still Image Cding JPEG Example: JPEG Transfrm Cefficient Level Cding Example fr an 8 8 luma blck: transfrm cefficient levels Last DC level: DC(N 1) = 178 Use recmmended luma tables Cding f DC transfrm cefficient level Predictin difference: DIFF = = 7 Categry C = 3: Cdewrd 100 Fixed-length cde (lwest 3 bits f 7 ): 111 Final bit representatin (6 bit): Cding f AC transfrm cefficient levels Zig-zag scanning and cnversin int (run,level) pairs yields (0, 3) (0, 1) (2, 1) (1, 1) (6, 3) (0, 1) (0, 2) (0, 1) (EOB) Representatin as (run,categry) [FLC bits] sequence (0, 2)[11] (0, 1)[1] (2, 1)[1] (1, 1)[0] (6, 2)[00] (0, 1)[0] (0, 2)[01] (0, 1)[0] (EOB) Bit sequence: VLC bits [FLC bits] (in ttal: 46 bits fr 63 AC levels) 01 [11] 00 [1] [1] 1100 [0] [00] 00 [0] 01 [01] 00 [0] 1010 Heik Schwarz Surce Cding and Cmpressin December 7, / 539

35 Still Image Cding JPEG JPEG Cmpressin Example Original (YCbCr 4:2:0, 12 bpp) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

36 Still Image Cding JPEG JPEG Cmpressin Example 1:10 Cmpressin (1.2 bpp) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

37 Still Image Cding JPEG JPEG Cmpressin Example 1:25 Cmpressin (0.48 bpp) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

38 Still Image Cding JPEG JPEG Cmpressin Example 1:50 Cmpressin (0.24 bpp) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

39 Still Image Cding JPEG JPEG Cmpressin Example 1:100 Cmpressin (0.12 bpp) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

40 Still Image Cding JPEG JPEG Cmpressin Example 1:200 Cmpressin (0.06 bpp) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

41 Still Image Cding JPEG Summary f JPEG JPEG Baseline Minimum f capabilities (required fr all DCT-based JPEG cdecs) Surce image: 1-4 clr cmpnents with 8-bit per sample Sequential prcessing f 8 8 blcks Transfrm: Separable 8 8 discrete csine transfrm (DCT) f type II Quantizer: Scalar unifrm recnstructin quantizer (using quantizatin table) DC cding: Predictin and cmbinatin f VLC and FLC AC cding: Zig-zag scan and run-level cding (cmbinatin f VLC & FLC) VLC cding: 2 DC tables (categry) & 2 AC tables (run/categry) Extended JPEG features Extended bit depth Adaptive binary arithmetic cding Prgressive and hierarchical cding Lssless cding mde Extended file frmats (e.g., EXIF) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

42 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Intra-Picture Cding in H.262 MPEG-2 Vide Heik Schwarz Surce Cding and Cmpressin December 7, / 539

43 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Intra-Picture Cding in H.262 MPEG-2 Vide H.262 MPEG-2 Vide Vide cding standard jintly develped by ITU-T and ISO/IEC JTC 1 Official name: ITU-T Rec. H.262 and ISO/IEC Standard was finalized in 1994 Still widely used in digital televisin and the DVD-Vide ptical disc frmat Three pictures types: I (intra), P (predictive) and B (bi-directinal) Includes tls fr interlaced vide Mst imprtant cnfrmance pint: Main Prfile Clr frmat: YCbCr 4:2:0 Bit depth: 8 bit per sample Intra-picture cding in H.262 MPEG-2 Vide Cnceptually very similar t JPEG Baseline Details and actual syntax are different Fixed variable-length entrpy cding tables Heik Schwarz Surce Cding and Cmpressin December 7, / 539

the chrma cmpnents In 4:2:0 chrma sampling frmat, a macrblck crrespnds t ne 16 16 luma blck tw 8 8 chrma blcks Cding f macrblcks Different cding mdes, als

44 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Macrblcks and Blcks in H.262 MPEG-2 Vide Picture partitining int macrblcks Picture is partitined int fixed-size macrblcks, which cnsist f luma samples and the crrespnding areas in the chrma cmpnents In 4:2:0 chrma sampling frmat, a macrblck crrespnds t ne luma blck tw 8 8 chrma blcks Cding f macrblcks Different cding mdes, als referred t as macrblck mdes Intra picture: 2 cding mdes Intra Intra+Q (quantizer change) Intra mde: Transfrm cding fr all six 8 8 blcks f a macrblck (4 luma and 2 chrma blcks) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

45 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Cding f 8 8 Intra Blcks in H.262 MPEG-2 Vide Transfrm cding f 8 8 blcks Orthgnal blck transfrm + scalar quantizatin + entrpy cding Very similar t JPEG (but sme differences in details) Orthgnal blck transfrm 2-d discrete cding transfrm (DCT) same as in JPEG Scalar quantizatin Quantizatin step size is specified by a quantizatin matrix and a quantizatin parameter (scaling fr all cefficients, can be mdified n macrblck basis) Entrpy cding f transfrm cefficient levels DC cefficient: Differential cding similar t JPEG AC cefficients: Zig-zag scan and run-level cding with EOB symbl Heik Schwarz Surce Cding and Cmpressin December 7, / 539

46 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Quantizatin in H.262 MPEG-2 Vide Standard specifies nly cnstructin f transfrm cefficients frm levels Inverse quantizatin f intra DC cefficients (intra dc precisin in range 8..11) t 00 = q intra dc precisin (594) Inverse quantizatin f AC cefficients in intra blcks t qik w ik QP ik = sgn(q ik ) 16 (595) with q ik : Quantizatin index fr cefficient at lcatin (i, k) inside blck w i,k : Entry f quantizatin matrix fr cefficient at lcatin (i, k) QP : Quantizatin parameter (als called quantizer scale ) t i,k : Recnstructed transfrm cefficient at lcatin (i, k) Inverse quantizatin is fllwed by clipping t range [ 2048, 2047] and, thereafter, the s-called mismatch cntrl peratin t t 77 : s is dd = t 77 1 : s is even and t 77 is dd with s = t t : s is even and t ik (596) 77 is even i=0 k=0 Heik Schwarz Surce Cding and Cmpressin December 7, / 539

47 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Quantizatin Matrices in H.262 MPEG-2 Vide Quantizatin matrices Quantizatin matrices define variatin f quantizer step sizes amng frequencies = Can be used fr psychvisual ptimizatin (t sme extend) Quantizatin parameter QP is used fr scaling the quantizatin matrices = Operatin pint can be mdified n macrblck basis with a few bits Fr 4:2:0 data, tw matrices are used: ne fr intra and ne fr nn-intra Default quantizatin matrices can be replaced by user-defined matrices default matrix fr intra blcks default matrix fr nn-intra blcks Heik Schwarz Surce Cding and Cmpressin December 7, / 539

48 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Cding f Transfrm Cefficients Levels fr Intra 8 8 Blcks DC transfrm cefficient level in intra blcks Very similar t JPEG Predictin using last cded DC level (reset at start f slice r nn-intra MB) Difference is cded by a categry (called dct dc size) a fixed-length cde Entrpy cding table is fixed in standard (cannt be mdified) AC transfrm cefficient levels Levels f a blck are cnverted int vectr using a zig-zag scan (same as in JPEG) Vectr f levels is cded using run-level cde Entrpy cding table fr mst frequent cmbinatins f run and level (actual levels including sign, nt categries as in JPEG) Includes end-f-blck (EOB) symbl Includes escape symbl fr less likely cmbinatins, fr which the actual run and level are transmitted with 6 and 12 bit, respectively Entrpy cding tables are fixed in standard (cannt be mdified) Fr intra, ne f tw defined tables can be selected n a picture basis Heik Schwarz Surce Cding and Cmpressin December 7, / 539

49 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Encder Cntrl fr Intra Pictures in H.262 MPEG-2 Vide What parameters can be chsen in encder? On sequence/picture level: Quantizatin matrix, intra vlc table, intra DC precisin On macrblck level: Quantizatin parameter QP On blck level: Transfrm cefficient levels = Transfrm cefficient levels have largest impact n cding efficiency D the parameters f different blcks influence each ther? Only last DC cefficient is used fr predictin Typically very small impact n cding efficiency = Interdependencies between blcks can be neglected Hw can the selectin f transfrm cefficient levels be ptimized? Selectin determines distrtin and rate! Runding t next level minimizes distrtin, but typically prduces a large rate Fr an peratin pint given by a Lagrange parameter λ, the cmbined cst measure D + λ R shuld be minimized Nt straightfrward due t run-level cding = Need t cnsider dependencies in cding f successive levels = Fixed decisin levels cannt be ptimal Heik Schwarz Surce Cding and Cmpressin December 7, / 539

50 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Example: Impact f Cnsidering Rate in Quantizatin Quantizatin example with = 10 and λ = 10 Cnsider quantizatin f the fllwing vectr f transfrm cefficients Runding t nearest quantizatin level accrding t q = rund(c/ ) yields = Sequence f (run,level) values: (0,4)(0,2)(0,2)(1,1)(3,-1)(EOB) = Bit sequence: ( )(01000)(01000)(0110)(001111)(10) = Distrtin D = 87, rate R = 30 = J = D + λ R = 387 Alternative quantizatin (cnsidering rate by minimizing J = D + λ R) = Sequence f (run,level) values: (0,4)(0,2)(0,2)(1,1)(EOB) = Bit sequence: ( )(01000)(01000)(0110)(10) = Distrtin D = 107, rate R = 24 = J = D + λ R = 347 (< 387) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

51 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Rate-Distrtin Optimized Quantizatin (RDOQ) General idea f rate-distrtin ptimized quantizatin fr run-level cding Evaluate all pssible vectrs f transfrm cefficient levels Chse vectr that minimizes J = D + λ R, with D: Distrtin fr blck (can be measured in transfrm dmain) R: Rate fr transmitting levels using given entrpy cding tables λ: Lagrange parameter, e.g., given as functin f quantizatin parameter QP = Very cmplex: Require restrictin fr levels and suitable algrithm Select reasnable set f ptential levels fr each cefficient Fllwing bservatins can be made (cnsidering abslute levels) Levels greater than the level btained by mathematically crrect runding dn t need t be cnsidered (larger distrtin and larger rate) Levels that are significantly smaller than the level btained by mathematically crrect runding dn t need t be cnsidered (very large distrtin) Gd cmprmise is btained by fllwing set f 1-3 levels (abslute values) Level q 0 btained by mathematically crrect runding, q 0 = rund(c/ ) Level q 1 btained by runding twards zer, q 1 = c/ If q 1 > 1, level q 2 = q 1 1 Heik Schwarz Surce Cding and Cmpressin December 7, / 539

52 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide RDOQ Algrithm fr Run-Level Cding Start with first psitin k = 0 in scanning rder Fr all ptential levels {q 0} determine distrtin D 0 = (c 0 q 0 ) 2 fr nn-zer levels q 0, rate R 0 using run-level cding tables Amng all nn-zer levels q 0, keep nly the ne that minimizes J 0 = D 0 + λ R 0 At mst tw candidate vectrs q 0 = [q 0] are cnsidered fr the further steps, ne with q 0 = 0 and ne with q 0 0 Fr each f the remaining psitins k in scanning rder Cmbine all ptential levels {q k } fr current psitin k with the selected candidate vectrs q k 1 t new candidate vectrs q k and determine distrtin D k = D k 1 + (c k q k ) 2 fr nn-zer levels q k, rate R k using run-level cding tables Amng all vectrs q k with the last level q k unequal t 0, discard all vectrs except the ne that minimizes D k + λ R k After last scanning psitin k Determined the final cst J = D + λ R fr all remaining candidate vectrs q (including EOB symbl) and chse the ne that minimizes J Heik Schwarz Surce Cding and Cmpressin December 7, / 539

53 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide RDOQ Example Simple RDOQ example using quantizatin step size = 10 and λ = 10 Cnsider transfrm cefficient vectr f 9 cefficients Cefficient 36 at psitin k = 0 with ptential levels {4, 3, 2}: [4]: D = 4 2 = 16, R = 8 J = 96 [3]: D = 6 2 = 36, R = 6 J = 96 [discard] [2]: D = 16 2 = 256, R = 5 J = 306 [discard] Cefficient 18 at psitin k = 1 with ptential levels {2, 1, 0}: [4, 2]: D = = 20, R = = 13 J = 150 [4, 1]: D = = 80, R = = 11 J = 190 [discard] [4, 0]: D = = 340, R = 8 (des nt include last zer) Cefficient 23 at psitin k = 2 with ptential levels {2, 1}: [4, 2, 2]: D = = 29, R = = 18 J = 209 [4, 2, 1]: D = = 189, R = = 16 J = 349 [discard] [4, 0, 2]: D = = 349, R = = 15 J = 499 [discard] [4, 0, 1]: D = = 509, R = = 12 J = 629 [discard] Cefficient 3 at psitin k = 3 with ptential levels {0}: [4, 2, 2, 0]: D = = 38, R = 18 (withut trailing zers) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

54 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide RDOQ Example (cntinued) Cefficient 12 at psitin k = 4 with ptential levels {1, 0}: [4, 2, 2, 0, 1]: D = = 42, R = = 22 J = 262 [4, 2, 2, 0, 0]: D = = 182, R = 18 (withut trailing zers) Cefficient -4 at psitin k = 5 with ptential levels {0}: [4, 2, 2, 0, 1, 0]: D = = 58, R = 22 (withut trailing zers) [4, 2, 2, 0, 0, 0]: D = = 198, R = 18 (withut trailing zers) Cefficient -3 at psitin k = 6 with ptential levels {0}: [4, 2, 2, 0, 1, 0, 0]: D = = 67, R = 22 (withut trailing zers) [4, 2, 2, 0, 0, 0, 0]: D = = 207, R = 18 (withut trailing zers) Cefficient 2 at psitin k = 7 with ptential levels {0}: [4, 2, 2, 0, 1, 0, 0, 0]: D = = 71, R = 22 (withut trailing zers) [4, 2, 2, 0, 0, 0, 0, 0]: D = = 211, R = 18 (withut trailing zers) Last cefficients -6 with ptential levels { 1, 0} (including 2 bits fr EOB): [4, 2, 2, 0, 1, 0, 0, 0, 1]: D = = 87, R = = 30 J = 387 [4, 2, 2, 0, 1, 0, 0, 0, 0]: D = = 107, R = = 24 J = 347 [4, 2, 2, 0, 0, 0, 0, 0, 1]: D = = 227, R = = 27 J = 497 [4, 2, 2, 0, 0, 0, 0, 0, 0]: D = = 247, R = = 20 J = 447 = Selected transfrm cefficient levels: [4, 2, 2, 0, 1, 0, 0, 0, 0] with J = 347 = Fr cmparisn, runding wuld yield: [4, 2, 2, 0, 1, 0, 0, 0, 1] with J = 387 Heik Schwarz Surce Cding and Cmpressin December 7, / 539

55 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Experimental Analysis f RDOQ fr Intra-Picture Cding Cding experiment with H.262 MPEG-2 Vide cmparing Encder with simple quantizatin using mathematically crrect runding Encder using rate-distrtin ptimized quantizatin Cding cnditins 6 vide cnferencing sequences with a reslutin f mre cmplex vide sequences with a reslutin f pictures f each sequence have been cded (intra nly) Flat quantizatin matrices (since quality is measured using PSNR) Same quantizatin parameter fr all macrblcks Bitstreams with different quantizatin parameters PSNR and rate have been measured Lagrange parameter λ has been cupled t quantizatin step size using the experimentally determined relatinship λ = cnst 2 Heik Schwarz Surce Cding and Cmpressin December 7, / 539

56 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Quantizatin Cmparisn Sequence Jhnny Jhnny, 1280x720, 60Hz Y-PSNR [db] Simple Quantizatin 35 RD-pt. Quantizatin bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

57 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Quantizatin Cmparisn Sequence Cactus Cactus, 1920x1080, 50Hz Y-PSNR [db] Simple Quantizatin 31 RD-pt. Quantizatin bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

58 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Quantizatin Cmparisn Summary Bit-rate savings f RDOQ versus simple quantizatin Bit-rate saving at a PSNR value is btained by interplating the r-d curves Average bit-rate savings are btained by averaging the savings fr 100 PSNR values Average bit-rate saving fr all sequences: 9% Jhnny, 1280x720, 60Hz Cactus, 1920x1080, 50Hz 100 % 100 % RD-pt. Quantizatin RD-pt. Quantizatin 90 % 90 % rate savings vs. Simple Quantizatin 80 % 70 % 60 % 50 % 40 % 30 % 20 % rate savings vs. Simple Quantizatin 80 % 70 % 60 % 50 % 40 % 30 % 20 % 10 % 10 % 0 % 0 % Y-PSNR [db] Y-PSNR [db] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

59 Still Image Cding Intra-Picture Cding in H.262 MPEG-2 Vide Summary f Intra-Picture Cding in H.262 MPEG-2 Vide Intra-picture cding in H.262 MPEG-2 Vide Vide cding standard f ITU-T and ISO/IEC JTC 1 Still widely used in digital televisin and DVD-Vide Intra-picture cding is very similar t JPEG Baseline Picture partitining in macrblcks and blcks Scalar quantizatin f transfrm cefficients Quantizatin matrices are cmbined with quantizatin parameter Run-level cding fr transfrm cefficients Rate-distrtin ptimized quantizatin Quantizatin with fixed decisin levels cannt be ptimal due t dependencies in run-level entrpy cding Actual rate fr cding transfrm cefficient levels need t be cnsidered Optimal quantizatin can be achieved by a trellis-like prcedure Rate-distrtin ptimized quantizatin yields average bit-rate savings f abut 10% relative t simple runding (fr intra-nly cding) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

60 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Intra-Picture Cding in H.263 and MPEG-4 Visual Heik Schwarz Surce Cding and Cmpressin December 7, / 539

61 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual ITU-T Recmmendatin H.263 Overview ITU-T Recmmendatin H.263 Vide cding standard f ITU-T Develped by Visual Cding Experts Grup (VCEG ITU-T SG16/WP3/Q6) Primarily designed fr lw bit-rate vide cnferencing Example applicatins: Vide cnferencing Was used fr Flash Vide cntent RealVide cdec (befre RealVide 8) was based n H.263 First versin (1995) Very similar structure as H.262 MPEG-2 Vide Several imprvements relative t H.262 MPEG-2 Vide Fr intra: Run-level-last cding and ptimized cding tables Secnd versin H.263+ (1998) Imprvements and new features Fr intra: AC level predictin, adaptive scans, specialized quantizatin and entrpy cding tables fr intra blcks Third versin H (2000) Multiple reference pictures fr mtin-cmpensated cding Heik Schwarz Surce Cding and Cmpressin December 7, / 539

62 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Intra-Picture Cding in H.263 Baseline Basic design fr intra-picture cding similar t H.262 MPEG-2 Vide Partitining int macrblcks as in H.262 MPEG-2 Vide Transfrm cding f 8 8 blcks (DCT + scalar quantizatin + entrpy cding) Orthgnal blck transfrm Same separable 8 8 DCT as in H.262 MPEG-2 Vide and JPEG Scalar quantizatin f transfrm cefficients (nly recnstructin is specified) Step size determined by quantizatin parameter QP (can be mdified n MB basis) DC cefficient (unifrm recnstructin quantizer) t 00 = 8 q 00 AC cefficients (unifrm recnstructin quantizer with extra-wide deadzne) 0 : q ik = 0 t ik = sgn(q ik ) QP (2 q ik + 1) : q ik 0 and QP is dd sgn(q ik ) QP (2 q ik + 1) 1 : q ik 0 and QP is even (597) (598) = Recnstructed levels are always dd-valued numbers (except zer) = Has been fund t prevent accumulatin f IDCT mismatches (fr inter) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

63 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Cding f Transfrm Cefficient Levels in H.263 Baseline Cded blck pattern (CBP) Signal which 8 8 blcks f a macrblck cntain nn-zer levels (AC levels) Cncept was als used in H.262 MPEG-2 Vide, but nly fr inter macrblcks Required fr run-level-last cding f AC levels In H.263, split int tw cmpnents: CBPC (tw bits fr the chrma blcks): Cded tgether with MB type CBPY (fur bits fr the luma blcks): Cded as separate cdewrd MB type CBPC cdewrd Intra 00 1 Intra Intra Intra Intra+Q Intra+Q Intra+Q Intra+Q stuffing CBPY cdewrd CBPY cdewrd Cding f DC transfrm cefficient level N predictin f DC cefficient (in cntrast t JPEG and H.262 MPEG-2 Vide) Fixed-length cde: 8 bit per DC level Heik Schwarz Surce Cding and Cmpressin December 7, / 539

64 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Cding f AC Transfrm Cefficient Levels in H.263 Baseline Cding f AC transfrm cefficient levels Cnvert matrix f AC levels t vectr using the zig-zag scan Vectr f AC levels is cded using run-level-last cde Entrpy cding table fr mst cmmn cmbinatins f Run: Number f preceding levels equal t zer Level: Value f the next nn-zer level Last: Flag indicating if the nn-zer level is the last nn-zer level in blck Entrpy cding table includes an escape symbl fr less likely cmbinatins, fr which the run, level and last are cded using fixed-length cdes (6+8+1 bit) Entrpy cding tables have been ptimized fr lw rates last run level cdewrd (s = sign) last run level cdewrd (s = sign) 0 0 ±1 10s 0 0 ± s 1 31 ± s 0 0 ± s 1 32 ± s 0 0 ± s 1 33 ± s 0 0 ± s 1 34 ± s 0 0 ± s 1 35 ± s 0 0 ± s 1 36 ± s 0 0 ± s 1 37 ± s 0 0 ± s 1 38 ± s 0 0 ± s 1 39 ± s 0 0 ± s 1 40 ± s 0 0 ± s escape Heik Schwarz Surce Cding and Cmpressin December 7, / 539

65 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Advanced Intra Cding Mde in Annex I f H.263+ Advanced intra-picture cding mde (in ptinal Annex I) specifies Adaptive predictin f transfrm cefficients in intra blcks Adaptive scanning f transfrm cefficient levels (depending n predictin) Mdified inverse quantizatin fr intra Separate entrpy cding table fr intra blcks (ptimized fr intra) Adaptive predictin and scanning Predict part f the transfrm cefficients using recnstructed transfrm cefficients f neighbring blcks f same clr cmpnent Quantizatin and entrpy cding f predictin residuals t ik ˆt ik t ik = ˆt ik + Q 1 (q ik ) (Q 1 : inverse quantizatin) (599) 3 predictin mdes (signaled at macrblck level) DC predictin: Predict DC using left and abve neighbring blck Vertical predictin: Predict first rw f cefficients using abve blck = Particularly suitable fr vertical structures Hrizntal predictin: Predict first clumn f cefficients using left blck = Particularly suitable fr hrizntal structures Heik Schwarz Surce Cding and Cmpressin December 7, / 539

at macrblck level Mdified quantizatin fr intra macrblcks Use same quantizatin fr all cefficients (including DC cefficient)

66 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Predictin Mdes fr Intra Blcks in Advanced Intra Cding Predictin f transfrm cefficients: 3 mdes signaled at macrblck level Mdified quantizatin fr intra macrblcks Use same quantizatin fr all cefficients (including DC cefficient) Unifrm recnstructin quantizer withut extra-wide deadzne and withut mismatch cntrl (nly imprtant fr inter macrblcks) t ik = 2 QP q ik, i, k = 0..7 (600) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Gal: Cncentrate zer levels at end f scanning pattern Nn-zer cefficient distributin is depending n predictin mde

Hrizntal predictin: Vertical scan (suitable fr hrizntal structures) Cding f vectr f transfrm cefficient levels N

67 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Entrpy Cding in Advanced Intra Cding Mde Scanning pattern is chsen based n predictin mde Gal: Cncentrate zer levels at end f scanning pattern Nn-zer cefficient distributin is depending n predictin mde DC predictin: Cnventinal zig-zag scan Vertical predictin: Hrizntal scan (suitable fr vertical structures) Hrizntal predictin: Vertical scan (suitable fr hrizntal structures) Cding f vectr f transfrm cefficient levels N separate cding f DC cefficient = included in run-level-last cde Entrpy cding table fr run-level-last cde ptimized fr intra statistics Heik Schwarz Surce Cding and Cmpressin December 7, / 539

68 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Further Extensins in H.263 fr Imprving Intra Cding Annex E: Syntax-based arithmetic cding Specifies nn-adaptive arithmetic cding fr syntax elements Rarely used in practice Annex J: Deblcking filter mde Specifies deblcking filter fr reducing blck-edge artifacts Strength f smthing filter is cntrlled by quantizatin parameter Mst useful fr cding f fllwing inter pictures, but als imprves quality f intra pictures at lw rates Annex T: Mdified quantizatin mde Imprves ability t cntrl bit rate (finer steps fr mdifying QP) Extends range f representable transfrm cefficient values Imprves chrma fidelity by chsing smaller quantizatin step size fr chrma than fr luma (particularly fr lw rates) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

69 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Intra-Picture Cding in MPEG-4 Visual Internatinal standard ISO/IEC (MPEG-4 Visual) Vide cding standard f Mving Pictures Experts Grup (MPEG) Includes H.263 Baseline decder, cntains several extensins Was used in digital cameras, DivX,... Cding f intra macrblcks Predictin f transfrm cefficients (similar t H.263 Annex I) DC level is always predicted frm left r abve blck (directin is determined by differences f neighbring DC levels) First rw/clumn f ACs can be ptinally predicted using same predictin directin as fr DC (usage is signaled at macrblck level) Tw methds fr quantizatin MPEG-style: Quantizatin as in MPEG-2 Vide (including weighting matrix) H263-style: Quantizatin as in H.263 Baseline Scanning f transfrm cefficients 3 scans: Zig-zag, hrizntal, vertical (chsen based n predictin mde) Cding f vectr f transfrm cefficient levels Cded blck pattern and run-level-last cding Heik Schwarz Surce Cding and Cmpressin December 7, / 539

70 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Encder Cntrl fr Intra Cding in H.263 and MPEG-4 Visual Increased degree f freedm relative t H.262 MPEG-2 Vide In additin t transfrm cefficient levels, the methd fr transfrm cefficient selectin can be selected n macrblck level H.263 (Annex I): DC, vertical r hrizntal predictin mde MPEG-4 Visual: DC r DC and AC predictin Rate-distrtin ptimized encder cntrl Determine bitstream b s that the distrtin D(s, s ) between riginal picture s and recnstructed picture s is minimized given a particular target rate R R target With B c being the set f cnfrming bitstreams with R R target, we can write b = arg min b B c D ( s, s (b) ) (601) = Nt feasible due t huge parameter space = Split int smaller ptimizatin prblems by partially ignring dependencies Cnsider blck f samples s k (e.g., picture r macrblck) and ptimize with respect t cding parameters p k (e.g., mdes and transfrm cefficient levels) min p k D ( s k, s k(p k ) ) subject t R(p k ) R c (602) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

71 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Rate-distrtin ptimized encder cntrl Lagrangian encder cntrl Cnstrained ptimizatin prblem fr a blck f samples s k min p k D ( s k, s k(p k ) ) subject t R(p k ) R c (603) can be refrmulated as uncnstrained ptimizatin prblem min p k D ( s k, s k(p k ) ) + λ R(p k ) (604) Cnsider partitin f s k int a number f subsets s k,i (e.g. macrblcks) If cding parameters p k,i are independent f each ther and an additive distrtin measure is used, we can write the ptimizatin prblem as min D ( s k,i, s k,i(p p k,i ) ) + λ R(p k,i ) (605) k,i i = Independent selectin f cding parameters p k,i Fr cding decisins in image and vide cding Cding decisins are typically nt independent (e.g., due t predictin) Fr practical applicability: Cnsider past decisins, but ignre impact n future Heik Schwarz Surce Cding and Cmpressin December 7, / 539

72 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Lagrangian Encder Cntrl in Image and Vide Cding Applicatin f Lagrangian encder cntrl Can be applied t basically all decisins in an encder Quantizatin: Select vectr q f transfrm cefficient levels accrding t q = arg min q D(q) + λ R(q) (606) with D(q): SSD distrtin fr chsing transfrm cefficient level vectr q R(q): Number f bits required fr representing q = Rate-distrtin ptimized quantizatin (as cnsidered fr run-level cding) Mde decisin: Select cding mde c fr a macrblck r blck c = arg min c D(c) + λ R(c) (607) with D(c): SSD distrtin fr chsing cding mde c fr the blck R(c): Number f bits fr blck when cded with mde c = Can be applied fr selecting intra predictin mde Mtin search: Will be cnsidered later in lecture Heik Schwarz Surce Cding and Cmpressin December 7, / 539

73 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Cmparisn f H.263 and MPEG-4 Visual with MPEG-2 Vide Cmparisn f cding efficiency fr intra-picture cding Selectin f all features that cntribute t cding efficiency H.262 MPEG-2 Vide cnfrming t Main prfile H.263+ with advanced intra cding, deblcking filter, mdified quantizatin MPEG-4 Visual with MPEG-style quantizatin Apply same level f encder ptimizatin fr fair cmparisn Best pssible cding efficiency fr given syntax Ignre cnstraints such as real-time peratin = Use rate-distrtin ptimized quantizatin fr all standards = Apply rate-distrtin ptimized mde decisin where applicable General cding cnditins Encde 10 pictures f 12 vide sequences (6 in 720p, 6 in 1080p) Flat quantizatin matrices (quality is measured using PSNR) Same quantizatin parameter fr all macrblcks Select Lagrangian parameter accrding t λ = cnst 2 (with experimentally determined factr) (608) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

74 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Intra Cding Cmparisn Sequence Jhnny Jhnny, 1280x720, 60Hz Y-PSNR [db] H.262 MPEG-2 Vide H MPEG-4 Visual bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

75 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Intra Cding Cmparisn Sequence Cactus Cactus, 1920x1080, 50Hz Y-PSNR [db] H.262 MPEG-2 Vide 32 H MPEG-4 Visual bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

76 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Intra Cding Cmparisn Summary Bit-rate savings f H.263 and MPEG-4 Visual versus H.262 MPEG-2 Vide Bit-rate saving at a PSNR value is btained by interplating the r-d curves Average bit-rate savings are btained by averaging the savings fr 100 PSNR values Average bit-rate saving fr all sequences H.263+ versus H.262 MPEG-2 Vide: 29% MPEG-4 Visual versus H.262 MPEG-2 Vide: 25% Highest savings are btained fr lw bit rates 100 % Jhnny, 1280x720, 60Hz 100 % Cactus, 1920x1080, 50Hz H.263+ MPEG-4 Visual H.263+ MPEG-4 Visual 90 % 90 % rate savings vs. H.262 MPEG-2 Vide 80 % 70 % 60 % 50 % 40 % 30 % 20 % rate savings vs. H.262 MPEG-2 Vide 80 % 70 % 60 % 50 % 40 % 30 % 20 % 10 % 10 % 0 % 0 % Y-PSNR [db] Y-PSNR [db] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

77 Still Image Cding Intra-Picture Cding in H.263 and MPEG-4 Visual Summary f Intra-Picture Cding in H.263 and MPEG-2 Visual Intra-picture cding in H.263+ Transfrm cding using 8 8 DCT Predictin f DC and, partly, AC cefficients Cded blck pattern Adaptive scanning f transfrm cefficient levels Scalar quantizatin Run-level-last cding f transfrm cefficient levels Optinal deblcking filter Intra-picture cding in MPEG-4 Visual Similar tls as in advanced intra cding mde f H.263+ (Annex I) Additinally includes quantizatin weighting matrices Rate-distrtin ptimized encder cntrl Split verall ptimizatin prblem int smaller prblems Encder decisin by minimizing D + λ R Rate-distrtin ptimized quantizatin (RDOQ) Rate-distrtin ptimized mde decisin Heik Schwarz Surce Cding and Cmpressin December 7, / 539

78 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Intra-Picture Cding in H.264 MPEG-4 AVC Heik Schwarz Surce Cding and Cmpressin December 7, / 539

79 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC H.264 MPEG-4 AVC Overview ITU-T Rec. H.264 ISO/IEC (MPEG-4 Advanced Vide Cding) Vide cding standard jintly develped by ITU-T VCEG and ISO/IEC MPEG Widely used tday in many applicatin spaces Digital televisin, blu-ray ptical disc, digital cameras, mbile phnes, vide streaming, vide cnferencing Supprted in mre than 1 billin devices Every secnd bit in the Internet is part f a H.264 MPEG-4 AVC bitstream Versin 1 (2003): Three prfiles: Baseline, Main, Extended (4:2:0, 8 bit) Fidelity range extensins (2005) Imprvements fr large picture sizes, ther chrma frmats, higher bit depth High, High 10, High 4:2:2, High 4:4:4 prfiles (remved later) Later: Additin f High 4:4:4 Predictive and intra-nly prfiles Scalable vide cding extensin (2007) Extensin fr scalable vide cding (SVC) Multiview vide cding extensin (2009) Extensin fr multiview vide (MVC) Used fr 3d-blu-ray discs Heik Schwarz Surce Cding and Cmpressin December 7, / 539

80 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Intra-Picture Cding in H.264 MPEG-4 AVC Main features f intra-picture cding Spatial intra predictin with multiple predictin mdes Transfrm cding with 4 4 integer transfrm Optinal 8 8 integer transfrm (High prfile) Scalar quantizatin Tw entrpy cding methds: Cntext-adaptive variable length cding (CAVLC) Cntext-adaptive binary arithmetic cding (CABAC) Main/High prfile Deblcking filter (cnceptually similar t Annex J f H.263) Picture partitining and intra cding mdes Pictures are partitined int macrblcks 4 intra cding mdes are supprted (selectin n MB level) Intra-4 4 Intra-8 8 (High prfile) Intra Intra-PCM (direct cding f samples) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

81 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Cding f Intra-4 4 macrblcks Spatial predictin f blcks DC, hrizntal, vertical predictin f transfrm cefficients (as in H.263, MPEG-4 Visual) can similarly als be realized in spatial dmain Predictin in spatial dmain ffers mre pssibilities Cding f luma cmpnent luma blck is partitined int blcks 4 4 blcks are spatially predicted 9 intra predictin mdes are supprted Predictin errr f 4 4 blcks is transfrm-cded Cding f chrma cmpnents Bth 8 8 chrma blcks are spatially predicted 4 intra predictin mdes are supprted Predictin errr is cded using transfrm cding 4 4 transfrm and secnd level transfrm f DCs Heik Schwarz Surce Cding and Cmpressin December 7, / 539

82 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Spatial Intra Predictin fr 4 4 Blcks Spatial intra predictin Predict blck using already cded neighbring samples DC predictin (mean value) and 8 directinal predictin mdes Cding rder and bundary cnditins Blcks are cded in z-scan rder Samples used fr predictin have t be available (already recnstructed) Nt all mdes are available fr all blcks Samples E, F, G and H can be replaced with sample D Heik Schwarz Surce Cding and Cmpressin December 7, / 539

83 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Illustratin f Intra Predictin Mdes Heik Schwarz Surce Cding and Cmpressin December 7, / 539

84 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Efficiency f Intra Predictin Sequence Jhnny Bit rate savings vs DC-nly: 15.5% vs DC+H+V: 8.0% Jhnny, 1280x720, 60Hz Y-PSNR [db] DC nly 36 DC, Hr, Ver DC, Hr, Ver, 45 degree 35 all predictin mdes bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

85 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Efficiency f Intra Predictin Sequence Cactus Bit rate savings vs DC-nly: 7.2% vs DC+H+V: 6.6% Cactus, 1920x1080, 50Hz Y-PSNR [db] DC nly DC, Hr, Ver 32 DC, Hr, Ver, 45 degree 31 all predictin mdes bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

86 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Transfrm Cding f 4 4 Blcks Separable 4 4 integer transfrm Orthgnal blck transfrm with integer apprximatin f DCT u 4 4 = A 4 4 s 4 4 A T 4 4 with A 4 4 = Heik Schwarz Surce Cding and Cmpressin December 7, / Inverse is specified by exact integer peratins = N accumulatin f transfrm mismatches Easy implementatin (nly additins and bit shift peratins) Basis vectrs have different nrms = Cmpensated by mdifying the quantizer step size accrdingly Scalar quantizatin f transfrm cefficients Unifrmly distributed recnstructin levels Lgarithmic quantizatin step size cntrl ( α 2 QP/6 ) Smaller quantizatin step sizes fr chrma (as in Annex T f H.263) Supprt fr quantizatin weighting matrices (High prfile) Quantizatin parameter can change at macrblck level (609)

87 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Cding f Transfrm Cefficient Levels Cntext-adaptive variable length cding (CAVLC) Cded blck pattern (fr all six 8 8 blck) using VLC table Zig-zag scan fr mapping matrix int vectr Syntax element ceff tken fr 4 4 blcks Specifies number f nn-zer cefficients and number f trailing nes Chsen VLC table depends n number f nn-zer cefficients in already cded neighbring blcks Additinally cde runs and levels as well as signs fr trailing nes Cntext-adaptive binary arithmetic cding (CABAC) [Main, High prfile] Binary arithmetic cding f all lw-level syntax elements Cded blck pattern (flag fr each f the six 8 8 blcks) Flag fr 4 4 blcks indicating whether nn-zer levels are present Cding f a significance map significance flag indicating whether level is nn-zer if nn-zer, last flag indicating whether last nn-zer level Cding f abslute levels (minus 1) and signs Prbability mdels are adapted t statistics during encding and decding Heik Schwarz Surce Cding and Cmpressin December 7, / 539

88 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Intra and Intra-8 8 Macrblcks Intra macrblck mde Predictin f luma blck Fur predictin mdes: DC, hrizntal, vertical, planar 4 4 transfrm f all sixteen 4 4 blcks Additinal 4 4 Hadamard transfrm f DC cefficients DC blck is treated separately in entrpy cding Similar cncept with 2 2 Hadamard transfrm f DC cefficients is als used fr chrma blcks (fr all intra cding mdes) Intra-8 8 macrblck mde (High prfile) Predictin f 8 8 blcks f luma cmpnent Same predictin mdes as fr Intra-4 4 (but extended t larger blck size) Reference samples are lw-pass filtered befre they are used fr predictin 8 8 integer transfrm fr the fur luma sub-blcks Entrpy cding is extended t 8 8 transfrm blcks Heik Schwarz Surce Cding and Cmpressin December 7, / 539

89 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Cding Efficiency f Intra Mdes Sequence Jhnny Jhnny, 1280x720, 60Hz Y-PSNR [db] Intra4x4 37 Intra8x8 36 Intra16x16 Main prfile (4x4, 16x16) 35 High prfile (all mdes) bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

90 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Cding Efficiency f Intra Mdes Sequence Cactus Cactus, 1920x1080, 50Hz Y-PSNR [db] Intra4x4 33 Intra8x8 Intra16x16 32 Main prfile (4x4, 16x16) 31 High prfile (all mdes) bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

91 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Cding Efficiency Cmparisn with Older Standards Cmparisn f cding efficiency fr intra-picture cding Selectin f all features that cntribute t cding efficiency H.262 MPEG-2 Vide cnfrming t Main prfile H.263+ with advanced intra cding, deblcking filter, mdified quantizatin MPEG-4 Visual with MPEG-style quantizatin H.264 MPEG-4 AVC High prfile with CABAC Apply same level f encder ptimizatin fr fair cmparisn Best pssible cding efficiency fr given syntax Ignre cnstraints such as real-time peratin = Use rate-distrtin ptimized quantizatin fr all standards = Apply rate-distrtin ptimized mde decisin where applicable General cding cnditins Encde 10 pictures f 12 vide sequences (6 in 720p, 6 in 1080p) Flat quantizatin matrices (quality is measured using PSNR) Same quantizatin parameter fr all macrblcks Select Lagrangian parameter accrding t λ = cnst 2 (with experimentally determined factr) (610) Heik Schwarz Surce Cding and Cmpressin December 7, / 539

92 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Intra Cding Cmparisn Sequence Jhnny Jhnny, 1280x720, 60Hz Y-PSNR [db] H.262 MPEG-2 Vide 36 MPEG-4 Visual H H.264 MPEG-4 AVC HP bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

93 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Intra Cding Cmparisn Sequence Cactus Cactus, 1920x1080, 50Hz Y-PSNR [db] H.262 MPEG-2 Vide MPEG-4 Visual 32 H H.264 MPEG-4 AVC HP bit rate [kbit/s] Heik Schwarz Surce Cding and Cmpressin December 7, / 539

94 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Intra Cding Cmparisn Summary Bit-rate savings f H.264 MPEG-4 AVC versus lder vide cding standards Bit-rate saving at a PSNR value is btained by interplating the r-d curves Average bit-rate savings are btained by averaging the savings fr 100 PSNR values Highest savings are btained fr lw bit rates Average bit-rate saving fr all sequences are summarized belw average bit rate savings relative t... cdec H.263+ MPEG-4 MPEG-2 H.264 / AVC 21.1 % 28.2 % 45.6 % H % 32.5 % MPEG % Heik Schwarz Surce Cding and Cmpressin December 7, / 539

95 Still Image Cding Intra-Picture Cding in H.264 MPEG-4 AVC Summary f Intra-Picture Cding in H.264 MPEG-4 AVC Intra-picture cding in H.264 MPEG-4 AVC Fur intra macrblck mdes Intra-4 4, Intra-8 8, Intra-16 16: Predictin & transfrm cding Intra-PCM: Direct cding f samples Intra predictin in spatial dmain using neighbring samples Intra-4 4 and Intra-8 8: Eight directinal mdes & DC predictin Intra and chrma: Fur intra predictin mdes Transfrm: Integer apprximatin f DCT Intra-4 4: Transfrm f 4 4 blcks Intra-8 8: Transfrm f 8 8 blcks Intra and chrma: Transfrm f 4 4 blcks + DC transfrm Scalar quantizatin Unifrm recnstructin quantizer (with ptinal weighting matrices) Nrms f basis vectrs are taken int accunt in quantizatin Tw methds fr entrpy cding Cntext-adaptive variable length cding (CAVLC) Cntext-adaptive binary arithmetic cding (CABAC) Deblcking filter Heik Schwarz Surce Cding and Cmpressin December 7, / 539

96 Still Image Cding Intra-Picture Cding in H.265 MPEG-H HEVC Intra-Picture Cding in H.265 MPEG-H HEVC Heik Schwarz Surce Cding and Cmpressin December 7, / 539

97 Still Image Cding Intra-Picture Cding in H.265 MPEG-H HEVC H.265 MPEG-H HEVC Overview ITU-T Rec. H.265 ISO/IEC (MPEG-H High Efficiency Vide Cding) Jintly develped by ITU-T VCEG and ISO/IEC MPEG Last vide cding standard with fcus n cding f high-reslutin vide Significantly increased cding efficiency, particularly fr high-reslutin vide First versin was finalized in January 2013 First versin specifies three prfiles Main prfile (4:2:0 chrma frmat, 8 bit per sample) Main 10 prfile (4:2:0 chrma frmat, 10 bit per sample) Main Still Picture prfile (intra-nly cding subset f Main prfile) Extensins are under develpment Fidelity range extensin (ther chrma samplings, higher bit depth) Scalable vide cding extensin Multiview vide cding extensin Multiview vide plus depth cding extensin Heik Schwarz Surce Cding and Cmpressin December 7, / 539

98 Still Image Cding Intra-Picture Cding in H.265 MPEG-H HEVC Main Features f H.265 MPEG-H HEVC Main imprvements relative t H.264 MPEG-4 AVC Larger blck sizes fr transfrm cding and mtin cmpensatin Increased flexibility fr partitining a picture int blcks Imprved interplatin filters and mtin vectr cding Increased number f intra predictin mdes Imprved cding f transfrm cefficient levels Additinal in-lp filter: Sample-adaptive ffset filter Intra-Picture Cding in H.265 MPEG-H HEVC Spatial intra predictin and transfrm cding f predictin residual Increased number f intra predictin mdes cmpared t H.264/AVC Larger transfrm sizes Mre flexible partitining f a picture Imprved cding f transfrm cefficient levels (fr larger blcks) Deblcking filter and additinal sample-adaptive ffset filter Heik Schwarz Surce Cding and Cmpressin December 7, / 539

99 Still Image Cding Intra-Picture Cding in H.265 MPEG-H HEVC Picture Partitining in H.265 MPEG-H HEVC Picture partitining int cding tree blcks Cding tree blcks (CTBs): Fixed size f 16 16, r luma samples Size f CTBs chsen by encder Luma and chrma CTBs tgether with syntax are called cding tree unit (CTU) Partitining f cding tree blcks Quad-tree partitining int cding blcks (CBs) Luma and chrma CBs tgether with syntax are called cding unit (CU) Maximum CU size: Size f the CTB Minimum CU size: Selected by encder, but equal t r larger than 8 8 luma samples Cding mde (intra r inter) is chsen fr CU CU is similar t macrblck in lder standards Cding rder: Z-scan Heik Schwarz Surce Cding and Cmpressin December 7, / 539

100 Still Image Cding Intra-Picture Cding in H.265 MPEG-H HEVC Example: Picture Partitining int Cding Units Example fr picture partitining int cding units Picture with luma samples f HEVC test sequence Traffic Quadtree-based partitining int cding unit represents a simple scheme fr lcally adapting the blck sizes t the image structure Heik Schwarz Surce Cding and Cmpressin December 7, / 539

101 Still Image Cding Intra-Picture Cding in H.265 MPEG-H HEVC Intra Cding f Cding Units in H.265 MPEG-H HEVC Partitining f a CB int transfrm blcks (TBs) Nested quad-tree partitining TB crrespnds t a single blck transfrm Min. and max. TB size are selected by encder Supprted transfrms: 4 4, 8 8, 16 16, Luma and chrma TBs tgether with syntax frm a transfrm unit (TU) Special case: Chrma 4 4 blcks are nt split Intra predictin and mde signaling One r fur luma intra predictin mdes per cding unit One chrma predictin mde per CU Actual intra predictin is perfrmed transfrm blck by transfrm blck = Imprved predictin accuracy Heik Schwarz Surce Cding and Cmpressin December 7, / 539

Video Encoder Control

Video Encoder Control Vide Encder Cntrl Thmas Wiegand Digital Image Cmmunicatin 1 / 41 Outline Intrductin Encder Cntrl using Lagrange multipliers Lagrangian ptimizatin Lagrangian bit allcatin Lagrangian Optimizatin in Hybrid