Inter-Picture Coding. Inter-Picture Coding. o Thomas Wiegand Digital Image Communication 1 / 62

Transcription

1 Inter-Picture Cding Thmas Wiegand Digital Image Cmmunicatin 1 / 62

2 Outline Intrductin Accuracy f Mtin-Cmpensated Predictin Theretical Cnsideratins Chice f Interplatin Filters Mtin Vectr Accuracy Mtin Mdels Blck Sizes fr Mtin-Cmpensated Predictin Blck Sizes in Vide Cding Standards Experimental Analysis Advanced Mtin-Cmpensated Predictin Adaptive Reference Picture Selectin Multi-Hypthesis Predictin Cding f Mtin Parameters Cding Structures In-lp Filters Thmas Wiegand Digital Image Cmmunicatin 2 / 62

3 Intrductin Review Basic Cncept f Inter-Picture Cding s sƹ u transfrm transfrm u s channel encder decder transfrm decder u bitstream sƹ encder MCP s decder MCP Inter-Picture Cding mdes Mtin-cmpensated predictin (MCP) f current blck ŝ[x, y] = s ref(x + m x, y + m y ) Transfrm cding f predictin errr signal u[x, y] = s[x, y] ŝ[x, y] Thmas Wiegand Digital Image Cmmunicatin 3 / 62

4 Intrductin Review Mtin-Cmpensated Predictin already cded reference picture FRESH FOOD mving bject best-matching blck in reference picture current picture displacement vectr fr current blck FRESH FOOD FRESH FOOD m = m x m y current blck x displaced bject y Mtin-cmpensated predictin Predictin signal is determined by mtin vectr and chsen ref. picture ŝ[x, y] = s ref(x + m x, y + m y ) = Determine mtin vectr m = (m x, m y ) in encder: Mtin estimatin = Determine reference picture: Fixed pattern r adaptive selectin = Transmit mtin data (mtin vectr & reference index) inside bitstream Thmas Wiegand Digital Image Cmmunicatin 4 / 62

5 Intrductin Effectiveness f Mtin-Cmpensated Predictin Accuracy f mtin descriptin Mtin mdels (translatinal mdel r higher rder mdels) Accuracy f mtin vectrs (r mtin parameters) Chice f interplatin filters Selectin f regins with cnstant mtin Rectangular blcks r mre flexible regins Adaptive blck size selectin Cding f mtin parameters Predictin f mtin parameters Inference f mtin parameters Advanced cncepts fr mtin-cmpensated predictin Blck-adaptive selectin f reference picture Multi-hyptheses predictin Cding structures Thmas Wiegand Digital Image Cmmunicatin 5 / 62

6 Theretical Analysis f MCP Theretical Analysis Mdel fr Hybrid Vide Cding s[x] u[x] r-d ptimal intra-picture encder bitstream input pictures intra-picture decder s x = s ref (x + m) u [x] s [x] mtincmpensated predictin s ref [x] decded picture buffer utput pictures Assume r-d ptimal intra-picture cding fr predictin errr signal Cmpare infrmatin rate-distrtin functins fr intra and residual pictures Simple mdels fr riginal and predictin errr signals Simple mdel fr mtin vectr accuracy Thmas Wiegand Digital Image Cmmunicatin 6 / 62

7 Theretical Analysis f MCP Theretical Analysis Gaussian Randm Signals Assumptin: Original and residual picture are realizatins f statinary Gaussian prcess Nt realistic, but still prvides useful insights = Keeps prblem mathematically tractable = Rate-distrtin functin f Gaussian prcesses represent upper bund fr rate-distrtin functin f all prcesses with same pwer spectral density Rate-distrtin functin fr Gaussian prcesses Fr given pwer spectral density Φ SS (ω x, ω y ) Parametric frmulatin with θ > 0 D(θ) = 1 π π ( ) 4π 2 min θ, Φ SS (ω x, ω y ) dω x dω y π π R(θ) = 1 π π ( 4π 2 max 0, 1 π π 2 lg 2 ) Φ SS (ω x, ω y ) dω x dω y θ Thmas Wiegand Digital Image Cmmunicatin 7 / 62

8 Theretical Analysis f MCP Pwer Spectral Density f Predictin Errr Signal Pwer spectral density Φ UU (ω) f predictin errr signal u[x] = s[x] ŝ[x] Mtin-cmpensated predictin with mtin vectr m = (m x, m y ) Sub-sample interplatin with cntinuus interplatin kernel g(x) ŝ[x] = (g s ref)(x + m) 2D band-limited Furier transfrm yields Ŝ(ω) = G(ω) S ref(ω) e j ωt m Assumptin: Using the crrect displacement vectr d = (d x, d y ), we can predict the current image signal s[x] up t a nise term n[x] S(ω) = S ref(ω) e j ωt d + N(ω) = S ref(ω) = ( S(ω) N(ω) ) e j ωt d Nise term: All differences nt caused by bject mtin (includes camera nise, quantizatin nise, lighting changes,...) Thmas Wiegand Digital Image Cmmunicatin 8 / 62

9 Theretical Analysis f MCP Pwer Spectral Density f Predictin Errr Signal Band-limited Furier transfrm U(ω) f predictin errr signal u[x] = s[x] ŝ[x] Using displacement errr = ( x, y ) = d m U(ω) = S(ω) Ŝ(ω) = S(ω) G(ω) S ref(ω) e j ωt m = S(ω) G(ω) (S(ω) N(ω)) e j ωt (m d) ( = S(ω) 1 G(ω) e j ωt ) + N(ω) G(ω) e j ωt Cnditinal pwer spectral density Φ UU (ω ) fr given displacement errr Assume: Input signal s[x] and nise term n[x] are statistically independent ( { Φ UU (ω ) = Φ SS (ω) 1 + G(ω) 2 2 Re G(ω) e j ωt }) + Φ NN (ω) G(ω) 2 where Re{ } specifies the real part f its argument Thmas Wiegand Digital Image Cmmunicatin 9 / 62

10 Theretical Analysis f MCP Pwer Spectral Density f Predictin Errr Signal Pwer spectral density Φ UU (ω) f predictin errr signal Randm displacement errr, independent f s[x] and n[x] Φ UU (ω) = E { Φ UU (ω ) } ( { }) = Φ SS (ω) 1 + G(ω) 2 2 Re G(ω)F (ω) + Φ NN (ω) G(ω) 2 with F (ω) being the expectatin value E {e j ωt } Nte: F (ω) is the Furier transfrm f the pdf f ( ) = f ( x, y ) fr the displacement errr F (ω) = E {e j ωt } = f ( x, y ) e j(ωx x+ωy y) d x d y Thmas Wiegand Digital Image Cmmunicatin 10 / 62

11 Theretical Analysis f MCP Theretical Analysis Interplatin Filter Optimal Interplatin Filter Wiener filter that minimizes Φ UU (ω) fr each spatial frequency ω = (ω x, ω y ) Given by frequency respnse (see derivatin in Gird, 1987) G pt (ω) = F (ω) Φ SS (ω) Φ SS (ω) + Φ NN (ω) with F (ω) being the cmplex cnjugate f F (ω) Yields pwer spectral density ( ) Φ pt UU (ω) = Φ SS(ω) 1 F (ω) 2 Φ SS (ω) Φ SS (ω) + Φ NN (ω) Ideal interplatin filter Reference fr evaluating the impact f the Wiener filter Frequency respnse G int (ω) = 1 yields pwer spectral density Φ int UU(ω) = 2 Φ SS (ω) ( 1 Re { F (ω) }) + Φ NN (ω) Thmas Wiegand Digital Image Cmmunicatin 11 / 62

12 Theretical Analysis f MCP Theretical Analysis Mtin Vectr Accuracy Furier transfrm F (ω) f displacement errr pdf f ( ) = f ( x, y ) Mdel: Displacement errrs = d m are unifrmly distributed inside an interval [ max, max ] [ max, max ] Furier transfrm F (ω) f displacement errr pdf F (ω) = F (ω x, ω y ) = max max max max max = sinc( max ω x ) sinc( max ω y ) e j(ωx x+ωy y) d x d y Assume: Displacement errrs result frm limited mtin vectr accuracy max = 2 (1+β) Yielding Furier transfrm with β = 0 integer-sample precisin β = 1 half-sample precisin β = 2 quarter-sample precisin F (ω) = F (ω x, ω y ) = sinc( 2 (1+β) ω x ) sinc( 2 (1+β) ω y ) Thmas Wiegand Digital Image Cmmunicatin 12 / 62

13 Theretical Analysis f MCP Theretical Analysis Signal Mdel Mdel fr pwer spectral density Φ SS (ω) Istrpic autcrrelatin functin { } R SS (ζ, η) = E S(x, y) S(x ζ, y η) Pwer spectral density Φ SS (ω) is given by = Yields mdel Φ SS (ω) = Φ SS (ω x, ω y ) = Φ SS (ω) = Φ SS (ω x, ω y ) = K = σs 2 ϱ ζ 2 +η 2 R SS (ζ, η) e j (ωx ζ+ωy η) dζ dη Thmas Wiegand Digital Image Cmmunicatin 13 / 62 ( 1 + ω2 x + ω 2 y (ln ϱ) 2 Cnsider band-limited signals sampled at the Nyquist rate = Fr given variance σs 2, chse K s that σ 2 S = 1 4π 2 π π π π Φ SS (ω x, ω y ) dω x dω y ) 3 2

14 Theretical Analysis f MCP Theretical Analysis Nise Mdel Remember: Nise term n[x] Represents differences between riginal picture s[x] and recnstructed reference picture s ref [x] that are nt caused by mtin Includes camera nise Includes quantizatin nise (depends n bit rate) Includes ther effects such as lighting changes... Simple mdel: White nise Cnstant pwer spectrum Φ NN (ω) inside base band Φ NN (ω) = σ 2 N = σ 2 S Θ Signal-t-nise parameter Θ characterizes rati f signal energies Θ = 10 lg 10 σ 2 S σ 2 N Thmas Wiegand Digital Image Cmmunicatin 14 / 62

15 Theretical Analysis f MCP Theretical Analysis Basic Apprach Pwer spectral density Φ UU (ω) and rate-distrtin functin R(D) depend n Signal variance σ 2 S = Only a scaling factr Crrelatin cefficient ϱ = Use ϱ = 0.9 (apprx. fr natural image) Signal-t-nise parameter Θ Mtin vectr accuracy β Chice f interplatin filter: Wiener filter r ideal interplatr Cmparisn f intra-picture cding and inter-picture cding Chse mdel parameters Θ and β and interplatin filter Calculate pwer spectral densities Φ SS (ω) and Φ UU (ω) fr riginal signal s[x] and predictin errr signal u[x] = s[x] ŝ[x] = Use derived frmulas fr F (ω), Φ SS (ω), Φ NN (ω), and Φ UU (ω) Calculate rate-distrtin functins R(D) fr ptimal cding f riginal signal s[x] and predictin errr signal u[x] = s[x] ŝ[x] = Use parametric frmula fr Gaussian randm signals = Assume Θ f(r) = Cnsider high bit rates Thmas Wiegand Digital Image Cmmunicatin 15 / 62

16 Theretical Analysis f MCP Theretical Results Small Signal-t-Nise Rati slid lines : Wiener filter dashed lines : ideal interplatr Φ(ω) / σ S 2 [db] ρ = 0.9, Θ = 5 db 20 signal spectrum Φ SS 10 Φ UU (integer-sample accuracy) Φ UU (quarter-sample accuracy) SNR: 10 lg 10 ( σ S 2 / D ) [db] 35 ρ = 0.9, Θ = 5 db 30 MCP (quarter-sample accuracy) intra-picture cding 5 MCP (integer-sample accuracy) spatial frequency ω x (ω y =0) bit rate [bits/sample] Theretical results fr a small signal-t-nise parameter f Θ = 5 db = Mtin vectr accuracy has a rather small effect = Chice f interplatin filter is very imprtant = N cding efficiency gain withut suitable interplatin filter Thmas Wiegand Digital Image Cmmunicatin 16 / 62

17 Theretical Analysis f MCP Theretical Results Large Signal-t-Nise Rati slid lines : Wiener filter dashed lines : ideal interplatr Φ(ω) / σ S 2 [db] ρ = 0.9, Θ = 20 db 20 signal spectrum Φ SS 10 Φ UU (integer-sample accuracy) Φ UU (quarter-sample accuracy) SNR: 10 lg 10 ( σ S 2 / D ) [db] 35 ρ = 0.9, Θ = 20 db intra-picture cding MCP (quarter-sample accuracy) 5 MCP (integer-sample accuracy) spatial frequency ω x (ω y =0) bit rate [bits/sample] Theretical results fr a large signal-t-nise parameter f Θ = 20 db = Interplatin filter has smaller effect = Higher mtin vectr precisin increases cding efficiency = Inter-picture cding yields large gain in cding efficiency Thmas Wiegand Digital Image Cmmunicatin 17 / 62

18 Theretical Analysis f MCP Theretical Results Summary slid lines : Wiener filter dashed lines : ideal interplatr SNR gain rel. t intra cding ρ = 0.9 Θ = 20 db Θ = 15 db Θ = 10 db Θ = 5 db 1/1 1/2 1/4 1/8 1/16 mtin vectr accuracy in samples 20 lg 10 G pt (ω) [db] Θ = 5 db -10 Θ = 10 db -15 Θ = 15 db -20 ρ = 0.9, β = 2 Θ = 20 db spatial frequency ω x (ω y =0) Strnger lw-pass filter fr smaller signal-t-nise parameters Θ Wiener filter appraches ideal interplatr fr large Θ Inter-picture cding gain increases with SNR parameter Θ Cding gain increases with mtin vectr accuracy, but appraches a limit Thmas Wiegand Digital Image Cmmunicatin 18 / 62

19 Chice f Interplatin Filters Interplatin Filters in Practical Vide Cdecs Translatinal mtin vectrs with given accuracy Cntinuus interplatin kernel g(x) nt required Sufficient: Set f discrete filters h[k, l] Separable Finite Impulse Respnse (FIR) Filters Runded crdinates x i = x and y i = y Intermediate hrizntally interplated signal s hr (x, y i ) = k h hr [k] s ref[x i k, y] Final interplated reference picture s ref (x, y) = k h ver [k] s hr (x, y i k) integer-sample lcatins half-sample lcatins quarter-sample lcatins Chice f filter h hr [k] and h ver [k] depends n phases φ x = x x i and φ y = y y i Thmas Wiegand Digital Image Cmmunicatin 19 / 62

20 Chice f Interplatin Filters Examples f Interplatin Filters 2-tap 4-tap 6-tap phase filter cefficients (results are divided by 64) 1/ / / / / / tap/ 1/ tap 1/ Interplatin filters in vide cding standards 2-tap: H.262 MPEG-2 Vide, H.263, MPEG-4 Visual (Simple prfile) 4-tap: Chrma interplatin filter in H.265 MPEG-H HEVC 6-tap: Luma interplatin filter in H.264 MPEG-4 AVC = Includes simple, nt separable filters fr sme psitins 7/8-tap: Luma interplatin filter in H.265 MPEG-H HEVC Thmas Wiegand Digital Image Cmmunicatin 20 / 62

21 Chice f Interplatin Filters Interplatin Filters Frequency Respnses Magnitude 20 lg 10 G(ω) [db] 2 half-sample lcatins tap filter -4 4-tap filter -6 6-tap filter -8 8-tap filter spatial frequency ω Magnitude 20 lg 10 G(ω) [db] 2 quarter-sample lcatins tap filter -4 4-tap filter -6 6-tap filter -8 7-tap filter spatial frequency ω Frequency characteristics f cnsidered interplatin filters Lnger filters have less lw-pass characteristics Strnger lw-pass filtering fr half-sample psitins Nte: N filtering is (typically) applied fr integer-sample psitins = Chsen sub-sample psitin des nt nly determine the phases f the filters, but als the strength f lw-pass filtering Thmas Wiegand Digital Image Cmmunicatin 21 / 62

22 Chice f Interplatin Filters Interplatin Filters Cding Efficiency bit-rate saving vs 2-tap filters [%] BQSquare ( , 60 Hz) /4-sample acc. (IPPP, MC blcks) 60 8/7-tap filters tap filters 6-tap filters PSNR (Y) [db] bit-rate saving vs 2-tap filters [%] Jhnny ( , 60 Hz) /4-sample acc. (IPPP, MC blcks) 30 6-tap filters tap filters 8/7-tap filters PSNR (Y) [db] First cding experiment: IPPP cding with blcks Quarter-sample precisin vectrs, blcks, IPPP cding Sequence BQSquare : High signal-t-nise rati Sequence Jhnny : Mderate signal-t-nise rati Best filter depends n sequence characteristics Thmas Wiegand Digital Image Cmmunicatin 22 / 62

23 Chice f Interplatin Filters Interplatin Filters Cding Efficiency bit-rate saving vs 2-tap filters [%] BQSquare ( , 60 Hz) /4-sample acc. (IBBB, all MC blck sizes) 60 8/7-tap filters tap filters 6-tap filters PSNR (Y) [db] bit-rate saving vs 2-tap filters [%] Jhnny ( , 60 Hz) /4-sample acc. (IBBB, all MC blck sizes) 30 4-tap filters tap filters 8/7-tap filters PSNR (Y) [db] Secnd cding experiment: Quarter-sample accuracy and additinal cding tls Quarter-sample precisin vectrs, variable blck sizes, bi-predictin enabled Same test sequences as in previus experiment 7/8-tap filters f H.265 MPEG-H HEVC give best cding efficiency Larger differences fr sequence with high signal-t-nise rati Thmas Wiegand Digital Image Cmmunicatin 23 / 62

24 Chice f Interplatin Filters Interplatin Filters Summary Experimental results Advantageus t chse filters that apprximate ideal interplatr = Large gains fr cntent with high signal-t-nise rati = Small r n lsses fr cntent with mderate signal-t-nise rati Lnger filters prvide mre degrees f freedm fr designing filter respnse Reasns fr discrepancy between experimental results and theretical analysis Blck-adaptive selectin between intra- and inter-picture cding = Inter-picture cding is chsen nly when it prvides a cding gain = Cding efficiency cannt becme wrse than that fr intra-nly cding Cmparably high SNR parameter fr inter-picture cded blcks Bi-predictin has similar effect as lw-pass filtering = Optimal filters have weaker lw-pass character In-lp filters (e.g., deblcking filter) = Signal-adaptive lw-pass filters fr large quantizatin nise Thmas Wiegand Digital Image Cmmunicatin 24 / 62

25 Mtin Vectr Accuracy Mtin Vectr Accuracy PSNR (Y) [db] BQSquare ( , 60 Hz) MCP (1/4 sample) 30 IPPP, MC blcks bit rate [Mbit/s] MCP (1/1 sample) MCP (1/2 sample) Intra PSNR (Y) [db] Jhnny ( , 60 Hz) MCP (1/1 sample) 40 MCP (1/2 sample) 38 MCP (1/4 sample) Intra 36 IPPP, MC blcks bit rate [Mbit/s] Cding experiment with H.265 MPEG-H HEVC, IPPP cding, blcks Cmparisn f integer-sample, half-sample, and quarter-sample accuracy Sub-sample interplatin with 7/8-tap filters f H.265 MPEG-H HEVC = Cding efficiency increases with mtin vectr precisin = Very large gains fr sequences with high SNR characteristics Thmas Wiegand Digital Image Cmmunicatin 25 / 62

26 Mtin Vectr Accuracy Mtin Vectr Accuracy entertainment-quality vide cntent cdec bit-rate savings relative t Intra IS-MCP HS-MCP IS-MCP % HS-MCP % % QS-MCP % % 8.88% cdec vide cnferencing cntent bit-rate savings relative t Intra IS-MCP HS-MCP IS-MCP % HS-MCP % % QS-MCP % % 11.75% Average bit-rate savings fr tw classes f HD vide sequences IPPP cding with blcks Cmparisn f Intra-picture cding Inter-picture cding with integer-sample accurate mtin vectrs Inter-picture cding with half-sample accurate mtin vectrs Inter-picture cding with quarter-sample accurate mtin vectrs = Quarter-sample accurate vectrs are typically a reasnable chice Usage f eighth-sample accuracy smetimes prvides additinal cding gain (depends n characteristics f vide sequence) Thmas Wiegand Digital Image Cmmunicatin 26 / 62

27 Mtin Mdels Higher Order Mtin Mdels General: Represent mtin field f a blck/regin with a set f parameters Translatinal mtin mdel (2 parameters) used in mst vide cdecs m x (x, y) = m x m y (x, y) = m y Affine mtin mdel (6 parameters) m x (x, y) = a 0 + a 1 x + a 2 y m y (x, y) = a 3 + a 4 x + a 5 y Planar perspective mdel (8 parameters) m x (x, y) = (a 0 + a 1 x + a 2 y) / (1 + a 6 x + a 7 y) x m y (x, y) = (a 3 + a 4 x + a 5 y) / (1 + a 6 x + a 7 y) y Parablic mdel (12 parameters) m x (x, y) = a 0 + a 1 x + a 2 y + a 3 x 2 + a 4 xy + a 5 y 2 m y (x, y) = a 6 + a 7 x + a 8 y + a 9 x 2 + a 10 xy + a 11 y 2 Thmas Wiegand Digital Image Cmmunicatin 27 / 62

28 Mtin Mdels Higher Order Mtin Mdels affine mdel planar perspective mdel parablic mdel Usage f higher rder mtin mdels Can (ptentially) better apprximate real mtin f bjects Require higher bit rate fr signaling mtin parameters Difficult t estimate mtin parameters (exhaustive search infeasible) = Iterative Gauss-Newtn methds Require mre general interplatin methds Observed cding gains typically d nt justify additinal cmplexity Thmas Wiegand Digital Image Cmmunicatin 28 / 62

29 Blck Sizes fr Mtin-Cmpensated Predictin Blck Sizes fr Mtin-Cmpensated Predictin Partitining f a picture int regins with cnstant mtin Small blcks: Enables mre accurate predictin signals Large blcks: Less bit rate fr transmitting mtin data Large blcks: Higher degree f freedm fr chsing transfrm size (typically disadvantageus t apply a transfrm acrss mtin bundaries) Nn-statinary character f vide signals = Variable blck sizes Blck sizes fr mtin-cmpensated predictin in vide cding standards H.262 MPEG-2 Vide: Single fixed blck size f luma samples H.263, MPEG-4 Visual: Tw mdes fr a macrblck Macrblck is cded as single blck Macrblck is split int fur 8 8 blcks H.264 MPEG-4 AVC: Macrblck: One 16 16, tw 16 8, tw 8 16, r fur sub-macrblck: One 8 8, tw 8 4, tw 4 8, r fur 4 4 H.265 MPEG-H HEVC: (see next slide) Thmas Wiegand Digital Image Cmmunicatin 29 / 62

30 Blck Sizes fr Mtin-Cmpensated Predictin Predictin Blck Sizes in H.265 MPEG-H HEVC M M M (M/2) (M/2) M (M/2) (M/2) M (M/4) M (3M/4) (M/4) M (3M/4) M Partitining f a Cding Tree Unit (CTU) int Predictin Units (PUs) Quadtree partitining int cding units (CU) Cding unit: Decisin between intra-picture and inter-picture cding Cding unit can be further partitined int up t fur predictin units (PU) Fur PUs nly allwed fr minimum CU size (if greater than 8 8) Asymmetric mdes nly supprted fr CU sizes greater than 8 8 = Predictin blck sizes range frm 8 4 / 4 8 t Thmas Wiegand Digital Image Cmmunicatin 30 / 62

31 Blck Sizes fr Mtin-Cmpensated Predictin Blck Sizes fr MCP Cding Efficiency PSNR (Y) [db] Cactus ( , 50 Hz) adaptive bit-rate saving f 8 8 adaptive vs 16 16: % n average 4 4 IPPP, 4 4 transfrm bit rate [Mbit/s] PSNR (Y) [db] Jhnny ( , 60 Hz) adaptive IPPP, 4 4 transfrm bit rate [Mbit/s] bit-rate saving f adaptive vs 32 32: 34.8 % n average First cding experiment: Exclude impact n residual cding (transfrm sizes) Use flexible H.265 MPEG-H HEVC syntax and disable certain blck sizes Fixed predictin blck sizes (4 4 t 64 64) and quadtree partitining Residual cding with 4 4 transfrm = Significant cding gain fr variable blck sizes (quadtree-based apprach) Thmas Wiegand Digital Image Cmmunicatin 31 / 62

32 Blck Sizes fr Mtin-Cmpensated Predictin Blck Sizes fr MCP Cding Efficiency PSNR (Y) [db] adaptive Cactus ( , 50 Hz) IPPP, all transfrm sizes bit rate [Mbit/s] bit-rate saving f adaptive vs 16 16: 21.7 % n average PSNR (Y) [db] Jhnny ( , 60 Hz) adaptive IPPP, all transfrm sizes bit rate [Mbit/s] bit-rate saving f adaptive vs 32 32: 29.2 % n average Secnd cding experiment: Adaptive transfrm blck sizes Fixed predictin blck sizes (4 4 t 64 64) and quadtree partitining Residual cding with variable blck sizes (4 4 t 32 32) = Imprves cding efficiency fr large MCP blck sizes = Significant cding gain fr variable blck sizes (quadtree-based apprach) Thmas Wiegand Digital Image Cmmunicatin 32 / 62

33 Blck Sizes fr Mtin-Cmpensated Predictin Cding Efficiency fr Nn-Square Predictin Blcks bit-rate saving vs square blcks [%] Cactus ( , 50 Hz) all supprted PU sizes (average: 2.2 %) square and sym. PUs (average: 1.5 %) PSNR (Y) [db] bit-rate saving vs square blcks [%] Jhnny ( , 60 Hz) 6 all supprted PU sizes (average: 4.0 %) square and sym. PUs (average: 2.5 %) PSNR (Y) [db] Experiment: Impact f nn-square predictin blcks (H.265 MPEG-H HEVC) Reference: Quadtree partitining with blck sizes frm 8 8 t Additinally enabled symmetric & asymmetric partitining mdes = Further imprvement f cding efficiency = Smaller effect than quadtree partitining Thmas Wiegand Digital Image Cmmunicatin 33 / 62

34 Blck Sizes fr Mtin-Cmpensated Predictin Summary: Blck Sizes in Vide Cding Standards H.262 MPEG-2 Vide Intra/Inter decisin & MCP: macrblcks Transfrm cding: 8 8 blcks H.263 & MPEG-4 Visual Intra/Inter decisin: macrblcks MCP: r 8 8 blcks Transfrm cding: 8 8 blcks H.264 MPEG-4 AVC Intra/Inter decisin: macrblcks MCP: 16 16, 16 8, 8 16, 8 8, 8 4, 4 8, r 4 4 blcks Transfrm cding: (lw cmplexity varaiant), 8 8, r 4 4 blcks H.265 MPEG-H HEVC Intra/Inter decisin: 64 64, 32 32, 16 16, r 8 8 cding units MCP: Frm t 8 4 / 4 8 (including nn-square blcks) Transfrm cding: 32 32, 16 16, 8 8, r 4 4 blcks Thmas Wiegand Digital Image Cmmunicatin 34 / 62

35 Blck Sizes fr Mtin-Cmpensated Predictin Cding Efficiency Impact f Supprted Blck Sizes entertainment-quality vide cntent cnfig. bit-rate savings relative t H.262 H.263 H.264 H % H % 6.04 % H % % % cnfig. vide cnferencing cntent bit-rate savings relative t H.262 H.263 H.264 H % H % 3.80 % H % % % Cding experiment fr HD test sequences Use H.265 MPEG-H HEVC with restricted blck sizes Only mdify supprted blck sizes fr simulating lder standards (use H.265 MPEG-H HEVC fr all ther aspects f cding) = Increased flexibility fr partitining a picture int blcks fr predictin and residual cding significantly cntributes t cding efficiency = Large blck sizes imprtant fr high-reslutin vide Thmas Wiegand Digital Image Cmmunicatin 35 / 62

36 Multiple Reference Pictures Multiple Reference Pictures s k 3 [x] s k 2 [x] s k 1 [x] s k [x] recnstructed reference pictures current picture Mtin-cmpensated predictin with multiple reference pictures Stre multiple recnstructed in a decded picture buffer Sliding windw buffer (stre mst recently decded pictures) Adaptive buffer management (transmit cmmands) Cnstruct reference picture list fr current picture Chse reference picture in additin t mtin vectr Transmit reference picture index in additin t mtin vectr Thmas Wiegand Digital Image Cmmunicatin 36 / 62

37 Multiple Reference Pictures Multiple Reference Pictures Cding Efficiency bit-rate saving vs 1 ref. pic. [%] Cactus ( , 50 Hz) 20 8 ref. pics. (avg. 7.6%) 15 4 ref. pics. (avg. 5.7%) 10 2 ref. pics. (avg. 3.0%) PSNR (Y) [db] bit-rate saving vs 1 ref. pic. [%] Jhnny ( , 60 Hz) 25 8 ref. pics. (avg. 10.1%) 20 4 ref. pics. (avg. 9.0%) 15 2 ref. pics. (avg. 5.3%) PSNR (Y) [db] Cding experiment with H.265 MPEG-H HEVC IPPP cding: Use N previus pictures as reference pictures Vary number N f available reference pictures frm 1 t 8 Reference: Single reference picture = Cding efficiency increases with number f reference pictures Thmas Wiegand Digital Image Cmmunicatin 37 / 62

38 Multi-Hypthesis Predictin Multi-Hypthesis Predictin s k 3 [x] s k 2 [x] s k 1 [x] s k [x] recnstructed reference pictures current picture Mtin-cmpensated predictin with multiple mtin hyptheses Weighted summatin f multiple displaced reference blcks s r k (x + m k ) Simplest and mst cmmnly used variant ŝ[x] = 1 K K 1 k=0 s r k (x + m k ) Thmas Wiegand Digital Image Cmmunicatin 38 / 62

39 Multi-Hypthesis Predictin Multi-Hypthesis Predictin Mtivatin f multi-hypthesis predictin Idealized assumptin: Individual predictin errrs u k [x] = s[x] s r k (x + m k ) are uncrrelated = Reduced residual pwer spectrum cmpared t single-hypthesis predictin Φ UU (ω) 1 K Φ U 0U 0 (ω) = Imprved cding efficiency fr residual signal But: Bit rate required fr transmitting mtin data is increased Blck adaptive selectin f number f mtin hyptheses Multi-hypthesis predictin des nt imprve cding efficiency fr all blcks = Adaptive selectin f number f mtin hyptheses Vide cding standards: Only up t tw mtin hyptheses (bi-predictin) Mtin estimatin: Iterative refinement f mtin vectrs Thmas Wiegand Digital Image Cmmunicatin 39 / 62

40 Multi-Hypthesis Predictin Multi-Hypthesis Predictin in Vide Cding Standards frward predictin backward predictin bi-predictin I/P B (2) B (3) I/P (1) H.262 MPEG-2 Vide, H.263, MPEG-4 Visual I (intra nly), P (uni-predictin), and B (bi-predictin) pictures B pictures are nly supprted as part f an BBP r BBI grup B pictures supprt the fllwing basic MCP mdes Frward: Uni-predictin frm preceding I/P picture Backward: Uni-predictin frm succeeding I/P picture Bi-directinal: Bi-predictin frm preceding and succeeding I/P picture Thmas Wiegand Digital Image Cmmunicatin 40 / 62

41 Multi-Hypthesis Predictin Multi-Hypthesis Predictin in Vide Cding Standards H.264 MPEG-4 AVC and H.265 MPEG-H HEVC Generalized cncept f bi-predictin and multiple reference pictures Decupling f cding rder and picture types I, P, and B slices instead f I, P, and B pictures B slices: Tw reference picture lists (list 0 and list 1) = Bth lists can include pictures frm past and future = Each stred picture can be included in either r bth lists Allws new (mre efficient) cding structures (discussed later) Basic inter-picture cding mdes supprted in B slices List 0 predictin: Uni-predictin with picture frm list 0 List 1 predictin: Uni-predictin with picture frm list 1 Bi-predictin: Bi-predictin with picture frm list 0 and picture frm list 1 = Bth hyptheses can use the same picture = Enables bi-predictin withut mdifying the cding rder (IBBB) Thmas Wiegand Digital Image Cmmunicatin 41 / 62

42 Multi-Hypthesis Predictin Multi-Hypthesis Predictin Cding Efficiency bit-rate saving vs uni-pred. [%] Cactus ( , 50 Hz) 20 2 ref. pics. (avg. 5.9%) 15 8 ref. pics. (avg. 7.3%) 10 4 ref. pics. (avg. 6.5%) 5 1 ref. pic. (avg. 5.3%) PSNR (Y) [db] bit-rate saving vs uni-pred. [%] Jhnny ( , 60 Hz) ref. pics. (avg. 12.1%) 20 4 ref. pics. (avg. 11.4%) 15 2 ref. pics. (avg. 10.5%) ref. pic. (avg. 8.3%) PSNR (Y) [db] Cding experiment with H.265 MPEG-H HEVC, IPPP/IBBB cding structure Blck-adaptive bi-predictin with different number f reference pictures Reference: Uni-predictin with same number f reference pictures = Significant cding gain fr blck-adaptive bi-predictin = Bi-predictin gain slightly increases with number f reference pictures Thmas Wiegand Digital Image Cmmunicatin 42 / 62

43 Multi-Hypthesis Predictin Multi-Hypthesis Predictin Cding Efficiency average bit-rate saving [%] entertainment-quality vide cntent adaptive bi-predictin uni-predictin number f reference pictures average bit-rate saving [%] vide cnferencing cntent adaptive bi-predictin uni-predictin number f reference pictures Cding experiment with H.265 MPEG-H HEVC, IPPP/IBBB cding structure Average bit rate savings relative t uni-predictin with single reference picture Average bit rate savings fr tw classes f HD test sequences = Bit-rate savings increase with number f reference pictures = Significant cding gains by cmbining multi-hypthesis predictin and multiple reference pictures Thmas Wiegand Digital Image Cmmunicatin 43 / 62

44 Cding f Mtin Parameters Cding f Mtin Parameters Cding parameters fr inter-picture cding mdes Number f mtin hyptheses r predictin type Reference index fr each hypthesis Mtin vectr fr each hypthesis Cnventinal inter cding mdes Transmit predictin type and reference index (unless it can be inferred) Transmit mtin vectr difference ( m x, m y ) = (m x, m y ) ( m x, m y ) = Cding efficiency depends n chice f mtin vectr predictr ( m x, m y ) Cding mdes with inferred mtin parameters All mtin parameters are derived at the decder side Explit data f already cded blcks (spatial & tempral neighbrs) = Suitable fr cnsistently mving regins Thmas Wiegand Digital Image Cmmunicatin 44 / 62

45 Cding f Mtin Parameters Mtin Vectr Predictin Simple variant H.262 MPEG-2 Vide Use mtin vectr f left blck as predictr m x m y = m (left) x = m (left) y Large mtin vectr differences at bject bundaries Median predictin H.263, MPEG-4 Visual, H.264 MPEG-4 AVC Cmpnent-wise median f three neighbring blcks m x = median ( m A x, m B x, m C ) x m y = median ( m A y, m B y, m C ) y A B current blck C On average, smaller mtin vectr differences Thmas Wiegand Digital Image Cmmunicatin 45 / 62

46 Cding f Mtin Parameters Mtin Vectr Predictin B 2 B 1 B 0 c-lcated blck current blck T 1 A 1 A 0 T 0 Switched mtin vectr predictin (H.265 MPEG-H HEVC) Idea: Adaptively chse mst suitable neighbring mtin vectr Cnstruct list f spatially and tempral neighbring mtin vectrs Select best candidate predictr Transmit index int candidate list in additin t mtin vectr difference H.265 MPEG-H HEVC: Tw candidate predictrs = Bit rate saving fr mtin vectr differences is typically larger than bit rate required fr signaling the chsen predictr Thmas Wiegand Digital Image Cmmunicatin 46 / 62

47 Cding f Mtin Parameters Cding Mdes with Inferred Mtin Parameters Tempral direct mde Mtin parameters fr bi-predictin Scale mtin vectr m cl f c-lcated blck accrding t time differences m 0 = t c t 0 t 1 t 0 m cl t 0 m 0 m cl current blck m 1 t c t 1 c-lcated blck time m 1 = t c t 1 t 1 t 0 m cl Merge mde (H.265 MPEG-H HEVC) Similar cncept as switched mtin vectr predictin Candidate list with mtin parameters f spatial and tempral neighbrs Select best candidate Transmit index int candidate list current blck Thmas Wiegand Digital Image Cmmunicatin 47 / 62 E A D B C c-lcated blck T 1 T 0

48 Cding Structures Cnventinal Cding Structures I P P P P P P P P P 8 9 I B B P B B P B B P 9 7 IPPP / IBBB cding All pictures are cded in acquisitin/display rder Pictures can be cded as P/B pictures (r P/B slices) Usage f multiple reference pictures can be enabled Cnventinal B pictures Pictures are cded using BBP r BBI grups P pictures can be replaced with B pictures Usage f multiple reference pictures can be enabled Tw hierarchy levels: I/P pictures & B pictures Thmas Wiegand Digital Image Cmmunicatin 48 / 62

49 Cding Structures Hierarchical Cding Structures grup f pictures (GOP) grup f pictures (GOP) grup f pictures (GOP) I 0 B 3 B 2 B 3 B 1 B 3 B 2 B 3 B 0 B 3 B 2 B 3 B 1 B 3 B 2 B 3 B B 3 B 2 B 3 B 1 B 3 B 2 B 3 B Hierarchical B pictures Vide sequence is partitined int s-called grups f pictures (GOPs) Multiple hierarchy levels (mst cmmn design: dyadic structures) Key pictures: Pictures f lwest hierarchy level Recnstructin quality f hierarchy levels have different imprtance = Cascading f quantizatin parameters QP k = QP 0 + δ 1 + (k 1) (typically: δ 1 = 4) Thmas Wiegand Digital Image Cmmunicatin 49 / 62

50 Cding Structures Hierarchical Cding Structures Cding Efficiency IBBB GOP2 GOP4 (simple) I/B B B B B B B B B I/B B 1 B 0 B 1 B 0 B 1 B 0 B 1 B 0 avg. bit-rate saving vs IBBB [%] cascaded QP assignment with δ 1 =4 GOP2 Average Cactus ParkScene BQTerrace GOP4 (simple) Kimn BasketballDrive GOP4 (hier.) GOP8 (hier.) GOP4 (hier.) GOP8 (hier.) I/B B 1 B 1 B 1 B 0 B 1 B 1 B 1 B 0 I/B B 2 B 1 B 2 B 0 B 2 B 1 B 2 B 0 avg. bit-rate saving vs IBBB [%] average f entertainment-quality vide cntent GOP2 GOP4 (simple) GOP4 (hier.) same QP GOP8 (hier.) I/B B 3 B 2 B 3 B 1 B 3 B 2 B 3 B 0 Thmas Wiegand Digital Image Cmmunicatin 50 / 62 δ 1 =2 δ 1 =4 δ 1 =3 δ 1 =1

51 Cding Structures Hierarchical Cding Structures Effect f QP Cascading PSNR (Y) [db] Cactus ( , 50 Hz) GOP8 (cascaded QP, δ 1 =4, QP 0 =28): Average = db at 4052 kbit/s 37.0 GOP8 (same QP, QP 0 =30): Average = db at 5080 kbit/s IBBB (same QP, QP 0 =30): Average = db at 5504 kbit/s 34.5 GOP8 (cascaded QP, δ 1 =4, QP 0 =30): Average = db at 3077 kbit/s picture number (in display rder) Same QP fr all hierarchy levels = Rughly same PSNR ver all pictures (similar as fr IBBB cding) QP Cascading ver hierarchy levels = Higher PSNR fr key pictures (nnetheless, vide appears smth) = PSNR lsses fr nn-key pictures are utweighed by bit rate savings Thmas Wiegand Digital Image Cmmunicatin 51 / 62

52 Cding Structures Lw-Delay Hierarchical Cding Structures I B B B B B B B B bit-rate saving vs IBBB [%] 0 I 0 B 2 B 1 B 2 B 0 B 2 B 1 B 2 B PSNR (Y) [db] FurPeple Vidy3 interactive vide cntent average: 15.9% Vidy4 KristenAndSara Jhnny Vidy1 Lw-delay hierarchical cding structures Lw-delay applicatins: Pictures are cded in acquisitin/display rder Can still change reference picture lists & QP cascading Usage f pictures with higher quality (lwer QP) ften imprves predictin quality fr ther pictures (utweighs additinal bit rate) = Cding efficiency can ften be imprved Thmas Wiegand Digital Image Cmmunicatin 52 / 62

53 In-Lp Filters In-Lp Filters Filtering f recnstructed pictures Reduce visual impact f typical cding artifacts = Blcking artifacts (blck-wise predictin and transfrm cding) = Ringing artifacts (lng transfrm basis functins & interplatin filters) Pst filters r in-lp filters Bth imprve subjective quality f utput pictures In-lp filters als imprve predictin signal fr fllwing pictures In-lp filters are nrmative In-lp filters Deblcking filter (H.263, H.264 MPEG-4 AVC, H.265 MPEG-H HEVC) Deringing filter (H.265 MPEG-H HEVC = Sample adaptive ffset) Adaptive Wiener filter (linear filter minimizing distrtin) Thmas Wiegand Digital Image Cmmunicatin 53 / 62

54 In-Lp Filters Deblcking Filter Basic Principle blck P blck Q recnstructed samples befre deblcking p' 1 p 3 p 2 p1 p 0 p' 0 q' 0 q 0 q 1 q 2 q 3 q' 1 p 0 = p mdified samples after deblcking blck bundary q 0 = q 0 0 p 1 q 1 = p 1 + p1 = q 1 + q1 Adaptive smthing f 1D line segments acrss blck bundaries Crrectin values 0, p1, and q1 depend n differences between samples values and cding parameters (QP, mtin, intra/inter) Different filtering mdes selected based n sample differences Strnger lw-pass filtering fr higher QP values (i.e., strnger quantizatin) Thmas Wiegand Digital Image Cmmunicatin 54 / 62

55 In-Lp Filters Deblcking Filter Example withut deblcking filter with deblcking filter Thmas Wiegand Digital Image Cmmunicatin 55 / 62

56 In-Lp Filters Deblcking Filter In-Lp Filter vs Pst Filter pst filter (GOP8 cding structure) in-lp filter (GOP8 cding structure) bit-rate saving [%] average: 1.5% BQTerrace BasketballDrive Kimn Cactus ParkScene bit-rate saving [%] average: 4.2% BasketballDrive Cactus BQTerrace Kimn ParkScene PSNR (Y) [db] PSNR (Y) [db] Cding experiment with H.265 MPEG-H HEVC Nn-nrmative pst filter vs nrmative in-lp filter Cding with hierarchical B pictures = In-lp filter als imprves predictin signal fr fllwing pictures = In-lp filter prvides larger cding gains Thmas Wiegand Digital Image Cmmunicatin 56 / 62

57 In-Lp Filters Sample Adaptive Offset (SAO) in H.265 MPEG-H HEVC sample value riginal signal recnstructed signal class 0 a c b class 1 a c b class 2 a class 3 a c c b b sample psitin First peratin mde: Edge ffset mde Gal: Reduce ringing artifacts arund edges = Select ne f fur edge classes fr a CTU (specifies edge directin) = Classify each sample int ne f five categries = Transmit ffset fr each categry Thmas Wiegand Digital Image Cmmunicatin 57 / 62

58 In-Lp Filters Sample Adaptive Offset (SAO) in H.265 MPEG-H HEVC categry cnditin 0 nne f the fllwing 1 (c < a) (c < b) ((c = a) (c < b)) 2 ((c < a) (c = b)) ((c = a) (c > b)) 3 ((c > a) (c = b)) 4 (c > a) (c > b) value value categry 1 a c b value categry 3 a c b a c b categry 2 a c b a c b value categry 4 a c b Edge ffset mde Each sample is assigned t ne f five categries Categrizatin depends n relatin t neighbring samples in edge directin Transmit psitive ffset fr fur categries = Smth recnstructed signal in gradient directin = Reduce ringing artifacts Thmas Wiegand Digital Image Cmmunicatin 58 / 62

59 In-Lp Filters Sample Adaptive Offset (SAO) in H.265 MPEG-H HEVC sample value band k+3 band k+2 PSNR = 42.1 db sample value Δ 3 = -2 PSNR = 48.4 db Δ 2 = -2 band signals befre crrectin band k+1 riginal signal Δ 1 = 1 riginal signal band k recnstructed signal befre ffset crrectin Δ 0 = 1 recnstructed signal after ffset crrectin sample psitin sample psitin Secnd peratin mde: Band ffset mde Divide range f sample values int 32 equally sized bands Transmit ffset fr fur cnsecutive bands (signal first band index) Selectin f peratin mde n CTU level Three chices: Edge ffset mde, band ffset mde, n filtering Lagrangian mde decisin can be applied Thmas Wiegand Digital Image Cmmunicatin 59 / 62

60 In-Lp Filters SAO Filter Cding Efficiency bit-rate saving [%] IPPP cding structure 30 average: 13.6% 25 Jhnny 20 Vidy4 15 Vidy1 10 Vidy3 5 KristenAndSara 0 FurPeple PSNR (Y) [db] bit-rate saving [%] IBBB cding structure average: 2.3% KristenAndSara 8 FurPeple 6 Vidy Jhnny Vidy1 Vidy PSNR (Y) [db] Cding experiments with H.265 MPEG-H HEVC Tw cnfiguratins: IPPP and IBBB cding = Significant cding gains fr IPPP cding = Smaller cding gains fr IBBB cding = Supprt f bi-predictin already reduces ringing artifacts Thmas Wiegand Digital Image Cmmunicatin 60 / 62

61 In-Lp Filters Adaptive Wiener Filter Signal-adaptive linear filter Apply linear filter t blcks f a picture r t the cmplete picture m n s [x, y] = h k,l s [x + k, y + l]. k= m l= n Encder: Linear filter that minimizes distrtin can be determined by slving linear equatin system Filter cefficients have t be transmitted Applicatin Different variant: Separable r nn-separable filter Nt included in any vide cding standards Cnsidered as candidate fr future standards H.264 MPEG-4 AVC and H.265 MPEG-H HEVC specify SEI message Allws signaling f filter cefficients Can be applied as nn-nrmative pst filter Thmas Wiegand Digital Image Cmmunicatin 61 / 62

62 Summary Summary Accuracy f mtin-cmpensated predictin Translatinal mtin mdel: Simple and effective Mtin vectr accuracy: Quarter-sample accuracy is suitable chice Interplatin filters: Gd apprximatin f ideal interplatr Blck sizes fr mtin-cmpensated predictin Variable blck sizes based n quadtree apprach Supprt f nn-square blcks further imprves cding efficiency Mtin parameter cding Mtin vectr predictin: Switched predictin imprves cding efficiency Additinal mdes with inferred mtin parameters: Merge mde Advanced mtin-cmpensated predictin & cding structures Multiple reference pictures: Blck-adaptive selectin Multi-hypthesis predictin: Typically adaptive bi-predictin Enable mre advanced cding structures: Hierarchical B pictures In-lp filters imprve subjective quality & cding efficiency Thmas Wiegand Digital Image Cmmunicatin 62 / 62