Lec 13 Review of Video Coding

Size: px

Start display at page:

Download "Lec 13 Review of Video Coding"

Evelyn Fowler
6 years ago
Views:

1 Sprng 017 Multmeda Communcaton Lec 13 Revew of Vdeo Codng Zhu L Course Web: Z. L Multmeda Communcaton, Sprng 017 p.1

2 Outlne About the Quz Monday, 03/09, 309 Format: Close book, no cell phone, no computer, calculator allowed, One page summary sheet encouraged and wll be graded as extra credt. Revew of Part I: Vdeo Codng Info Theory & Entropy Huffman and Golomb Codng Arthmetc Codng Transforms Quantzaton Vdeo Sgnal Processng Vdeo Codng Systems Rate-Dstorton Optmzaton Z. L Multmeda Communcaton, Sprng 017 p.

3 Entropy, Condtonal Entropy, Mutual Info Self Info of an event X = x k = log Pr X = x k = log(p k ) Entropy of a source H X = k p k log( 1 p k ) Condtonal Entropy, Mutual Informaton H X 1 X = H X 1, X H(X ) I X 1, X = H X 1 + H X H X 1, X Relatve Entropy Total area: H(X, Y) Man applcaton: Context Modelng a b c b c a b c b a b c b a D(p q = k p k log p k q k H(X Y) I(X; Y) H(Y X) H(X) H(Y) Z. L Multmeda Communcaton, Sprng 017 p.3

4 Lossless Prefx Codng Prefx Codng Codes on leaves No code s prefx of other codes Smple encodng/decodng Root node 0 1 Internal node leaf node Kraft- McMllan Inequalty: For a codng scheme wth code length: l 1, l, l n, k l k 1 Gven a set of nteger length {l 1, l, l n } that satsfy above nequalty, we can always fnd a prefx code wth code length l 1, l, l n Proof: by countng the number of offsprng leaf nodes Z. L Multmeda Communcaton, Sprng 017 p.4

5 Hoffmann Codng Hoffman Codng: teratve sort and merge to create a bnary tree, assgnng the bt value along the way, reverse for code Example Source alphabet A = {a 1, a, a 3, a 4, a 5 }, Probablty dstrbuton: {0., 0.4, 0., 0.1, 0.1} Sort merge Sort merge Sort merge Sort merge 1 a (0.4) a1(0.) a3(0.) a4(0.1) a5(0.1) Assgn code Z. L Multmeda Communcaton, Sprng 017 p.5

6 Golomb Code q: Quotent, n used unary code q Codeword qm r n m m r r: remander, fxed-length code K bts f m = ^k m=8: 000, 001,, 111 If m ^k: (not desred) log m log m bts for smaller r bts for larger r m = 5: 00, 01, 10, 110, 111 Z. L Multmeda Communcaton, Sprng 017 p.6

JPEG Transform Codng Block (8x8 pel) based

Level Category Fxed on the Level wth n the

7 JPEG Transform Codng Block (8x8 pel) based codng DCT transform to fnd sparse representaton Quantzaton reflects human vsual system Zg-Zag scan to convert D to 1D strng Run-Level pars to have even more compact representaton Hoffman Codng on Level Category Fxed on the Level wth n the category = * Quant Table: Z. L Multmeda Communcaton, Sprng 017 p.7

Codng of AC Coeffcents Zgzag scannng: Example 8 4-0 0 0 0 0-31 -4 6-1 0 0 0 0 0-1 -1 0 0 0 0 0 0 - -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Example: zgzag scannng

8 Codng of AC Coeffcents Zgzag scannng: Example Example: zgzag scannng result EOB (Run, level) representaton: (0, 4), (0, -31), (1, -4), (0, -), (1, 6), (0, -1), (3, -1), (0, -1), (3, ), (0, -), (5, -1), EOB Z. L Multmeda Communcaton, Sprng 017 p.8

9 Encodng: CDF: F x ()=[ ] T(X)=( )/=0.553 P(X)= 0.014; l(x)=6 bts Code: (0.553) = Essental of Arthmetc Codng LOW=0.0, HIGH=1.0; whle (not EOF) { n = ReadSymbol(); RANGE = HIGH - LOW; HIGH = LOW + RANGE * CDF(n); LOW = LOW + RANGE * CDF(n-1); } output LOW; T(X) Z. L Multmeda Communcaton, Sprng 017 p.9

10 Adaptve Bn Arthmetc Codng Bnary sequence: Intal counters for 0 s and 1 s: C(0)=C(1)=1. P(0)=P(1)=0.5 After encodng 0: C(0)=, C(1)= P(0)=/3, P(1)=1/ After encodng 01: C(0)=, C(1)=. P(0)=1/, P(1)=1/ After encodng 011: C(0)=, C(1)=3. P(0)=/5, P(1)=3/5 After encodng 0111: C(0)=, C(1)= P(0)=1/3, P(1)=/3. Encode Z. L Multmeda Communcaton, Sprng 017 p.10

11 Adaptve Bn Arthmetc Decodng Input Intal counters for 0 s and 1 s: C(0)=C(1)=1 P(0)=P(1)=0.5 Decode 0 After decodng 0: C(0)=, C(1)=1. P(0)=/3, P(1)=1/3 Decode 1 After decodng 01: C(0)=, C(1)=. P(0)=1/, P(1)=1/ Decode 1 After decodng 011: C(0)=, C(1)=3. P(0)=/5, P(1)=3/5 Decode 1 After decodng 0111: C(0)=, C(1)=4. P(0)=1/3, P(1)=/3. Decode 1 Z. L Multmeda Communcaton, Sprng 017 p.11

12 Untary Transforms y=ax, x,y n R d, A: dxd y = A= y = x [a 1T, a T,, a T d ] d k=1 a k T x Inner product<x, a k > Untary Transforms: A s untary f: A -1 =A T, AA T = I d The bass of A s orthogonal to each other Examples: < aj, ak > = 0 < a k, a k > = cos sn sn cos Z. L Multmeda Communcaton, Sprng 017 p.1

13 Untary Transform Propertes Preserve Energy: y = Ax A T A = I x = y y = Ax = Ax T Ax = x T A T Ax = x T Ix = x Preserve Angles: the angles between vectors are preserved untary transform: rotate a vector n R n,.e., rotate the bass coordnates DoF of Untary Transforms k-dmenson projectons n d-dmensonal space: kd k. Above example: 3x-x = ; normal ponts to the unt sphere n Z. L Multmeda Communcaton, Sprng 017 p.13

Energy Compacton and De-correlaton Energy

$to pack a large fracton of sgnal energy nto$ De-correlaton Hghly correlated nput elements

Covarance matrx R yy = Cov y = E{ y E y y

there an optmal transform that do best n ths?

dsplay scale: g x: columns of mage pxels

14 Energy Compacton and De-correlaton Energy Compacton Many common untary transforms tend to pack a large fracton of sgnal energy nto just a few transform coeffcents De-correlaton Hghly correlated nput elements qute uncorrelated output coeffcents Covarance matrx R yy = Cov y = E{ y E y y E{y} T } DCT Example: y=dct(x), Queston: Is there an optmal transform that do best n ths? x 1,x,, x 600 R xx y 1,y,, y 600 R yy lnear dsplay scale: g x: columns of mage pxels dsplay scale: log(1+abs(g)) Z. L Multmeda Communcaton, Sprng 017 p.14

15 Karhunen-Loève Transform (KLT)/PCA a untary transform wth the bass vectors n A beng the orthonormalzed egenvectors of R xx y = Ax, x = A T y A T = [a 1, a,, a d ] R xx a k = λ k a k, k = 1,,, d assume real nput, wrte A T nstead of A H denote the nverse transform matrx as A, AA T =I R x s symmetrc for real nput, Hermtan for complex nput.e. R xt =R x, R xh = R x R x nonnegatve defnte,.e. has real non-negatve egen values Attrbutons Kar Karhunen 1947, Mchel Loève 1948 a.k.a Hotellng transform (Harold Hotellng, dscrete formulaton 1933) a.k.a. Prncple Component Analyss (PCA, estmate R x from samples) Z. L Multmeda Communcaton, Sprng 017 p.15

16 Propertes of K-L Transform Decorrelaton by constructon: R y = E{yy T } = E Axx T A T = AR xx A T = λ 1 λ λ 3 a j T R xx a k = 0, f j! = k λ k, f j = k Mnmzng Error under lmted coeffcents reconstructon Bass restrcton: Keep only a subset of m transform coeffcents and then perform nverse transform (1 m N) Keep the coeffcents w.r.t. the egenvectors of the frst m largest egenvalues (ndcaton of energy) Z. L Multmeda Communcaton, Sprng 017 p.16

17 Scalar Quantzaton Unform Quantzer & Dstortons Dstorton Metrcs: MSE Unform Q MSE: d xˆ x f ( x) dx y x 0 M b 1 b 1 f ( x) dx d M x dx M / Z. L Multmeda Communcaton, Sprng 017 p.17

18 Scalar Quantzaton Non-Unform Scalar Quantzaton Intuton: denser samplng at hgher pdf regon Formulaton: mnmze d() over {b k, y k }, take Lagrangan and va KKT condton: Z. L Multmeda Communcaton, Sprng 017 p.18 x f(x) M k b b k k k k k dx x f x y dx x f x x y b d 1 1 ) ( ) ( ˆ }), ({ 0 ) ( ) ( y y b b f b y b f b y b d b b b b dx x f dx x f x I X X E y y d 1 1 ) ( ) ( 0 y s the centrod!

19 Vector Quantzer A more optmal soluton, better approx. R-D nfo theoretcal boundary kmeans() % desred rate R=8; [ndx, vq_codebook]=kmeans(x, ^R); kd-tree mplementaton [kdt.ndx, kdt.leafs, kdt.mbox]=buldvsualwordlst(x, ^R); [node, prefx_code]=searchvsualwordlst(q, kdt.ndx, kdt.leafs); Z. L Multmeda Communcaton, Sprng 017 p.19

20 Quantzaton Error Metrcs: MSE, SNR Varance of a random varable unformly dstrbuted n [- L/, L/]: Let M = R, each bn ndex can be represented by R bts. SNR( db) 10log 10log R db X X max x 10 max / M Sgnal Energy Nose Energy L/ L/ x 0 dx L 10log 1 L 10 M 10log log 1/1 X 1/1 10 R max (0 log 10 ) R PSNR = 10 log 10 x max nose energy = 10 log MSE Z. L Multmeda Communcaton, Sprng 017 p.0

21 Vdeo Sgnal Processng: Color Space Samplng RGB components of an mage have strong correlaton. Can be converted to YUV space for better compresson. HVS s more senstve to the detals of brghtness than color. Can down-sample color components to mprove compresson. Luma sample Chroma sample YUV 4:4:4 No downsamplng Of Chroma MPEG-1 MPEG- YUV 4:: YUV 4::0 :1 horzontal downsamplng :1 horzontal downsamplng of chroma components of chroma components chroma samples for 1 chroma sample for every every 4 luma samples 4 luma samples Z. L Multmeda Communcaton, Sprng 017 p.1

22 Vdeo Sgnal Processng: Moton Estmaton Implemented as nt pel resoluton exhaustve search n moton_estmaton.m Z. L Multmeda Communcaton, Sprng 017 p.

23 Vdeo Sgnal Codng Structure: I, P, and B Frames I P P P P P P GoP I B B P B B P I frames (Key frames): GoP Intra-coded frame, coded as a stll mage. Can be decoded drectly. Used at GOP head, or at scene changes. Allow random access, mproves error reslence. P frames: (Inter-coded frames) Predcated from the prevous frame. B frames: B-drectonal nterpolated predcton frames Predcted from both the prevous frame and the next frame: more flexbltes better predcton. Useful when new objects come nto the scene. Z. L Multmeda Communcaton, Sprng 017 p.3

24 Sub-pxel Moton Estmaton Sx-tap flter for half-pxel samples: h = [ (E 5F + 0G + 0H 5 I + J) + 16 ] / 3 Smlar operaton for v n vertcal drecton. x = [ (v1 5 v + 0 v3 + 0 v4 5 v5 + v6) + 16 ] / 3 E F G h H I J v1 v v3 x v4 v5 v6 B-lnear flter for 1/4-pxel samples: Matlab: blnearinterpoloaton.m Z. L Multmeda Communcaton, Sprng 017 p.4

25 Fast Search n Moton Estmaton Damond Patter Search Ref: [11] Renxang L, Bng Zeng, Mng L. Lou, A new three-step search algorthm for block moton estmaton, IEEE Trans. Crcuts Syst. Vdeo Tech vol.4(4): (1994). [top 10 cted T-CSVT paper] [1] S. Zhu, K.-K. Ma, A new damond search algorthm for fast block-matchng moton estmaton, IEEE Transactons on Image Processng vol.9(): (000). Z. L Multmeda Communcaton, Sprng 017 p.5

Bossen, Woojn Han, Junghye Mn, Kemal Ugur, Intra Lke a sparse transform bass!

26 Vdeo Sgnal Processng: Intra Predcton Much more modes DC mode: copy DC values from neghbor Planar mode: top row or left col average Angular: pxels on certan lne Ref: Jan Lanema, Frank Bossen, Woojn Han, Junghye Mn, Kemal Ugur, Intra Lke a sparse transform bass! Codng of the HEVC Standard. IEEE Trans. Crcuts Syst. Vdeo Tech. (1): (01) Z. L Multmeda Communcaton, Sprng 017 p.6

In H.64 (and H.63v), ths s used n the predcton loop to mprove moton estmaton accuracy. Decoder needs to do the same.

27 Vdeo Sgnal Processng: Deblockng Flter Reduce blockng artfact n the reconstructed frames Can mprove both subjectve and objectve qualty Flter n H.61: [1/4, 1/, 1/4]: Appled to non-block-boundary pxels n each block. A low-pass smoothng flter. In H.64 (and H.63v), ths s used n the predcton loop to mprove moton estmaton accuracy. Decoder needs to do the same. Also called loop flter. H.64: 4x4 block level Before. and After H.65: 8x4 block level Z. L Multmeda Communcaton, Sprng 017 p.7

28 Vdeo Sgnal Processng: Sample Adaptve Offset (SAO) Classfy pxels on block edge as one of the four categores Offset ts pxel value accordngly Z. L Multmeda Communcaton, Sprng 017 p.8

29 HEVC Codng Structure Quad Tree Decomposton: Slde Credt: Vvenne Sze & Madhukar Budagav, ISCAS 014 Tutoral Ref: G. Schuster, PhD Thess, 1996: Optmal Allocaton of Bts Among Moton, Segmentaton and Resdual Z. L Multmeda Communcaton, Sprng 017 p.9

30 HEVC Codng Tools HEVC (H.65) vs AVC (H.64) Credt: Vvenne Sze & Madhukar Budagav, ISCAS 014 Tutoral Z. L Multmeda Communcaton, Sprng 017 p.30

31 Predcton Unt (PU) PU: Basc Unt for Transform & Quantzaton Z. L Multmeda Communcaton, Sprng 017 p.31

32 Transform Unt (TU) TU Square blocks for transform & quantzaton/codng, sze: 4x4, 8x8, 16x16, 3x3 MaxTU sze: 3 for luma, 16 for chroma Mn TU sze: 4x4 for both luma and chroma TU sze can be larger than PU for nter PU case (MV merge) Z. L Multmeda Communcaton, Sprng 017 p.3

33 Lagrangan Relaxaton Method Prmal Problem: Lagrangan: KKT Condton: Statonary: mn x Complementary Slackness: D(x) s. t. R x C L x, λ = D x + λ(r x C) L(x, λ ) x = 0, λ = D(x ) R(x ) Slope s D/R gradent λ D x = 0, λ = 0, R x < C Constrant not tght λ > 0, R x = C, λ = δd(x )/δr(x ) Constrant tght Z. L Multmeda Communcaton, Sprng 017 p.33

34 Operatonal Rate-Dstorton Theory Operatonal R-D optmzaton: D C RD functon ORD functon operatng ponts ORD convex hull Gves the operatonal R-D performance curve. {Q j }: operatng ponts assocated wth codng decsons and parameters Optmzaton: select operatng ponts that mnmzes dstorton for a gven rate, or mnmzng rate for a gven dstorton. Lagrangan method reduced the DoF to 1 mn x k D x k, s. t, R x k C mn x k R x k, s. t, D x k D max R Z. L Multmeda Communcaton, Sprng 017 p.34

35 Codng Model Decson H.63 Example Possble modes: 4 INTRA: code the MB as ntra block, no moton estmaton. SKIP: Use the co-located MB n the prevous frame as the reconstructon for the current MB. INTER 1MV: Use 1 moton vector for the MB, encode the MV and resdual. INTER 4MV: Use 4 moton vectors for the MB. One for each 8x8 block. Model decson objectve functon J D REC ( MB, MODE Q) R ( MB, MODE Q) k MODE REC k Lagrangan Estmaton from QP Rate Model R REC () Dstorton Model D REC () MODE Q) ( Q Z. L Multmeda Communcaton, Sprng 017 p.35

36 Dstorton Model Unform Quantzer Model: D( Q) 1 N N Q Dstorton from Encodng: MSE n mse = 1 n k x k x k PSNR psnr = 10log 10 x max mse Z. L Multmeda Communcaton, Sprng 017 p.36

37 ρ-doman Lnear Rate Model It has been observed that the bt rate of all typcal transform codng systems, such as JPEG, SPIHT, H.63 and MPEG- can be modeled as a lnear functon of ρ: R( ) (1 ) Ө s related to source content. R(ρ) 1 ρ Z. L Multmeda Communcaton, Sprng 017 p.37

38 Summary Entropy Measure of uncertanty of a source Condtonal reduces entropy foundaton for context modelng n lossless codng (HW-, Q.c) Golomb and Arthmetc Codng ExpGolomb Codng s a good approx. of Geometrc Dstrbuton source, whch s usually true for resdual error ExpGolomb s lke a bnarzaton soluton, x R d [ ] Arthmetc Codng wth Prob Adaptaton: Update the P.M.F on the fly Context Aware Arthmetc Codng: Condtonal Entropy Codng Transform & Quantzaton Transform de-correlate the source, becomes a collecton of scalar sgnals Quantzaton tes to the human vsual system Optmal energy compacton/de-correlatng transform: KLT Z. L Multmeda Communcaton, Sprng 017 p.38

39 Summary Vdeo Sgnal Processng Moton Estmaton: Block Based, Sub-Pel Accuracy ME Fast Moton Estmaton Intra Predcton: specal dctonary lke codng De-Blockng: remove Codng block boundary effects SAO: bas removal, not really flterng Vdeo Codng Systems HEVC: CTU, CU, PU, TU, how they together provdes codng effcency? R-D Optmzaton: how to avod exhaustve search of all operatng ponts? Lagrangan method: Convex-Hull Optmal Soluton Rate Model Lnear Transform Doman Stats Based Dstorton Model and Metrc Lagrangan Estmaton. Z. L Multmeda Communcaton, Sprng 017 p.39

40 Part II: Vdeo Streamng and Networkng Tentatve Topcs: QoE Metrcs: Referenced, Lght Reference, and Reference-less QoE metrcs Vdeo over Multple Access Networks : Resource Prcng Soluton, DP+Lagrangan Framework MPEG Systems: Fle Format (MP4Box), Streamng Soluton (DASH.js), MMT Meda Transport: RTP/RTSP, HTTP/WebSocket, WebRTC, and QUIC Congeston Measure and Modelng n Meda Networkng Error Correcton Codng PP Systems Content Identfcaton and Info Centrc Networkng Z. L Multmeda Communcaton, Sprng 017 p.40

Lec 13 Review of Video Coding

Lec 13 Review of Video Coding CS/EE 5590 / ENG 401 Specal Topcs (17804, 17815, 17803) Lec 13 Revew of Vdeo Codng Zhu L Course Web: http://l.web.umkc.edu/lzhu/teachng/016sp.vdeo-communcaton/man.html Z. L Multmeda Communcaton, 016 Sprng