Lec 12 Rate-Distortion Optimization (RDO) in Video Coding-II

Transcription

1 Sprng 07: Multmeda Communcaton Lec ate-dstorton Optmzaton (DO) n Vdeo Codng-II Zhu L Course Web: Z. L Multmeda Communcaton, Sprng 07 p.

2 Outlne Lec ecap Lagrangan Method HW-3 Lagrangan Optmzed Mode Decson and ate COntrol Mode Decson n Vdeo Codng ate Control n Vdeo Codng Z. L Multmeda Communcaton, Sprng 07 p.

3 Lagrangan elaxaton Method Prmal Problem: Lagrangan: KKT Condton: Statonary: mn x Complementary Slackness: D(x) s. t. x C L x, λ = D x + λ( x C) L(x, λ ) x = 0, λ = D(x ) (x ) Slope s D/ gradent λ D x = 0, λ = 0, x < C Constrant not tght λ > 0, x = C, λ = δd(x )/δ(x ) Constrant tght Z. L Multmeda Communcaton, Sprng 07 p.3

4 Operatonal ate-dstorton Theory Operatonal -D optmzaton: D C D functon OD functon operatng ponts OD convex hull Gves the operatonal -D performance curve. {Q j }: operatng ponts assocated wth codng decsons and parameters Optmzaton: select operatng ponts that mnmzes dstorton for a gven rate, or mnmzng rate for a gven dstorton. Lagrangan method reduced the DoF to mn x k D x k, s. t, x k C mn x k x k, s. t, D x k D max Z. L Multmeda Communcaton, Sprng 07 p.4

5 Dualty Gap Dualty problem solves the prmal problem when dualty gap s zero. Feasble regon: (u, t) n F F :{( u, t) u f( x), t f0( x), x DOM f0} Dual problem: max g( ) 0 Searchng on a set of supportng hyper planes wth slope - Dualty gap: p*- q* max{nf ( t 0 u, tf u)} p* p* mn f0( x), s. t. f( x) 0 xc f 0 (x) F q* f (x) Convex Hull approxmaton typcal n most Lagrangan relaxed -D Optmzaton solutons. Penalty s ths dualty gap * Z. L Multmeda Communcaton, Sprng 07 p.5

6 HW-/3 HW-3: Any questons? Tentatvely due next Monday. evew of Multmeda Communcaton Part I Entropy and Entropy Codng Huffman Codng, Golomb Codng, Arthmetc Codng Transforms, Quantzaton Vdeo Sgnal Processng: Moton, Inter and Intra Predcton ate-dstorton Optmzaton Quz week: Quz - (03/08) o Q. Ture/False o Q. Multple Choces o Q3~Q5. Computng and Program Solvng o Close book but allow one-page cheatng sheet, whch wll be graded. Z. L Multmeda Communcaton, Sprng 07 p.6

7 Outlne Lec ecap Lagrangan Method HW-3 Mode Decson n Vdeo Codng ate Control n Vdeo Codng Z. L Multmeda Communcaton, Sprng 07 p.7

8 Mode Decson n H.63 TMN0 G. Sullvan and T. Wegand, ate-dstorton optmzaton for vdeo compresson, IEEE Sgnal Processng Magazne, Vol. 5, No.6, pp , Nov T. Wegand and B. Grod, Lagrange multpler selecton n hybrd vdeo coder control, Proceedngs of 00 Internatonal Conference on Image Processng, Vol. 3, pp , Oct. 00. Possble MB Codng Modes n H.63: INTA: code the MB as ntra block, no moton estmaton. SKIP: Use the co-located MB n the prevous frame as the reconstructon for the current MB. INTE MV: Use moton vector for the MB, encode the MV and resdual. INTE 4MV: Use 4 moton vectors for the MB. One for each 8x8 block. Quantzaton Parameter (QP) for each MB: QP of prevous MB wth a mnor adjustment: only 5 choces o QP, QP+, QP-, QP+, QP-. Fndng the best QP for each MB s part of the rate control. Z. L Multmeda Communcaton, Sprng 07 p.8

9 Mode Decson n H.63 TMN0 -D optmzed Codng Mode Decson for the k-th MB: Assume QP has been determned by rate control algorthm: Mnmze the Lagrangan codng mode cost: J D EC EC D EC ( MB ( MB ( MB k k k, MODE, MODE, MODE Q) : Q) : Q) MODE EC Dstorton of ( MB k, MODE Q) the k - th MBwth MODE ate for the MBwth MODE and stepq. and step Q. Possble values of MODE: INTA, SKIP, INTE MV, INTE 4MV λmode: Lagrangan multpler D functon OD functon operatng ponts OD convex hull 5 How to determne λ MODE from the Q step? λ MODE reflects the general slope of the -D functon Q provdes operatng range Z. L Multmeda Communcaton, Sprng 07 p.9

10 Mode Decson n H.63 TMN0 mn MODE J, J D EC ( MB k, MODE Q) MODE EC ( MB k, MODE Q) At hgh rate, the relatonshp between EC and DEC can be wrtten as ( D) a ln Larger λmode gves hgher prorty to the reducton of rate, leadng to soluton wth lower rate (and therefore larger dstorton). Smaller λmode gves hgher prorty to the reducton of dstorton, leadng to soluton wth less dstorton, but hgher rate. b D D Pont A: Larger λmode, larger Q step Pont B: Smaller λmode, smaller Q step Z. L Multmeda Communcaton, Sprng 07 p.0

11 Mode Decson n H.63 TMN0 Further nterpretaton of the Lagrangan multpler λ: ( D) a ln b D When J s mnmzed: J D D a ln b D D a dd d (The optmal λ s the neg slope of the curve) Proof: J 0 a D b( ) D 0 or D D b a b a aln / D D D Z. L Multmeda Communcaton, Sprng 07 p.

12 Mode Decson n H.63 TMN0 But the slope s dfferent at dfferent ponts of the curve It s better to represent the slope by Q step. In H.63, λmode s chosen as [Sullvan 98]: MODE Q) ( Q Justfcaton: At hgh rate, the dstorton has unform dstrbuton: Plug nto the (D) formula: ( D) a ln b D D( Q) Q ( Q) b a ln Q dd dd( Q) / dq / 6Q Q d d( Q) / dq a / Q a (can also be obtaned from 0.85 s an emprcal result n H.63. D / a drectly) Z. L Multmeda Communcaton, Sprng 07 p.

13 Mode Decson n H.63 TMN0 Code Mode Cost: Possble optons for MODEk: INTA, SKIP, INTE MV, INTE 4MV INTA mode cost: DEC(MBk, INTA Q): The sum of squared errors (SSE) between the orgnal MB and the reconstructed MB (after DCT, Quant, de-quant, nverse DCT). EC(MBk, INTA Q): Ths ncludes the bts needed to encode the mode decson, and the quantzed DCT coeffcents of the current MB. Entropy codng has to be called slow. Any better rate producton model? Later we wll study rho-doman rate control, can avod entropy codng. SKIP mode cost: DEC(MBk, INTA Q): For SKIP mode, the co-located MB n the prevous frame s used drectly as the reconstructed MB, so the dstorton s the sum of squared errors between the two MBs. The dstorton does not depend on the QP of the current frame. J D EC ( MB, MODE Q) ( MB, MODE Q) k MODE EC(MBk, INTA Q): Only ncludes the bt to sgnal the SKIP mode ( bt n H.63 standard). Z. L Multmeda Communcaton, Sprng 07 p.3 EC k

14 Mode Decson n H.63 TMN0 INTE MV and INTE 4MV cost: Need to fnd the best moton vector frst. Another Lagrangan optmzaton problem for the moton estmaton: Moton Est. Cost: v arg mn v D SAD ( S k, v) MOTION MOTION ( S k, v) S k : The block of nterest (6x6 n INTE MV and 8x8 n INTE 4MV). v: Moton vector canddate n the search wndow. DSAD: The Sum of Absolute Dfference (SAD) between S k and ts reference (as a measure of the moton estmaton effcency) MOTION: The bts to encode the moton vector (does not nclude the codng of the resdual). In H.63, λmotion s chosen heurstcally as: MOTION MODE n H.63. Z. L Multmeda Communcaton, Sprng 07 p.4

15 Mode Decson n H.63 TMN0 v arg mn v D SAD ( S, v) MOTION, v) The search of moton vector that mnmzes the moton estmaton cost above proceeds frst over nteger-pxel locatons. Half pxel moton estmaton s then performed around the optmal nteger moton vector. k MOTION After the MV (or MVs) for the current MB s selected, we can compute the codng mode cost for INTE MV or 4MV: ( S k J D EC ( MB, MODE Q) ( MB, MODE Q) k MODE EC k DEC for INTE modes: econstructon error: Sum of Squared Error between the orgnal MB and the reconstructed MB (usng the best MV found above). EC for INTE modes: Includes the bts to encode the MV, the quantzed DCT coeffcents of the moton estmaton resdual, and the mode decson. Z. L Multmeda Communcaton, Sprng 07 p.5

16 H.63 Model Decson Summary Possble modes: 4 INTA: code the MB as ntra block, no moton estmaton. SKIP: Use the co-located MB n the prevous frame as the reconstructon for the current MB. INTE MV: Use moton vector for the MB, encode the MV and resdual. INTE 4MV: Use 4 moton vectors for the MB. One for each 8x8 block. Model decson objectve functon J D EC ( MB, MODE Q) ( MB, MODE Q) k MODE EC k Lagrangan Estmaton from QP MODE Q) ( Q Z. L Multmeda Communcaton, Sprng 07 p.6

17 Outlne Mode decson: Mode decson n H.63+ TMN0 Mode decson n H.64 ate Control: ate Control n MPEG-4 VM5 ate Control n H.63+ TMN8 and MPEG4 VM8 ρ-doman rate control Z. L Multmeda Communcaton, Sprng 07 p.7

18 Mode Decson n H.64 Based on H.63 TMN0, but more complcated. Summary of Codng Modes n H.64: Intra slces: o INTA 4X4 (9 cases), INTA 6X6 (4 cases). o Intra Chroma mode (4 cases) P-slces: o INTA 4X4 (9 cases), INTA 6X6 (4 cases), o SKIP, 6x6, 6x8, 8x6, Tree 8x8 (4 cases). o For each MB, fnd the best ref frame after ME wrt all ref frames. B-slces: o DIECT mode o INTA 4X4 (9 cases), INTA 6X6 (4 cases), o FWD: 6x6, 6x8, 8x6, Tree 8x8 (4 cases) o BWD: 6x6, 6x8, 8x6, Tree 8x8 (4 cases) o B-drectonal predcton. o For each partton, fnd the best fwd and best bwd ref frames wrt all ref frames, then decde the best predcton drectons: fwd, bwd, or b-dr. Z. L Multmeda Communcaton, Sprng 07 p.8

19 Mode Decson n H.64 General formula: J Dstorton MODEate For INTA mode testng and P slce INTE mode testng: For B slce testng: MODE, B Moton vector decson: When SSE s When SAD s ( QP)/3 MODE, I, P 0.85 max, mn 4, used n ME, Between [, 4] used n ME, QP- 6 MOTION MOTION MODE, I, P MODE, I, P. MODE, I, P Z. L Multmeda Communcaton, Sprng 07 p.9.

20 ate Control for Vdeo Encodng Goal: Maxmze the vdeo qualty under a gven rate constrant. ate control at dfferent levels: GOP level Pcture/Frame level Macro Block (MB)/Codng Tree level Vdeo bt rate may be adapted by: Varyng the quantzaton: most useful Varyng the frame rate Varyng the spatal resoluton Addng/droppng layers (for scalable codng) Z. L Multmeda Communcaton, Sprng 07 p.0

21 ate Control n MPEG4 VM5 T. Chang and Y.-Q. Zhang, A new rate control scheme usng quadratc rate dstorton model, IEEE Trans. Crcuts and Systems for Vdeo Technology, Vol. 7, No., pp , Feb Adopted by MPEG Verfcaton Model (VM) 5.0 n Nov The ate-quantzaton step model: atonale: At hgh rate: aq bq D ( ) ln D Q α: content dependent Taylor expanson: Let x ln( x) D ' / x '' / x. ( D) 0 3 ( D) D D 3 D D 3( D) Z. L Multmeda Communcaton, Sprng 07 p.

22 ate Control n MPEG4 VM5 Use three sets of parameters, {a, b }, {a, b }, and {a 3, b 3 }, for I, P, B frames, respectvely. Use lnear regresson to get a, b : Collect j, Q j : the bts and Q step of each prevously encoded frame n each category... n n n Q b a Q b Q a Q b Q a Q r or, b a A b a Q Q Q Q Q Q n n n r A A A b a T T Drawbacks:. Only good for hgh rate;. Frame-level QP control only. 3. hgh cost n computng Z. L Multmeda Communcaton, Sprng 07 p. [a, b ] wll be the best parameter for ths partcular seq s I frames

23 Outlne Mode decson: Mode decson n H.63+ TMN0 Mode decson n H.64 ate Control: ate Control n MPEG-4 VM5 ate Control n H.63+ TMN8 and MPEG4 VM8 ρ-doman rate control Z. L Multmeda Communcaton, Sprng 07 p.3

24 ate Control n H.63+ TMN8 and MPEG4 VM8 TMN: Test Model Near-term, ef: J. bas-corbera and S. Le, ate control n DCT vdeo codng for low-delay communcatons, IEEE Trans. Crcuts and Systems for Vdeo Technology, Vol. 9, No., pp. 7-85, Feb Advantages: Macro-block level rate control Sutable for low bt rate ate model for low bt rate: Algorthm: B ( Q) A K Q c Q C, : varance of a DCT coeffcent. Assgn QP based on the standard devaton σ of each macroblock: σ Q B : The total bts generated by the -th macro-block. A: 56, # of pxels n each MB C: overhead by moton vectors. K, C can be estmated, content dependent. Z. L Multmeda Communcaton, Sprng 07 p.4

25 ate Control n H.63+ TMN8 and MPEG4 VM8 Dstorton Model: Assumng unform quantzer Average dstorton: D N N Q N: total number of macroblocks. Problem formulaton: argmn Q, Q,... Q N subject to N N B N B. Select Q for each MB such that the total bts for the current frame s B. Z. L Multmeda Communcaton, Sprng 07 p.5 Q

26 ate Control n H.63+ TMN8 and MPEG4 VM8 Use Lagrangan multpler: B C Q K A Q N B B Q N Q J N N N N ) ( Set the dervatve wth respect to Q to be 0: AKN Q Plug n to the bt rate constrant B B N ANC B AKN AK N N k k ANC B AK Q Z. L Multmeda Communcaton, Sprng 07 p.6

27 ate Control n H.63+ TMN8 and MPEG4 VM8 Problems: Does not model the cost of moton vectors accurately. Modern vdeo codng uses deadzone quantzer. The dstorton model s not very accurate ether. Z. L Multmeda Communcaton, Sprng 07 p.7

28 Outlne Mode decson: Mode decson n H.63+ TMN0 Mode decson n H.64 ate Control: ate Control n MPEG-4 VM5 ate Control n H.63+ TMN8 and MPEG4 VM8 ρ-doman rate control o Orgnal algorthm o Generalzaton to H.64 Z. L Multmeda Communcaton, Sprng 07 p.8

29 ρ-doman ate Control Z. He, S.Mtra, "A Unfed ate-dstorton Analyss Framework for Transform Codng," IEEE Transactons on Crcuts and Systems for Vdeo Technology, vol., no., Dec. 00, pp. -36 (00 Best Paper Award) Z. He, S. Mtra, A lnear source model and a unfed rate control algorthm for DCT vdeo codng, IEEE CSVT, (), pp , Nov. 00. Problems of prevous rate-control algorthms: Complcated non-lnear (Q) and D(Q) expressons Large control errors Performance degradaton at scene changes Targeted for a specfc vdeo codng system ρ: The percentage of zero coeffcents after quantzaton. ρ-doman rate control: break up (Q) nto (ρ) and ρ(q). (ρ) s lnear of ρ. ρ(q) can be easly obtaned. More accurate and robust rate control. No need to run entropy codng faster Z. L Multmeda Communcaton, Sprng 07 p.9

30 ρ-doman ate Control ρ: The percentage of zeros n all quantzed coeffcents. There exsts -to- mappng between Q and ρ: Can be obtaned from the dstrbutons of coeffcents. Example: deadzone quantzer -3q0 -q0 -q0 0 q0 q0 3q0 q( X ) 0 q Pr 0 If the hstogram of transform coeffcents, D(x), s known: ( q0) M x q A lookup table can be created to map Q to ρ. 0 q 0 q 0 f ( x) dx D( x), M : total# of coeffs. Z. L Multmeda Communcaton, Sprng 07 p.30

31 ate Producton Model: q-doman vs p-doman quantzer doman rate model: Dffcult to estmate, very content dependent X: quantzer, Y: ate (q) Z. L Multmeda Communcaton, Sprng 07 p.3

32 ate Producton Model: q-doman vs p-doman Tranform doman rate model: Very good lnearty, easy to estmate X: p, Y: ate (p) Z. L Multmeda Communcaton, Sprng 07 p.3

33 ρ-doman Lnear ate Model It has been observed that the bt rate of all typcal transform codng systems, such as JPEG, SPIHT, H.63 and MPEG- can be modeled as a lnear functon of ρ: ( ) ( ) (ρ) Ө s related to source content. How to estmate Ө? The lne passes the pont (,0). Only need another pont n order to estmate Ө. Quantze the mage or moton estmaton resdual usng one QP value, compute the correspondng ρ and (ρ), then ( ) ρ Can also adaptvely estmate Ө MB by MB. Z. L Multmeda Communcaton, Sprng 07 p.33

34 ρ-doman Lnear ate Model Estmaton of ө usng statstcs of prevously encoded MBs: N m : Number of coded MBs n the current frame. m : Number of bts used to encode the prevous Nm MBs. ρ m : Number (not rato) of zeros produced by the prevous Nm MBs. Snce each 6x6 MB has 384 pxels (56 luma, 64 Cr-chroma, 64 Cb-chroma) ө for the current MB s estmated as: m /(384Nm) /(384N Determnng QP for each MB: t : bt budget for the current frame t m : bt budget for the remanng MBs M: total number of MBs per frame Bt rate for the remanng MBs: m m ) 384N t m 384( M Nm) m m Z. L Multmeda Communcaton, Sprng 07 p.34 m

35 ρ-doman Lnear ate Model From ( ) ( ) The fracton of zeros that need to be produced by the remanng MBs s / Gven the desred fracton of zeros, and usng the relatonshp between QP and the fracton of zeros, we can fnd the desred QP for the current MB. Z. L Multmeda Communcaton, Sprng 07 p.35

36 ρ-doman ate Control Wth the adaptve estmaton of ө and the lnear rate mode, the rate control becomes straghtforward. Defne: T: Target bt rate (bts) per frame BT: Encoder buffer sze B0: Number of bts n the buffer α: Target buffer level percentage. αbt: Target buffer level n bts. The avalable bts for codng the current frame can be found by: B 0 T B T T B0 B T Z. L Multmeda Communcaton, Sprng 07 p.36

37 ρ-doman ate Control for H.64 Z. He and D. O. Wu, Lnear rate control and optmum statstcal multplexng for H.64 vdeo broadcast, IEEE Trans. Multmeda, 0(7), 37-49, Nov H.64 poses some dffcultes for rho-doman rate control:. H.64 has sgnfcant amount of overhead bts for moton vectors, MB modes, pcture headers, QP parameters Can be as hgh as 50% of all bt streams The fracton of overhead bts changes dramatcally from frame to frame. The ntra-predcton n H.64 creates recursve dependency between dfferent MBs. Orgnal rho-doman rate control assumes ndependent MB codng. Therefore the frame-level statstcs of coeffcent dstrbutons can be collected before the rate control. It s found n the paper above that n H.64, the overhead bts are also lnear functon of rho. Therefore the lnear source model s stll vald. Z. L Multmeda Communcaton, Sprng 07 p.37

38 ρ-doman ate Control for H.64 To collect the frame-level statstcs requred by the rho-doman rate control, a two-loop method s proposed. In the frst loop, moton compensaton of all MB modes and ntra-predcton are performed for all MBs. However, the ntra-predcton uses orgnal neghborng pxels nstead of reconstructed pxels. The purpose s to break up the dependency between the codng of neghborng MBs. Block transform s then appled to the moton compensaton resdual and ntra predcton resdual, and the frame-level statstcs s collected. The second loop uses the standard H.64 encodng for each MB: The ntra predcton s checked agan usng reconstructed neghborng pxels. If codng mode of the MB s selected to be ntra, then the block transform s appled agan. The added complexty s the ntra-predcton and the correspondng block transform. Note: H.64 apples dfferent deadzone quantzers for ntra and nter MBs, so we need to mantan dfferent statstcs for ntra and nter MBs. Z. L Multmeda Communcaton, Sprng 07 p.38

39 ρ-doman ate Control for H.64 Step : Collectng frame-level statstcs usng the smplfed method. Step : Intalzaton Nm = m = ρm = 0. Set ө to be ts value n the prevous frame of the same type. Step 3: t m Use the followng to fnd : Use the map between QP and ρ to fnd the QP for the current MB. Step 4: Update Fnd number of zeros (ρ0) and number of bts (0) produced by the current MB. Set Update ө by emove the frequences of the DCT coeffcents of the current MB from the statstcs. Step 5: Loop. epeat Step 3 and 4 for all MBs. 384( M Nm) m m 0 m m 0 /( 384 m m N m / N m N m ) Z. L Multmeda Communcaton, Sprng 07 p.39

40 λ-doman rate control for HEVC λ s the key factor to determne both the btrate and dstorton λ s the slope of the -D curve and thus there exsts a one-to-one correspondence between D and λ λ can both determne the resdue and non-resdue bts, whle QP s unable to determne the non-resdue bts B. L, H. L, L L, J.Zhang: λ Doman ate Control Algorthm for Hgh Effcency Vdeo Codng. IEEE Trans. on Image Processng 3(9): (04) Z. L Multmeda Communcaton, Sprng 07 p.40

41 λ-doman rate control algorthm Bt allocaton Pcture level bt allocaton o The λ rato of dfferent pctures should be nversely proportonal to ts nfluence to the whole sequence Block level bt allocaton o The λ of dfferent BUs should be equal to acheve the best -D performance Achevng of the target btrate D C K D CK K Z. L Multmeda Communcaton, Sprng 07 p.4

42 Expermental esults How well the rates stck to target Z. L Multmeda Communcaton, Sprng 07 4

43 Summary Mode Decson QP s gven by outer loop of control (ate Control) Lagrangan relaxed formulaton of the model objectve functon Lagrangan multpler reflects the operatng range on the -D curve ate-control Accurate ate Producton model? Lnear model va rho doman work Dstorton model: quantzer related Herarchcal allocaton: GoP level, Frame Level, and then Block level, lke a long term short term hgh frequency tradng company. Next Class: evew of the Part I: Vdeo Codng. Z. L Multmeda Communcaton, Sprng 07 p.43