MOTION-GUIDED QUANTIZATION FOR VIDEO TONE MAPPING

Transcription

1 MOTION-GUIDED QUANTIZATION FOR VIDEO TONE MAPPING Ronan Boitard, Dominique Thoreau Tehniolor 975 Av. des Champs Blans Cesson-Sevigne, Frane Rémi Cozot, Kadi Bouatouh IRISA 263 Avenue du Général Leler Rennes, Frane ABSTRACT Tone Mapping Operators (TMOs) transform High Dynami Range (HDR) ontents to address Low Dynami Range (LDR) displays. However, before reahing the end-user, these ontents are usually ompressed using a ode (oderdeoder) for broadasting or storage purposes. Ahieving the best trade-off between rendering and ompression effiieny is of prime importane. Any TMO inludes a rounding quantization to onvert floating point values to integer ones. In this work, we propose to modify this quantization to inrease the ompression effiieny of the tone mapped ontent. By using a motion ompensation, our tehnique preserves the rendering intent of the TMO while maximizing the orrelations between suessive frames. Experimental results show that we an save up to 12% of the total bit-rate as well as an average bit-rate redution of 8.5% for all the test sequenes. We show that our tehnique an be applied to other appliations suh as denoising. Index Terms HDR Video, Video Tone Mapping, Video Compression, HEVC, Adaptive Quantization. 1. INTRODUCTION Tone Mapping Operators (TMOs) transform High Dynami Range (HDR) images or video sequenes to a lower dynami range. Tone mapping still images has been a field of ative researh over the last deade and several satisfying solutions exist [1, 2]. With the reent developments in the HDR video aquisition field [3], more and more HDR video ontents are available. Consequently, video tone mapping has drawn a lot of attention reently [4, 5, 6]. These tehniques usually fous on the subjetive quality of the tone mapped video ontents. However, before reahing the end-user, these ontents need to be ompressed using a ode (oder-deoder) for broadasting or storage purposes. By tuning the tone map urve of a TMO, one an inrease the quality of the deoded ontents [7, 8]. However, the tone map urve is not the only aspet that an influene the ompression effiieny. Indeed, the last operation performed by a TMO onsists in quantizing floating point values to integer ones. We believe that by making this quantization temporally oherent a higher ompression effiieny an be ahieved. To this end, we propose to adapt the quantization to inrease the temporal orrelations between suessive frames. This paper is organized as follows. In setion 2, we provide the neessary bakground to understand the proposed tehnique. Then, we present our method whih adapts the quantization based on a motion estimation/ompensation. In setion 4, we ompare, in terms of ompression effiieny, our method with a rounding-based quantization. Then we show how we extend our approah to handle denoising. Before onluding, we provide a detailed explanation on how our tehnique an be improved to ahieve an even higher ompression effiieny. 2. BACKGROUND We present in this paper a tehnique whih onsists in adapting the quantization of any TMOs to inrease the video ompression effiieny. To this end, we first define what we mean by quantization before presenting tone mapping. Then we briefly introdue some key aspets of the ITU-T H.265 / MPEG-H Part 2 High Effiieny Video Code (HEVC) [9] Quantization In this paper, the term quantization refers to onverting a floating point value to an integer one. When quantizing a value, one has only three hoies: floor ( ), eil ( ) or round ( ). This quantization is different from the adaptive quantization, usually onsidered in imagery, that onsists in optimizing the distribution of the quantization bins in regard to an image s umulative distributive funtion [10]. To the best of our knowledge, no work has been performed on adapting the quantization (the onversion) to an appliation Tone Mapping TMOs onvert HDR ontents to Low Dynami Range (LDR) ones. In HDR imaging, the pixels represent the physial sene

2 Fig. 1. Workflow of the three steps needed to perform a tone mapping operation. luminane (expressed in d/m 2 ) stored as floating point values. In the ase of LDR imaging, the pixels are assigned ode values orresponding to a olor spae standard representation, the most ommon one being the BT.709. Fig. 1 illustrates the HDR to LDR onversion. First, the mapping operation, whih is the ore of a TMO, ompresses HDR values to fit in the range [0-1]. As there is a multitude of TMOs and their desription is not relevant to this paper, we refer the reader to [1, 2] for further details. Seondly, the gamma enoding redistributes the tonal level loser to how our eyes pereive them (usually γ = 1/2.2). Finally, the quantization step onverts floating point values to integer ode values orresponding to the used bit-depth (i.e. [0; 2 n 1] for n bits). This operation onsists in saling the gamma enoded values to the maximum value desired (i.e. 2 n 1) and then rounding them to the nearest integer: I = Q(I u ) = (2 n 1)I u (1) where Q( ) represents the quantization operation, n the used bit-depth and the rounding to the nearest lower integer. I u (respetively I) represents the unquantized (respetively quantized) gamma enoded image. For the sake of larity, saling by a fator of (2 n 1) will be omitted in the quantization equations. The quantization is performed for eah hannel of the image I u separately, regardless of its representation, i.e. RGB, YUV et. When dealing with video sequenes, TMOs apply the desribed operations to eah frame separately. As for video TMOs, they deal with temporal ohereny only by modifying the mapping operation [4] or by post-proessing I u [11] High Effiieny Video Code HEVC is the suessor of the ITU-T H.264 / MPEG-4 Part 10 Advaned Video Coding (AVC) ode. Developed by the Joint Collaborative Team on Video Coding (JCT-VC) group, it was released in January 2013 and is reported to double AVC ompression ratio. The HEVC test Model (HM) is urrently in its version 12. HEVC is a blok-based ode that exploits both spatial and temporal orrelations between the ode values of the pixels to ahieve a high ompression ratio. To exploit these orrelations, bloks are predited using two proesses. Intra-predition relies on spatial orrelation to predit the urrent blok using bloks already deoded in the urrent frame. Inter-predition exploits the temporal orrelation by prediting the urrent blok using bloks from a set of previous/subsequent deoded frames. The predited blok is then subtrated from the original blok leaving only the residuals to be enoded. As this paper fouses on temporal ohereny, a detailed explanation of the inter-predition is given hereafter. To predit the urrent blok, a blok-based motion estimation is performed to find the best temporal predition. It onsists in finding a blok that minimizes the distortion with the urrent blok to be enoded. An example of distortion metris is the Sum of Absolute Differenes (SAD) or the Mean Square Error (MSE). This motion estimation is performed for different blok-sizes (from 64 to 4 organized in a quad-tree in the HM). The seleted motion vetor is the one that minimizes a rate-distortion funtion. The inter-predition results from performing a motion ompensation on a referene frame using the seleted motion vetors. In video ompression, it is generally onsidered that the losest the predition, the more effiient the ompression. In the next setion, we present a tehnique to adapt the quantization at the tone mapping stage so as to improve a ode s inter-predition. 3. MOTION-GUIDED QUANTIZATION (MGQ) 3.1. Motivations Before broadasting it to the end-user, any video ontent needs to be enoded using a ode. Tone mapped HDR video ontents annot skip this proess. By optimizing the tone mapping/ompression ombination, one ould inrease the quality of the rendered ontents while reduing the broadaster s bandwidth use. In setion 2.2, we mentioned that the quantization performed by a TMO onsists in a rounding to the nearest integer (Eq. 1). In the meanwhile, we stated that inreasing the quality of a ode s inter-predition results in higher ompression ratios (beause the resulting residuals are smaller). From the two above observations, we believe that by adapting the quantization to the predition proess of a ode, its ompression effiieny an be inreased. To this end, we propose a quantization method that adapts to any TMO and inreases the orrelation between suessive frames. With this aim in view, the quantization of the gamma enoded urrent frame to be tone mapped (F u, represented by floating point values) should adapt to the previously tone mapped frame (F r whih values have been already quantized). Fig. 2 illustrates the tone mapping of two suessive frames of an HDR video sequene with and without using our tehnique. The following setion details how our tehnique,

3 Fig. 2. Tone mapping two onseutive frames with and without applying the MGQ. The range of eah input/output as well as the type of data is indiated (N = uint, R = float). alled Motion-Guided Quantization (MGQ), adapts the quantization Our Approah Reall that the aim of our tehnique is to inrease the ompression effiieny of a tone mapped ontent by adapting the quantization operation. As a ode greatly relies on the interpredition to remove redundant data, inreasing the quality of this predition should provide a higher ompression effiieny. As mentioned in setion 2.3, the inter-predition relies on a motion estimation/ompensation operation to remove redundant data between frames of a video sequene. That is why we first perform a Motion Estimation (ME) between Fr and Fu to obtain, for eah pixel loation (x, y), a motion vetor (δx, δy). We then ompute the Motion Compensation (MC) whih provides the inter-predited frame Fp : Fp (x, y) = Fr (x + δx, y + δy) (2) To be onsistent with the predition proess used in HEVC, the motion estimation is only performed on the luma hannel and the resulting motion vetors are used for eah hannel of a YUV frame. Our tehnique uses the predited frame Fp to adapt the quantization of the urrent frame Fu : FMGQ = M GQ(Fu ) = bfu dfu e if Fu Fp 0 if Fu Fp < 0 (3) where M GQ( ) represents the Motion-Guided Quantization operation while b (respetively d e) represents the rounding to the nearest lower (respetively higher) integer. Reall that Fu is expressed with floating point values while Fp with integer ones. Both frame s values range from 0 to 2n 1. As in setion 2.2, the M GQ is applied to eah hannel of the frame Fu separately. The workflow of the MGQ tehnique is illustrated in Fig. 3. Our method effiiently inreases the quality of the inter-predition by reduing the distortion between the predited frame Fp and the urrent frame Fu. Fig. 3. Details on the MGQ. AQ stands for Adaptive Quantization (Eq. 3 or Eq. 4). Fr is the tone mapped referene frame, Fu the tone mapped urrent frame before quantization. The MGQ provides the quantized tone mapped frame FMGQ. However, with our tehnique the distortion between Fu and FMCQ is always higher than or equal to the rounding quantization. To tune the trade-off between the quantization distortion and inter-predition effiieny, we add a parameter δ that enables our tehnique to adapt to the differene between Fp and Fu : if 0 Fu bfu Fp < δ MGQ u df e if δ < Fu (4) F = Fp < 0 bfu otherwise To better understand the way this trade-off behaves, let us onsider three ases: δ = 0, δ = 1 and δ =. When δ = 0, the MGQ behaves as a rounding quantization while for δ = it orresponds to Eq. 3. For δ = 1, we define Ω as the set of pixels to whih the MGQ has been applied, say those that satisfy kfp Fu k < δ. After applying the quantization, we obtain FMGQ (Ω) = Fp (Ω) sine the distortion was lower than δ (i.e. 1). Consequently, when prediting FMGQ (Ω) using Fp (Ω) the resulting residuals are equal to 0. All the other pixels are quantized using the rounding operation. Table 1 illustrates different quantizations orresponding to different pixels onditions. Table 2 summarizes the tradeoff between the distortion to the original unquantized values Fu and that of the predited values Fp. To sum up, fixing δ allows the user to balane the number of pixels quantized using the MGQ or the rounding, based on the distortion between the unquantized values and the predition. A higher δ means a higher distortion between FMGQ and Fu as well as a higher quality of the predition Fp, thereby reduing the amount of residuals to enode. 4. RESULTS Our tehnique aims at inreasing the ompression effiieny while adapting to any TMO without altering its intent. In this setion, we show that the distortion obtained with our tehnique and the rounding quantization are very lose. Finally,

4 Table 1. Example of the different quantization tehniques. F u is the unquatized tone mapped frame, F p the predited frame and F MGQ the urrent tone mapped frame quantized using different values of δ (f. Eq. 4). F u F p F u F p , δ = , δ = , δ = Table 2. Sum of distortion resulting from the different quantization of table 1. F u FMGQ F p, δ = , δ = , δ = Table 3. PSNR in db by quantizing with and w/o using our tehnique (59 db orrespond to a MSE of 0.081). Quantization Sun Tunnel Students TunnelHD Rounding MGQ MGQ Inf we report the ompression effiieny of tone mapped ontents with and without using our tehnique Quantization Loss Integer quantization assigns several floating point values to the same integer. This proess obviously results in a loss of information in the quantized signal. We assess the loss due to the quantization by omputing the Peak Signal to Noise Ratio (PSNR) between the unquantized urrent frame F u and the quantized one F. Table 3 reports the PSNR using three different quantizations: Rounding, MGQ with δ = 1 (noted MGQ 1 ) and MGQ with δ = (named MGQ Inf ). As mentioned before, our quantization tehnique provides a slightly more distorted sequene than the rounding quantization. This distortion is no greater than one ode value for all the quantized pixels (ontrary to the rounding quantization that entails a maximum distortion of half a ode value). For omparison, the distortion due to a lossy ode is always greater than or equal to one ode value. Fig. 4. Rate-distortion results for the Sun sequene Compression Effiieny For our experiments on ompression effiieny, we used the HM 12.0 with the Random Aess Main Profile. To assess the ompression effiieny, one usually ompares the PSNR between the input video and its deoded ounterpart. This omparison an be performed in two different ways. First, eah input video is enoded at targeted bit-rates. A diret omparison of the PSNR allows to assess the inreased quality of the ontent for these bit-rates. Fig. 4 and 5 plot the results with and without using the M GQ quantization. The two sequenes used (Sun and Tunnel [12]) are of VGA resolution (640x480) while the targeted bit-rates are 125, 250, 500 and 1000 kbps. We used Ramsey et al. TMO [13] to tone map both HDR sequenes. Results show that we ahieve a higher quality of reonstrution (between 0.15 db and 0.4 db gain) at the deoding stage using the MGQ Inf. We an also notie that the higher the bit-rate, the higher the gain. The ase MGQ 1 provides only a small improvement over the rounding operation. The trade-off between distortion and ompression effiieny is illustrated through Table 3 and Fig. 4 and 5. Note that by tuning the δ parameter, one an shift the MGQ δ urve from the Rounding to the MGQ Inf urve. The seond tehnique omputes the average perentile bit-rate redution under the same PSNR. Table 4 reports the Bjontegaard Distortion rate (BD-rate) [14] for the tested video sequenes. The sequene TunnelHD is of HD resolution (1920x1080) and has been also tone mapped using Ramsey et al. TMO [13]. The Students sequene [6] however is of resolution 1280x720 and has been tone mapped using Farbman et al. TMO [15]. The results show that for the same quality, the MGQ Inf provides an average bit-rate redution of 8.5% for all the test sequenes. Note that the two VGA sequenes perform better than the two other ones. This is due to the fat that these sequenes are relatively noisy and our quantization tehnique redues some of the temporal noise. In the next setion, we show that our tehnique an be extended to other appliations suh as denoising.

5 Table 5. Average PSNR between a denoised version of a tone mapped video sequene and this sequene quantized either with a rounding quantization or the MGQ. Quantization Sun Tunnel Students TunnelHD Rounding MGQ Inf Fig. 5. Rate-distortion results for the Tunnel sequene. Table 4. Average perentile bit-rate redution under the same PSNR when omparing the Rounding and the MGQ Inf quantization tehniques. The BD-rate is omputed using piee-wise ubi interpolation. Sequene Y U V Sun -12.8% -40.1% -40.6% Tunnel -10.4% -31.7% -32.7% Students -5.6% -18.8% -17.5% TunnelHD -5.4% -21.7% -24.5% Average -8.5% -28.1% -28.8% 5. DENOISING APPLICATION As mentioned above, our method performs better for the Sun and Tunnel sequenes beause it redues the temporal noise. When ompression is not the targeted appliations, our method an redue the noise present in a tone mapped video sequenes. Indeed, in the previous setion, the MGQ was guided by the value of the differene between F u F p. Instead of adapting to the inter-predited frame F p, we adapt our quantization to a denoised frame F d. The way F d is omputed is not relevant to this paper and any existing denoising tehniques an be used [16]. For our experiments, we will onsider a simple temporal-filtering with motion ompensation: M F k d (x, y) = l= N F k l (x δx k,k l, y δy k,k l ) w(l) where (δx k,k l, δy k,k l ) is a motion vetor obtained through a motion estimation between frames F k l and F k. N (respetively M) represents the number of non-ausal (respetively ausal) extents of the averaging window and w(l) are the weights or the filter oeffiients. Note that ausal frames are expressed with integer values while non-ausal ones with (5) floating point values (inluding the urrent one whih is in our ase F u ). For our experiments we used only two frames in the filter bank: the previous one F r and the urrent one F u. We tested our method on the same set of sequenes and TMOs as in setion 4.2. To assess the performane of our method when ompared to the rounding quantization, we ompute the PSNR between the quantized frame (either F or ) and the desired denoised frame F d. We report those PSNR in Table 5. For all test sequenes, we ahieve at least 1 db of gain using the MGQ tehnique when ompared to the rounding tehnique. The main advantage of using our tehnique rather than performing a denoising after the tone mapping resides in the fat that our tehnique do not introdue additional artifats to the sequene. Indeed, denoising usually results in a smoothing whih is soure of problems when performed on edges. However, our tehnique fails for really noisy sequenes. In a more generalized manner, we believe that our method an adapt to any appliation where tone mapping is required provided that the right test value is hosen (e.g. F p for ompression and F d for denoising). 6. FUTURE WORK Our method has some limitations, the main one being its omputational omplexity due to the motion estimation. Furthermore, the proposed implementation is sub-optimum with respet to the predition proess performed in HEVC. Beause, first our tehnique does not follow the Group Of Pitures (GOP) hierarhial pattern that is used in a ode. Indeed, it only uses motion ompensation between suessive frames. Seond, the intra-predition proess should also benefit from our quantization. Third, a blok-based ode uses a ratedistortion ost funtion to selet the best preditor for eah blok while our solution only relies on a distortion metris. To sum up, the MGQ tehnique, if implemented in the oding loop instead of being a pre-proessing, should provide an even higher ompression effiieny. It would allow to tune the quantization separately for eah of the available predition mode. The seleted mode and its assoiated quantization would depend on the ode s rate-distortion funtion rather than solely on the distortion. Regarding the trade-off parameter δ, it ould be linked to the rate-distortion fun-

6 tion to ahieve a higher ompression effiieny while reduing the quantization distortion. Furthermore, the omputational omplexity would no longer be an issue as the motion estimation is already performed for the inter-frame predition. However, to implement our approah inside the oding loop of a blok-based enoder (say HEVC), the omputations within the ode should be performed with floating point values rather than integer. Finally, addressing more appliations suh as olor reprodution or traking should be investigated. 7. CONCLUSION In this paper, we pointed out that, when performing tone mapping, rounding quantization is not effiient. We hose a quantization that aimed at improving the ompression effiieny of tone mapped video ontents. Our tehnique relies on the motion ompensation between two suessive frames of a sequene to adapt the quantization during the tone mapping. Results showed an average bit-rate redution under the same PSNR ranging from 5.4% to 12.8%. The proposed method allows a trade-off between ompression effiieny and quantization distortion of the original video. We also applied our tehnique to denoising to show that our method an be generalized to deal with any appliations where tone mapping is needed. Aknowledgment The TunnelHD sequene was produed by Binole and Tehniolor within the framework of the frenh ollaborative projet NEVEx. 8. REFERENCES [1] E. Reinhard, W. Heidrih, P. Debeve, S. Pattanaik, G. Ward, and K. Myszkowski, High Dynami Range Imaging, 2nd Edition: Aquisition, Display, and Image- Based Lighting, [2] F. Banterle, A. Artusi, K. Debattista, and A. Chalmers, Advaned High Dynami Range Imaging: Theory and Pratie, AK Peters (CRC Press), Natik, MA, USA, [3] M. D. Toi, C. Kiser, N. Toi, and P. Sen, A versatile HDR video prodution system, ACM Transations on Graphis, vol. 30, no. 4, pp. 1, [4] R. Boitard, D. Thoreau, K. Bouatouh, and R. Cozot, Temporal Cohereny in Video Tone Mapping, a Survey, in HDRi First International Conferene and SME Workshop on HDR imaging, mapping, in HDRi Seond International Conferene and SME Workshop on HDR imaging, [6] G. Eilertsen, R. Wanat, R. Mantiuk, and J. Unger, Evaluation of Tone Mapping Operators for HDR- Video, Computer Grahpis Forum, vol. 32, no. 7, pp , 2013, Students sequene available at [7] Z. Mai, H. Mansour, R. Mantiuk, P. Nasiopoulos, R. Ward, and W. Heidrih, Optimizing a tone urve for bakward-ompatible high dynami range image and video ompression., IEEE Transations on Image Proessing (TIP), vol. 20, no. 6, pp , [8] R. Boitard, D. Thoreau, R. Cozot, and K. Bouatouh, Impat of Temporal Coherene-Based Tone Mapping on Video Compression, in Proeedings of the 21st European Signal Proessing Conferene (EUSIPCO), Marrakeh, Moroo, [9] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, Overview of the High Effiieny Video Coding (HEVC) Standard, IEEE Transations on Ciruits and Systems for Video Tehnology, vol. 22, no. 12, pp , [10] L. Yang and D. Wu, Adaptive quantization using pieewise ompanding and saling for Gaussian mixture, J. Vis. Comun. Image Represent., vol. 23, no. 7, pp , [11] R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouh, Zonal brightness ohereny for video tone mapping, Signal Proessing: Image Communiation, vol. 29, no. 2, pp , [12] G. Krawzyk,, Sun and Tunnel sequenes available at [13] S. Ramsey, J. J. III, and C. Hansen, Adaptive temporal tone mapping, Computer Graphis and Imaging ,, no. 3, pp. 3 7, [14] G. Bjontegaard, Improvement of the bd-psnr model, ITU-T SG16, VCEG-AI11, [15] Z. Farbman and D. Lishinski, Tonal stabilization of video, ACM Transations on Graphis, vol. 30, no. 4, pp. 1, [16] J. Brailean, R. Kleihorst, S. Efstratiadis, A. Katsaggelos, and R. Lagendijk, Noise redution filters for dynami image sequenes: a review, Proeedings of the IEEE, vol. 83, no. 9, pp , [5] R. Boitard, R. Cozot, D. Thoreau, and K. Bouatouh, Survey of temporal brightness artifats in video tone