no reference video quality assessment metrics for multimedia: state of the art of signal-based

Transcription

1 Effect of transmission performance on Multimedia Quality of Service June Prague, Czec Republic no reference video quality assessment metrics for multimedia: state of the art of signal-based approaches Orange Labs Nicolas Ramin, Ricardo Pastrana-Vidal {nicolas.ramin}{ricardo.pastrana}@orange-ftgroup.com Université de Nantes - France Hakim Saadane

2 contents problem statement signal-based NRQM: state of the art summary/conclusion 2

3 problem statement: multimedia context quality perceived by the end-user communication chain degradations distortions production coding transmission - temp. sampling - temp. sampling - quantization - packet loss - jitter - delay spatial blurriness blockiness ringing temporal jerky motion fluidity break restitution - post. processing - buffer limit? 3

4 problem statement: quality assessment subjective assessment gathers human evaluation advantage most recognized drawback impossible monitoring MOS objective assessment subjective assessment objective quality metric MOS ~ MOS performances study 4

5 problem statement: objective quality metrics approaches: QoS-parameter vs signal? QoS-parameter signal hybrid input network parameters reconstructed signal both disadvantag e link parameters/effects non unequivocal computational cost technology dependant advantage low computational cost close to the signal received by end-user compromise (to be shown) signal-based: needed for accurate quality estimation 5

6 problem statement: objective quality metrics signal-based metrics full reference (FR) S ^S FR metric ~ MOS reduced reference (RR) S ^S features extraction features extraction RR metric ~ MOS no reference (NR) S ^S NR metric ~ MOS greater application context: live services 6

7 contents problem statement signal-based NRQM: state of the art global multimedia NRQM temporal NRQM spatial NRQM summary/conclusion 7

8 signal-based NRQM: global multimedia proprietary solutions comparative study by VQEG: in progress [Caviedes, 2003][Cheng, 2005][Farias,2005][Barland, 2007] estimation of different effects (blocking, ringing, sharpness ) limitation: lack of temporal integration not well adapted in multimedia context VIP [Pastrana, 2007] tackling with fluidity and clearness-sharpness perceptual modeling of the interactions limitation: lack of artifact integration extending sharpness measure 8

9 contents problem statement signal-based NRQM: state of the art global multimedia NRQM temporal NRQM spatial NRQM summary/conclusion 9

10 signal-based NRQM: temporal effects jerky motion freezes fluidity break signal features mean motion abrupt variation freeze duration main contribution image index signal-perceptual approach: [Pastrana, 2006] features signal processing detection primary perception human processing judgment model ~ MOS 10

11 signal-based NRQM: temporal based on psychovision sporadic freeze perception: [Pastrana, 2004a] temporal masking effect: [Pastrana, 2004b] density of freezes: [Pastrana, 2004c] abrupt temporal variation: [Pastrana, 2006] performance [Pastrana, 2006] 11 refs ; several fluidity distributions (covering real impairment profiles) all database: Rp = 0.92 ; Rs = 0.92 all \ videoconference: Rp = 0.95 ; Rs = 0.95 limitations refined content influence: freeze on talking-head lips vs hand motion related work: [Yang, 2007] 11

12 contents problem statement signal-based NRQM: state of the art global multimedia NRQM temporal NRQM spatial NRQM artifact-based blur-based summary/conclusion 12

13 signal-based NRQM: artifact-based blockiness: perception effects blockiness = extended blocks effect + false contouring effect perceptual and cognitive involvement same gradient at borders? false contouring extended blocks contrast masking dynamics dynamics false contouring / real edges interpretation false homogenous / real homogenous region meaning influence meaning influence same impairment? same impairment? low level perception Cognitive 13

14 signal-based NRQM: artifact-based blockiness metrics: principle and diversity false contouring effect measure grid localization border gradient neighbor. gradient masking local contrast Alleviate artifact/ content confusion spatial integration inter ~ effects integration MOS extended blocks measure grid localization [Muijs, 2005] domain - spatial [Wang,2002] masking [Wang,2002] alleviate artifact/content confusion [Gao,2002] inter effects Integration [Wang,2002] - frequency [Wang,2000] local contrast [Babu,2005] extended blocks measure [Wang,2002] 14

15 signal-based NRQM: artifact-based blockiness metrics: review of contributions localization false contouring consideration strength measure domain luminance contrast activity masking confusion artifact/real edges extended block consideration meaning influence integration [Wu, 1997] grid 8x8 spatial attempt attempt no no no [Wang, 2000] grid 8x8 frequency attempt attempt no no no [Gicquel, 2000] grid 8x8 spatial no attempt no no no [Wang, 2002] grid 8x8 spatial no attempt no attempt no [Gao, 2002] grid 8x8 spatial no attempt attempt no no [Liu, 2002] grid 8x8 frequency attempt attempt no no no [Suthaharan, 2003] grid 8x8 spatial attempt attempt no no no [Venkatesh, 2004] grid 8x8 spatial no attempt no no no [Pan, 2004a] grid NxN gradient orientation histogram no attempt no attempt no [Pan, 2004b] grid 8x8 spatial attempt attempt no attempt no [Babu, 2005] grid 8x8 spatial attempt attempt attempt attempt no [Muijs, 2005] grid NxN spatial no attempt no no no [Crete, 2007] grid 8x8 spatial no attempt attempt attempt no 15

16 signal-based NRQM: artifact-based ringing effect ringing metric used in [Cheng, 2005] normalized variance of near by far edge neighborhood low performance alone 21 refs MPEG2: Rp = 0.46 used in [Barland, 2006] limitation: edge detection in contrasted edge neighborhood low performance alone 5 refs - JPEG: Rp=0.37 ; Rs=0.39 confusion between edge neighborhood activity/ringing 16

17 signal-based NRQM: artifact-based summary: low level perception validation: insufficient interest: not always demonstrated high level perception blocking: confusion between false contouring/original edges: insufficient confusion between extended blocks on important/poor area: missing ringing: confusion between edge neighborhood activity/ringing: not validated meaning influence integration: open question contextual limitations post/loop filtering (ex: H264/WM9) 17

18 contents problem statement signal-based NRQM: state of the art global multimedia NRQM temporal NRQM spatial NRQM artifact-based blur-based summary/conclusion 18

19 signal-based NRQM: blur-based effects illustration natural blur signal feature: edge spreading detail loss edge spreading L Blurred edge s Sharp edge 19

20 signal-based NRQM: blur-based approaches edge spread measure [Marziliano, 2002] edge spectral kurtosis [Caviedes, 2004] other contributions perceptual edge detection [Pastrana, 2005] oriented edge spreading [Ong, 2003] noise robustness measure [Ferzli, 2005][Ladjal, 2006][Ferzli, 2007] performances good intra-content correlation [Marziliano, 2004][Caviedes, 2004] bad inter-content correlation [Marziliano, 2004][Caviedes, 2004] 20

21 signal-based NRQM: blur-based evaluation of outliers causes visibility involves cognitive phenomenon biasing blurs stimulus A focal blur stimulus B motion blur shading blur images blur map strong limitations - potential lack of cognitive integration 21

22 contents problem statement signal-based NRQM: state of the art summary/conclusion 22

23 summary automatic quality assessment: signal-based NRQM greater application context needed for accurate quality estimation global multimedia signal-based NRQM: more work needs to be done temporal NRQM ready to use proposition spatial NRQM artifact-based NRQM main limitations: confusion between artifact/original signal meaning influence integration main inconvenience: technology dependant blur-based NRQM advantage: technology independent strong limitation: potential lack of cognitive integration 23

24 conclusions state of signal-based NRQM research temporal: maturity spatial: open issue overall metric for multimedia: open issue open questions alleviating the confusion artifact/original signal? potential lack of cognitive integration? 24

25 references [Babu, 2005] Babu, R. & Perkis, A. (2005),An HVS-Based No-Reference Perceptual Quality Assessment of JPEG Coded Images Using Neural Networks, in ICIP. [Barland, 2006] Barland, R. & Saadane, A. (2006),A reference free quality metric for compressed images, in WVPQM. [Barland, 2007] Barland, R. (2007),'Evaluation objective sans référence de la qualité perçue: applications aux images et vidéos compressées', PhD thesis, Université de Nantes. [Caviedes, 2003] Caviedes, J. & Oberti, F. (2003),No-reference quality metric for degraded and enhanced video, in SPIE VCIP. [Caviedes, 2004] Caviedes, J. & Oberti, F. (2004), 'A New Sharpness Metric Based on Local Kurtosis, Edge and Energy Information', in Signal Processing: Image Communication 19, [Cheng, 2005] Cheng, H. & Lubin, J. (2005),Reference-free objective quality metrics for MPEG-coded video, in SPIE HVEI. [Crete,2007] Crete, F. (2007),'Estimer, mesurer et corriger les artefacts de compression pour la télévision numérique', PhD thesis, Université Joseph Fourier. [Farias, 2005] Farias, M.C.Q. and Mitra, S.K. (2005), No-Reference Video Quality Metric Based on Artifact Measurements, in ICIP [Ferzli, 2005] Ferzli, R. & Karam, L. (2005),No-Reference Objective Wavelet Based Noise Immune Image Sharpness Metric, in ICIP. [Ferzli, 2007] Ferzli, R. & Karam, L. (2007),A No-Reference Objective Image Sharpness Metric Based on Just-Noticeable Blur and Probability Summation, in ICIP, pp [Gao, 2002] Gao, W.; Mermer, C. & Kim, Y. (2002), 'A De-Blocking Algorithm and Blockiness Metric for Highly Compressed Images.', IEEE TCSV 12(12), [Ladjal, 2006] Ladjal, S. (2006), Estimation du flou dans les images naturelles, in RFIA. [Gicquel, 2000] Gicquel, J.; Blin, J. & Wyckens, E. (2000),'Procédé de contrôle de la qualité d'images numériques distribuées par détection de faux contours.', France Télécom, Patent [Liu, 2002] Liu, S. & Bovik, A. (2002), 'Efficient DCT-Domain Blind Measurement and Reduction of Blocking Artifacts', IEEE TCSV 12(12), [Marziliano, 2002] Marziliano, P.; Dufaux, F.; Winkler, S. & Ebrahimi, T. (2002),A no-reference perceptual blur metric, in ICIP, pp [Marziliano, 2004] Marziliano, P. and Dufaux, F. and Winkler, S. and Ebrahimi, T. (2004), Perceptual blur and ringing metrics: Application to JPEG2000, in SPIC 19(2), pp

26 references [Muijs, 2005] Muijs, R. & Kirenko, I. (2005),A No-Reference Blocking Artifact Measure for Adaptive Video Processing, in 'European Signal Processing Conference'. [Ong, 2003] Ong, E.; Lin, W.; Lu, Z.; Yao, S.; Yang, X. & Jiang, L. (2003),No-reference JPEG-2000 image quality metric, in ICME. [Pan, 2004a] Pan, F.; Lin, X.; Rahardja, S.; Ong, E. & Lin, W. (2004),Measuring blocking artifacts using edge direction information [image and video coding], in ICME. [Pan, 2004b] Pan, F.; Lin, X.; Rahardja, S.; Lin, W.; Ong, E.; Yao, S.; Lu, Z. & Yang, X. (2004),A locally adaptive algorithm for measuring blocking artifacts in images and videos., in ISCAS. [Pastrana, 2004a] Pastrana-Vidal, R.; Gicquel, J.; Colomes, C. & Cherifi, H. (2004), Sporadic Frame Dropping Impact on Quality Perception, in SPIE HVEI. [Pastrana, 2004b] Pastrana-Vidal, R.; Gicquel, J.; Colomes, C. & Cherifi, H. (2004),Temporal masking effect on dropped frames at video scene cuts, in SPIE HVEI. [Pastrana, 2004c] Pastrana-Vidal, R.; Gicquel, J.; Colomes, C. & Cherifi, H. (2004),Frame dropping effects on user quality perception, in WIAMIS. [Pastrana, 2005] Pastrana, R. (2005),'Vers une Métrique Perceptuelle de Qualité Audiovisuelle dans un Contexte à Service Non Garanti', PhD thesis, Université de Bourgogne, France Telecom RD. [Pastrana, 2006] Pastrana-Vidal, R. & Gicquel, J. (2006),Automatic quality assessment of video fluidity impairments using a no-reference metric, in WVPQM. [Pastrana, 2007] Pastrana-Vidal, R. R. & Gicquel, J. (2007),A no-reference video quality metric based on a human assessment model, in WVPQM. [Suthaharan, 2003] Suthaharan, S. (2003),A Perceptually Significant Block-Edge Impairment Metric for Digital Video Coding, in ICASSP. [Venkatesh, 2004] Venkatesh Babu, R.; Bopardikar, A.; Perkis, A. & Hillestad, O. (2004),No-Reference Metrics for Video Streaming Applications, in Packet Video Workshop. [Wang, 2000] Wang, Z.; Bovik, A. & Evans, B. (2000),Blind Measurement of Blocking Artifacts in Images., in ICIP. [Wang, 2002] Wang, Z.; Sheikh, H. & Bovik, A. (2002), No Reference Perceptual Quality Assessment of JPEG Compressed Images., in ICIP. [Wu, 1997] Wu, H. R. & Yuen, M. (1997), 'A Generalized Block-Edge Impairment Metric for Video Coding', IEEE Signal Processing Letters 4(11), [Yang, 2007] Yang, K.; El-Maleh, Orange K.; Guest, Labs C. - NRQM C. & Das, for P. multimedia: K. (2007), Perceptual state of the temporal art of quality signal-based metric for compressed approachesvideo, in 'WVPQM'. 26