Lecture 2: Introduction to Audio, Video & Image Coding Techniques (I) -- Fundaments

Transcription

1 Lecture 2: Introduction to Audio, Video & Image Coding Techniques (I) -- Fundaments Dr. Jian Zhang Conjoint Associate Professor NICTA & CSE UNSW COMP9519 Multimedia Systems S

2 Acknowledgement and References for lectures 1 to 5 Special thanks should go to Prof. Henry Wu (Prof. & Director of School of ECE,RMIT) for his great support to help me draft lecture slides Special thanks should go to Prof. John Arnold and A/Prof. Michael Frater (ADFA@UNSW) for their great support during my PhD study -- video coding and communication. Reference Books: M. Ghanbari, Video coding: an introduction to standard codecs, Barry G. Haskell, Digital video: an introduction to MPEG -2, Yun Q. Shi, Image and video compression for multimedia engineering, F. Pereira, The MPEG-4 book, 2002 COMP9519 Multimedia Systems Lecture 2 Slide 2 J Zhang

3 Tutorial 1

4 1.5.3 Pixel Representation Tutorial 1 Y C C Colour Space d b r For digital component signal (CCIR Rec 601), 8-bit digital variables are used, however: 1. Full digital range is not used to give working margins for coding and filtering. 2. RGB to Yd CbCr conversion is given by Yd Rd 16 C b G d 128 = + C r B d 128 Rd Yd 16 G d Cb 128 = B d Cr 128 The positive/negative values of U and V are scaled and zero shifted in a transformation to the Cb and Cr coordinates. where digital luminance, Yd, has a rang of (16-235) with 220 levels starting at 16, and digital chrominance difference signals, Cb and Cr, have a range of (16-240) with 225 levels centered at 128. COMP9519 Multimedia Systems Lecture 2 Slide 4 J Zhang

5 1.5.4 Chrominance sub-sampling Tutorial 1 Human vision is relatively insensitive to chrominance. For this reason, chrominance is often sub-sampled. Chrominance sub-sampling is specified as a three-element ratio. COMP9519 Multimedia Systems Lecture 2 Slide 5 J Zhang

6 1.5.4 Chrominance sub-sampling Tutorial 1 In 4:4:4 format: Y, Cr & Cb 720 x 576 pixels per frame In 4:2:2 format: Y 720 x 576 and Cr & Cb 360 x 576 pixels per frame In 4:2:0 format: Y 720 x 576 and Cr & Cb 360 x 288 pixels per frame A commonly used format is 4:2:0 which is obtained by sub-sampling each colour component of 4:2:2 source vertically to reduce the number of lines to 288; COMP9519 Multimedia Systems Lecture 2 Slide 6 J Zhang

7 1.5.5 Digital Video Formats Tutorial 1 Common Intermediate Format (CIF): This format was defined by CCITT (TSS) for H.261 coding standard (teleconferencing and videophone). Several size formats: SQCIF: 88x72 pixels. QCIF: 176x144 pixels. CIF: 352x288 pixels. 4CIF: 704x576 pixels. Non-interlaced (progressive), and chrominance sub-sampling using 4:2:0. Frame rates up to 25 frames/sec COMP9519 Multimedia Systems Lecture 2 Slide 7 J Zhang

8 2. Introduction to audio, video & image coding techniques (I) 2.1. Spatial Redundancy in Images 2.2. Lossless & Differential Coding (Entropy coding) 2.3 Introduction to Image Quantization 2.4 Summary COMP9519 Multimedia Systems Lecture 2 Slide 8 J Zhang

9 2.1.1 Spatial Statistical Redundancy Spatial redundancy is existed among pixels within a single frame of image. Especially, neighboring pixels are highly correlated. Ref: H. Wu Lena image COMP9519 Multimedia Systems Lecture 2 Slide 9 J Zhang

10 2.1.2 Information Measurement -- Review Information Measure Consider a symbol x with an occurrence probability p, its info. content (i.e. the amount of info contained in the symbol) I = i( x) = log[ ] = log p( x) bits p( x) The smaller the probability, the more info. the symbol contains The occurrence probability somewhat related to the uncertainty of the symbol A small occurrence probability means large uncertainty or the info. Content of a symbol is about the uncertainty of the symbol. Average Information per Symbol Consider a discrete memoriless information source By discreteness, the source is a countable set of symbols By memoriless, the occurrence of a symbol in the set is independent of that of its preceding symbol. COMP9519 Multimedia Systems Lecture 2 Slide 10 J Zhang

11 2.1.2 Information Measurement -- Review Look at a source that contains m possible symbols: {si, i=1,2..m} The occurrence probabilities: {Pi, i=1,2..m} The info. content of a symbol si; = i( s) = log p bits Information Entropy Ii i 2 The Entropy is defined as the average information content per symbol of the source. The Entropy, H, can be expressed as follows: H = m i = 1 p i log 2 p i bits From this definition, the entropy of an information source is a function of occurrence probabilities. The entropy reaches the Max. when all symbols in the set are equally probable. i COMP9519 Multimedia Systems Lecture 2 Slide 11 J Zhang

12 2.1.2 Information Measurement -- Review Information Content Consider the two blocks of binary data shown below which contains the most information? COMP9519 Multimedia Systems Lecture 2 Slide 12 J Zhang

13 2.1.2 Information Measurement -- Review Definition-- Image mean Given a two-dimensional (2-D) image field with pixel value, x[n.m], n=1,2,,n and m=1,2,,m, the mean of the image is defined as the spatial average of the luminance values of all pixel, i.e., x = 1 x[ n, m] (2-5) Definition--Image variance Given a two-dimensional (2-D) image field with pixel value, x[n.m], n=1,2,,n and m=1,2,,m, the variance of the image is defined as the average value of the squared difference between the value of an arbitrary pixel and the image mean, i.e., N M σ = ( x[ n, m] x) N M (2-6) N M N M n = 1 m = 1 n= 1 m= 1 COMP9519 Multimedia Systems Lecture 2 Slide 13 J Zhang

14 2.1.2 Information Measurement -- Review Image quality measurement (MSE,MAE,SNR and PSNR) Assume symbol xrepresents the original image and ˆx the reconstructed image, M and N the width and the height of respectively Mean squared error (MSE): M 1 N 1 1 MSE = [ x( m, n) xˆ ( m, n)] MN m= 0 n= 0 Mean absolute error (MAE): M 1 N 1 1 MAE = x( m, n) xˆ ( m, n) MN m= 0 n= 0 Peak signal to noise ration (PSNR): 2 (2-7) (2-8) PSNR = 10log db MSE COMP9519 Multimedia Systems Lecture 2 Slide 14 J Zhang (2-9) Peak pixel value is assumed 255

15 2.2 Lossless & Predictive Coding (Entropy Coding) Introduction to Entropy Coding Introduction to Predictive Coding Introduction to DPCM coding COMP9519 Multimedia Systems Lecture 2 Slide 15 J Zhang

16 2.2.1 Introduction to Entropy Coding Application of information content to a real image COMP9519 Multimedia Systems Lecture 2 Slide 16 J Zhang

17 2.2.1 Introduction to Entropy Coding Image Histogram Entropy = 7.63 bits/pixel COMP9519 Multimedia Systems Lecture 2 Slide 17 J Zhang

18 2.2.1 Introduction to Entropy Coding The number of bits required to represent an image can be made based on the information content using an entropy (variable length coding) approach such as a Huffman code Highly probable symbols are represented by short code-words while less probable symbols are represented by longer code-words The result is a reduction in the average number of bits per symbol COMP9519 Multimedia Systems Lecture 2 Slide 18 J Zhang

19 2.2.1 Introduction to Entropy Coding Example Fixed length coding Symbol Probability Codeword Codeword Length A B C D Average bits/symbol = 0.75* * * *2 = 2.0 bits/pixel COMP9519 Multimedia Systems Lecture 2 Slide 19 J Zhang

20 2.2.1 Introduction to Entropy Coding Example Entropy (Variable Length) Coding Symbol Probability Codeword Codeword Length A B C D Average bits/symbol = 0.75* * * *3 = bits/pixel (A 30% saving with no loss) COMP9519 Multimedia Systems Lecture 2 Slide 20 J Zhang

21 2.2.1 Introduction to Entropy Coding Generation of Huffman Codewords If the symbol probabilities are known, Huffman codewords can be automatically generated Details are introduced in next two slides Note on Merging (left to right) 1. Reorder in decreasing order of probability at each step 2. Merge the two lowest probability symbols at each step Note on Splitting (right to left) 1. Split the symbol merged at that step into two symbols COMP9519 Multimedia Systems Lecture 2 Slide 21 J Zhang

22 2.2.1 Introduction to Entropy Coding 1. From Right to Left, 2. Two bottom-most branches are formed a node 3. Reorder probabilities into descending order Tree Construction process COMP9519 Multimedia Systems Lecture 2 Slide 22 J Zhang

23 2.2.1 Introduction to Entropy Coding 1) Re-arrange the tree to eliminate crossovers, 2) The coding proceeds from left to right, 3) 0 step up and 1 step down. Code generation COMP9519 Multimedia Systems Lecture 2 Slide 23 J Zhang

24 2.2.1 Introduction to Entropy Coding Truncated and Modified Huffman Coding Given the size of the code book is L, the longest codeword will reach L bits For a large quantities of symbols, the size of the code book will be restricted Truncated Huffman coding For a suitable selected L1<L, the first L1 symbols are Huffman coded and the remaining symbols are coded by a prefix code, following by a suitable fixed-length code Second Order Entropy Instead of find the entropy of individual symbols, they can be grouped in pairs and the entropy of the symbol pairs calculated. This is called the SECOND ORDER ENTROPY. For correlated data, this will lead to an entropy closer to the source entropy. COMP9519 Multimedia Systems Lecture 2 Slide 24 J Zhang

25 2.2.1 Introduction to Entropy Coding Limitations of Huffman Coding Huffman codewords have to be an integer number of bits long. If the probability of a symbol is 1/3, the optimum number of bits to encode that symbol is -log2 (1/3) =1.6. Assigning either one or two bits leads to a longer code message than is the theoretically necessary The symbol probabilities must be known in the decoder size. If not, they must be generated and transmitted to the decoder with the Huffman coded data A larger of number of symbols results in a large codebook Dynamic Huffman coding scheme exists where the code words are adaptively adjusted during encoding and decoding, but it is complex for implementation. COMP9519 Multimedia Systems Lecture 2 Slide 25 J Zhang

26 2.2.1 Introduction to Entropy Coding Arithmetic Coding It overcomes limitation of Huffman coding: non-integer length coding, and probability distribution can be derived in real-time It operates by replacing a stream of input symbols with a single floating point output number. Consider the following symbols with probabilities A sub-interval cab be defined by its lower end point and its width or lower and upper end points The sum of preceding Probabilities known as Cumulative Probability CP( s i ) = 1 p( s ) i i j= 1 Where CP(S1)=0 is defined COMP9519 Multimedia Systems Lecture 2 Slide 26 J Zhang

27 2.2.1 Introduction to Entropy Coding Arithmetic Coding Suppose we wish to encode the string: S1; S2; S3; S4; S5; S6; We start with the interval [L,H) and set to [0,1) for 6 symbols. Since the first symbol is S1, we pick up its subinterval [L,H) = [0.0,0.3), and any real symbols can be considered as disjoint to be divided in the same way. To encode the S2, we use the same procedure as used in above to divide the interval [0,0.3) into six sub-intervals. We pick up the S2 subinterval [0.09,0.12) The subinterval recursion is equivalent to the two recursions End point recursion and width recursion L new = L + W CP current current new Low end point recursion W new = W P( S ) current i The width recursion COMP9519 Multimedia Systems Lecture 2 Slide 27 J Zhang

28 2.2.1 Introduction to Entropy Coding COMP9519 Multimedia Systems Lecture 2 Slide 28 J Zhang

29 2.2.1 Introduction to Entropy Coding Arithmetic Decoding For encoding, the input is a source symbol string and the output is a subinterval (called the final subinterval). For our case, it is [ , ). Decoding sort of reverses what encoding has done. The decoder knows the encoding procedure and therefore has the information contained in the following figure It compares the lower end point of the final subinterval with all the end points in the above figure 0< <0.3!!! COMP9519 Multimedia Systems Lecture 2 Slide 29 J Zhang

30 2.2.1 Introduction to Entropy Coding The lower end falls into the subinterval associated with the symbol S1 which is first decoded. After the first symbol is decoded, 0.09< <0.12!!!. The lower end is contained in the subinterval covers the S2 Repeat the process until the symbols S4; S5; S6 are subsequently decoded. Using Huffman coding to encode the same symbols COMP9519 Multimedia Systems Lecture 2 Slide 30 J Zhang

31 2.2.1 Introduction to Entropy Coding Human coding converts each source symbol into a fixed codeword (but variable length). The output of S1; S2; S3; S4; S5; S6 are: 00,101,11,1001,1000,01 which is a 17-bit code string Arithmetic coding converts a source symbol to a code symbol string = ~ [ , ). Which is 15- bit code string This is a simple example about the arithmetic coding is more efficient than Human coding It is obvious that the width of the final subinterval becomes smaller when the length of the source symbol string become larger and larger. This was a big problem to widely apply the arithmetic coding Arithmetic coding now becomes an increasingly important coding COMP9519 Multimedia Systems Lecture 2 Slide 31 J Zhang

32 2.2.1 Introduction to Entropy Coding Run-length Coding Ref: H.Wu In run-length coding, a run of consecutive identical symbols is combined together and represented by a single codeword. The coding of facsimile information is one application of run length coding. The various run-lengths are then represented by a variable length (e.g. Huffman) codeword. In the case of facsimile transmission, separate VLCs are defined for white and black Use an escape code (out side of symbol set or 0 ), followed by the run-length coded symbol and number of repetitions of symbol. COMP9519 Multimedia Systems Lecture 2 Slide 32 J Zhang

33 2.2.1 Introduction to Entropy Coding Run-length Coding (Example) [Ref: H. Wu] Run-length (using 0 as the escape code) coding DCT Coefficients COMP9519 Multimedia Systems Lecture 2 Slide 33 J Zhang

34 2.2.2 Introduction to Differential Coding Strong correlation exists between adjacent pixel spatially Spatial redundancy is existed among pixels within a single frame of image. Especially, neighboring pixels are highly correlated. A pixel is coded based on the difference between its value and a predicted value. Ref: H. Wu Lena image Spatial Statistical Redundancy COMP9519 Multimedia Systems Lecture 2 Slide 34 J Zhang

35 2.2.2 Introduction to Differential Coding By exploring spatial/temporal inter-pixel correlation, prediction and quantization coding scheme achieve efficiency and yet computationally simple coding technique. When the prediction error (difference error) is quantized, the differential coding is called Differential Pulse Code Modulation (DPCM) Where zi is current input pixel Where z i = n ai z j= 1 i j is a linear prediction function of N previously reconstructed samples (Summation) z = f ( z,... 1 z 2 z n ) COMP9519 Multimedia Systems Lecture 2 Slide 35 J Zhang

36 2.2.2 Introduction to Differential Coding In this approach, the preceding pixel on the video/image line is used to predict the next pixel. The prediction error is then transmitted Example (+7) 101 (+8) 103 (+2) 96 (-3) 97 (-4) 104 (+1) This approach achieves compression because the entropy of the prediction error is less than the entropy of the original image. The better the prediction, the lower the entropy of the prediction error. COMP9519 Multimedia Systems Lecture 2 Slide 36 J Zhang

37 2.2.2 Introduction to Differential Coding Lena image 1-D prediction error of Lena image Entropy = 3.06 bits/pixel COMP9519 Multimedia Systems Lecture 2 Slide 37 J Zhang

38 2.2.2 Introduction to Differential Coding This approach can be applied to a two dimensional predictor by taking advantage of vertical correlation as well. Example 2-D prediction error of Lena image Entropy = bits/pixel COMP9519 Multimedia Systems Lecture 2 Slide 38 J Zhang

39 2.2.2 Introduction to Differential Coding Ref: H, Wu Coding efficiently Code assigner is a strategy to code the symbols of coefficients after the transform. Improve accuracy Prediction error: Where Reconstruction: e( n) = s( n) sˆ ( n) sˆ( n) = f [ s '( n 1), s '( n 2), s '( n 3),...] s '( n) = sˆ ( n) + e'( n) COMP9519 Multimedia Systems Lecture 2 Slide 39 J Zhang

40 2.3 Introduction to Image Quantization Quantization The major mechanism for loss in image/video codecs (encoder/decoder) is as a result of quantization However, when quantization is performed carefully in frequency domain, it is difficulty for a viewer to notice any degradation. Original JPEG (compressed 70%) COMP9519 Multimedia Systems Lecture 2 Slide 40 J Zhang

41 2.3 Introduction to Image Quantization Linear Quantization The simplest form of quantize is a linear quantize where the quantize step is constant Maximum error = ± Half quantize step size COMP9519 Multimedia Systems Lecture 2 Slide 41 J Zhang

42 2.3 Introduction to Image Quantization Linear/Non-linear Quantization For the case of the linear quantizer, a simple relationship exits between the number of symbols after quantization and the error introduced by quantization This approach is sensible for a signal with a uniform distribution of values such as an original image However, it makes very little sense for difference image where the differences are non-uniformly distributed about zero, For these types of signals, a non-linear quantization is needed. COMP9519 Multimedia Systems Lecture 2 Slide 42 J Zhang

43 2.3 Introduction to Image Quantization Linear Quantizer Applied to a Difference Image COMP9519 Multimedia Systems Lecture 2 Slide 43 J Zhang

44 2.3 Introduction to Image Quantization Non-Linear Quantizer Applied to a Difference Image COMP9519 Multimedia Systems Lecture 2 Slide 44 J Zhang

45 2.3 Introduction to Image Quantization Non-Linear Quantizer Transfer Function COMP9519 Multimedia Systems Lecture 2 Slide 45 J Zhang

46 2.3 Introduction to Image Quantization Quantization and HVS In most of the codecs, the quantization stepsize is controlled in such a way that distortion is as imperceptible as possible Much study has been conducted for the characteristics of the HVS (human visual system). It is well known that various features in video/image tend to mask the visibility of errors introduced by coding. These include: Edges, Motion and high luminance values Codec try to focus quantization errors in these regions. COMP9519 Multimedia Systems Lecture 2 Slide 46 J Zhang