Image Compression. Universidad Politécnica de Madrid Dr.-Ing. Henryk Richter

Size: px

Start display at page:

Download "Image Compression. Universidad Politécnica de Madrid Dr.-Ing. Henryk Richter"

Lesley Pierce
5 years ago
Views:

Image Compression Universidad Politécnica de Madrid 2010 Dr.-Ing. henryk.richter@comlab.

1 Image Compression Universidad Politécnica de Madrid 2010 Dr.-Ing. University of Rostock Institute of Communications Engineering

2 Literature Rao K.R.: Techniques & Standards for Image, Video & Audio Coding, Prentice Hall 1996 Ohm, J.-R.: Multimedia Communications Technology, Springer-Verlag 2004 Wang, Y. et al.: Video Processing and Communications, Prentice Hall 2002 Rao K.R. et al.: The transform and data compression handbook, CRC Press 2001 Watkinson: MPEG-2, Focal Press 1999 Pennebaker, W.B. et al.: JPEG still image compression standard, NY 1993 Mitchell J. L. et al.: MPEG Video Compression Standard. Chapman and Hall 1997 Taubman, D.S. et al.: JPEG2000, Kluwer Academics Publishers, 2002 Richardson I.: H.264 and MPEG-4 Video Compression, Wiley & Sons 2003 Gersho A. and Gray R. M.: Vector Quantization and Signal Compression, Kluwer Academics Publishers 1992 INT 2

3 Literature Journals IEEE Transactions on Image Processing IEEE Transactions on Circuits and Systems for Video Technology IEEE Signal Processing Magazine IEEE Communications Magazine Conferences International Conference on Image Processing (ICIP) Picture Coding Symposium (PCS) International Conference on Visual Communications and Image Processing (VCIP) INT 3

4 Transmission System source General Transmission Model source encoding Sender channel coding (error protection) modulation Noise, Fading, Interference Errors channel (transmission/storage) Receiver sink source decoding error correction demodulation data compression = source coding INT 4

5 Necessity of data compression Width Height RGB FPS Data rate = 31.1 MB/s = MB/s Capacity CD 700 MB DVD 8.5 GB Blue-ray 50 GB HDD 1 TB Storage time 22 s 4.5 s 4.5 min 54 s 26 min 5 min 8.9 h 1.7 h Bandwidth DVB-H 384 kbit/s UMTS 1.4 MBit/s ADSL2+ 16 MBit/s Ethernet 1 GBit/s Transmission 90 min film 40 d 202 d 11 d 55 d 1 d 4.8 d 0.37 h 1.8 h INT 5

Aspect ratio and non square pixels Cinematic recordings

contraction on film (anamorphic representation) typical

(100:54) 2,35 : 1 1,85 : 1 1,33 : 1 (4:3) Application to

DVD 720x576 pixels (1,25:1) Letterbox Cropping

6 Aspect ratio and non square pixels Cinematic recordings traditionally recorded on 35 mm film horizontal image contraction on film (anamorphic representation) typical frame aspect ratios 2,35:1 (21:9, Cinemascope) 1,85:1 (100:54) 2,35 : 1 1,85 : 1 1,33 : 1 (4:3) Application to non-widescreen standards e.g. DVD 720x576 pixels (1,25:1) Letterbox Cropping (Pan/Scan) Anamorphic Representation of digital video commonly requires appropriate resampling INT 6

General data compression tools data compression decorrelation data reduction coding reversible concentration of signal energy reduction of

7 General data compression tools data compression decorrelation data reduction coding reversible concentration of signal energy reduction of symbol count increased information per symbol irreversible removal of irrelevancy reversible removal of redundancy optimization / adaptation FUN 7

coding transmission / storage output Decoder inverse

8 General lossy codec input decorrelation prediction transformation Encoder data reduction quantization coding precoding entropy coding transmission / storage output Decoder inverse transformation prediction data reconstruction inverse quantization decoding FUN 8

9 General data compression tools data compression decorrelation data reduction coding reversible concentration of signal energy reduction of symbol count increased information per symbol irreversible removal of irrelevancy reversible removal of redundancy optimization / adaptation FUN 9

precision vector quantization scalar quantization mapping of similar vectors to a common

10 General data compression tools data reduction subsampling quantization reduction of spatial resolution reduction of temporal resolution reduction of signal amplitude precision vector quantization scalar quantization mapping of similar vectors to a common representative vector mapping of similar values/ amplitudes to a common representative value RED 10

11 Color Spaces: RGB (1) primaries Red Green Blue additive compositing model range of components 0,0...1,0 typical normalization: 8 bit per component video with integer range of ³ = 16.7 Mio. available colors levels zero level of all components equals black full level of all components equals white well suited for capture and display television sets computer displays / monitors digital cameras red magenta white blue cyan yellow green INT 11

Color Spaces: RGB (2) compositing between 3

red green 0 0 0 052 02 02 052 002 04 051 001

001 05 021 0 051 06 08 05 021 04 0 051 001 052

12 Color Spaces: RGB (2) compositing between 3 gray planes representing the color components red green blue 0 INT

13 YUV / YIQ Color Spaces History for the introduction of color television, backwards compatibility to existing black/white receiver was demanded separation of luma (brightness) from color components (chroma) limited bandwidth for additional color component signals (~1/6 of total video bandwidth) Practical advantages luma bandwidth/resolution not tied to chroma resolution controlling brightness, color saturation and contrast easy to perform in comparison to the RGB color model Applications NTSC color television (U.S.A., Japan) YIQ color model SMPTE-140M PAL color television (Europe) YUV color model ITU-R Rec. BT.709, SMPTE 170M digital image and video compression formats (e.g. JPEG, MPEG) INT 13

14 YCbCr Color Space (1) YCbCr is the quantized and scaled digital correspondence to analog YUV analog RGB input: ER,E G,E B 0,...,1 ITU-R Rec. BT.601, SMPTE 170M: E Y = E G E B E R E PB = E G E B E R E PR = E G E B E R ITU-R Rec. BT.709 (HDTV): E Y = E G E B E R E PB = E G E B E R E PR = E G E B E R quantized YCbCr at n bit quantization: a) limited video dynamic range (BT.601) b) full video dynamic range (JPEG, MPEG-4) Y = n 8 E Y + 2 n 4 Cb= n 8 E PB + 2 n 1 Cr= n 8 E PR + 2 n 1 Y =((2 n 1) E Y ) Cb=((2 n 1) E PB )+2n 1 Cr=((2 n 1) E PR )+2n 1 Y, Cb, Cr are unsigned integers with grayscale point at Cb=Cr=2 n-1 INT 14

15 YCbCr Color Space (2) Grayscale image if only Y is present Cb corresponds to blue/yellow balance Cr corresponds to red/green balance INT 15

16 YCbCr chroma subsampling (1) Specific chroma subsampling variants have standard names first digit: number of luma pixels second digit: horizontal chroma pixels relative to luma third digit: definition dependent 4:4:4 4:2:2 4:1:1 4:2:0 luma sample chroma sample (MPEG-2 and later) chroma sample (MPEG-1) INT 16

17 YCbCr chroma subsampling (2) Subsampling of chroma components: YCbCr 4:2:0 50% data reduction Y 4 Blocks Y Cb Cr 1 Block Cb 1 Block Cr Subsampling on Sender side Upsampling on Receiver side RED 17

18 Sampling 1 Sub-Sampling: also: down-sampling simplest form of data reduction by decreasing the data rate parameter: sub-sampling factor M 2 M N only each M th sample is captured: y [m] =x [m M ] Sub-Sampling with rational accuracy M>1 M R decimation using nearest neighbor sample decimation using antialiasing filter k(h) y[m] =x[m M] y[m] =x[m M] k(h) RED 18

19 Sampling 2 x[n] M y[m] x[n] y[m] n m RED 19

20 Sampling 3 Spectrum ( X(f) x[n] ) is periodic with sampling frequency f s X(f) 0 f s /2 M f s 2 Sub-sampling: periodicity interval of spectrum reduced by factor M repetitions of spectral components pushed into lower frequency range disobeying the Nyquist barrier leads to aliasing f s f Sampling frequency is changed by factor 1 / M RED 20

21 Sampling 4 X(f) 0 f (x) s /2 M f (x) s 2 f (x) s f Y (f) 0 f (y) s = f s (x) M 2 f s (x) M 3 f (x) s M f RED 21

22 Comparison: decimation* by factor 2 All signal components beyond the normalized frequency 0.5 contribute to aliasing in this downsampling example windowed sinc (Lanczos-2,Lanczos-3 shown) exhibits quick cutoff in the stopband all other algorithms considered here leave considerable energy in the stopband Magnitude Response (db) 0 Magnitude (db) Box FIR, 2 Tap Tent (linear), 3 Tap Cubic, A=-0.5, 9 Tap Lanczos-2, 9 Tap Lanczos-3, 13 Tap 8 Bit limit -48 db *zero phase filtering Normalized Frequency (!! rad/sample) Lanczos-16 RED 22

23 Subsampling examples Bi-cubic Original Windowed Sinc (Lanczos) Nearest Neighbor (reduction to 30% plus enlargement for display) RED 23

24 Quantization I Scalar Quantization partitioning of the signal dynamic range into sub-intervals of reduced accuracy algorithm let x be a random signal amplitude, falling into interval q assign a reconstruction amplitude y q, with y q interval q quantization: x q -- sender reconstruction: q y q, where y q =[x] Q -- receiver quantization error: e q = x [x] Q Unrecoverable loss of information choice of intervals application specific, leading to different types of scalar quantizers RED 24

25 Quantization 2 Uniform Quantization Property: equal width of all sub-intervals q q x x y q y q e q 2 e q 2 x x RED 25

26 Quantization 3 Uniform Quantization special variants of uniform quantizers q q limitation x x dead zone y q e q e q x x RED 26

..1 1 bit plane 2 bit planes 3 bit planes 4 bit

27 Quantization 5 Example: successive approximation using bit planes y q = x 2 n 2 n with n = bit plane 2 bit planes 3 bit planes 4 bit planes 5 bit planes 6 bit planes 7 bit planes 8 bit planes (ori) RED 27

28 General data compression tools data compression decorrelation data reduction coding reversible concentration of signal energy reduction of symbol count increased information per symbol irreversible removal of irrelevancy reversible removal of redundancy optimization / adaptation FUN 28

General data compression tools entropy coding code word based arithmetic symbols are substituted by a discrete series of bits minimal length of output symbols is 1 some basic algorithms: Shannon

29 General data compression tools entropy coding code word based arithmetic symbols are substituted by a discrete series of bits minimal length of output symbols is 1 some basic algorithms: Shannon Shannon-Fano Golomb-Rice Huffman symbols are regarded as part of a code interval, narrowing with each symbol binary representation of the final code interval as bit stream some basic algorithms: arithmetic coding binary arithmetic coding range coding prefix codes, which can be handled by a generic decoder optimal coding is a symbol representation with I i l i RED 29

30 Information theoretic basics (1) information content of symbol with probability I i x i P (x i ) I i = ld 1 P (x i ) [bit] (Claude E. Shannon, 1948) information content is reciprocal to the probability of symbols the more surprising a symbol is, the higher its information content highly probable symbols contribute less information information content is expressed in binary units FUN 30

31 Information theoretic basics (2) entropy H of source symbols n x i (i =1,...,n) H = P (x i ) I i [bit/symbol] i=1 H entropy of source denotes the minimum average number of bits to code a message comprised of symbols x i for statistically independent symbols, it is not possible to provide a lossless message with less bits than given by H multiplied with the number of transmitted symbols x i FUN 31

32 entropy example Symbols Probabilities A B C D a) 1/4 1/4 1/4 1/4 b) 1/2 1/4 1/8 1/8 Entropy H a = H max =4 1 ld 4=2 [bit/symbol] 4 H b = 1 2 ld ld 4+2 ld 8=1.75 [bit/symbol] 4 8 Redundancy of source R a = H max H a =0 (redundancy free source) R b = H max H b =0.25 [bit/symbol] FUN 32

33 entropy example (2) H a =2bit/symbol What about these two distributions below? Probabilities are equal to examples a and b. On the first glance, the formally calculated entropy would be equal to the random patterns on the left side. H b =1.75 bit/symbol Symbols in the examples on the right are obviously correlated. FUN 33 Entropy formula for uncorrelated source symbols does not apply in such cases. Dependencies not always obvious, hence success of coding dependent on modeling.

34 Huffman Code (optimal binary valued code) Construction algorithm: (Huffman D. 1952) 1. find two symbols x j, xk with lowest probability within input symbols xi, where pi = P(xi ) pj pk pi with i=(1,2,...,n), i j, i k 2. assign a new symbol xjk to the compound of xj, xk 2.1. remove xj, xk from input symbols (n=n-2) 2.2. assign 0 to xj,1 to xk, remember xj, xk as leaves of xjk 2.3. add xjk with probability pjk = pj + pk to input symbols xi (n=n+1) 3. return to 1) until pjk = 1,0 4. collect leaves of the tree, generate code bits and lengths along the path from root to leaves Features bottom-up method, starting with the leaves of the tree optimum code with code words of binary length l i N despite optimum code, the coding redundancy Rcod is only zero in case p i = 1 2 n,n N minimum code length li is 1 bit, hence even when the entropy Hsrc is less than one bit per symbol, all code words contain at least one bit COD 34

obtained by following a path from root to the leaves car color P(xi) ci li black 0,5

35 Huffman Code - Example Construction by using least probabilities, compound symbol represents the former individual symbols or compounds code words / lengths can be obtained by following a path from root to the leaves car color P(xi) ci li black 0, green 0,25 1 1,0 root 01 2 red 0,125 blue 0, ,25 0 0, COD 35

36 VLC tree options/properties Identical length of code words, different naming scheme root 1,0 1 0 root 1, ,5 black 0, , ,5 black 0,25 0,25 green 1 0 0,25 0,25 green 1 0 0,125 0,125 H =? 0,125 0,125 red blue l =? red blue COD 36

37 Summary: Variable Length Codes Huffman Codes are optimal bit based codes for a given probability distribution Advantages fast encoding and decoding possible by using look-up tables prefix codes have an implicit length do not need further control information have no restrictions on symbol count and allow random symbol order can be inter-mixed with fixed length codes in a single bit stream Drawbacks l 1.0, even with Hsrc < 1.0 bit/symbol if signal statistics change (i.e. non stationary input signals) compression gets sub-optimal due to fixed code trees adaptive coding feasible but code tree regeneration (at each symbol) is complex and with varying tree adaptation algorithms not necessarily unique in most cases, either the code tree is transmitted within data stream or known a priori in form of a constant table in both encoder and decoder coding redundancy Rcod is only zero in case p i = 1 2 n,n N COD 37

Precoding method overview precoding run length coding strings and blocks of identical symbols are substituted by a smaller number of symbols from a new alphabet block sorting strings of symbols are

38 Precoding method overview precoding run length coding strings and blocks of identical symbols are substituted by a smaller number of symbols from a new alphabet block sorting strings of symbols are sorted to match other sets in the current scope more closely in order to increase correlation dictionary coding arbitrary symbol strings are mapped into new symbols of another alphabet match arbitrary symbol strings against an encoder maintained dictionary encode match parameters instead of explicit string coding other any method providing information about symbol relationship can aid in reduction of inter-symbol redundancy RED 38

39 General data compression tools data compression decorrelation data reduction coding reversible concentration of signal energy reduction of significant symbol count increased information per symbol irreversible removal of irrelevancy reversible removal of redundancy RED 39

Tools for concentration of signal energy / information decorrelation prediction spectral decomposition forecast of signal values and side information knowledge about previous symbols

40 Tools for concentration of signal energy / information decorrelation prediction spectral decomposition forecast of signal values and side information knowledge about previous symbols incorporated into prediction of other elements transformation decomposition of signals into basis functions filter banks decomposition of signals into (possibly overlapping) frequency domains RED 40

41 Prediction Transmitter Receiver x [n] + e[n] e[n] + x [n] - + ˆx [n] x [n] ˆx [n] T x [n 1] T predictor T x [n 1] x [n 2] predictor T x [n 2] T x [n k] T x [n k] Recursive RED 41

42 Example: e[x,y] = s[x,y] - s[x,y-1] Histogram reveals concentration of signal energy towards RED 42

43 2D prediction two-dimensional signals can be predicted in the same way as 1D signals, however correlation between direct neighbors (vertical, horizontal) higher than for predictors of greater distance general 2D linear prediction ˆx[i, j] = k a k,l x[i k, j l] l for grayscale intensity x[i, j] at position (i, j) causal prediction requires k, l > 0; k, l N (processing in raster scan order) i i A C B X D ˆX = f(a, B, C, D) j j DEC 43

2D prediction example ˆx[i, j] =x[i, j] x[i 1,j] ˆx[i,

x[i 1,j] horizontal vertical horizontal+vertical 0.3 0.

44 2D prediction example ˆx[i, j] =x[i, j] x[i 1,j] ˆx[i, j] =x[i, j] x[i, j 1] ˆx[i, j] =x[i, j] 1 2 x[i, j 1] + x[i 1,j] horizontal vertical horizontal+vertical RED 44

45 Tools for concentration of signal energy / information decorrelation prediction spectral decomposition forecast of signal values and side information knowledge about previous symbols incorporated into prediction of other elements transformation decomposition of signals into basis functions filter banks decomposition of signals into (possibly overlapping) frequency domains RED 45

46 Transforms Karhunen-Loève Transform (KLT) Discrete Fourier Transform (DFT) Discrete Cosine Transform (DCT) / Discrete Sine Transform (DST) Walsh-Hadamard Transform (WHT), Hadamard Transform (HT) Haar Transform Wavelet Transform (WT) lapped transforms [MAL92] Overlapping local trigonometric bases Modified DCT (MDCT) Modulated Complex Lapped Transform (MCLT) [MAL99][MAL03] DEC 46

47 Karhunen-Loève Transform Decorrelate elements of a vector x basis functions are the eigenvectors of the covariance matrix of the input signal KLT is the optimal decorrelation transform, providing optimum energy concentration. Drawbacks of KLT dependent on accurate modeling of signal statistics re-computation for each processed image eigenvector decomposition is computationally complex requires side information transmission of signal statistics or transform matrix negative impact on total coding gain not separable for images (or blocks) high complexity transform matrix can not be factored into sparse matrices high complexity DEC 47

48 Discrete Cosine Transform real valued signal transform from space/time domain to frequency domain [ANR74] concept the discrete fourier transform (DFT) provides energy compaction suitability for compression is limited due to complex spectral components (in typical cases) however: DFT of real even (symmetric) signals results in purely real valued spectrum key idea is to generate a symmetric signal from the input signal whose DFT results in a real spectrum 8 DCT variants available differences are in the way of symmetric extension direct extension (mirroring around x[0], x[n-1]) half shifted extension (mirroring including x[0], x[n-1]) even/odd mirroring (+/-) DCT-I to DCT-IV perform either direct or half shifted extension DCT-V to DCT-VIII perform direct extension one one, half shifted extension on the other boundary (rarely used in practice) DCT Transform kernel is purely real valued, resulting in real valued DCT coefficients DCT is signal independent. DEC 48

49 Example: Symmetric extension DCT-I...DCT-IV DCT-I x[n] 2N-2 N=8 DCT-III x[n] 4N N=8 0 7 n 0 7 n periodicity: 2N-2 symmetry: n=0; n=n-1 x[n] N=8 periodicity: 4N symmetry: n=0; n=2n DCT-II x[n] 2N N=8 0 7 n DCT-IV x[n] 4N N=8 n 0 7 n 0 7 periodicity: 2N symmetry: n=-0.5; n=n-0.5 periodicity: 2N symmetry: n=-0.5; n=2n-0.5 RED 49

50 DCT-I...DCT-IV transform kernels Transform kernels for forward and inverse transform are the basis functions i.e. cosine functions of different frequency Type I 2 knπ w[n]w[k] cos k, n =0, 1,...,N N N Type II 2 N k(n +0.5)π w[k] cos N k, n =0, 1,...,N 1 Type III 2 N (k +0.5)nπ w[n] cos N k, n =0, 1,...,N 1 Type IV 2 (k +0.5)(n +0.5)π N cos N k, n =0, 1,...,N 1 2 factors and weights w[m] make the DCT orthogonal and orthonormal N 0 if m<0 or m>n w[m] = 1 2 if m = 0 or m = N 0 if 0 <m<n DEC 50

51 Example: 1D DCT basis functions DCT-II DCT-IV k= example shows first 5 (non normalized) basis functions of 1D transform with N=8 DCT-II has constant as basis function k=0 image/video coding DCT-IV has low frequency wave as basis function k=0 audio coding RED 51

52 DCT-II transform DCT-II offers favorable energy concentration in images [RY90] first base function is a constant, important for flat image regions for first order stationary Markov-processes near optimal decorrelation (AR(1) models) DCT-III is the inverse transform of DCT-II used in all relevant standards for image/video compression (JPEG,MPEG,H.264*) Commonly used block size for image coding is 8x8 Coefficients are real valued, perfect reconstruction requires FP calculation integer arithmetic possible, however with limited accuracy IEEE standard 1180 enforces maximum permissible drift among different implementations Special variants shape adaptive DCT (SADCT) in MPEG-4 Part 2 4x4 / 8x8 integer DCT in H.264 / MPEG-4 AVC * heavily quantized form DEC 52

53 Reason for block based DCT Full frame DCT-II 16x16 block DCT-II 8x8 block DCT-II Quantization errors are distributed over size of basis functions. quant: q[x, y] = 0 for f> X[x, 1 24 y] < 50 X[x, y] for f X[x, 1 24 y] 50 RED 53

54 DCT-II of images input image 8x8 block DCT-II 8x8 basis functions horizontal frequency DC AC AC AC AC... AC AC AC... AC AC... vertical frequency AC frequency Images are divided into blocks of NxN samples every block is transformed individually transformed blocks consist of DC and AC components frequency progression within block in diagonal direction RED 54

55 2D DCT-II / DCT-III formal for NxN samples DCT (DCT-II) S[u, v] = 2 N N 1 x=0 N 1 y=0 w[x]w[y]s[x, y]cos with u, v, x, y =0, 1,...,N 1 x, y coordinates in spatial domain (2x + 1)uπ 2N u, v coordinates in frequency domain w[x],w[y] = 1 2 for u, v =0 1 else cos (2y + 1)vπ 2N IDCT (DCT-III) s[x, y] = 2 N N 1 u=0 N 1 v=0 w[u]w[v]s[u, v]cos with u, v, x, y =0, 1,...,N 1 x, y coordinates in spatial domain (2x + 1)uπ 2N u, v coordinates in frequency domain w[u],w[v] = 1 2 for u, v =0 1 else cos (2y + 1)vπ 2N RED 55

56 References [DSV2] Müller, E.: Digital Signal Processing 2, winter semester lecture, Strutz T.: Bilddatenkompression, 3. Auflage, Vieweg-Verlag 2005 [Mitchell] Mitchell, P.; Netravali, A.: Reconstruction Filters in Computer Graphics. ACM Computer Graphics, Vol. 22:4, 1988, (also proc. Siggraph 88) [Keys] Keys, R,: Cubic convolution interpolation for digital image processing. IEEE Transactions on Signal Processing, Acoustics, Speech, and Signal Processing. Vol. ASSP-29:6, 1981, [Turk] Turkowski, K.: Filters for Common Resampling Tasks, Graphics Gems I, Academic Press, 1990, [LBG80] Linde, Y.; Buzo, A.; Gray, R. M.: An algorithm for vector quantizer design. IEEE on Comm. Vol. 28, no. 1, [GG92] Gersho, A.; Gray, R.M.: Vector Quantization and Signal Compression, Kluwer Academics Publishers, 1992, ISBN [DC] Nam Phamdo: Vector Quantization, RED 56

57 References [WBSS04] Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. 4, pp , Apr [WLB04] Z. Wang, L. Lu and A. C. Bovik, "Video Quality Assessment Based on Structural Distortion Measurement", Signal Processing: Image Communication, special issue on Objective Video Quality Metrics, vol. 19, no. 2, pp , Feb [WSBA03] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, Overview of the H.264/AVC Video Coding Standard, IEEE Transactions on Curcuits and Systems for Video Technology, Vol. 13, No. 7, Jul 2003 [x264] L. Aimar, L. Merritt, E. Petit et al., "x264 - a free h264/avc encoder ", FUN 57

SYDE 575: Introduction to Image Processing. Image Compression Part 2: Variable-rate compression

SYDE 575: Introduction to Image Processing Image Compression Part 2: Variable-rate compression Variable-rate Compression: Transform-based compression As mentioned earlier, we wish to transform image data