ENGR 027 Image Compression The Discrete Cosine Transform & JPEG 02/16/17

Transcription

1 ENGR 027 Image Compression The Discrete Cosine Transform & JPEG 02/16/17

2 Data Compression What is data compression Squeezes out redundancy in order to encode data using fewer bits. Image storage: Consider a 12 Mpixel image on your new smart phone 3,000x4,000 pixels, 3 color channels 8 bit/s per channel 36 Mbyte/picture 32 GB memory => Can store about 900 pictures Image transmission: FIOS is about 56 Mbps = 7 MB/sec About 5 seconds to transmit one 12 Mpixel image 02/16/2017 2

3 Data Compression This is quite phenomenal compared to the state of things in 1986 Disk drives ~500 MB Memory ~4 MB Internet speed was ~56 Kbps Monitors were only 800 x 600 pixels; didn t expect much There were no digital cameras, smart phones HD TV s etc. Image data compression enabled the development of these types of devices. 02/16/2017 3

4 Data Compression Compression Lossless No informaeon is lost Only limited amount of compression possible due to entropy limit Lossy Some informaeon lost Aim is minimum distoreon with maximum bit rate 02/16/2017 4

5 Informa0on Entropy Informaeon entropy Introduced by Claude Shannon at Bell labs A Mathemaecal Theory of Communicaeon, Bell System Technical Journal, 1948 Analogous to staesecal mechanics concept of entropy If symbol (i) occurs with probability p i, the amount of informaeon carried by that symbol is: I i = log 2 1 p i 02/16/2017 5

6 Informa0on Entropy Informaeon entropy Average informaeon entropy per symbol in a message with an alphabet of n symbols is: n H = pilog2 p i= 1 i This sets the absolute minimum for the average number of bits per symbol in a message. Sets limit on how much lossless data compression you can achieve 02/16/2017 6

7 Image Histogram Example: Image histogram: entropy= entropy=0 02/16/2017 7

8 Lossless Compression Example: Previous image histogram looks like you need all 8 bits, but if you subtract adjacent pixels, to reduce correlaeon, you can get the entropy down to about 4 bits/pixel The pracecal limit for lossless image data compression is roughly 2:1 Not so good. 02/16/2017 8

9 Lossy Compression Lossy compression: If you are willing to sacrifice some informaeon, you can get much greater compression raeos How much depends on how much you are willing to sacrifice Rate-distoreon curve 02/16/2017 9

10 Lossy Compression Lossy compression: 02/16/

11 The Origins of JPEG Back to the future where things stood in 1986 There was a need for image compression people were looking at pictures on computers storing images on disk transminng images over the networks (internet just beginning) Maybe someday there would be things like digital cameras, digital radiography, video on demand no way! There were all kinds of methods for compression Huffman coding (lossless) Differeneal Pulse Code Modulaeon (DPCM) Vector Quanezaeon Frequency domain techniques 02/16/

12 The Origins of JPEG Standards: Standards are great. Everyone should have one. The birth of JPEG: Photographic Experts Group formed under: ISO/TC97/SC2 Working Group 8 Coded Representaeon of Picture and Audio Informaeon ISO stands for Internaeonal Standards Organizaeon Objeceve was to be able to send images at a bit rate of 64 Kbs that would be recognizable in one second» progressively improve as more informaeon was transmired. 02/16/

13 The Origins of JPEG Another organizaeon: CCITT (Consultaeve Commiree for Internaeonal Telephony and Telegraphy --- now known as ITU-T; Internaeonal Telecommunicaeons Union) Also interested in new forms of image communicaeons Formed Joint effort with ISO Photographic Experts Group Hence the name Joint Photographic Expert Group JPEG 02/16/

14 The Origins of JPEG This was an internaeonal standards effort Each country has standards organizaeons that parecipate, provide input, vote on whether to adopt a standard, etc. Lots of countries on ad hoc commiree: Canada, Denmark, France, Germany, Israel, Italy, Japan, Republic of Korea, The Neatherlands, United Kingdom, United States Joined by other countries on final votes: China, Czechoslovakia, Switzerland, Iran, Turkey, Yugoslavia, USSR Who do you think the troublemaker was that voted no? 02/16/

15 The Origins of JPEG The US body is ANSI American Naeonal Standards Insetute Formed the ANSI Accredited Standards Commiree X3L3 for image compression Individual companies parecipated: IBM, AT&T, Eastman Kodak, 3M, DuPont, lots of small companies hoping to have inside track for developing compression hardware.» DuPont s interest was digital radiography, since one of their big products was X-ray film. 02/16/

16 The Origins of JPEG Compeeeon for methods in methods proposed Goal was: Recognizable images at bits/pixel Useful images at 0.25 bits/pixel Excellent quality images at 0.75 bits/pixel Indisenguishable from original at 2.25 bits/pixel So, we re talking about compression raeo of about 32:1 for useful images 02/16/

17 The Origins of JPEG Skipping all the gory details: For lossy image compression, an adapeve discrete cosine transform method was chosen A couple of opeons for lossless Huffman coding & Arithmeec coding Other standards as well JBIG Joint Bi-level Experts Group (binary images) MPEG Moving Picture Experts Group New standard: JPEG More on that later 02/16/

18 The Origins of JPEG Standard was issued and approved in 1992 Is JPEG and image format? No At least that was never the inteneon Was only supposed to provide specificaeon for compression, but despite the best efforts of the commiree, most people regard it as image format. 02/16/

19 Overview of JPEG Overview of JPEG JPEG modes of operaeon: Lossless mode (prediceve, entropy coding) Sequeneal mode, DCT-based coding Progressive mode, DCT-based coding Hierarchical mode The Baseline system (what everyone uses now) Sequeneal mode DCT-based coding Huffman coding for entropy encoding 02/16/

20 Overview of JPEG Schemaec of JPEG DCT-Based Encoder/Decoder From arecle by Greg Wallace, IEEE Trans. On Consumer Electronics, /16/

21 Overview of JPEG More details on baseline JPEG encoder: Huffman Table Color components (Y, C b, or C r ) 8 8 FDCT Quantizer AC DC Zig-zag reordering Difference Encoding Huffman coding Huffman coding JPEG bit-stream Quantization Table Huffman Table 02/16/

22 Color Space Conversion Color space conversion JPEG is actually color blind There is no specificaeon for the color space Pracecally speaking, everyone uses luminance and chrominance channels (rather than RGB) Eye is more sensieve to changes in luminance so less compression on this channel Less sensieve to changes in chrominance, so more compression on these channels You can downsample and not loose much informaeon 02/16/

23 Color Space Conversion Color space Color mixing Primary parameters RGB Additive Red, Green, Blue CMYK Subtractive Cyan, Magenta, Yellow, Black YCbCr YPbPr additive Y(luminance), Cb(blue chroma), Cr(red chroma) YUV additive Y(luminance), U(blue chroma), V(red chroma) YIQ additive Y(luminance), I(rotated from U), Q(rotated from V) Used for Printer Video encoding, digital camera Video encoding for NTSC, PAL, SECAM Video encoding for NTSC Pros and cons Easy but wasting bandwidth Works in pigment mixing Bandwidth efficient Bandwidth efficient Bandwidth efficient 02/16/

24 Original image Luminance Chrominance Cb Chrominance Cr 02/16/

25 Color Space Conversion Y R 0 C b G 128 = + C r B 128 (a) translate from RGB to YC b C r R Y G Cb 128 = B Cr 128 (b) translate from YC b C r to RGB 02/16/

26 Color Space Conversion Y I Q = R G B Y R U = G V B 02/16/

27 Color Space Conversion W W W Y H Y H Y H W C b H W C b H W C b H W W W C r H C r H C r H (a) 4:4:4 (b) 4:2:2 (c) 4:2:0 02/16/

28 Step 1: Divide image into 8x8 blocks Next step: Divide image into 8x8 blocks Why? The overall image size can be anything. For JPEG to work on any image, you need to create a uniform block size Could have chosen larger blocks, but at the eme, people were thinking in terms of images on a computer screen which were typically 640x480 pixels! As it turns out, 8x8 is prery good for todays larger images because you can t see block arefacts 02/16/

29 Step 1: Divide image into 8x8 blocks Matrix image taken from Wikipedia.com 02/16/

30 Level shih pixel values (0,255) Level (-128,127) Shift Done on each individual 8x8 block for each color plane 02/16/

31 Discrete Cosine Transform Next step: Discrete Cosine transform for each block This is the heart of the compression method Maps highly correlated input elements into reasonably uncorrelated coefficients There is an ideal way to do this called the Karhunen-Loeve transform (or Hotelling transform) which is closely related to principal component analysis. This transform opemally decorates pixels, however, the transform depends on the data The DCT comes close to this, but is a staec transform 02/16/

32 Discrete Cosine Transform 1-D DCT Why the DCT instead of Discrete Fourier transform? Periodicity of DFT results in disconenuiees that usually occur at the boundaries For DCT, both boundaries are even so always creates conenuous extension at boundaries 4 types of DCT s JPEG uses DCT II 02/16/

33 Discrete Cosine Transform 1-D DCT for length 8 vector: Fk () Ck ( ) 2 7 = n= 0 (2n+ 1) kπ f( n)cos 16 1 C(0) =, C( k) = 1 for k 0 2 f() n 7 = k = 0 (2n+ 1) kπ Ck ( ) 1 Fk ( )cos C(0) =, C( k) = 1 for k f( n) is 1-D sample value Fk ( ) is 1-D DCT value 02/16/

34 Discrete Cosine Transform 2-D DCT for 8x8 block: 7 7 Ck ( ) Cl ( ) (2m+ 1) kπ (2n+ 1) l Fkl (, ) f( mn, )cos cos = m= 0 n= 0 Transform is separable Fast algorithms exist In fact, for 8x8 blocks you can factorize algorithm to make it very fast Note that JPEG doesn t specify how you do the DCT, just that you get the right result 02/16/ π

35 Discrete Cosine Transform What does space versus DCT transform look like for one block? Noece that only a few of the pixels in the transform domain have high amplitude. 02/16/

36 Discrete Cosine Transform Basis vectors (blocks) for 8x8 DCT 02/16/

37 Quan0za0on Next step: Quanezaeon This is where the loss of informaeon occurs You can think of this as something like rounding, but how values are mapped from original (e.g. floaeng point) numbers to quanezed values can be tailored to specific applicaeon For JPEG, different quanezers are used for luminance and chrominance channel Again, JPEG does not specify what quanezers to use But everyone uses the same ones that have been empirically determined to best match the human visual system 02/16/

38 Quan0za0on Quanezaeon Fuv (, ) Fq ( u, v) = Round Quv (, ) /16/

39 Quanezaeon Example F(u,v) 8x8 DCT coefficiences Q(u,v) Quantization matrix 02/16/

40 Quanezaeon Example Fuv (, ) Quv (, ) F ( u, v) = q Fuv (, ) Round Quv (, ) 02/16/

41 Encoding Quan0zed Coefficients Next step: Encoding of the quanezed coefficients AC and DC components are handled differently AC components are re-ordered in a zig-zag scan Many of the AC components will be zero, so you compactly encode long runs of the same value DC components have the average value of the block. If mean value doesn t change much from block to block, difference will be a small number 02/16/

42 Zig-Zag Reordering of AC coefs /16/

43 Difference Encoding of DC coefs Encode : Diff i = DC i - DC i-1 Decode : DC i = DC i-1 + Diff i Diff 1 =DC 1 Diff i-1 = DC i-1 -DC i-2 Diff i = DC i -DC i-1 DC 0 DC 1 DC i-1 DC i 0 block 1 block i-1 block i 02/16/

44 Encoding Quan0zed Coefficients The AC run lengths and DC coefficients are losslessly encoding using a Huffman coder The idea behind Huffman coding is that frequently occurring symbols use a small number of bits, while infrequently occurring symbols use large numbers of bits. 02/16/

45 Final Result of JPEG encoding The end result of all of these processes is a bit stream that contains the encoded image data and all of the informaeon needed to decode the data Quanezaeon tables, Huffman code tables, etc. 02/16/

46 Final Result of JPEG encoding 02/16/

47 JPEG Compression Ar0facts What kind of arefacts will JPEG compression produce? You can see 8x8 blocks if you magnify. 02/16/

48 02/16/

49 02/16/

50 JPEG Compression Ar0facts What kind of arefacts will JPEG compression produce? Not good for images containing text because edges of text may span blocks 02/16/

51 02/16/

52 Lossless JPEG Compression Standard Lossless JPEG There was concern that some imaging applicaeons would not use lossy compression Medical images Liability if radiologist misdiagnosed based on lossy compression arefacts Lossless algorithm does not use DCT DCT is invereble, but rounding could result in some mismatch of original image from reconstructed image 02/16/

53 Lossless JPEG Compression Standard Lossless coding: Subtraceon of adjacent pixels 8 prediceon models for x using various combinaeons of adjacent pixels C A B X For example, predict X from A+B-C, so encode: X [A+B-C] 02/16/

54 Lossless JPEG Compression Standard Lossless coding: Two types of encoders Huffman Arithmeec We talked about Huffman Arithmeec encoding is another entropy encoder Clever method that maps message into a single, binary fraceon between 0 and 1. Berer than Huffman coding, but probably everyone uses Huffman although actually, no one uses lossless JPEG! 02/16/

55 Wavelets About same eme as JPEG was started, a revolueon in frequency-domain analysis Wavelets 02/16/

56 Wavelets There is a new version of JPEG JPEG 2000 Uses the wavelet transform Technically superior, but very few people use it 02/16/

57 02/16/

58 Time-frequency transforms Another look at the Fourier Transform + jωt X( ω) = F [ x( t)] = x( t) e dt (Analysis Equation) jωt ( ) = F [ ( ω)] = ( ω) ω (Synthesis Equation) xt X X e d 2π Time Frequency Equivalent representation in terms of information in a signal In order to find content of signal at one frequency, you must integrate over all times. In order to find content of signal at one time, you must integrate over all frequencies.

59 Time-frequency transforms More general view of signal transforms: + X( ω) = x( t) ψ( ω, t) dt (Transform from t-domain to ω-domain) + xt ( ) = X( ωψ ) ( ω, td ) ω (Inverse transform from ω-domain to t-domain) Transformation kernel jωt 1 ψω (, t) = e ψω (, t) = e 2π + jωt Equivalently sines and cosines. Not localized in time.

60 Time-frequency transforms For interesting signals, frequency content changes with time. Music is a good example log(freq) So is an ECG stress test time

61 Time-frequency transforms If you take the Fourier transform of a signal, all time information is lost Waveform Time (Seconds)

62 Time-frequency transforms Information on both time and frequency What does time-frequency display look like? Spectrogram Frequency (HZ) Time (Seconds) SKC-2009

63 Time-frequency transforms How would you get this type of information? One way is the short-time Fourier transform (STFT) Windowing function: + iωt X( τω, ) = x( t) W( t τ) e dt Estimates the amplitude of the sinusoid with frequency ω around the time τ The windowing function should have local support. A good choice is the Gaussian (Gabor window) 1 W (t τ ) = e 2 2πσ ( t τ ) 2 2σ 2

64 Time-frequency transforms Gaussian window is optimal for time and frequency localization. Achieves the equality condition in time-frequency localization inequality: ( t)( ω) Δ Δ 1 2 But shortcoming is that that window size is fixed. You have to pick your resolution and stick with it.

65 Time-frequency transforms You can have good resolution in time or good resolution in frequency, but you can t have both. x( t) F F 0 good localization in time... t t o t t o or F good localization in freq. F 0 t o t

66 Can you do beqer than using a fixed window? Imagine many windows; with different resolutions in time and frequency Multiresolution analysis The wavelet transform decomposes a signal into representations at multiple resolutions using a basis function that is scaled and translated 1 t ψs, τ (t) ψ τ = s s

67 Wavelet Transform A bit of history Fourier Transform (~1807) Jean Baptiste Joseph Fourier Gabor Transform (STFT) (1946) Dennis Gabor Fast Fourier Transform (FFT) (1965) Cooley & Tukey Wavelets sort of crept in from many directions First wavelet was Haar function (1909), but not called that Math Physics (Quantum mechanics) Engineering (sub-band coding) All these disparate notions came together in mid 80 s and unified into what we know today as wavelets Term was coined by Jean Morlet (working for Elf-Aquitaine, oil company)

68 Wavelet Transform The wavelet transform (continuous case) 1 x τ X(s, τ) = x( t) ψ dt s s 1 1 x τ xt ( ) = X( s, τψ ) dτds 2 Cψ s s 1 2 Cψ = 2π ψ ( ξ) dξ ξ But, what is ψ? That is the great thing it can be almost any function as long as it is zero mean and square integrable: ( ) dξ = 0 ψ ξ

69 Wavelet Transform Wavelet transform does not have to be orthogonal, but from a signal processing perspective, it s nice if it is. What are some real wavelets. Wavelets that scale by factors of two and shift to cover interval in even steps: Orthogonal wavelet Discrete wavelet transform

70 Wavelet Transform The discrete wavelet transform for a time signal Forward transform cn ( ) = xt ( ) φ ( tdt ) k 2 k ( ) ( )2 ψ 2 Signal reconstruction n k ( ) d n = x t t n dt k 2 () ( ) φ( ) k ( )2 ψ 2 n n k k ( ) xt = cn t n + d n t n ψ is the wavelet, φ is the scaling (father wavelet) function

71 Wavelet Transform Also discrete wavelet transform (discrete sequence) Forward transform k + 1 k( ) = ( ) k(2 ) 0-2 m= y n x m h n m k M M 1 M 1( ) = ( ) M 1(2 ) m= y n x m h n m Signal reconstruction k + 1 k( ) = ( ) k(2 ) 0-2 m= y n x m h n m k M M 1 M 1( ) = ( ) M 1(2 ) m= y n x m h n m

72 Wavelet Transform What do some common wavelets look like? (Continuous) Meyer wavelet

73 Wavelet Transform What do some common wavelets look like? (Continuous) Morlet wavelet

74 Wavelet Transform What do some common wavelets look like? (Continuous) Mexican hat wavelet

75 Wavelet Transform What do some common wavelets look like? (Discrete) Daubechies (D4)

76 Wavelet Transform What do some common wavelets look like? (Discrete) Haar wavelet

77 Wavelet Transform Multiresolution analysis Scaling functions Signals can be represented as: xt () = aφ( t k) where φ is a scaling function k k At infinitesimal scale, φ is the δ-function Imagine a function space at a coarser resolution φ could be something like a pulse of a certain width that averages over the signal in that pulse In fact, this is the Haar wavelet scaling function

78 Wavelet Transform You can increase the resolution by changing the time scale xt () = aφ(2 t k) doubles the resolution. k k Another way to think of this as that the space itself has a certain resolution You can represent a coarse function in a low resolution space You can represent functions that have more detail in a higher resolution space, and so on. A space that contains a high resolution version of the signal would also contain those with lower resolution. A multiresolution representation of a signal would be it s representation in a nested set of subspaces

79 Mul0resolu0on analysis L V k V V V V V L L V k+ 1 V = {0}, V = L 2 2 φ k ( t n) k/2 kn, = 2 φ 2 are the basis functions of these spaces. For a signal represented at some level of resolution k, x ( t) V k ( ) x ( t) V x ( t) = a φ 2 t n k k k k n k k

80 Mul0resolu0on analysis Rather than creating a multiresolution analysis from the scaling functions, you can add the detail from the differences between the spaces. Call the orthogonal complement of V k in V k+1 W k The W s are wavelet spaces. Form the next higher resolution space from: V = V W V = V W W L V0 W0 W1 = L You don t necessarily have to start from V 0 You can start from any resolution level you wish: 2 L V10 W10 W11 or = L 2 L V 5 W 5 W 4 = L

81 Mul0resolu0on analysis Using the basis functions in these vector spaces spanned by the scaling basis functions and the wavelet basis functions, any square integrable function x(t) can be represented as: 2 L V0 W0 W1 xt = L 2 ( ) L (Square integrable functions) k 2 () ( ) φ( ) ( ) φ( ) k ( )2 ψ 2 n= n n k c( n) = x(t), φ ( t) = x( t) φ ( t) dt n d ( n) = x(t), ψ ( t) = x( t) ψ ( t) dt k k, n k, n n k ( ) xt = cn t n cn t n + d n t n

82 Mul0resolu0on analysis More generally, if you start at some other resolution level k 0 which sets the coarsest scale for the space spanned by φ k0,n (t) 2 L Vk0 Wk0 Wk0+ 1 xt 2 ( ) L (Square integrable functions) k k, n k, n k k, n k, n k 2 k kn, () t = 2 φ(2 t n) k 0 2 k0 2 0 k ( ) xt () = c ( n)2 φ(2 t n) + d( n)2 ψ 2t n k0 n= n k= k c ( n) = x(t), φ ( t) = x( t) φ ( t) dt d ( n) = x(t), ψ ( t) = x( t) ψ ( t) dt φ = L k 2, () 2 ψ 2 k ( ) ψ t = t n kn k k

83 Wavelet Transform Why is the wavelet analysis effective? For a large class of signals, the wavelet coefficients a k,n drop off rapidly with k and n. Means that by eliminating small ones, you make the representation of the signal sparser. Useful for compression, denoising, and feature extraction for classification Allows a more accurate local representation of signal. Wavelet coefficients correspond to local-in-time features at different scales Wavelet basis is adjustable and adaptable. There is not just one wavelet possible (unlike Fourier basis) Calculation of wavelet transform can be very efficient With orthogonal discrete wavelets, can be O(N) compared to O(NlogN) for FFT

84 Summary as to wavelets are so useful They can represent smooth functions They can represent discontinuities The basis functions are local. Good for nonstationary signals Nonlinear thresholding near optimal for denoising Truncation of coefficient vector is near optimal for data compression

85 Some wavelets Scales and locations

86 Some wavelets Daubechies Father & Mother wavelets

87 Wavelet transform coefficients 1024 samples only 32 non-vanishing WT coefs.

88 Frequency content

89 Frequency content

90 2-D Wavelets (s0ck figure at different levels)

91 2-D Wavelets (s0ck figure at different levels)

92 Progressive transmission of image

93 Progressive transmission of image

94 Progressive transmission of image

95 Progressive transmission of image Reconstructed from 5% of wavelet coefs.

96 Progressive transmission of image Reconstructed from 5% of wavelet coefs.