Multiresolution image processing

Multiresolution image processing Laplacian pyramids Some applications of Laplacian pyramids Discrete Wavelet Transform (DWT) Wavelet theory Wavelet image compression Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 1

Image pyramids [Burt, Adelson, 1983] Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 2

Image pyramid example original image Gaussian pyramid Laplacian pyramid Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 3

Overcomplete representation Number of samples in Laplacian or Gaussian pyramid = 1 1 1 4 1 + + +... + x number of original image samples 2 P 4 4 4 3 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 4

LoG vs. DoG Laplacian of Gaussian Difference of Gaussians Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 5

Image processing with Laplacian pyramid Input picture + Processed picture - Analysis Filtering Interpolator Interpolator Synthesis Subsampling + - Filtering Subsampling Interpolator Interpolator Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 6

Expanded Laplacian pyramid Input picture + Processed picture - Analysis Filtering Synthesis Filtering + - Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 7

Mosaicing in the image domain Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 8

Mosaicing by blending Laplacian pyramids Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 9

Expanded Laplacian pyramids Original: begonia Gaussian level 2 Laplacian level 2 Original: dahlia Gaussian level 2 Laplacian level 2 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 10

Blending Laplacian pyramids Level 0 Level 2 Level 4 Level 6 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 11

Image analysis with Laplacian pyramid Input picture + - Analysis Filtering Interpolator Subsampling Filtering + - Recognition/ detection/ segmentation result Subsampling Interpolator Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 12

Multiscale edge detection Zero-crossings of Laplacian images of different scales Spurious edges removed Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 13

Multiscale face detection Input image pyramid Extracted window (20 by 20 pixels) Correct lighting Histogram equalization Receptive fields Hidden units subsampling Network Input Output 20 by 20 pixels Preprocessing Neural network [Rowley, Baluja, Kanade, 1995] Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 14

Example: multiresolution noise reduction Original Noisy Noise reduction: 3-level Haar transform no subsampling w/ soft coring Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 15

1-d Discrete Wavelet Transform Recursive application of a two-band filter bank to the lowpass band of the previous stage yields octave band splitting: frequency Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 16

Cascaded analysis / synthesis filterbanks h 0 g 0 h 1 g 1 h 0 g 0 h 1 g 1 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 17

2-d Discrete Wavelet Transform ω y ω y ω y ω x ω x ω x ω y ω y ω x ω x ω y ω y...etc ω x ω x ω y ω y ω x ω x Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 18

2-d Discrete Wavelet Transform example Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 19

2-d Discrete Wavelet Transform example Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 20

2-d Discrete Wavelet Transform example Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 21

2-d Discrete Wavelet Transform example Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 22

2-d Discrete Wavelet Transform example Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 23

Review: Z-transform and subsampling Generalization of the discrete-time Fourier transform n ( ) [ ] ; C ; x z = x n z z r < z < r n= + Fourier transform on unit circle: substitute Downsampling and upsampling by factor 2 z = j e ω x( z ) 2 2 1 ( ) ( ) 2 x z + x z Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 24

Two-channel filterbank x( z) h 0 2 2 g 0 ˆx ( z) h 1 2 2 g 1 1 1 xz ˆ( ) = g0( z) h0( z) x( z) + h0( z) x( z) + g1( z) h1( zxz ) ( ) + h1( zx ) ( z) 2 2 1 1 = 0 0 + 1 1 + 0 0 + 1 1 2 2 Aliasing cancellation if : [ ] [ ] [ h ( z) g ( z) h( z) g ( z) ] x( z) [ h ( z) g ( z) h( z) g ( z) ] x( z) g ( z) = h( z) 0 1 g ( z) = h ( z) 1 0 Aliasing Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 25

Example: two-channel filter bank with perfect reconstruction Impulse responses, analysis filters: Lowpass highpass 1 1 3 1 1,,,, 4 2 2 2 4 1 1 1,, 4 2 4 Impulse responses, synthesis filters Lowpass highpass 1 1 1,, 4 2 4 1 1 3 1 1,,,, 4 2 2 2 4 Biorthogonal 5/3 filters LeGall filters Mandatory in JPEG2000 Frequency responses: Frequency response 2 1 0 0 h 0 g 0 π 2 Frequency g 1 h 1 π Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 26

Lifting Analysis filters [ n] even samples x 2 Σ Σ K 0 low band y 0 λ λ λ 1 2 L 1 λ L odd samples [ 2 1] x n+ Σ Σ K 1 high band y 1 L lifting steps [Sweldens 1996] First step can be interpreted as prediction of odd samples from the even samples Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 27

Lifting (cont.) Synthesis filters [ n] even samples x 2 Σ - Σ - 1 K 0 low band y 0 odd samples [ 2 1] x n+ λ λ λ 1 2 L 1 Σ - Σ - λ L 1 K 1 high band y 1 Perfect reconstruction (biorthogonality) is directly build into lifting structure Powerful for both implementation and filter/wavelet design Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 28

Example: lifting implementation of 5/3 filters [ n] even samples x 2 Σ low band y 0 ( 1 z) + 1 2 1 + 4 z odd samples [ 2 1] x n+ Σ 1/2 high band y 1 Verify by considering response to unit impulse in even and odd input channel. Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 29

Conjugate quadrature filters Achieve aliasing cancelation by 1 0( ) 0( ) Impulse responses h k = g k = f k With perfect reconstruction: orthonormal subband transform! Perfect reconstruction: find power complementary prototype filter ( ) h z = g z f z 1 ( ) ( 1 ( ) ) h z = g z = zf z [Smith, Barnwell, 1986] 1 1 [ ] [ ] [ ] k + 1 [ ] = [ ] = ( ) ( + ) 0 0 h1 k g1 k 1 f k 1 2 ( ) 2 ( jω ) j( ) F e ω ± π F e + = 2 Prototype filter Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 30

Wavelet bases ( ) ( ) x = xt ( ) 2 Consider Hilbert space of finite-energy functions. Wavelet basis for that span L 2 ψ L : family of linearly independent functions ( m ) m () 2 ( m 2 ) n t = ψ t n 2 2 L ( ). Hence any signal x L ( ) x = m= n= y ( m ) [ ] n ψ ( m) n mother wavelet can be written as Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 31

Multi-resolution analysis Nested subspaces ( ) ( ) ( ) ( ) ( ) ( ) 2 1 0 1 2 2 V V V V V L Upward completeness Downward complete ness m Z m Z () ( m) ( m) ( 0) 2 {} 0 ( ) m Self-similari t y x t V iff x 2 t V Translation invariance () V V = L = ( ) ( m) ( ) ( ) ( ) 0 0 x t V iff x t n V for all n Z { ϕ } () t () t = ( t n) There exists a "scaling function" ϕ with integer translates ϕ ϕ - such that forms an orthonormal basis for V n n ( 0) n Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 32

Multiresolution Fourier analysis span span span ( p 3) ( p 3) { ϕ n } = V ( p 2) ( p 2) { ϕ n } = V ( p 1) ( p 1) { ϕ n } = V ( p) ( p) span{ ϕ } = V n n n n n span span ( p 1) ( p 1) { ψ n } = W n ( p 2) ( p 2) { ψ n } = W n span ( p 3) ( p 3) { ψ n } = W n ( p) ( p) span{ ψ } = W n n ω Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 33

Since V Relation to subband filters ( 0) ( 1) V Orthonormality ( 1 () [ ] ) () ( 0) ( 0) [ n ] =, 0, recursive definition of scaling function linear combination ( 1 of scaling functions in V ) [ ] ( ) ϕ t = g0 n ϕn t = 2 g0 n ϕ 2t n n= n= δ ϕ ϕ n ( 1 [] ) ( 1) = g0 i ϕi () t g0[ j] ϕj+ 2n() t dt i j ( 1) ( 1) = g0[] i g0[ j 2 n] ϕi, ϕj = g0[] i g0[ i 2n] i, j i g0 [ k ] unit norm and orthogonal to its 2-translates: corresponds to synthesis lowpass filter of orthonormal subband transform Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 34

Wavelets from scaling functions ( p) ( p) ( p 1) W is orthogonal complement of V in V ( p) ( p) ( p) ( p) ( p 1) W V and W V = V Orthonormal wavelet basis for ψ ( 1 () t g [ n] ϕ ) () t = 1 n n= linear combination of scaling functions ( 1 in V ) n+ 1 [ ] = ( 1) ( 1) ( 0) ( 0) ( 1) { ψ n } W V 1 [ ] ϕ ( ) = 2 g n 2t n Using conjugate quadrature high-pass synthesis filter g1 n g0 n The mutually orthonormal n= ( ) ( ) { ψ n } ϕn ( ) { } V 0 0 1 functions, and, together span. n Z n Z () Easy to extend to dilated versions of ψ t to construct orthonormal wavelet basis ( m) 2 { ψ } L ( ) n nm, Z for. n Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 35

Calculating wavelet coefficients for a continuous signal Signal synthesis by discrete filter bank Suppose continuous signal ( ) () ( ) ( ) () () () [] ( ) [ ] ϕ( ) () () () 0 1 1 1 1 = 0 ϕn + 1 ψn i Z j Z () 1 () 1 x () t V () 1 () 1 w () t W Signal analysis by analysis filters h 0 [k], h 1 [k] Discrete wavelet transform () () [ ] () () ( ) [ ] ( ) ( ) 0 0 0 0 0 = 0 = 0 ϕn n Z n Z 1 1 1 1 x t y n t n y n V Write as superposition of x t V and w t W x t y i y j ( 0) ( 1 ) () 1 = ϕn y0 [ n] g0[ n 2 i] + y1 [ j] g1[ n 2i] n Z i Z j Z y ( 0 ) [ n] 0 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 36

1-d Discrete Wavelet Transform y () 1 [ n] 0 h 0 y ( 2 ) [ n] 0 g 0 y ( 1 ) [ n] 0 h 1 y ( 2 ) [ n] 1 g 1 xt () Sampling y ( 0 ) [ n] 0 h 0 g 0 y ( 0 ) [ n] 0 Interpolation ϕ ( t) xt ( ) h 1 y ( 1 ) [ n] 1 g 1 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 37

Example: Daubechies wavelet, order 2 ϕ ψ h 0 h 1 g 0 g 1 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 38

Example: Daubechies wavelet, order 9 ϕ ψ h 0 h 1 g 0 g 1 Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 39

Comparison JPEG vs. JPEG2000 Lenna, 256x256 RGB Baseline JPEG: 4572 bytes Lenna, 256x256 RGB JPEG-2000: 4572 bytes Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 40

Comparison JPEG vs. JPEG2000 JPEG with optimized Huffman tables 8268 bytes JPEG2000 8192 bytes Bernd Girod: EE368 Digital Image Processing Multiresolution Image Processing no. 41