Fourier, Wavelets and Beyond: The Search for Good Bases for Images

Transcription

1 ICIP-04, Singapore, October Fourier, Wavelets and Beyond: The Search for Good Bases for Images Martin Vetterli EPFL & UC Berkeley 1. Introduction: Representations, Approximations and Compression 2. Fourier versus Wavelets 3. Wavelets and Approximation Theory 4. Wavelets and Compression 5. Going to Two Dimensions: Non-Separable Constructions 6. Beyond the Standard Model: Non-BL Signals, New Images, DSP Conclusions ICIP04 1

2 Acknowledgements Collaborations: Sponsors: NSF Switzerland T.Blu, EPFL M.Do, UIUC P.L.Dragotti, Imperial College M.Gastpar, UCBerkeley P.Marziliano, NTU Singapore I.Maravic, B. Beferull-Lozano, R.Shukla, V.Velisavljevic EPFL C.Weidmann, TRC Vienna Discussions and Interactions: R.Baraniuk, Rice A.Cohen, Paris VI I. Daubechies, Princeton R.DeVore, Carolina D. Donoho, Stanford V.Goyal, MIT J. Kovacevic, CMU S. Mallat, Polytech. & NYU M.Orchard, Rice M.Unser, EPFL ICIP04 2

3 Outline 1. Introduction: Representations, Approximations and Compression From Rainbows to Spectras Signal Representations Approximations Compression 2. Fourier versus Wavelets 3. Wavelets and Approximation Theory 4. Wavelets and Compression 5. Going to Two Dimensions: Non-Separable Constructions 6. Beyond the Standard Model: Non-BL Signals, New Images Conclusions and Outlook ICIP04 3

4 From Rainbows to Spectras Von Freiberg, 1304: Primary and secondary rainbow Newton and Goethe ICIP04 4

5 Signal Representations 1807: Fourier upsets the French Academy... f(t) t = + + Fourier Series: Harmonic series, frequency changes, f 0, 2f 0, 3f 0,... But : Gibbs paper 1899: Gibbs correction Orthogonality, convergence, complexity ICIP04 5

6 1910: Alfred Haar discovers the Haar wavelet dual to the Fourier construction f(t) time = Haar series: Scale changes S 0, 2S 0, 4S 0, 8S 0... orthogonality m = 1 m = 0 t t m = -1 t ICIP04 6

7 Theorem 1 (Shannon-48, Whittaker-35, Nyquist-28, Gabor-46) If a function f(t) contains no frequencies higher than W cps, it is completely determined by giving its ordinates at a series of points spaced 1/(2W) seconds apart. [if approx. T long, W wide, 2TW numbers specify the function] It is a representation theorem: {sinc(t-n)} n in Z, is an orthogonal basis for BL[-π,π] f(t) in BL[-π,π] can be written as ft) ( ) = fn ( ) sinct ( n) 1 10 n slow...! Note: Shannon-BW, BL sufficient, not necessary. many variations, non-uniform etc Kotelnikov-33, Raabe-38 ICIP04 7

8 Representations, Bases and Frames Ingredients: as set of vectors, or atoms, { ϕ n } an inner product, e.g. a series expansion ϕ n, f = ( ϕ n f ) ft () = ϕ n, f ϕ n () t Many possibilities: orthonormal bases (e.g. Fourier series, wavelet series) biorthogonal bases overcomplete systems or frames e e 1 1 = ϕ 1 ϕ ϕ 1 1 e 1 ϕ 0 n e 0 e 0 = ϕ 0 e 0 = ϕ 0 OB ϕ 1 BOB ~ ϕ 0 ϕ 2 UTF Note: no transforms, uncountable ICIP04 8

9 The linear approximation method Approximations, aproximation... Given an orthonormal basis {g n } for a space S and a signal f = n f, g n g n, the best linear approximation is given by the projection onto a fixed subspace of size M (independent of f!) The error (MSE) is thus fˆm = n J M 2 εˆm = f fˆ = fg, n n J M g n fg, n 2 Ex: Truncated Fourier series project onto first M vectors corresponding to largest expected inner products, typically LP ICIP04 9

10 The Karhunen-Loeve Transform: The Linear View Best Linear Approximation in an MSE sense: Vector processes., i.i.d.: Consider linear approximation in a basis Then: X = [ X 0, X 1,, X N 1 ] T EX [ i ] = 0 EX [ X T ] = R X Xˆ M M 1 = Xg, n g n M < N n = 0 Karhunen-Loeve transform (KLT): For 0<M<N, the expected squared error is minimized for the basis {g n } where g m are the eigenvectors of R X ordered in order of decreasing eigenvalues. Proof: eigenvector argument inductively. E[ εˆm ] = R X g n, g n Note: Karhunen-47, Loeve-48, Hotelling-33, PCA, KramerM-56, TC N 1 n = M ICIP04 10

11 Geometric intuition: Principal axes of distribution: Shapes: ellipsoids To first approximation, keep all coefficients above a threshold: λ i This can be used in many settings, classification, denoising, and compression (inverse waterfilling theorem) N ICIP04 11

12 Compression: How many bits for Mona Lisa? <=> {0,1} ICIP04 12

13 A few numbers... D.Gabor, September 1959 (Editorial IRE) "... the 20 bits per second which, the psychologists assure us, the human eye is capable of taking in,..." Index all pictures ever taken in the history of mankind 100 years bits Huffman code Mona Lisa index a few bits (Lena Y/N?, Mona Lisa...), what about R(D)... Search the Web! billion images online, or bits JPEG 186K... There is plenty of room at the bottom! JPEG2000 takes a few less, thanks to wavelets... Note: 2 (256x256x8) possible images (D.Field) Homework in Cover-Thomas, Kolmogorov, MDL, Occam etc (from a contemporary: 0 bits, I don t care for this modern stuff) ICIP04 13

14 Source Coding: some background Exchanging description complexity for distortion: rate-distortion theory [Shannon:58, Berger:71] known in few cases...like i.i.d. Gaussians (but tight: no better way!) distorsion N0σ (, 2 ) DR ( ) = σ 2 2 2R or -6dB/bit D(R) complexity typically: difficult, simple models, high complexity (e.g. VQ) high rate results, low rate often unknown ICIP04 14

15 Limitations of the Standard Models Splendeurs et misères de la fonction débit-distortion (after Balsac) Precise results beautiful (maybe too much for its own good) upper and lower bounds constructive Problems complexity: exponential in code length code construction: finding good codes is hard Paradox: Best codes used in practice are suboptimal (Effros) transform codes dominate the scene of real compression So: unlike in lossless compression, lossy compression uses IT in a limited way;) Audio/Image/Video: distortion measures? ICIP04 15

16 New image coding standard... JPEG old JPEG (DCT) -6 db/bit new JPEG (wavelets) ICIP04 16

17 The Swiss Army Knife Formula of Transform Coding [Goyal00] Model: iid vector process of size N, µ=0, R x, MSE, high rate vector quantizer, entropy code x α γ γ 1 β xˆ transform and scalar quantizers, entropy code y 1 i 1 i 1 y 1 α 1 β 1 x T y 2 α 2 i 2 γ bits γ 1 i 2 β 2 y 2 T 1 xˆ y N α N i N i N β N y N DR ( ) = 2 ---Σh( y i ) tr T 1 ( T 1 ) T N ( ) 2 12N 2 2R Trace min: orthonormal; differential entropy min: independence Gaussian case: coincide! but in general not... ICIP04 17

18 Representation, Approximation and Compression: Why does it matter anyway? Parsimonious or sparse representation of visual information is key in storage and transmission indexing, searching, classification, watermarking denoising, enhancing, resolution change But: it is also a fundamental question in information theory signal/image processing approximation theory vision research Successes of wavelets in image processing: compression (JPEG2000) denoising enhancement classification Thesis: Wavelet models play an important role Antithesis: Wavelets are just another fad! ICIP04 18

19 Outline 1. Introduction: Representations, Approximations and Compression 2. Fourier versus Wavelets Fourier and Local Fourier Transforms Wavelet transforms Piecewise Smooth Signal Representations 3. Wavelets and Approximation Theory 4. Wavelets and Compression 5. Going to Two Dimensions: Non-Separable Constructions 6. Beyond the Standard Model: Non-BL Signals, New Images Conclusions and Outlook ICIP04 19

20 A Tale of Two Representations: Fourier versus Wavelets Orthonormal Series Expansion f = α n ϕ n α n = ϕ n, f <ϕ n, ϕ m > = δ n m f 2 = α 2 n Z Time-Frequency Analysis and Uncertainty Principle Then ft () F( ω) 2 t = t 2 ft () dt 2 ω = ω 2 F ( ω ) dω 2 t 2 ω π -- 2 ω not arbitrarily sharp in time and frequency! t ICIP04 20

21 Local Fourier Basis? The Gabor or Short-time Fourier Transform ϕ mn, () t wt ( nt)e jmω 0 = t nt ( ) Time-frequency atoms localized at ( nt, mω 0 ) frequency time When T, ω 0 small enough ft () c F mn, ϕ mn, t Example: Spectrogram () where F mn, = <ϕ mn,, f> ICIP04 21

22 The Bad News... Balian-Low Theorem ϕ mn, is a short-time Fourier frame with critical sampling ( Tω 0 = 2π) then either 2 t = or 2 ω = or: there is no good local orthogonal Fourier basis! Example of a basis: block based Fourier series sinω ω T 0 T 2T t Note: consequence of BL Thm on OFDM, RIAA ICIP04 22

23 There exist good local cosine bases. Replace complex modulation ( ϕ mn, () t = wt ( nt) The Good News! e jmω 0 t where wt () is a power complementary window ) by appropriate cosine modulation cos π -- 2 m T t nt Wt ( nt ) = 1 n wt () 2 T T t Result: MP3! Many generalisations... ICIP04 23

24 Replace (shift, modulation) by (shift, scale) Another Good News! or Ψ mn, () t 2 m 2 Ψ t 2m n = nm, Z 2 m then there exist good localized orthonormal bases, or wavelet bases frequency time ICIP04 24

25 Examples of bases h (1) 1 0 h (2) h (3) g (3) Haar Daubechies, D 2 ICIP04 25

26 Wavelets and representation of piecewise smooth functions Goal: efficient representation of signals like: f(t) Scaling functions Noise time Wavelets where: Wavelet act as singularity detectors Scaling functions catch smooth parts Noise is circularly symmetric Note: Fourier gets all Gibbs-ed up! ICIP04 26

27 Key characteristics of wavelets and scaling functions Daubechies-88: Wavelets derived from filter banks, ortho-lp with Scaling function: N zeroes at π, φω ( ) = G e j ω 2i i = 1 Orthonormal wavelet family: Gz ( ) = ( 1+ z 1 ) N Rz ( ) ( ( )) ψ mn, () t = 2 m/2 ψ( 2 m t n) Scaling function and approximations scaling function ϕ() t spans polynomials up to degree N-1 n c n ϕ( t n) = k = 0, 1, N 1 Strang-Fix theorem: if φω ( ) has N zeros at multiples of 2π (but the origin), then { ϕ( t n) } n Z spans polynomials up to degree N-1 Two scale equation: ϕ() t smoothness: follows from N, α = 0,203 N = t k g n ϕ( 2t n) 2 n ICIP04 27

28 Lowpass filters and scaling functions reproduce polynomials Iterate of Daubechies L=4 lowpass filter reproduces linear ramp scaling function linear ramp Scaling functions catch trends in signals ICIP04 28

29 Wavelet approximations wavelet ψ has N zero moments, kills polynomials up to deg. N-1 wavelet of length L= 2N-1, or 2N-1 coeffs influenced by singularity at each scale, wavelet are singularity detectors, wavelet coefficients of smooth functions decays fast, e.g. f in c p,m << 0 ψ mn,, f = c2 m p Note: all this is in 1 dimension only, 2D is another story ICIP04 29

30 How about singularities? If we have a singularity of order n at the origin (0: Dirac, 1: Heaviside,...), the CWT transform behaves as Xa0 (, ) = c n a 1 n -- 2 large Amplitude Time Scale Time small In the orthogonal wavelet series: same behavior, but only L=2N-1 coefficients influenced at each scale! e.g. Dirac/Heaviside: behavior as 2 and 2 m 2, m<<0 2 m Wavelets catch and characterize singularities! ICIP04 30

31 Thus: a piecewise smooth signal expands as: x (1) X h 0.5 X h (2) (3) X h X g (3) lowpass catches trends, polynomials a singularity influences only L wavelets at each scale wavelet coefficients decay fast ICIP04 31

32 Outline 1. Introduction: Representations, Approximations and Compression 2. Fourier versus Wavelets 3. Wavelets and Approximation Theory Non-linear approximation Fourier versus wavelet, LA versus NLA 4. Wavelets and Compression 5. Going to Two Dimensions: Non-Separable Constructions 6.Beyond the Standard Model: Non-BL Signals, New Images Conclusions and Outlook ICIP04 32

33 From linear to non-linear approximation theory The non-linear approximation method Given an orthonormal basis {g n } for a space S and a signal f = n f, g n g n the best nonlinear approximation is given by the projection onto an adapted subspace of size M (dependent on f!), f M = n I M fg, n g n I M : f, g n n IM fg, m m IM set of M largest, The error (MSE) is thus 2 ε M = f f = n I M fg, n 2 and ε M εˆm. Difference: take the first M coeffs (linear) or take the largest M coeffs (non-linear) ICIP04 33

34 Nonlinear approximation This is a simple but nonlinear scheme Clearly, if A M (.) is the NL approximation scheme: A M ( x) + A M ( y) A M ( x+ y) This could be called adaptive subspace fitting From a compression point of view, you pay for the adaptivity in general, this will cost log bits which cannot be spent on coefficient representation anymore N k LA: pick a subspace a priori NLA pick after seeing the data ICIP04 34

35 Non-Linear Approximation Example Nonlinear approximation power depends on basis Example: f(t) cst 1/sqrt2 1 Two different bases for [0,1]: Fourier series { e j2πkt } k I Wavelet series: Haar wavelets Linear approximation in Fourier or wavelet bases εˆm 1 M Nonlinear approximation in a Fourier basis ε M 1 M Nonlinear approximation in a wavelet basis ε M 1 2 M t ICIP04 35

36 Fourier versus Wavelet bases, LA versus NLA N= 1024, M=64 Fourier (left): LA versus NLA does not matter Wavelets (right): NLA does orders of magnitude better! ICIP04 36

37 Nonlinear approximation theory and wavelets f(t) t Approximation results for piecewise smooth fcts between discontinuities, behavior by Sobolev or Besov regularity mk 1/2 k derivatives coeffs 2 ( ) when m«0 Besov spaces can be defined with wavelet bases. If then [DeVoreJL92]: = ( p ) 1 p < 0< p< 2 f Gp, fg, n ε M = om ( 1 2 p ) ICIP04 37

38 Approximation in Sobolev and Besov Spaces Linear Approximation, W s [0,N] Sobolev-s: uniformly smooth Fourier: εˆm = M 2s δ δ > 0 Wavelets: εˆm = M 2s δ δ > 0 Non-Linear Approximation Besov-s: smooth between a finite # of discontinuities Fourier: does not work, ε M = M 1 Wavelets: approximation power given by the smoothness! Key: effect of discontinuities limited, because wavelets are concentrated around discontinuties f(t) in W s (0,N) between finite # of discontinuities, then f(t) in B p (0,N) (wavelet of compact support) Then: ε M result can be refined to get = M ε M p M 2s δ 1 -- < s p = δ > 0 ICIP04 38

39 Smooth versus piecewise smooth functions: It depends on the basis and on the approximation method Linear Approximation Error for Smooth Functions Fourier Wavelet MSE x Number of retained coefficients (N) Nonlinear Approximation Error for Piecewise Smooth Functions Fourier Wavelet MSE s= 2, N= 2^16, D_3, 6 levels x ICIP04 39

40 Outline 1. Introduction: Representations, Approximations and Compression 2. Fourier versus Wavelets 3. Wavelets and Approximation Theory 4. Wavelets and Compression A small but instructive example piecewise polynomials and D(R) piecewise smooth and D(R) improved wavelet schemes 5. Going to Two Dimensions: Non-Separable Constructions 6. Beyond the Standard Model: Non-BL Signals, New Images Conclusions and Outlook ICIP04 40

41 4. Wavelets and Compression Compression is just one bit trickier than approximation... A small but instructive example: Assume x[n] = α δ[n-k], signal is of length N, k is U[0,N-1] and α is N(0,1). This is a Gaussian RV at location k ~N(0,1) Note: R x = I! k N Linear approximation: εˆm = M Non-linear approximation, M > 0: ε M = 0 ICIP04 41

42 Given budget R for block of size N: 1. Linear approximation and KLT: equal distribution of R/N bits DR ( ) = c σ 2 2 2R N ( ) This is the optimal linear approximation and compression! 2. Rate-distortion analysis [Weidmann:99] High rate case: Obvious scheme: pointer + quantizer DR ( ) = c σ 2 2 2R logn ( ) This is the R(D) behavior for R >> Log N Much better than linear approximation Low rate case: Hamming case solved, inc. multiple spikes: - there is a linear decay at low rates L 2 case: upper bounds that beat linear approx. ICIP04 42

43 Example 1: Binary, Hamming, 1 and k spikes 1 Single Spike in N = 2, 3, 4, 6, 10, 20 Positions 1 K = 16, 12, 8, 4, 2 Spikes in N = 32 Positions Hamming Distortion N=2 N=20 Hamming Distortion D/D max K=16 N=32 K= R/log(N) R/R max Example 2: Bernoulli-Gaussian 0 5 Bernoulli 0.11 x Gaussian Spike Blahut Arimoto Upper Bound Asymptote p=0.11 Distortion [MSE db] Rate [bits] ICIP04 43

44 Piecewise smooth functions: pieces are Lipschitz-α f(t) Scaling functions Noise time Wavelets The following D(R) behavior is reachable [CohenDGO:02]: DR ( ) c 1 R 2α c 3 R 2 c 4 = + R There are 2 modes: R 2α corresponding to the Lipschitz-α pieces R 2 c R corresponding to the discontinuities ICIP04 44

45 Rate-distortion bounds for piecewise polynomial functions D(R) behavior of nonlinear approximation with wavelets f(t) uniformly distributed random constant 1 t Consider the simplest case: Haar! Recall that ε M 2 M c j 2 j 2 and consider describing the significantcoefficients 2 j/2 ~ c j J j Choose a stepsize for a quantizer. Therefore number of scales J before coefs set to zero log( 1/ ) number of bits per coefficient log( 1/ ), thus R J 2 Distortion: number of scales times 2 J 2 J Thus D w ( R) C 3 R 2 c 2 = R ICIP04 45

46 Rate-distortion behavior using an oracle An oracle decides to optimally code a piecewise polynomial by allocating bits where needed : Consider the simplest case f(t), f^(t) f 1 f^ t Two approximation errors t : quantization of step location a : quantization of amplitude Rate allocation: R t versus R a Result: D p ( R) = C 1 2 R 2 ICIP04 46

47 Piecewise polynomial, with max degree N A. Nonlinear approximation with wavelets having N+1 zero moments D w ( R) = C w ( 1 + α C w R) 2 C w R B. Oracle-based method D p ( R) C p 2 C p = R ( ) Thus wavelets are a generic but suboptimal scheme oracle method asymptotically superior but dependent on the model Conclusion on compression of piecewise smooth functions: D(R) behavior has two modes: DR ( ) c 1 R 2α c 3 R 2 c 4 = + R 1/polynomial decay: cannot be (substantially) improved exponential mode: can be improved, important at low rates ICIP04 47

48 Can we improve wavelet compression? Key: Remove depencies accross scales: dynamic programming: Viterbi-like algorithm tree based algorithms: pruning and joining wavelet footprints: wavelet vector quantization Theorem [DragottiV:03]: Consider a piecewise smooth signal f(t), where pieces are Lipschitzα. There exists a piecewise polynomial p(t) with pieces of maximum degree α such that the residual r α () t = ft () pt () is uniformly Lipschitz-α. This is a generic split into piecewise polynomial and smooth residual = + function f(t) piecewise polynomial Lipschitz-α ICIP04 48

49 Footprint Basis and Frames Suboptimality of wavelets for piecewise polynomials is due to independent coding of dependent wavelet coefficients D w ( R) C R 2 R Compression with wavelet footprints Theorem: [DragottiV:03] Given a bounded piecewise polynomial of deg D with K discontinuities. Then,a footprint based coder achieves DR ( ) c 1 2 c 2 = R ( ) This is a computational effective method to get oracle performance What is more, the generic split piecewise smooth into uniformly smooth + piecewise polynomial allows to fix wavelet scenarios, to obtain R DR ( ) c 1 R 2α c 2 2 c 3 = + This can be used for denoising and superresolution ICIP04 49

50 Outline 1. Introduction: Representations, Approximations and Compression 2. Fourier versus Wavelets 3. Wavelets and Approximation Theory 4. Wavelets and Compression 5. Going to Two Dimensions: Non-Separable Constructions the need for truly two-dimensional constructions directional wavelet transforms tree based methods non-separable bases and frames: contourlets 6. Beyond the Standard Model: Non-BL Signals, New Images Conclusions and Outlook ICIP04 50

51 5. Going to Two Dimensions: Non-Separable Constructions Going to two dimensions requires non-separable bases Objects in two dimensions we are interested in texture smooth boundary c 2 smooth surface, polynomial textures: DR ( ) = C 0 2 2R per pixel smooth surfaces: DR ( ) = C 1 2 2R per object! ICIP04 51

52 Models of the world: Gauss-Markov Piecewise polynomial the usual suspect Many proposed models: mathematical difficulties one size fits all... Lena is not PC, but is she BV? But: Fourier, DCT, wavelets use a separable approach (line/column...) => geometry based image processing ICIP04 52

53 Recent work on geometric image processing Long history: compression, vision, filter banks Current affairs: Signal adapted schemes Bandelets [LePennec & Mallat]: wavelet expansions centered at at discontinuity as well as along smooth edges Non-linear tilings [Cohen, Mattei]: adaptive segmentation Tree structured approaches [Shukla et al, Baraniuk et al] Bases and Frames Wedgelets [Donoho]: Basic element is a wedge Ridgelets [Candes, Donoho]: Basic element is a ridge Curvelets [Candes, Donoho] Scaling law: width ~length 2 L(R 2 ) set up Multidirectional pyramids and contourlets [Do et al] Discrete-space set-up, l(z 2 ) Tight frame with small redundancy, Computational framework Directional wavelet transforms and frames [Velisavljevic et al] This is where the action is! ICIP04 53

54 Directional Wavelet Transforms: Compressing Mondrians! A class of images for which 2D wavelet transforms work well Size N by N K lines N K pixels two approaches: usual and fully separable wavelet transform K N ~ ~ K --- ( log N 2 2 ) 2 ICIP04 54

55 Analysis of Performance [Velisavljevic et al 04] Class: Mondrian ~ piecewise constant with straight edges along α,β Image: NxN K edges with average length O(N) Wavelet transform: Haar with J~log 2 N stages Number of nonzero coefficients: Standard wavelet transform: ~KN Fully separable wavelet transform: ~K/2(log 2 N) 2 Mean-square error Standard wavelet transform: e(m)=o(1/m) Fully separable wavelet transform: e(m)=o(2 -αm ) This can be generalized to many directions, with segmentation ICIP04 55

56 Nonseparable schemes and approximation Approximation properties: wavelets good for point singularities ridgelets good for ridges curvelets good for curves Consider c 2 boundary between two csts # wavelet coeffs O(2 j ) # curvelet coeffs O(2 j/2 ) W l t Rate of approximation, M-term NLA in bases, c 2 boundary Fourier: O(M -1/2 ) Wavelets: O(M -1 ) Curvelets: O(M -2 ) Xl t ICIP04 56

57 Compression of non-separable objects Objects we know how to compress... Basis element Approximation Wavelets Ridgelets E M M 1 2 M E M Rate/distortion Oracle Wavelets...poor DR ( ) = C 2 2R Ridgelets...suboptimal adaptive schemes: close to oracle fixed basis: under investigation ICIP04 57

58 Tree Based Geometric Compression [ShuklaDDV:03] Idea tree and quadtree algorithms popular, many pruning algorithms optimality proofs for wedgelets [Donoho:99] new pruning and joining algorithm Intuition: full tree dyadic tree pruned & joined tree (0,0) (0,0) (1,0) (1,1) (1,0) (1,1) (2,2) (2,3) (2,2) (2,3) (3,4) (3,5) (4,8) (4,9) Joint Encoding (3,4) (3,5) (4,8)(4,9) Joint Encoding N J R DR ( ) R 2 c 1 2 J DR ( ) R 1 N J J N J J 0 DR ( ) 2 c 2 R Results: Rate-distortion optimal for piecewise polynomials DR ( ) c 1 2 c 2 = R ( ) that is, like an oracle method (up to constants) ICIP04 58

59 Extension to Quadtree: Example Results: consider a piecewise polynomial 2D signal, with polynomial boundaries, the following rate-distortion behavior is achieved DR ( ) c 3 2 c 4 = R ( ) this is like an oracle method, and >> than prune algorithms which have a R penalty complexity: polynomial ICIP04 59

60 The prune-join quadtree algorithm polynomial fit to surface and to boundary on a quadtree rate-distortion optimal tree pruning and joining quadtree with R(D) pruning Note: careful R(D) optimization! R(D) Joining of similar leaves ICIP04 60

61 Geometric Compression versus JPEG2000 at 0.11 bits/pixel, PSNR: pruned-joined quadtree JPEG2000 ICIP04 61

62 Behavior of tree algorithms on piecewise smooth fcts ppf: piecewise polynomial functions psf: piecewise smooth functions Signal Class Oracle Coder Wavelet Coder Prune tree Coder Prune-join tree Coder 1-D PPF 2-D PPF 1-D PSF 2-D PSF 2 cr 2 c 1 R 2 c 2 R 2 c3r logr R 2 dr 2 c 4 R 2 c 5 R R 2p R 2p logr R logr p logr R R log R R p R p logr R 2p p at most log penalty with polynomial complexity (and a bit more work gets rid of logs...) Interesting scaling laws, good behavior in practice! ICIP04 62

63 Directional bases and contourlets [M.Do] Goal: find a discrete-space construction that has good approximation properties for smooth functions with smooth boundaries directional analysis as in a Radon transform multiresolution as in wavelets and pyramids computationally easy bases or low redundancy frames Background: curvelets [Candes-Donoho] indicate that good fixed bases do exist for approximation of piecewise smooth 2D functions a frequency-direction relationship indicates a scaling law d j 1 2 directional filter banks are available [BambergerS, DoV] Idea: directional analysis: directions are key multiresolution analysis Result: one-more-let: contourlets! ICIP04 63

64 Example of directional basis functions 64 channels, elementary filters are Haar filters orthonormal directional basis 64 equivalent filters, the 32 mostly horizontal ones are shown This ressembles a local Radon transform, or radonlets! changes of sign (for orthonormality) approximate lines (discretizations) ICIP04 64

65 Multiresolution directional pyramid 2,2 bandpass directional channels image bandpass directional channels Result: tight pyramid and orthogonal directional channels => tight frame low redundancy < 4/3, computationally efficient, ICIP04 65

66 Basis functions: wavelets versus contourlets ICIP04 66

67 Multiresolution directional pyramid: Expansion example Pepper image and its expansion Compression, denoising, inverse problems: if it is sparse, it is going to work! ICIP04 67

68 Example: denoising with contourlets original noisy wavelet contourlets 13.8 db 15.4 db ICIP04 68

69 Outline 1. Introduction: Representations, Approximations and Compression 2. Fourier and Wavelet Representations 3. Wavelets and Approximation Theory 4. Wavelets and Compression 5. Going to Two Dimensions: Non-Separable Constructions 6. Beyond the Standard Model: Non-BL Signals, New Images, DSP... Images are not bandlimited: Variation on a theme by Shannon, Sampling FRI signals Images are on manifolds: The plenoptic function Interactive Communication: Distributed schemes 7. Conclusions and Outlook ICIP04 69

70 Images are not bandlimited! Example: Edges: 1/ω decay of Fourier transform Yet: all of image processing is based on Shannon sampling... or blurred pictures! Can t we do better? A theory for sampling of non-bandlimited signals ICIP04 70

71 A Variation on a Theme by Shannon Shannon, BL case: ft () = fnt ( ) sinct ( T n) or 1/T degrees of freedom per unit of time n Z But: a single discontinuity, and no more sampling theorem... f(t) F (ω) F t ω Signals with finite number of degrees of freedom per unit of time allowing exact sampling results? => Finite rate of innovation Usual setup: x() t : signal, ht ():sampling kernel, yt ():filtering of x() t and y n :samples ICIP04 71

72 A Representation Theorem [VMB:02] For the class of periodic FRI signals which includes sequences of Diracs non-uniform or free knot splines piecewise polynomials there are sampling schemes with sampling rate at the rate of innovation which allow perfect reconstruction at polynomial complexity xt () ht () yt ()y, n x Method: Yule-Walker system, annihilating filter, Vandermonde system (related to spectral estimation, ECC) Extensions: 2D, noisy, local ICIP04 72

73 A local algorithm for FRI sampling The return of Strang-Fix: use scaling function with poly. reproduction a 0 (t t 0 ) a 0 (t t 0 ) a 1 (t t 1 ) a 1 (t t 1 ) 0 t 0 t n y n = a 0 + a 1 n ny n = a 0 t 0 + a 1 t 1 a 0 (t t 0 ) a 0 (t t 0 ) a 1 (t t 1 ) a 1 (t t 1 ) 0 t n n2 y n = a 0 t a 1t t n n3 y n = a 0 t a 1t 3 1 local reconstruction with polynomial complexity for diracs and piecewise polynomials ICIP04 73

74 Images are on a manifold: the plenoptic function A piece of the manifold... Define the plenoptic function [Adelson, Bergen:90] all possible pictures, e.g. in a room f(x,y,z,α,θ,λ,i) high dimensional function, mostly smooth... studied in computer vision/graphics, video processing ICIP04 74

75 On Plenoptic Sampling Model Plenoptic function is it bandlimited? (no... FM modulation!) how to approximate it, how to sample non-bl cases Plenoptic sampling how many cameras are required to reconstruct a view from any point? this is a sampling and interpolation problem approximate solutions [Shum et al, T.Chen et al] ICIP04 75

76 An Example: Super-resolution... take many pictures...make a higher resolution one sampling with unknown location image is non-bandlimited, or aliased result might be non-bandlimited This can be cast as a problem of sparse representation in a frame computationally difficult... example where wavelet thinking opens new possibilities ICIP04 76

77 X 1.. T 1 T 2 T 3 Interactive communication: distributed schemes What happens if the data is distributed? - 1 -^X... - X M1 encoder ^X M1 - ^X M1+1 M1+1 - X reconstruction - X M1+M2 encoder ^X M1+M2 - X M1+M2+1 - ^X M1+M X N encoder 3 - -^X N This is a common situation: distributed image data base sensor networks multi-camera systems Distributed compression: classic but hard IT problem Slepian-Wolf coding (lossless case: no penalty) Wyner-Ziv (lossless side information) lossy distributed case: mostly open, high rate understood ICIP04 77

78 X1. X N -^X N X1.. X N -^X N Distributed Karhunen-Loeve Transform -^X1. -^X T T X M encoder - - X M encoder - -^X M - ^X M+1 -^X M - ^X M+1 X M+1 M+1 - X. struction reconstruction recon Two special cases: partial KLT conditional KLT - - The solutions differ from the marginal KLT (which ignores X M+1...X N ) Both approximation and compression are studied Gaussian case: Partial KLT: minimizes total reconstruction error Conditional KLT: uses conditional independence Fully distributed KLT: iterative solution Application example two way communication <-> conditional KLT ICIP04 78

79 Conclusions Wavelets and the French revolution too early to say? from smooth to piecewise smooth functions Sparsity and the Art of Motorcycle Maintenance sparsity as a key feature with many applications denoising, inverse problems, compression LA versus NLA: approximation rates can be vastly different! To first order, operational, high rate, D(R) improvements still possible low rate analysis difficult Two-dimensions: really harder! and none used in JPEG approximation starts to be understood, compression mostly open Shift invariant subspaces: well understood ICIP04 79

80 Why image representation remains a fascinating topic... A lone student standing in front of four tanks The woman with the enigmatic smile The lightly clad lady with a hat A 3 year old saluting at his father s funeral just picture this in your head! ICIP04 80

81 Thank you for your attention! the plenoptic function from my window... Questions? ICIP04 81

82 Publications For overviews: D.Donoho, M.Vetterli, R.DeVore and I.Daubechies, Data Compression and Harmonic Analysis, IEEE Tr. on IT, Oct M. Vetterli, Wavelets, approximation and compression, IEEE Signal Processing Magazine, Sept Coming up: M.Vetterli, J.Kovacevic and V.Goyal, Fourier and Wavelets: Theory, Algorithms and Applications, Prentice-Hall, 200X ;) For more details, Theses, on line at C.Weidmann, Oligoquantization in low-rate lossy source coding, PhD Thesis, EPFL, M. N. Do, Directional Multiresolution Image Representations, Ph.D. Thesis, EPFL, P. L. Dragotti, Wavelet Footprints and Frames for Signal Processing and Communications, PhD Thesis, EPFL, R.Shukla, Rate-distortion optimized geometrical image processing, PhD Thesis, EPFL, ICIP04 82

83 Papers: A.Cohen, I.Daubechies, O.Gulieruz and M.Orchard, On the importance of combining wavelet-based non-linear approximation with coding strategies, IEEE Tr. on IT, 2002 P. L. Dragotti, M. Vetterli. Wavelets footprints: theory, algorithms and applications, IEEE Transactions on Signal Processing, May R. Shukla, P. L. Dragotti, M. N. Do and M. Vetterli, Rate-distortion optimized tree structured compression algorithms for piecewise smooth images, IEEE Transactions Image Processing, M.N.Do and M.Vetterli, Contourlets: A computational framework for directional multiresolution image representation, submitted, C.Weidmann, M.Vetterli, Rate-distortion behavior of sparse sources, IEEE Tr. on IT, under revision. M. Vetterli, P. Marziliano and T. Blu, Sampling signals with finite rate of innovation, IEEE Transactions on SP, June I. Maravic and M. Vetterli, Sampling and reconstruction of signals with finite rate of innovation in the presence of noise, IEEE Transactions on SP, 2004, to appear. M.Gastpar, P.L.Dragotti and M.Vetterli, The distributed Karhunen-Loeve transform, IEEE Tr. on IT, submitted, ICIP04 83