Image representation with multi-scale gradients

Transcription

1 Image representation with multi-scale gradients Eero P Simoncelli Center for Neural Science, and Courant Institute of Mathematical Sciences New York University

2 Visual image representation Clean edges are rare One scale s texture is another scale s edge Need seamless transitions from isolated features to dense textures Sparseness? explicit geometric quantities

3 Sparseness Seems intuitively appealing, but in practice: Hard to compute Unlikely to benefit compression (must code addresses!) Discontinuous under geometric distortion of image Relevance to neuroscience is questionable

4 Dissent (at the Republican National Convention)

5 Multi-scale gradient basis Multi-scale bases: efficient representation Derivatives: good for analysis Explicit geometry (orientation) Local Taylor expansion of image structures Combination: Explicit incorporation of geometry in basis Bridge between PDE / harmonic analysis approaches

6 Steerable pyramid Basis functions are Kth derivative operators, related by translation/dilation/rotation Tight frame (4(K-1)/3 overcomplete) Translation-invariance, rotation-invariance [Simoncelli et.al., 1992; Simoncelli & Freeman 1995]

7 First derivative of radial function: k+1 kth-order directional derivatives: [Freeman & Adelson, 91]

8

9 Filters Polar-separable: B k (r, θ) = H(r)G k (θ), k [0, K 1], H(r) = G k (θ) = cos π 2 log 2r 2 π, π 4 < r < π 2 1, r π 2 0, r π 4 α K cos(θ πk K ) K 1, θ πk K < π 2 0, otherwise,

10 Recursive computation H 0 (- ) H 0 ( ) y L 0 (- ) B 0 (- ) B 0 ( ) L 0 ( ) B 1 (- ) B 1 ( ) x B K (- ) B K ( ) L 1 (- ) 2 2 L 1 ( )

11 Example decomposition (3rd derivatives)

12 Bayes denoising Additive Gaussian noise: y = x + w P (y x) exp[ (y x) 2 /2σ 2 w] Bayes least squares solution is conditional mean: ˆx(y) = IE(x y) = dxp(y x)p(x)x/p(y)

13 Denoising: classical Assume signal is Gaussian Then estimator is linear: IE( x y) = C x (C x + C w ) 1 y

14 Wavelet statistics: Marginal #" " -*./01.*,6'.) :..'41,;*1.')5,, '(')5 #"!% #"!$!!"" "!"" &'()*+,-*./01.* [Field 87; Mallat 89; Daugman 89; etc]

15 Denoising: shrinkage Assume marginal distribution [Mallat, 89] P (x) exp x/s p Then estimator is generally nonlinear: p = 2.0 p = 1.0 p = 0.5 [Simoncelli & Adelson, 96]

16 Statistics: Joint [Simoncelli, 97]

17 GSM model Model generalized neighborhood of coefficients as a Gaussian Scale Mixture (GSM) [Andrews & Mallows 74]: x = z u, where - z and u are independent - x z is Gaussian, with covariance zc u - marginals are always leptokurtotic [Wainwright & Simoncelli, 99] IPAM, 9/04 16

18 GSM - prior on z Empirically, z is approximately lognormal Alternatively, can use Jeffrey s noninformative prior: P (z) 1/z

19 Simulation #"! Image data #"! GSM simulation #" "!!" "!" #" "!!" "!"

20 Denoising: Joint IE(x y) = dz P(z y) IE(x y, z) = dz P(z y) zcu (zc u + C w ) 1 y ctr where P(z y) = P( y z) P(z) P y, P( y z) = exp( yt (zc u + C w ) 1 y/2) (2π) N zc u + C w Numerical computation of solution is reasonably efficient if one jointly diagonalizes C u and C w... [Portilla, Strela, Wainwright, Simoncelli, 03] IPAM, 9/04 20

21 Example estimators ESTIMATED COEFF.! w NOISY COEFF. $%&'()*%+,,- +'1&2/1+3)*%+,,-!" "!!" #" " $%&'()./0+$1!#"!!" "!" Estimators for the scalar and single-neighbor cases

22 Comparison to other methods "'& "'& /456,(74-.)/-0123 "!"'&!!!!'&!#!#'&!$ "!"'&!!!!'&!#!#'&!$, :;<= :>6965#8*>?6<!$'&!" #" $" %" &" ()*+,-*.)/-0123!$'&!" #" $" %" &" ()*+,-*.)/-0123 Results averaged over 3 images

23 Original Noisy (22.1 db) Matlab s wiener2 (28 db) BLS-GSM (30.5 db)

24 Original Noisy (8.1 db) UndecWvlt HardThresh (19.0 db) BLS-GSM (21.2 db)

25 Real sensor noise 400 ISO denoised

26 GSM summary GSM captures local variance Excellent denoising results What s missing? Underlying Gaussian leads to simple computation Joint model of z variables [Wainwright etal 99; Romberg etal 99; Hyvarinen/Hoyer 02; Karklin/Lewicki 02; etc.] Explicit geometry...

27 Reminder:

28 Local orientation Lines normal to gradient, length proportional to magnitude

29 Importance of local orientation Randomized orientation Randomized magnitude Two-band, 6-level steerable pyramid

30 orientation magnitude orientation

31 Reconstruction from orientation Original Quantized to 2 bits Converges: projections onto convex sets Resilient to quantization Highly redundant, across both spatial position and scale [with David Hammond]

32 Spatial redundancy x?? y y Relative orientation histograms, at different locations See also: Geisler, Elder [with x Patrik Hoyer & Shani Offen]

33 Scale redundancy [with Clementine Marcovici]

34 Cast Local GSM model: Martin Wainwright, Javier Portilla, Vasily Strela Denoising: Javier Portilla, Martin Wainwright, Vasily Strela GSM tree model: Martin Wainwright, Alan Willsky Compression: Robert Buccigrossi Texture representation/synthesis: Javier Portilla Local phase: Zhou Wang Orientation: David Hammond, Clementine Marcovici

35