Part III Super-Resolution with Sparsity

Transcription

1 Aisenstadt Chair Course CRM September 2009 Part III Super-Resolution with Sparsity Stéphane Mallat Centre de Mathématiques Appliquées Ecole Polytechnique

2 Super-Resolution with Sparsity Dream: recover high-resolution data from lowresolution noisy measurements: Medical imaging Satellite imaging Seismic exploration High Definition Television or Camera Phones Can we improve the signal resolution? Sparsity as a tool to incorporate prior information.

3 Inverse Problems Measure a noisy and low resolution signal: Y = Uf + W with f C N and dim(imu) =Q < N. Inverse problems: compute an estimation and minimize the risk: F = DY r(d, f) =E{ F f 2 } Super-resolution estimation: is computed in a space of dimension Is it possible, how?! "# F

4 Regularized Inversion To estimate f from invert U! Y = Uf + W (NullU) U Ũ 1 ImU Pseudo inverse: Ũ 1 Uf = f if f (NullU) Ũ 1 g = 0 if g (ImU) Deconvolution: Uf = f h with Ũ 1 1 f = f h with h 1 (ω) ω ĥ(ω) ω

5 Regularization and Denoising Ũ 1 Y = Ũ 1 Uf + Ũ 1 W Problems: Ũ 1 Uf (NullU) no super-resolution Ũ 1 W is huge if Ũ 1 is not bounded. Regularized inversion includes a noise reduction with a projection in a space V : F = R(Ũ 1 Y ) V Optimizing R requires prior information. No super-resolution : dim(v) Q.

6 Singular Value Decompositions Basis of singular vectors {e diagonalizes U k } 1 k N U : U Ue k = λ 2 k e k Diagonal denoising over the singular basis: r k = F = R(Ũ 1 Y )= Since Ũ 1 Y, e k = λ 2 k Y, Ue k 1 1+σ 2 λ 2 k Linear filtering of deconvolution: R(Ũ 1 Y )=Ũ 1 Y r N 1 k=0 yields r k Ũ 1 Y, e k e k. N 1 F = k=0 ˆr(ω) Y, Ue k λ 2 k + σ2 e k. h 1 (ω) ω

7

8 Thresholding for Inverse Problems Remove noise from with a thresholding estimator. Ũ 1 Y = Ũ 1 Uf + Ũ 1 W Optimal in a basis {φ p } p Γ providing a sparse representation of f and which decorrelates the noise coefficients Ũ 1 W, φ p. The dictionary vectors φ p must be almost eigenvectors of, they must have a narrow spectrum: U U φ p = k S p φ p,e k e k with λ 2 k λ 2 p for k S p

9 Satellite Image Deconvolution Original Smoothed Linear Wiener Thresholding

10 Sparse Spike Deconvolution Seismic data: Y = f u + W with f[n] = p Λ a[p] δ[p n] Y [q] = p Λ a[p] u[q n]+w [q] #$%" #$% #$!" #$! #$#" #!#$#" u[n] #$!" #$! #$#" û(ω)!#$!!!"#!!##!"# # "#!##!"# #!!"#!!##!"# # "#!##!"# f Y F Super-resolution inversion by detection of the sparse support

11 Sparse Super-Resolution Prior information: f has a sparse approximation in a normalized dictionary of at least N vectors D = {φ p } p Γ f = a[p] φ p + ɛ Λ p Λ with a small error ɛ Λ. It results that Y = Uf + W = a[p] Uφ p + (Uɛ Λ + W ) p Λ has a sparse approximation in the redundant dictionary D U = {Uφ p } p Γ in the space ImU of dimension Q N

12 Sparse Super-Resolution A sparse approximation of Y is computed in D U = {Uφ p } p Γ Y Λ = p Λ ã[p] Uφ p with a pursuit algorithm. A basis pursuit minimizes the Lagrangian: Y ã[p] Uφ p 2 + λ ã[p] p Γ p Γ and Λ is the support of ã. It yields a signal estimator F = p Λ ã[p] φ p using prior information which recovers φ p from each Uφ p.

13 Error and Exact Recovery From the sparse decomposition of Y = f + W Y Λ = p Λ ã[p] Uφ p since f = p Λ a[p] φ p + ɛ Λ f F p Λ ã[p]φ p p Λ a[p] φ p + ɛ Λ. Small error if Λ includes Λ and if {Uφ p } p Λ is a Riesz basis. Exact recovery in the redundant dictionary D U = {Uφ p } p Γ Super-resolution: if Λ is not restricted to a space of dimension Q.

14 Sparse Spike Deconvolution Seismic data: Y = f u + W with f[n] = p Λ a[p] δ[p n] φ p [n] =δ[p n], Uφ p [q] =u[q n], F [n] = p Λ ã[p] δ[n n] 0.25 u[n] 0.15 û(ω) f[n] 0.4 Y [q] 1 F [n]

15 Comparison of Pursuits Original Seismic wavelet Seismic trace Basis Pursuit Matching Pursuit Orth. Match. Pursuit

16 Stability: Hence the to U U. Conditions for Super-resolution The signal approximation support {Uφ p } p Λ Uφ p φ p Λ should small. must be a Riesz basis should not be too smal. must have a spread spectrum relatively Support recovery: the dictionary be as incoherent as possible. D U = {Uφ p } p Γ must Exact recovery criteria: ERC(Λ) < 1.

17 Image Inpainting Uf[q] =f[q] for q Ω with Ω = Q<N Super-resolution in a wavelet dictionary D U = {Uφ p } p Γ Original Support of Ω Super-resolution

18 Image Inpainting Uf[q] =f[q] for q Ω with Ω = Q<N Wavelet and local cosinedictionary D U = {Uφ p } p Γ Y = Uf + W Linear estimation Super-resolution

19 Tomography U is a Radon transform which integrates along straight lines. Original Radon Fourier Support Linear Back Prop. Haar super-resol.

20 Super-Resolution Zooming Need to increase numerically acquired image resolution: Conversion to HDTV of SDTV, Internet and Mobile videos... Size increase: 60 images of 720 x 576 pixels = 320 Mega bit/s Spatial deinterlacing and up-scaling up to 8 times more pixels PAL/NTSC x 20 Frame rate conversion twice more images for LCD screens HD LCD screens > Vidéo processor in the TV : 120 images of 1080 x 1920 pixels = 7.5 Giga bit/s

21 Image and Video Zooming Image subsampling : Uf = f[n/s] is a linear projector. Linear inversion without noise: linear interpolation f[p] = n Uf[n] θ(p ns) Prior information: geometric regularity. Super-resolution by interpolations in the directions of regularity Sparse super-resolution marginally improves linear interpolations.

22 High Resolution Image Low Resolution Image Contourlet SuperResolution db Cubic spline Interpolation db

23 Aliased Interpolation pi pi 0 0 pi pi 0 pi pi pi 0 pi Original Subsampled Linear Interp. Direct. Interp. Super-resolution is not possible for horizontal and vertical edges.

24 Adaptive Directional Interpolations Linear Tikhonov estimation: F θ = I θ Y minimizes R θ I θ Y subject to UI θ Y = Y where R θ is a linear directional regularity operator. Adaptive directional interpolation adapt locally θ by testing locally the directional regularity with gradient operators. General class of mixing linear operators in a frame F = θ Θ I θ a(θ,p) Y,φ p φ p p Γ {φ p } p Γ Problem: how to optimize the a(θ,p)?

25 Wavelet Block Interpolation Wavelet transform on 1 scale, j = 1 f,ψj,n k = f(x)2 j ψ k (2 j (x n)) dx k =1 k =2 2 j =2 k =3 Low frequencies are linearly interpolated (no aliasing). Adaptive directional interpolation of fine scale wavelets.

26 Wavelet Block Interpolation Dictionary of blocks {B θ,q } θ,q To a wavelet block decomposition Y = ɛ(θ,q) P Bθ,q Y + Y r θ q with P Bθ,q Y = Y,ψ1,n k ψ1,n k (n,k) B θ,q we associate an interpolation estimation F = θ I θ ( q ɛ(θ,q) Y q,θ ) + I r (Y r ) How to optimize the ɛ(θ,q)?

27 To compute Y = θ Adaptive Tikhonov Estimation ɛ(θ,q) P Bθ,q Y + Y r q where ɛ(θ,q) is sparse and ɛ(θ,q) 1 if R θ I θ P Bθ,q Y is small: Lagrangian minimization L = Y θ,q ɛ(θ,q) P Bθ,q Y 2 + λ θ,q ɛ(θ,q) R θ I θ P Bθ,q Y 2 Standard l 1 minimization. Can be solved with a greedy pursuit. If there is only one ɛ(θ,q) 0then L is minimized by ɛ(θ,q) = max (1 λ R θi θ P Bθ,q Y 2 ) P Bθ,q Y 2, 0 and L = Y 2 e(θ,q) with e(θ,q)= P B θ,q Y 2 ɛ(θ,q) 2 2.

28 Wavelet Block Spaces Wavelet transform Blocks of oriented bars Block projection pursuit

29 Super-Resolution Block Interpolation I:wavelet transform II:block space pursuit IV: inverse wavelet transform III:block inverse transforms directional interpolations

30 Comparison with Cubic Splines Block pursuit on wavelet coefficients Block Interpolations over wavelet coefficients Cubic spline interpolations

31 Comparison with Cubic Splines Block pursuit on wavelet coefficients Block Interpolations SNR = db over wavelet coefficients SNR = db Cubic spline interpolations

32 Examples of Zooming Original Image Cubic Spline Interpolation Bandlet Super-Resolution SNR = db SNR = db

33 Super-Resolution Zooming Need to increase numerically acquired image resolution: Conversion to HDTV of SDTV, Internet and Mobile videos... Spatial deinterlacing and up-scaling up to 8 times more pixels Frame rate conversion twice more images for LCD screens

34 3rd. Concluion Super-resolution is possible for signals that are sparse in a dictionary D = {φ p } p Γ which has a spread spectrum and which is transformed in an incoherent dictionary D U = {Uφ p } p Γ Super-resolution is typically not possible for any class of signals Need to incoporate as much prior informaiton as possible: use of structured sparse representations. What if it was possible to choose the operator U? compressed sensing...