Image restoration: edge preserving

Size: px

Start display at page:

Download "Image restoration: edge preserving"

Mervyn White
5 years ago
Views:

1 Image restoration: edge preserving Convex penalties Jean-François Giovannelli Groupe Signal Image Laboratoire de l Intégration du Matériau au Système Univ. Bordeaux CNRS BINP 1 / 32

2 Topics Image restoration, deconvolution Motivating examples: medical, astrophysical, industrial,... Various problems: Fourier synthesis, deconvolution,... Missing information: ill-posed character and regularisation Three types of regularised inversion 1 Quadratic penalties and linear solutions Closed-form expression Computation through FFT Numerical optimisation, gradient algorithm 2 Non-quadratic penalties and edge preservation Half-quadratic approaches, including computation through FFT Numerical optimisation, gradient algorithm 3 Constraints: positivity and support Augmented Lagrangian and ADMM Bayesian strategy: a few incursions Tuning hyperparameters, instrument parameters,... Hidden / latent parameters, segmentation, detection,... 2 / 32

3 Convolution / Deconvolution y = Hx + ε = h x + ε x H + ε y x = X (y) Restoration, deconvolution-denoising General problem: ill-posed inverse problems, i.e., lack of information Methodology: regularisation, i.e., information compensation Specificity of the inversion / reconstruction / restoration methods Trade off and tuning parameters Limited quality results 3 / 32

4 Competition: Adequation to data Compare observations y and model output Hx Unknown: x Known: H and y Comparison experiment model x H Hx y Quadratic criterion: distance observation model output J LS (x) = y Hx 2 4 / 32

5 Competition: Smoothness prior Data insufficiently informative Account for prior information 1 Question of compromise, of competition 2 Specificity of methods Here: smoothness of images Quadratic penalty of the gray level gradient (or other linear combinations) P(x) = p q (x p x q ) 2 = Dx 2 5 / 32

6 Quadratic penalty: criterion and solution Criterion including quadratic penalty J PLS (x) = y Hx 2 + µ Dx 2 = y Hx 2 + µ (x p x q ) 2 p q Restored image x PLS = arg min J PLS (x) x (H t H + µd t D) x PLS = H t y x PLS = (H t H + µd t D) 1 H t y 6 / 32

7 Computational aspects and implementation Various options and many relationships... Direct calculus, compact (closed) form, matrix inversion Algorithms for linear system solution Gauss, Gauss-Jordan Substitution Triangularisation,... Special algorithms, especially for 1D case Recursive least squares Kalman smoother or filter (and fast versions,... ) Numerical optimisation gradient descent: descent in the direction opposite to the gradient and various modifications Diagonalization Circulant approximation and diagonalization by FFT 7 / 32

8 Solution from least squares and quadratic penalty True Observation Quadratic penalty Quadratic penalty Circulant approx Non-circulant 8 / 32

9 Edge preserving approaches Limited resolution of linear solutions Desirable: less smoothing around discontinuities Ambivalence: Smoothing (homogeneous regions) Heightening, enhancement, sharpening (discontinuities, edges)... and new compromise, trade off, conciliation Algorithmic structure Based on the linear solution (based on quadratic criterion) FFT (Wiener-Hunt) Kalman smoother or filter (and its fast versions, factorised,... ) Numerical optimisation (gradient,... ) 9 / 32

10 Edge preservation and non-quadratic penalties Restored image still defined as the minimiser of a penalised criterion... x = arg min J(x) x J(x) = y Hx 2 + µ P(x)... once again penalising small variations but less penalising for discontinuities P(x) = p q ϕ(x p x q ) ϕ(δ) = δ 2 ϕ(δ) =...? Ambivalence: new compromise, trade off, conciliation Smoothing (homogeneous regions) Heightening, enhancement, sharpening (discontinuities, edges) 10 / 32

11 Typical potentials Again ϕ(δ) δ 2 for small δ Behaviour for large δ 1 Horizontal asymptote 2 Horizontal parabolic behaviour 3 Oblique (slant) asymptote 4 Vertical parabolic behaviour Alpha = 0.50 et Seuil = 1.00 Q Huber H and L G and McC B and Z 1.5 Potentiel Variable Delta 11 / 32

12 Four major types of potentials 1 Horizontal asymptote ϕ(δ) 1 [Blake and Zisserman (87), Geman and McClure (87)] ϕ(δ) = αs 2 min ( 1, (δ/s) 2) ; ϕ(δ) = αs 2 (δ/s) 2 2 Horizontal parabolic behaviour ϕ(δ) log δ [Hebert and Leahy (89)] ϕ(δ) = αs 2 log [ 1 + (δ/s) 2] 1 + (δ/s) 2 3 Oblique (slant) asymptote ϕ(δ) δ Huber, hyperbolic, Log hyperbolic-cosine, fair function { δ 2 ) if δ s ϕ(δ) = α 2s δ s 2 ; ϕ(δ) = 2αs ( [δ/s] 2 1 if δ s 4 Vertical parabolic behaviour ϕ(δ) δ 2 Wiener-Hunt solution ϕ(δ) = αδ 2 12 / 32

13 Potentials with oblique asymptote (L 2 / L 1 ): details Potentiel Alpha = 0.50 et Seuil = Variable Delta { Huber : ϕ(δ) = αs 2 [δ/s] 2 if δ s 2 δ /s 1 if δ s ) Hyperbolic : ϕ(δ) = 2αs ( [δ/s] 2 1 LogCosh : ϕ(δ) = 2αs 2 log cosh ( δ /s) FairFunction : ϕ(δ) = 2αs 2 [ δ /s log (1 + δ /s)] 13 / 32

14 More general non-quadratic penalties (1) Differences, higher order derivatives, generalizations,... P(x) = n P(x) = n P(x) = n P(x) = n ϕ(x n+1 x n ) ϕ(x n+1 2x n + x n 1 ) ϕ(α x n+1 x n + α x n 1 ) ϕ(α t n x) Linear combinations (wavelet, other-stuff-in-et,... dictionaries,... ) P(x) = ϕ(wnx) t = ( ) ϕ w nm x m n n m Redundant or not Link with Haar wavelet and other 14 / 32

15 More general non-quadratic penalties (2) 2D aspects: derivative, finite differences, gradient approximations P(x) = p q ϕ(x p x q ) = n,m ϕ(x n+1,m x n,m ) + n,m ϕ(x n,m+1 x n,m ) Notion of neighborhood and Markov field Any highpass filter, contour detector (Prewitt, Sobel,... ) Linear combinations: wavelet, contourlet and other-stuff-in-et,... Other possibilities (slightly different) Enforcement towards a known shape P(x) = p ϕ(x p x p) Separable penalty P(x) = p ϕ(x p) 15 / 32

16 Penalised least squares solution A reminder of the criterion and the restored image J(x) = y Hx 2 + µ p q ϕ s (x p x q ) x = arg min J(x) x with ϕ s one of the mentioned (non-quadratic) potentials and two hyperparameters: µ and s Non-quadratic criterion Non-linear gradient No closed-form expression Two questions Practical computation: numerical optimisation algorithm,... Minimiser: existence, uniqueness,... continuity 16 / 32

17 Convexity and existence-uniquiness Convex set R N, R N +, intervals of R N,... Properties: intersection, convex envelope, projection,... Strictly convex criterion, convex criterion, Θ(u) = u 2, Θ(u) = u 2, Θ(u) = u, Huber,... Properties: sum of convex function, level sets,... Key result Set of minimisers of convex criterion on a convex set is a convex set Strict convexity unique minimiser Application ϕ convex = J convex In the following developments, potential ϕ s Huber or hyperbolic: convex (strict) guarantees In addition: non-convex no guarantee (although, sometimes... ) 17 / 32

18 Half-quadratic enchantement (start) Reminder of the criterion J(x) = y Hx 2 + µ p q ϕ(x p x q ) Minimisation based on quadratic Original idea of [Geman + Yang, 95] Set of auxiliary variables b pq so that: ϕ(δ pq) δ 2 pq ] ϕ(δ) = inf b With appropriate ζ(b): Extended criterion J(x, b) = y Hx 2 + µ p q [ 1 2 (δ b)2 + ζ(b) 1 2 [(x p x q ) b pq ] 2 + ζ(b pq ) und natürlich: J(x) = inf b J(x, b) 18 / 32

19 Legendre-Fenchel Transform (LFT) or Convex Conjugate (CC) Definition of TLF or CC Consider f : R R strictly convex once (or twice) differentiable The TLF or CC is the function f : R R defined by: f (t) = sup x R [ xt f (x) ] Remark f (0) = sup x R [ f (x) ] = inf x R [ f (x) ] t, x R, t, x R, xt f (x) f (t) f (t) + f (x) xt 19 / 32

20 LFT: some shift and dilatation / contraction properties f (t) = sup x R [ xt f (x) ] Vertical: shift-dilatation (α R and β R +) { g(x) = α + β f (x) g (t) = β f (t/β) α Horizontal: shift (x 0 R) and dilatation (γ R +) { { g(x) = f (x x 0 ) g(x) = f (γx) g (t) = f (t) tx 0 g (t) = f (t/γ) Specific case α = 0, β = 1 / x 0 = 0 / γ = 1 20 / 32

21 LFT: a first example Quadratic case (α R and β R +) Let us consider the function We look for the CC f (x) = α β(x x 0) 2 f (t) = sup x R [ xt f (x) ] Let us denote g t (x) = xt f (x) = xt ( α + β(x x 0 ) 2 / 2 ) The derivative reads: g t (x) = t β(x x 0) And the second derivative is: g t(x) = β By nullification of g t (x): x = x 0 + t/β Then by substitution: f (t) = g t( x) f (t) = 1 2β (t + βx 0) 2 ( α + 1 ) 2 βx / 32

22 LFT: a generic result for explicitation (a) A swiss army formula: Legendre formula f (t) = sup x R Let us denote g t (x) = xt f (x) [ xt f (x) ] The derivative reads: g t (x) = t f (x) And the second derivative is: g t(x) = f (x) By nullification of g t (x): Then by substitution: t f ( x) = 0 x = f 1 (t) = χ(t) f (t) = g t ( x) = t x f ( x) = tχ(t) f [χ(t)] 22 / 32

23 LFT: a generic result for explicitation (b) Derivatives Convex conjugate made explicit f (t) = tχ(t) f [χ(t)] with χ = f 1 The derivative reads: f (t) = χ(t) + tχ (t) χ (t) f [χ(t)] = χ(t) = f 1 (t) And the second derivative is: f (t) = χ(t) = 1 f [χ(t)] > 0 hence f is convex 23 / 32

24 LFT: a key result Double conjugate Transformed twice gives back the original f (x) = f (x) f (t) = sup [ xt f (x) ] Let us note h t (x) = xt f (x) and nullify the derivative: 0 = h t( x) = t f ( x) = t f 1 ( x) = t χ( x) x = χ 1 (t) = f (t) By substitution f (t) = h t ( x) = xt f ( x) = xt [ xχ( x) f (χ( x))] = xt xt + f (t) = f (t) 24 / 32

25 Outcome for LFT: un théorème vivant Definition Let us consider f : R R strictly convex once (or twice) differentiable Properties f (t) = sup x R [ xt f (x) ] f (t) = tχ(t) f [χ(t)] with χ = f 1 f = f 1 = χ f (t) = 1 / f [χ(t)] f is convex f (x) = f (x) 25 / 32

26 A theorem in action: half-quadratic (beginning) Problem statement Consider a potential ϕ, convex or not, and look for ζ such that [ ϕ(δ) = inf (δ b) 2 /2 + ζ(b) ] b R Let us define g such that it is strictly convex: Consider its CC: g (b) = sup δ R = sup δ R g(δ) = δ 2 /2 ϕ(δ) [ bδ g(δ) ] [ ϕ(δ) (δ b) 2 /2 ] + b 2 /2 Let us set (reason explained on the next slide): ζ(b) = g (b) b 2 [ /2 = sup ϕ(δ) (δ b) 2 /2 ] δ R 26 / 32

27 A theorem in action: half-quadratic (middle) Take advantage of g = g g(δ) = g (δ) δ 2 /2 ϕ(δ) = sup b R ϕ(δ) = δ 2 /2 sup [ bδ g (δ) ] b R [bδ g (δ)] [ = inf (δ b) 2 /2 + ζ(b) ] b R The icing on the cake, we have the minimiser: 0 = [ (δ b) 2 /2 + ζ(b) ] = ( b δ) + ζ ( b) = g ( b) δ then: b = g 1 (δ) = g (δ) = δ ϕ (δ) 27 / 32

28 A theorem in action: half-quadratic (ending) Reminder: original criterion and extended criterion J(x) = y Hx 2 + µ p q ϕ(x p x q ) J(x, b) = y Hx 2 + µ p q 1 2 [(x p x q ) b pq ] 2 + ζ(b pq ) Algorithmic strategy: alternating minimisation 1 Minimisation w.r.t. x for fixed b: x(b) = arg min x J(x, b) Quadratic problem 2 Minimisation w.r.t. b for fixed x: b(x) = arg min b J(x, b) Separated and explicit update 28 / 32

29 Image update Non-separable but quadratic J(x) = y Hx 2 + µ p q = y Hx 2 + µ Dx b 2 Multivariate system of equations Minimiser 1 2 [(x p x q ) b pq ] 2 (H t H + µd t D) x = H t y + µ D t b x = (H t H + µd t D) 1 (H t y + µ D t b) Numerical aspects Descent algorithm (including inner-iterations) Circulant case and FFT (without inner-iterations) 29 / 32

30 Auxiliary variables update Separable (parallel computation), explicit (no inner-iterations), easy Each b = b pq δ = δ pq = x p x q Update: b = δ ϕ (δ) Huber: b = δ [1 2α min (1; s/δ)] [ Geman & McClure: b = δ 1 2α ] (s/δ) 2... Alpha = 0.40 et Seuil = Facteur de contraction Huber H and L G and McC B and Z Variable Delta 30 / 32

31 Result L 2 / L 1 (Huber) True Observation Quadratic penalty Huber penalty 31 / 32

32 Synthesis and extensions Synthesis Image deconvolution Also available for non-invariant linear direct model And also available for the case of signal... Edge preserving and non-quadratic penalties Gradient of gray levels (and other transform) Convex (and differentiable) case and also some non-convex cases Numerical computations Half-quadratic approach: separable and FFT Numerical optimisation (gradient,... ) Extensions Including constraints better image resolution (next lecture) Hyperparameters estimation, instrument parameter estimation, / 32

Image restoration: numerical optimisation

Image restoration: numerical optimisation Short and partial presentation Jean-François Giovannelli Groupe Signal Image Laboratoire de l Intégration du Matériau au Système Univ. Bordeaux CNRS BINP / 6 Context