1 / 41 Journal Club Presentation Xiaowei Zhou Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology 2009-12-11
2 / 41 Outline 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
3 / 41 Outline Motivation Diffusion process Linear diffusion 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
4 / 41 What are the problems? Motivation Diffusion process Linear diffusion Figure: Left: a noisy picture; Right: ultrasound image to examine the heart of zebrafish
5 / 41 Why diffusion Motivation Diffusion process Linear diffusion How to reduce noises and restore the original image contents. Traditional linear filtering is unable to preserve "semantically meaningful" features while reducing the noise. Key idea of diffusion: evolve the image as a dynamic system to reduce system randomness. Some applications of diffusion Image restoration Scale space theory feature enhancement...
6 / 41 Outline Motivation Diffusion process Linear diffusion 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
Motivation Diffusion process Linear diffusion Definition of diffusion Diffusion In physics, diffusion is a process that equilibrates concentration differences without creating or destroying mass. Examples: heat diffusion. molecular motion. http://en.wikipedia.org/wiki/diffusion An animation of molecular diffusion In image processing, the idea of diffusion is borrowed to reduce the random variation and restore the image. Space: 2D image plane Concentration: Image intensity 7 / 41
8 / 41 Laws for diffusion Motivation Diffusion process Linear diffusion Fick s Law Intensity gradient u generates flux j. j = D u (1) u: Intensity (mass density, temperature, image intensity...) D: diffusion tensor, symmetric positive definite if D = ki: flux j is always parallel to u else: flux j is not always parallel to u Continuity Equation no mass generated or destroyed. tu = div(j) (2)
9 / 41 Diffusion Equation Motivation Diffusion process Linear diffusion Fick s law and continuity equation yeild: Diffusion Equation j tu = div(d u) u(x, 0) = f (x) The diffusion process is determined by intensity gradient and the tensor field. In image processing: boundary condition is needed: < D u, n >= 0 (no normal flow on boundary) D is critical to design a diffusion method. For linear diffusion, D is constant over image. For nonlinear diffusion, D is adapted to local image structure. (3)
10 / 41 Outline Motivation Diffusion process Linear diffusion 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
11 / 41 Solution for linear diffusion Motivation Diffusion process Linear diffusion Solve the simplest case: D = ki { t u = k div( u) = k u u(x, 0) = f (x) (4) D is independent with the image structure and isotropic. Solution: u(x, t) = φ(x, t) f (x) φ(x, t). = 1 4πkt n exp( x T x 4kt ) (5) http://en.wikipedia.org/wiki/heat_equation φ is a Gaussian function with σ = 2kt! That means the solution of linear diffusion process is Gaussian filtering with σ increasing with time.
12 / 41 Drawbacks of linear diffusion Motivation Diffusion process Linear diffusion the Gaussian process will distort the image structure Figure: Left: linear diffusion result at different time; Right: corresponding zero-crossing points. Dislocation of edges occurs at coarse scales. http://www.mia.uni-saarland.de/weickert/demos.html
13 / 41 Outline 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
14 / 41 Perona and Malik modified the diffusion coefficient D to be adapted { to the image: large, when u is small inside a homogeneous region D = small, when u is large near boundaries between regions Perona and Malik s Anisotropic Diffusion j tu = div(g( u 2 ) u) u(x, 0) = f (x) where, g( ) is a nonnegative monotonically decreasing function with g(0) = 1 P. Perona, J. Malik Scale space and edge detection using anisotropic diffusion IEEE Trans. Pattern Anal. Mach. Intell., Vol. 12, 629 C639, 1990. (6)
15 / 41 Diffusion coefficient Requirements on the diffusion coefficient function: nonnegative: satisfy the maximum principle to avoid generating spurious features in coarse levels of the scale space. monotonically decreasing: discourage the diffusion over edges where the image gradient is large. A such function popularly used: g(s 2 ) = 1 1 + s 2 /λ 2 (7)
16 / 41 Diffusivity and Flux function g(s 2 ) is usually called diffusivity, describing the diffusion or conduction property. s g(s 2 ) is named flux function, describing magnitude of flux. Figure: Left: diffusivity in Eq.(7); Right: the corresponding flux function
17 / 41 Example result of anisotropic diffusion Figure: Left: original noisy image; Middle: after linear diffusion; Right: after nonlinear diffusion
18 / 41 Example result of anisotropic diffusion Figure: Left: original noisy image; Middle: after 100 iterations; Right: after 1000 iterations
19 / 41 Edge enhancement property We can analyze the property of the nonlinear diffusion in Eq.(6) first from 1-D case. Define the flux function: φ(s) = s g(s 2 ). For certain type of g(s 2 ) (Page 16): In 1-D space, Eq.(6) reads: φ (s) 0, s λ φ (s) < 0, s > λ t u = φ (u x )u xx (8) If we check the derivative of the magnitude of gradient with respective to time: t (u 2 x ) = 2u x x (u t ) = 2φ (u x )u x u 2 xx + 2φ (u x )u x u xxx (9)
20 / 41 Edge enhancement property cont. At a edge in a bounded image, we have following assumputions: u xx = 0 u x u xxx < 0 From Eq.(9), it can be found that: if u x > λ,φ (u x ) < 0, t (u 2 x ) > 0 if u x < λ,φ (u x ) > 0, t (u 2 x ) < 0 It means the gradient at edges where u x > λ will get larger. The edge is enhanced.
21 / 41 Edge enhancement property cont. In 2D case: t u = div(g( u 2 ) u) = g( u 2 ) u + g( u 2 ) T u = g( u 2 ) u + 2g ( u 2 )u ηη = g( u 2 )u ξξ + [g( u 2 ) + 2 u 2 g ( u 2 )]u ηη = g( u 2 )u ξξ + φ ( u )u ηη (10) where u ξξ and u ηη are the second derivative along directions perpendicular and parallel to u x respectively: H u is the Hessian matrix. u ξξ = ξ T H u ξ u ηη = η T H u η
22 / 41 Edge enhancement property cont. Eq.(10) gives us more insights into 2D diffusion: The 2D diffusion can be decomposed to process along two orthogonal directions along the local edge and normal to the edge. if u x << λ, the diffusion is the ordinary isotropic diffusion driven by Laplacian. if u x λ, φ ( u ) 0, the diffusion u ηη parallel to the image gradient (normal to the edge) is suppressed. if u x >> λ, φ ( u ) < 0, backward diffusion happens normal to the edge!
23 / 41 Spatial regularization The original Perona-Malik model has the following problem: in the beginning of the diffusion, the diffusion coefficient can t be estimated robustly since the high variation in a noisy image. This issue can be numerically solved by adding a spatial filtering when estimating the gradient, and the diffusion coefficient is modified as: where G σ is a Gaussian mask. D = g( G σ u 2 ) (11)
24 / 41 Outline 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
25 / 41 An example: edge-enhancing Anisotropic Diffusion Using a anisotropic tensor D, such that D reflect the edge structure: Eigenvectors: v 1 u σ, v 2 u σ Eigenvalues: damped diffusion across edge: λ 1 = g( u σ 2 ) full diffusion parallel to edge: λ 2 = 1 J. Weickert Theoretical foundations of anisotropic diffusion in image processing Computing Supplement; Vol. 11 Proceedings of the 7th TFCV on Theoretical Foundations of Computer Vision, 1996.
26 / 41 result of edge-enhancing anisotropic diffusion Figure: Left: original noisy image; Right: after edge-enhancing anisotropic diffusion. Adopted from Weickert 1996
27 / 41 Another example J. Weickert Coherence-Enhancing Diffusion Filtering International Journal of Computer Vision, 1999. Make use of the image structure tensor K ρ ( u σ u T σ )
28 / 41 Outline 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
29 / 41 SRAD Yu, Y. and Acton, ST Speckle reducing anisotropic diffusion IEEE Transactions on Image Processing, 2002, vol. 11, issue 11, pp. 1260-1270. Basic formulation: the same with Eq.(6) modification:the diffusion coefficient is modified to be: g(s 2 ) = 1 1 + [s 2 s 2 0 ]/[s2 0 (1 + s2 0 )] (12) s 0 and s are local statistics of the homogeneous region and the region of interest respectively. s 2 = var(u) ū 2 (13)
30 / 41 Comments on SRAD SRAD is intrinsically the same with the original model of Perona and Malik. the gradient magnitude estimator G σ u 2 in the Perona-Malik model is replaced by local statistics. This idea is borrowed from the Lee and Frost filters, which are popular adaptive filters in the radar community to reduce speckle noises in radar images. In the Lee and Frost filters, this statistics is used to determine the coefficient of smoothing. To determine s 0 in Eq.(12), a homogeneous region needs to be chosen. For fully automatic implementation, in this paper s 0 is set to be decreasing exponentially with the diffusion process going on. s 0 (t) = s 0 (0) exp( ρt).
31 / 41 Example results of SRAD For the phantom data: Figure: Left: original speckle image; Middle: result of SRAD; Right: result of the Perona-Malik model
32 / 41 Example results of SRAD For the zebrafish data: Figure: Left: original speckle image; Middle: result of SRAD; Right: result of the Perona-Malik model
33 / 41 Outline 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
34 / 41 Energy minimization The diffusion process can be viewed as minimizing a potential function evaluating the total variation of the image. If we can find a potential function ψ( ) such that: ( u) (ψ( u )) = g( u 2 ) u (14) ( u) means gradient respect to u. Then minimizing the energy functional: E(u) =. ψ( u )dxdy (15) Ω with the Euler-Lagrange equation leads to the gradient descent: t u = div( u (ψ( u ))) = div(g( u 2 ) u) (16)
35 / 41 diffusivity vs. potential Linear diffusion: g(s 2 ) = 1, ψ(s) = s2 2 the penalty function is exactly 2 norm. Perona-Malik model: g(s 2 ) = 1 1+s 2 /λ 2, ψ(s) = λ2 2 log(1 + ( s λ) 2) the penalty function is a robust estimator. Generally, only need to find ψ (s) = sg(s 2 ) Thus, diffusion is minimizing the total variation by gradient descent. The diffusion coefficient is determined by the penalty function (estimator) for the variation.
36 / 41 Some other comments The time of diffusion controls the fidelity to the original image. It is corresponding to the coefficient balancing the data term and regularization in ordinary regularized energy functions. Instead aiming at the objective function, designing a diffusion function (tensor) is more direct and easy to understand in the view of physics. (e.g. the diffusion with anisotropic tensor is difficult to be expressed as an energy minimization form) Why in robust estimation theory ψ (s) is usually called the flux function? Because it is exactly the flux function of diffusion defined in Page 16.
37 / 41 Outline Other applications of PDE based methods 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
38 / 41 Segmentation Other applications of PDE based methods Chan and Vese s active contour without edges: E CV (c 1, c 2, φ) = λ 1 c 1 I 0 (x, y) 2 H(φ(x, y))dxdy Ω +λ 2 Ω c 2 I 0 (x, y) 2 H( φ(x, y))dxdy +µ H(φ(x, y)) dxdy + ν H(φ(x, y))dxdy (17) Ω The gradient descent flow: [ ( ) ] φ φ = δ ε (φ) µ div ν λ 1 (u 0 c 1 ) 2 + λ 2 (u 0 c 2 ) 2 t φ φ(0, x, y) = φ 0 (x, y) (18) Ω
39 / 41 Motion tracking Other applications of PDE based methods Brox, T. et al. High accuracy optical flow estimation based on a theory for warping Proc. 8th European Conference on Computer Vision The energy functional with respect to the displacement field: E(d = (u, v)) = ρ( I 2 (x + d) I 1 (x) 2 + γ I 2 (x + d) I 1 (x) 2 )dx Ω +α ρ( u 2 + v 2 )dx (19) Ω The gradient descent flow (two complicated, please refer to the paper): u t = x E ux + y E uy E u v t = x E vx + y E vy E v
40 / 41 Outline Other applications of PDE based methods 1 Motivation Diffusion process Linear diffusion 2 3 Other applications of PDE based methods
41 / 41 Other applications of PDE based methods Why PDE based methods model and explore the image structure mathematical foundation, concise representation physically meaningful and understandable numerically implementable...