CN780 Final Lecture. Low-Level Vision, Scale-Space, and Polyakov Action. Neil I. Weisenfeld

Transcription

1 CN780 Final Lecture Low-Level Vision, Scale-Space, and Polyakov Action Neil I. Weisenfeld Department of Cognitive and Neural Systems Boston University chapter May 9, 2005 p.1/25

2 Introduction The term early vision refers to the processing that happens, or is thought to happen, at the beginning of the visual analysis chain close to the retinal image. At least since Hubel and Wiesel s discovery of orientation sensitive cells in V1, scientists have been obsessed with models of early vision that center around the extraction of features, especially edges, from a scene. Computational strategies focus on context-free and therefore local image-processing operations. As Sochen points out, this locality leads to differential operators. chapter May 9, 2005 p.2/25

3 Laplacian Edge Detection Image on the left is the original, blurred image. Image in the middle is the log-magnitude of the Laplacian applied to the image. Image on the right is the zero-crossing of the Laplacian image. elec539/projects97/morphj chapter May 9, 2005 p.3/25

4 Scale-Space: Isotropic Diffusion Some authors propose a pyramid where images are smooth at various resolutions. But this can be generalized to a continuum: I(x, y, t) = I 0 (x, y) G(x, y; t) Perona-Malik note that this is also the solution to the heat equation and therefore has a physical interpretation: isotropic diffusion: I t = I = (I xx + I yy ) chapter May 9, 2005 p.4/25

5 Scale-Space: Isotropic Diffusion But isotropic diffusion has two problems: edges are not preserved and they move at different scales: chapter May 9, 2005 p.5/25

6 Scale-Space: Anisotropic Diffusion Perona and Malik introduced the idea of anisotropic diffusion where the constant in the heat equation is now variable with space. I t = div(c(x, y, t) I) = c(x, y, t) I + c I if c(x, y, t) = c then we have I t = c I now choose c in a way that takes the gradient into account: c(x, y, t) = g( I(x, y, t) ) chapter May 9, 2005 p.6/25

7 Scale-Space: Anisotropic vs. Isotropic Diffusion chapter May 9, 2005 p.7/25

8 Sochen Unifying Theory Sochen, 1996, From High Energy Physics to Low Level Vision (Lawrence Berkeley National Labs Tech Report LBNL-39243) Images modeled as a surface embedded in a higher dimensional space (e.g. a 2D image as a height-field in R 3 ). Image processing operations that generate a scale space are modeled as a metric in the embedded space and a flow. The Polyakov action, from high-energy physics, is used to generate the flow. A number of classic operations fall into this framework and the mathematics are in place for extension to other problems. Sochen, for example, shows how to use the theory to extend some classical operations on grayscale images to RGB-color images. chapter May 9, 2005 p.8/25

9 Image as a Surface 2D, single-channel image I(x, y) to R 3 as (x, y, I) I(x, y) (x, y, I) chapter May 9, 2005 p.9/25

10 Where are we going with this? We re interested in differential operations on, say, image intensity values with respect to image coordinates. The surface embedding allows us to visualize this in two ways, either in (σ 1, σ 2 ) space (with a local metric) or (x, y, I): chapter May 9, 2005 p.10/25

11 Differential Geometry Intro How do we measure distances between points within our coordinate systems? Consider the map X : Σ R 3 X is explicitly [X 1 (σ 1, σ 2 ), X 2 (σ 1, σ 2 ), X 3 (σ 1, σ 2 )] Squared distance ds 2 on Σ is local coordinates adjusted by g µν (σ 1, σ 2 ): ds 2 = g µν dσ µ dσ ν Einstein summation = g 11 (dσ 1 ) 2 + 2g 12 dσ 1 dσ 2 + g 22 (dσ 2 ) 2 chapter May 9, 2005 p.11/25

12 Connect Metrics Consider a more general map X : Σ M and metric g on Σ and metric h on M. If we know h and X, we can construct g: g µν (σ 1, σ 2 ) = h ij (X) µ X i ν X j note we re summing over i, j and µ X i means X i / σ µ For our image map: X : (σ 1, σ 2 ) [x = σ 1, y = σ 2, z = I(σ 1, σ 2 )] we have: ( ) 1 + Ix 2 I x I y (g µν ) = I x I y 1 + Iy 2 chapter May 9, 2005 p.12/25

13 Where are we? In the grand tradition, and hopefully more successfully that Sun Microsystems campaign The Network is the Computer : note that The Map is the Image. Or, at least, it contains the image. We want to (remember) define a scale-space and flows (transitions) from one image to another. The framework calls for a measure on the maps and equations will be derived by minimizing the measure with respect to something. chapter May 9, 2005 p.13/25

14 Polyakov Action: The Measure Again, we have (Σ, g) and (M, h) and our mapping X : Σ M A measure of the weight of the map is: S[X i, g µν, h ij ] = d m σ gg µν µ X i ν X j h ij (X) m = dimension(σ), g = g µν, g µν = g 1 µν rememember that g µν = µ X i ν X j h ij (X) and g µν g νγ = δ µ γ, so the weight is related to the surface area. This functional is thought to have been introduced by Polyakov for m = 2. chapter May 9, 2005 p.14/25

15 Simple Example Embed a surface in R 3 and use Cartesian coordinates, so: ( ) 1 0 (g µν ) =, x = σ 1, y = σ 2, h ij = δ ij 0 1 The functional is now the Euclidean L2 norm: S[I, g µν = δ µν, h ij = δ ij ] = d 2 σ( x 2 + y 2 + I 2 ) and minimizing with respect to I gives the heat operator (c.f. isotropic diffusion, but we don t have a time factor, yet. Part of the flexibility here is deciding what to minimize with respect to and scaling of intensity via h ij chapter May 9, 2005 p.15/25

16 Minimizing a Functional The Euler-Lagrange equations can be used to find an extremum. Given: I = x2 f[y(x), ẏ(x), x]dx x1 Euler-Lagrange gets you: f y d dx ( f ẏ ) = 0 chapter May 9, 2005 p.16/25

17 Minimizing and Adding Time The details of how to do this are worked out in Appendix A of the difficult to find LBNL technical report and are eliminated from all of the published works. Minimizing the Polyakov functional with respect to the embedding space yields: 1 2 g δs hil =... (see paper) δxl Add time (scale-space) by viewing this as a gradient descent problem: X i = 1 t 2 δs hil g δx l chapter May 9, 2005 p.17/25

18 Perona-Malik Flow Perona-Malik Diffusion can be derived within this framework and has the advantage of extending to n-dimensions. The Euler-Lagrange equation simplifies, in a Euclidean space to: I t = µ gg µν ν I with the n-dimensional image embedded in R n+1 if we choose (g µν ) = fi d where I d is the identity, we have: I t = n µ=1 µ f n 2 1 µ I chapter May 9, 2005 p.18/25

19 Perona-Malik Flow cnt d For n 2, we can choose f n 2 1 = C(I) and we have Perona-Malik Flow: for C(I) = f(i 0) I, we have: I t = div(c(i) I) I t = div(f(i 0 ) I I ) which is the fundamental equation for geodesic active contours (c.f. level set methods) chapter May 9, 2005 p.19/25

20 Beltrami Flow The authors propose a new flow, operating in multiple dimensions, which the dub the Beltrami flow: I i t = g I g µ ( gg µν ν I i ) = H i where H i is the mean curvature in the ith direction. Geometrically, each point moves with a velocity proportional to the mean curvature in each direction. This looks nasty, but it is the induced flow when using the trivial embedding, Euclidean metrics, and minimizing with respect to I. See examples next page. chapter May 9, 2005 p.20/25

21 Beltrami Flow Examples chapter May 9, 2005 p.21/25

22 Multiscale Active Contours Bresson, et. al. used the framework to extend active contours to multiple scales: chapter May 9, 2005 p.22/25

23 Final Words on the Framework The framework provides: an embedding of images in a higher dimensional space via a mapping a connection between the metric in the image manifold and the (n + 1)D space an action functional on the maps themselves (which encode the images) The connection to established methods seems somewhat forced I need to understand the geometry better The technique opens up avenues for analysis and derivation: metrics can be chosen in either space and derived for the other and the action can be minimized with respect to the metrics or the mapping. chapter May 9, 2005 p.23/25

24 Concluding Ideas The authors cite 1994 work of Florac, et. al., in which they present linear heat flow scale-space in the log-polar coordinate system. They point out that the Beltrami operator is a parameterization invariant differential operator which can work in this capacity and is edge preserving. The generalization of Perona-Malik flow to n-dimensions may be of interest in multi-channel MRI. I have never seen precisely the same text recycled into so many different publications. Many of the details are worked out only in the LBNL tech report. The generalization of various techniques to n-dimensions may be of great interest in and of itself. chapter May 9, 2005 p.24/25

25 References Primary reference is Sochen, et. al.: A General Framework for Low Level Vision. IEEE Trans Image Proc 7(3): , The LBNL tech report is not available online from LBNL and the version of Sochen s web site is stripped down for one of the many re-publications of the text. The full version was on Kimmel s website at: ron/pub.html Sochen has a talk at sochen/beltrami2.pdf Multiscale Active Contours was from Bresson, et al. at EPFL, Lausanne, Switzerland. It does not appear to have been published yet. Malik and Perona is in IEEE PAMI 12(7): , chapter May 9, 2005 p.25/25