Geometry-based Image Segmentation Using Anisotropic Diffusion TR November, 1992

Transcription

1 Geometry-based Image Segmentation Using Anisotropic Diffusion TR November, 1992 Ross T. Whitaker Stephen M. Pizer Medical Image Display Group Department of Computer Science Department of Radiation Oncology The University of North Carolina Chapel Hill, NC To appear in "Shape in picture: the mathematical description of shape in greylevel images", (Y.-L. 0, A. Toet, HJ.-A.M. Heijmans, D.H. Foster, P. Meer, eds.). Springer-Verlag, Hiedelherg, This research has heen supported by NIH grant #POl CA UNC is an Equal Opportunity!AjJi17111Jtive Action lnstitulion.

2 Geometry-based image segmentation* using anisotropic diffusion Ross T. Whitaker Stephen ill. Pizer Department of Computer Science, University of North Carolina, Chapel Hill, North Carolina U.S.A. Abstract. Segmentations which are based on the geometry of the intensity surface, or height function, of an image can be useful for describing visual features in the image. Some visual features can be characterized as boundaries between geometric patches. We define geometric patches as regions in an image where geometric descriptors vary slowly over the support of the region. The boundaries between these patches represent discontinuities in the higher order geometry of the intensity surface. Reliable and accurate characterizations of local geometry can be quite difficult because stable measurements of the differential structure of images are obtained only by averaging over a finite aperture, which introduces spatial uncertainty into such measurements. In this work, the scale of differential measurements is established by using an anisotropic diffusion process that generates 1 piece-wise constant approximations to geometrical descriptions of images. The types of the geometric features as well as the definition of boundaries depends on the types of visual features that one wishes to measure. This framework raises some questions regarding the type of discontinuities that can be preserved while maintaining piece-wise constant functions, and how the resulting segmentation can be used to detect visual features. Keywords: segmentation, differential geometry, anisotropic diffusion, evolutionary systems. 1 Introduction 1.1 Geometric Patches An image is a function, f, that maps points in the image space, I C IRn, to some subset the set of real numbers, so that f : IR", IR, where the range is called the intensity or sometimes luminance. The graph of this function is the intensity surface. While this surface is an-manifold in a n+l dimensional space, it is also a height function with a single distinquished direction or a.xis that represents *This research has been supported by NIH grant #POl CA47982

3 the intensity of the image. The specification of contiguous regions in the image space such that the regions depend on the shape properties, or geometry, of this height function is a geometry-based zmage segmentation. An example of this kind of segmentation is the partitioning of an image into regions that are based on slowly varying luminance or height where the resulting boundaries forming luminance 'edges', that are often detected as areas of high gradient magnitude. This example relies on the most basic property of the intensity surface - its height or position along the intensity axis. Another example is the segmentation of images on the basis of extremum in height and the flanks or hillsides associated with these 'peaks' and 'pits. Such segmentations have been created on the basis of gradient directions and water sheds [13] [10] [14]. Second order properties such as the mean curvature of the intensity surface or the second derivative in the gradient direction can also serve as the basis for such segmentations. X Fig.l. Local shape of the image intensity surface is repre~ent measurments made at every point in the image space. by a finite set of local We consider segmentations which group neighborhoods in images into regions on the basis of the homogeniety or similarity of the local shape within those neighborhoods. The local shape at a point in the image is the differential geometry of the intensity surface about that point. In the first example above, local shape has implicitly been reduced to the height of this intensity surface- a zero order property. In that case measuring the simliarity among points consists of measuring the intensity at various pla('.es in the image and then comparing these values in local neighborhoods. For higher order properties of local shape, the notion of similarity between the shape of two or more surface patches is ambiguous because there are many degrees of freedom which can be compared. This indicates that two important decisions must be made when constructing segmentations on the basis of geometry. The first decision is the order of the local geometry that is to be measured and compared. Representations of local

4 surface structure must be finite and so comparisons are made between approximating functions to the intensity surface at each point in the image. The order of this approximation dictates how extensively local geometry is represented for the purpos.e of comparison. The second decision is the nature of the comparison, or the metric, that is used in determining similarity. Such a metric quantifies the differences in shape between two surface patches. Regions are places in an image where variations in geometry are small within local neighborhoods and boundaries are the loci- of places where variations in geometry are large. Deciding how large variations should be in order to constitute a boundary can be difficult. We do not address this question directly but point out that there are several widely used methods for making this decision in a discrete fashion if, indeed, a discrete segmentation is the goal. One option is to quantify the variation in local shape and then threshold this measure. Points where geometry changes at rates above the threshold are defined as boundaries. Another option is to specify boundaries as critical points in the rate of change of local geometry, as does the Canny edge operator [1]. 1.2 Patch Boundaries And Visual Features In order to quantify differences in local shape we choose to represent_ shape as a position in a finite dimensional feature space. The set of measurements which determine this feature space is called a geometnc description of the local surface shape. The similarity of shape between two points in the image is inversely related to the distance between the t.wo corresponding points in this feature space. We define dissimilarity as the measure of the rate of change of geometric descriptions as represented in the feature space and, thus, boundaries are regions of high dissimilarity. A number of visual features can be expressed as the boundaries between geometric patches. This is true of luminance edges' described above. 'Ridges' or 'creases' can be expressed as the boundaries of geometric patches that contain similar first order information. Laplacian zero crossings (also considered a type of 'edge') are, of course, the boundaries of regions which are grouped on the basis of the sign of mean curvature. There exists a duality between features such as these and the geometrical properties of the patches which they enclose. Segmentation is a method for 'detecting' or 'extracting' the features that correspond to the boundaries of such patches. Furthermore, this analysis suggests that the appropriate geometry and dissimilarity measures should depend on the kind of features one wishes to characterize. The remainder of this paper will focus on a number of questions that arise when constructing segmentations using t.he above framework. We describe a segmentation method using an anisotropic diffusion process. This process is generalized to account for geometry and scale, yielding a technique which appears to be robust with respect to noise but which preserves high frequency information about the boundaries of regions that are of sufficient size.

5 2 Geometry-limited diffusion 2.1 Describing local geometry The local shape of the intensity surface at every point in the image is represented through a finite set of scalar values. These values constitute a set of geometric descriptors and an associated feature space. There are a variety of possibilities for such representations, but for this work we choose the Taylor series expansion of the local intensity surface in Cartesian coordinates. We do this for two reasons. First, the coefficients of the Taylor series can be obtained directly from derivatives of the intensity surface, and these can be measured from digital image using linear filters. Second, in analysing the behavior of this system it is useful to be able to rely on the linearity of these operators and the orthogonality of these measurements. A set of derivative measurements on the original image create a vector-valued function f : IR" >--!Rm where the bold f =!J,..., fm designates this function as vector-valued, n is the dimension of the image space, and m is the dimension of the feature space, which is also the number of derivative measurements used to represent or encode the shape of the surface. 2.2 Scale And Anisotropic Diffusion Stable differential measurements of digital images are obtained only through some neighborhood operator or kernel. Requiring the appropriate symmetries indicates that such kernels should be Gaussian blobs or derivatives of Gaussians, which have also been shown to be excellent models of the receptive fields in natural vision [18]. Koenderink [.5] [4] [6] has shown that the use of Gaussian kernels as test functions introduces a scale parameter into the measurements that result from these kernels and that different kinds of measurements require test functions of different scales. He suggests a scale space where measurements of the image are made over a continuous range of scales. The convolution of the image with a test function is equivalent to blurring the original image before taking the measurement. Gaussian kernels are the fundimental solutions to the diffusion equation also called the heat equation, where scale replaces the time or evolution parameter, t. 8f c\7. vf = (!) at This blurring is essentially a low pass filtering, where the bandwidth of the filter is determined by the choice of scale in the Gaussian kernel. Indeed, the signal to noise ratio of measurements made with Gaussians can improve with increasing scale [11]. However, low pass filters such as the Gaussian can have adverse effects on the characterization of objects or features whose shapes depend on high frequency information. This phenomenon is important when considering higher order measurements, which are susceptible to high frequency noise but degrade rapidly under low pass filtering. Nonuniform diffusion has been proposed [3] [8] [7] as an alternative to the uniform scale space as described by the diffusion equation. More specifically,

6 edge-effected diffusion incorporates a variable conductance term which limits the flow of intensity according to local gradient information. Solutions to this equation have been shown reduce unconelat.ed noise while preserving, and even enhancing, edges. Several authors have proposed edge-affected diffusion in the form of the equation, v g(lv JIJv 1 = ~;. (2) where f is an image; g, the conductance modulating term, is a bounded, positive, decreasing function of lv fl. and t is the time or evolution parameter. The solutions of this equation can be thought of as a kind of anistropic or nonuniform scale space. When the original image is used as the initial condition, this process produces, over time, a set of smooth regions with nearly constant luminance, separated by step edges. It has been shown that this process bears a strong mathematical resemblance to a class of relaxation algorithms which seek piece-wise constant approximations to images [7] [15]. These algorithms typically define energy functions which penalize those images for differing too much from the input image and for having intensity variations other than steps at edges. Minimizing such energy functions using iterative techniques such a-.; gradient descent yields processes which are similar, if not identical, to (2). This paper discusses a framework for anisotropic diffusion that generates piece-wise constant approximations to higher order information. Although we discuss primarily nonuniform diffusion equations as in (2), we do so with the understanding that this framework has similar implications for a broad range of regularization problems which are solved by means of a constrained smoothing process. 2.3 Scale within diffusion Our earlier work [16] has shown that uniform scale can be incorporated into the gradient measurement that is included in the conductance term of (2). This scale can account for the unreliability of local gradient measurements in the presence of correlated and uncorrelated noise. This approach is consistent with the notion that all image measurements have an associated scale, and that the appropriate scale or scales may depend on the data and the task at hand [6]. Furthermore, by decreasing the scale at which the gradient is measured over time, one can obtain boundary information that reflects both small scale and large scale gradient information. This results in the multi-scale anisotropic diffusion equation, V' g(iv'g(srfi)v'/= ~{, ( 3) in which ''G(s)*" denotes convolution with a Gaussian kernel of a particular size s(t), which is itself a function of the time parameter and generally decreases as the process evolves. This process describes a continuous 'trade off' between the isotropic and anisotropic scale, where the scale of isotropic measurements used in the conductance te!-m decreases a.'3 the image undergoes progressively more anisotropic blurring.

7 2.4 Multi-valued diffusion \Ve have generalize the anisotropic diffusion of (2) in order to incorporate representations of surface shape. Instead of a single intensity image, diffuse a multivalued {vector-valued) image which is a function f: IRn r- IRm. The diffusion process introduces a time or evolution parameter, t, into the function f: IRn x IR+ r- IRm so that there is a multi-valued function at each point In time or each level of processing. The multi-valued diffusion equation is or v g(d(g(s) * f))df = at, (4) where V: IRn r- rn.+ is a dissimilarity operator. The convolution, G(s) * f, incorporates the above notion (from ( 3)) of time varying, uniform scale. Df is the derivative off in the form of a matrix, also called the Jacobian. The conductance, g, is a scalar, and the operator "\7 " is a vector that is applied to the matrix gdf using the standard convention of matrix multiplication. Equation ( 4) is a system of separate single-valued diffusion processes, evolving simultaneously, and sharing a common conductance modulating term. The boundaries are not defined on any one image, but are shared among (and possibly dependent on) all of the images in the system. 2.5 Diffusion in a Feature Space The behavior of the system ( 4) is clearly dependent on the choice of the dissimilarity operator, D. ln the single feature case, m = 1, the gradient magnitude ' proves to be a useful measure. That is, D f = ('V f 'V f)' For higher dimensions we evaluate dissimilarity based on distances in the range off, also called the feature space. The dissimilarity operator is constructed to capture the manner in which neighborhoods in the image space (n dimensional) map into the feature space. The dissimilarity at a point ;z, 0 E IRn in the image space is a measure of the densdy of space in the neighborhood of x 0 after it is mapped to range, or feature space. If the resulting space is dense. it will indicate that the neighborhood of zo has low dissimilarity. We propose a dissimilarity that is the Frobenius (root sum of squares) norm of the Jacobian. If J is the Jacobian off, with elements hi, then the dissimilarity is (5) We choose this norm for several reasons. First, this matrix norm is induced from the Euclidean vector norm so that in the case of m = 1, this expression is the gradient magnitude, as in edge-affected diffusion. Second, the square of this norm (as it often appears in the conductance functions [7]) is differentiable. Finally, this norm when applied to geometric objects (tensors) always produces a scaler which is invariant to orthoganal coordinate transformations.

8 The dissimilarity can be generalized to account for local transformations in the feature space. If one considers coordinate transformations (rotations and rescaling of axis) to be changes in the relative importance of features in the calculation of distance, then the dissimilarity measure could allow the relative importance of various features to vary depending on the position in the feature space. If <P(y), for y E!Rm, is the local coordinate transformation. The dissimilarity operator is generalized: v.r = II<PJII (6) The local metric, if>(y), provijes a A.exibilit.y in defining distances in the features space. Each position in the feature space represents a different local shape. Thus, <P(y) is precisely the shape metric that is described in section 1.1. When considering feature spaces that consist of geometric features this metric is essential for comparing incommensurate quantities associated with axes of these spaces. It is also necessary in order to construct dissimilarity measures that are invariant to ones initial choice of spatial coordinates. 2.6 Diffusion of Geometric Features The differential measurements that comprise the Taylor series description of local surface shape form feature vectors that become the initial conditions of a multvalued diffusion equation. In the feature space of geometric features, each point corresponds to a different shape, and the metric cp quantifies these differences in shape. One should choose <P so that it creates boundaries that are of interest and choose the scale, s(t), in the diffusion equation so that unwanted fluctuations in the feature measurements (noise) are eliminated. As anistropic scale increases, measurements can be taken from the resulting set of features can be made with progressively smaller Gaussian kernels, and decisions about presence of specific geometric properties can be made on the basis of those measurements. This strategy is depicted in Figure 2. Our assertion is that information about (N+1)th order derivatives can be obtained in three steps. First, make differential measurements of the image up to Nth order using derivative of Gaussian filters. Second. regularize these derivatives in a way that breaks these 'images' into patches that are nearly piece-wise constant. Third, study the boundaries of these patches and make decisions about the presence of higher order features which include (N+l)th order derivatives. 2.7 An Example: First Order Geometry and Ridges We have constructed such a system in order to locate 'ridges' or 'creases' in digital images. We refer to the definition of ridges described by Gauch [2] as loci of extremum of curvature in the level curves of the intensity surface. Level set curvature is a second order property that describes the rate of change of direction of tangents to the level curves. Notice that the height of the intensity surface does not enter into this definition, so we can encode the local image geometry with

9 /7-g original image geometric measurements multi-valued diffusion decisions Fig. 2. Geometry-limited dlffu~ion on a single valued image. in which features are geometric measurements made two features /x and /y, the pair of first partial derivatives of intensity. Effectively we have modeled each local surface patch as a plane of a particular orientation which is specified by these two features. These two measurements comprise a two dimensional feature space. We use the tilde to denote the fact that these features are undergoing nonlinear diffusion, but have the image derivatives as their initial conditions. VVe define a dissimilarity operator \vhich captures only changes in the direction of the vector created by this pair of features. This can be done by using the local metric or by computing the Jacobian on a pair of intermediate features that have been normalized with respect to the Euclidean length of this vector (i.e. first map the features to the unit circle). Notice, this normalization makes explicit the fact that this distance measure, or dissimilarity, is invariant to any monotonic intensity transformation although the pair of features, fx and }y, are clearly not. Our experiments [17] show that this process forms sets of homogeneous regions which have similar gradient directions and which are the 'flanks' or 'hillsides' in the intensity surface of the original image. The boundaries of these regions appear to correspond well with the ridges in the original image. Because of the tendency of this process to produce piece-wise constant solutions, the boundaries are very local and allow for easy detection with one of the techniques already mentioned. The results of this process are robust with respect to noise, and the use of scale in measuring dissimilarity allows us to control the size of these pat.ches without distorting the shapes of the boundaries between patches. (7) 2.8 Time Behavior of Invariants We wish to construct a process which produces results that are independent of the coordinate system that is used to describe the image space. In particular, the results should not be influenced by translations and rotations of the original image. The encoding of local geometry in terms of coefficients of Taylor series forces the choice of a coordinate system. The absolute position of points in the

10 feature space will depend on this choice. because the axis of the feature space are sets of directional derivatives that are aligned with the coordinate axis in the image space. This presents a serious question. How can one construct a system which produces invariant results but which relies on a feature space that is intimatelj.tied to our choice of coordinate systems'? The answer lies in the dissimilarity measure, which defines distance in the feature space. Positions in the feature space need not be invariant to orthogonal transformations in order to obtain invariant results. It is essential, however, that the relative positions of points and the distances between those points be invariant. To ensure this in variance. express the dissimilarity as a geometric invariant of the intensity surface of the original image. Given this, the dissimilarity will be invariant at the start of the process because it is a geometric invariant of the intensity surface. As the process progresses, however, the terms of the invariant will change so that they no longer resemble derivatives of the intensity surface. Will dissimilarity remain invariant as the nonlinear, nonuniform diffusion progresses? The following proposition shows that geometric invariants constructed from the original geometric features (derivatives of the intensity surface) remain invariant throughout the diffusion procf'ss. It. concerns polynomial invariants as described by ter Haar Romeny and Florack [11] [12], where polynomials are expressed using the Einstein summation convention of summing expressions with like indices over the basis. For example, the square of the gradient magnitude of a function, f : IR.2...,_ IR. is expressed as: fdi = L [; h = fx' + fu 2 JE.r.y (8) while the Laplacian is: fii = L /ii = fxx + fyy ie.r.y (9) _Any expression of this form with the property that all indices are paired (no free indices) is a scalar that is invariant to orthogonal coordinate transformations (rotations and translations), and is thereby independent of ones choice of coordinates. Proposition 1. Give11 a geometry~ limited di}juston system as described in section 2. 6 wzth.: 1. a set of features, f = ft,....fm, with. milia/ values that are derivatives of a smooth zmage, 2. a dissimilarity measure, Vf that is an in1'anant expression of those features and their first order derivatives, and 8. solutions which are analytic m llme, then any function P that depends on theses jeatu1 es and has an initial value (t = 0) that can be expressed a.s a poh;nonuai i1u,arwnt of the intensity surface 1s invariant for all t 2: 0.

11 Proof The proof relies 011 t.lte ahalytic nature of the solutions to express the invariant as Taylor series. Using the chain rule and the diffusion equation, express each time derivative of the invariant in terms spatial derivatives of the initial values (t = 0) of the features. a m fj of m a atp(f,tj = {; aj/(f,tj 8 ; = {; aj/(f,t)'v. u(d(flj'v f, (11) Inductively each term in this series in (10) is an invariant. Consider the kth term ji.p(f,t) from (10), and assume (inductive assumption) it is invariant and is also a function of the features and higher order derivatives of those features. Then the time derivative %tk,; 1 1 P(f, t) must also be invariant. This is true because the time derivative of any of these terms can be converted as in (11) to a spatial derivative using the multi-value diffusion equation. With an invariant conductance term, the operator V g( D(f) )'V introduces indices into expressions only in pairs, thus maintaining the Einstein summation convention. The first term of the Taylor expansion, lip( f. 0), is invariant by assumption because it is a polynomial invariant. consisting of derivatives of the intensity surface. Inductively, the entire series of ( 10) is invariant, a.ud by analyticity so is P(f, t). This result is important. because it states that the dissimilarity measure as well as any other function of polynomial invariants, remains insensitive, throughout the process, to the original choice of coordinates. (10) 3 Conclusions Characterizing images in terms of the differential structure of the intensity surface allows one to examine the way shape varies across the image space and to create segementations on the basis of local shape. Regions in the image where local shape is homogeneous can are grouped on the basis of geometry. This approach requires a model of local shape that captures relevant properties with a finite representation as well as a definition of (distance' that allows one to quantify the similarity of two shapes. Anisotropic scale can be used as a means of localizi.rtg patch boundaries in a manner that appears to be accurate and robust. The geometry-limited diffusion process produces homogeneous patches with relatively well defined boundaries while maintaining the geometric invariance of the measurements that are used to detect and characterize these boundaries. References 1. Canny, J. (198i) A computational approach to edge detection. IEEE PAMI 8 (6), pp

12 2. Gauch, J. M. {1989) A multiresolution intensity axis of symmetry and its application to image segmentation, Report TR89-047, Department of Computer Science, University of North Carolina, Chapel Hill. North Carolina. 3. Grossberg, S. (1984) Neural dynamics of brightness perception: features, boundaries, diffusion, and resonance, Perception and Psychophysics 36 (5), pp Koenderink, J.J. (1984) The structure of images, Bioi. Cybern. 50, pp Koenderink, J. J., Van Doorn. A. J. (19!:10) Receptive field families, Biol. Cybern. 63, pp Koenderink, J. J. {1990) Solid Shape, MIT press, Cambridge, Mass. 7. Nordstrom, N. (1990) Biased anisotropic diffusion- a unified regularization and diffusion approach to edge detection, Image and Visual.Camp. 8 (4), pp Perona, P., Malik. J. (1990) Scale-space and edge detection using anisotropic diffusion, IEEE Transactions on Pattern Analysis Machine Intelligence 12, pp Pizer, S.M., Coggins, J. M., Burbeck. C. A. (1991) Formation of image objects in human vision, Proceedings of the Conference on Computer Assisted Radiology, Springer-Verlag, Berlin. 10. Pizer, S. M., et. al. (1990) Descriptions of intensity structure via scale and symmetry, Proceedings First Conference on Visualization in Biomedical Computing, IEEE Cat. #90TH , pp Romeny, T. H., Florack, L. (1991) A multiscale geometric approach to human vision. In: Hendee, B., Wells, P. N. T. (eds.) Perception of Visual Information, Springer Verlag, Berlin. 12. Romeny, T.H., Florack. L., (1991) Scale ~pace: its natural operators and differential invariants. In: Colchester, A. C. F., Hawkes, D. J. (eds.), Lecture Notes in Computer Science 511, Springer-Verlag, pp Rosin, P. 1., Colchester, A. C. F., Hawkes, D..I. (1991) Early image representation using regions defined by maximum gradient. profiles between singular points, To appear in Pattern Recognition. 14. Serra, J.P. (1982) Image Analysis and \lat.hematical Morphology, Academic Press, New York. pp Snyder, W., Han, Y.S., Bilbro, G., Whitaker. R.T., Pizer, S.M. (1992) Relaxation and image diffusion, unpublished. 16. Whitaker, R. T., Pizer, S.M. {1991) A multi-scale approach to nonuniform diffusion Report TR91-040, Department of Computer Science, University of North Carolina, Chapel Hill, North Carolina To appear in CVGIP: Image Understanding. 17, Whitaker, R. T. (1991) Geomet.ry-limit.ed Diffusion In The Characterization of geometric patches in images, Report TR91-039, Department of Computer Science, University of North Carolina, Chapel HilL North Carolina. To appear in CVGIP: Image Understanding. 18. Young, R. A. (1986) Simulations of human retinal function with the gaussian derivative model, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition,.June : This article was processed using L TEX with SHAPE.sty modified from LMAMULT