Invertible Apertured Orientation Filters in Image Analysis

Transcription

1 International Journal of Computer Vision 31(2/3), (1999) c 1999 Kluwer Academic Publishers. Manufactured in The Netherlands. Invertible Apertured Orientation Filters in Image Analysis STILIYAN N. KALITZIN, BART M. TER HAAR ROMENY AND MAX A. VIERGEVER Image Sciences Institute, University Hospital Utrecht, HP E01.334, P.O. Box 85500, 3508 GA, Utrecht, The Netherlands Received ; Accepted Abstract. A unitary approach for locally apertured orientation analysis of 2D and 3D scalar images is proposed. The size of the local aperture (the scale) needed for the orientation representation induces in general a lost of spatial acuity, or blur. Our construction permits a compensation of the blur by a reconstruction procedure. For this purpose, a special scale-dependent orientation bundle (map of the visual space into function of both position and orientation) is build from the local Gaussian-derivatives jet of a scalar image. In this construction there is an invertible relation between the orientation bundle and the original image. This invertible transformation is used to regain the original acuity in the spatial domain after analyzing orientation features at any given scale. The approach turns out to be highly effective for the detection of elongated structures and for removal of elongated artifacts in 2D images. Keywords: integral transformations, orientation analysis, scale space, complete systems 1. Introduction and Motivation Local apertured filters are one of the basic tools for studying structures in images in local pixel neighborhoods. In the simple cases the filter can represent just an averaging operation in an isotropic neighborhood of a given size taken around every image point (e.g., Gaussian filter). The size of the filter aperture normally is referred as the scale of the filter. In more advanced applications, for example in the steerable filters, the filter can include a variety of additional parameters such as orientation. Another example of local apertured methods is provided by the Gabor filter where both orientation and spatial frequency are locally measured. In the above examples the analysis is based on a finite window (typically defined via a Gaussian kernel) that represents the region around every pixel where features are being measured. The size of this window (aperture or scale) relates inevitably to the presence of a spatial blur, or loss of spatial acuity. The more complex the features, the wider the windows that have to be used and therefore, the lower the accuracy that can be guaranteed. In the case of a Gaussian filter this effect is simply manifested by the low-pass filtering properties of the Gaussian kernel. In practical applications, the finite aperture or scale, leads in general to loss of spatial resolution of the detected features. The goal of this paper is to construct locally apertured orientation filter that avoids the acuity loss. Our construction provides a complete representation of the image through the orientation degrees of freedom. Such filter allows for an inversion linear transformation to reconstruct the original, spatial degrees of freedom. We construct the filter by using the local Gaussian derivative jet. The harmonic components of the projected image are special linear combinations of the local derivatives. We call in what follows the filtered image an orientation bundle over the visual space. Such bundles are functions of both the position and the local orientation defined over the space of the original image. At every spatial point this bundle carries information about the orientation properties of the image in the given neighborhood. An essential feature of our orientation bundle is that it provides a complete representation of the original data for any aperture size. In more technical terms this means that the inverse transformation can be

2 146 Kalitzin, ter Haar Romeny and Viergever performed at any given scale so in fact we have a one parameter family of image representations labeled by the scale parameter. In more intuitive terms, our orientation filtering provides a way to redistribute spatial and angular information. If we collect data from a given angular sector, then the further we look, the higher is our angular resolution. But summing up data over long distances inevitably increases the spatial blur of our data mappings. If on the other hand we collect data from only the closest point neighborhoods, then the spatial acuity will be high, but the difference between the content in the adjacent angular sectors will be small. In this sense the proposed approach gives a mathematical construction that balances between spatial and orientation precision and permits the transitions between the two to be controlled by the aperture parameter. The introduction of proper orientation parameters into the image provides also a way to detect and possibly segment extended structures as curves, ends, curve bifurcation etc. Nonlinear differential invariants defined on both spatial and angular derivatives highlight the locations of the corresponding structures at the desired scale. Invertibility, on the other hand, can be used to restore the original spatial configuration and positional accuracy of the detected structures. In the present paper along with the general construction we show application to medical imaging. Our analysis provides also a method for removing elongated artifacts from images. In the last case the reconstruction properties of the filter are used to preserve the rest of the image structures as close to the original as possible. Our main theoretical analysis is based on the use of infinite number of local orientation degrees of freedom, just the same way as every continuous image or signal is presumed to contain all spatial frequencies. In practice, of course, only a finite number of spectral components can be handled by the numerical methods. In our case this means that we can only deal with a finite number of rotational harmonic components. Our construction offers in this case two possibilities. We can give up the exact invertibility and assume only approximate reconstruction scheme or instead, we can preserve the completeness of the representation by modifying the shape of the apertured window. This analysis represents the orientation-discrete version of our method. The applications and examples in the paper are of 2D images, but we have developed the scheme for 3D orientation analysis as well. In the last case the discrete version is labeled by the representation indices of the corresponding spherical harmonics. The paper is organized as follows. In the next section we first reference to results in related works in the areas of scale space, orientation filters and wavelets. In Section 3 we give a general definition of linear orientation bundles and subsequently introduce a class of invertible orientation filters for 2D and 3D images. We focus on the special multi-scale orientation bundles derived from the local Gaussian derivative jet. Applications are presented in Section 4, where the invertible Gaussian bundle is used in tasks of segmentation of thin elongated structures in noisy medical images. The method is also applied for removal of elongated artifacts in bore-hole geophysical data. Technical details along with the relevant notations and conventions used in the paper are summarized in the Appendix. 2. References to Related Works Local (differential) multi-scale analysis of gray-scale images has now been well understood within the frameworks of linear scale-space theory (Koenderink, 1984; Lindeberg, 1994; ter Haar Romeny, 1997). Point neighborhoods can be studied at a given scale by invariants constructed from the local jet of Gaussian derivatives (Florack et al., 1996). Multi-scale local orientation analysis provides another representation of the pixel neighborhoods that includes local orientation angle. Recently, much attention has been given to the construction of so-called steerable orientation filters (Perona, 1994; Freeman and Adelson, 1991; Simoncelli, 1994; Simoncelli et al., 1991; Michaelis, 1995). In these approaches, the dependence of the filter on the local orientation angle can be obtained as a linear superposition of its response over a finite set of angles. Sterability improves the speed of the method as well as it reduces the storage requirements. In our work the steerability restriction is realized by orientation functions that contain only a finite number of harmonic modes. This restriction is equivalent to the classical steerability condition, it is explicitly covariant and it can be generalized to any local group. Group theoretical approaches that generalize the local orientation analysis has been developed in the works (Michaelis, 1995; Segman and Zeevi, 1993; Hel Or and Teo, 1996). In our work we use only basic facts concerning the rotation group representations in 2D and 3D. As a fundamental group-theory reference we give Miller (1972), Conwell (1990). A central element in our paper is devoted to a class of invertible orientation filters constructed from Gaussian

3 Orientation Filters in Image Analysis 147 derivatives. This means that the system of all position and orientation values can be used to reconstruct the original image, despite the amount of space-blur caused by the finite-aperture filters used to construct the orientation bundle. Derivative constructs has been introduced in Simoncelli (1994). The invertibility property of our transformation puts it in a close analogy with the Gabor-frame methods (Zibulski and Zeevi, 1997), steerable pyramid (Greenspan et al., 1994) (also based on a discrete Gabor transform) and especially with the 2D Gabor wavelet model (Lee, 1996) (see also the extended reference list there). The last model addresses the same fundamental questions about the exchange between spatial and angular information. An essential difference between our approach and the wavelet one is that the last uses all scales to invert the filter, while in our case we can invert the local bundle for each scale separately. In more precise terms, the wavelet approach provides another overcomplete functional representation where the scale is also a spectral parameter. In our case the only integral spectral parameter is the orientation angle (or spherical angles in 3D) while the scale can be viewed as external dummy parameter. The last circumstance allows the information contained in the original data to be preserved at any scale. A band-pass orientation filter possessing invertibility property has been introduced and studied in Simoncelli and Freeman (1995), Karasaridis and Simoncelli (1996). The last construction uses direct partition of the Fourier filter domain while our method relies on smooth Gaussian-derivative functions with overlapping supports to cover the frequency volume. Application of local steerable wedge filter for detection of particular image structures has been introduced in Simoncelli (1996). A class of non-linear invariants derived from orientation filters are studied in Simoncelli (1996) and applied for highlighting of lines, end-points and junctions. The analysis there is based on a general over-complete set of invariants built from the local harmonic amplitudes. In our paper we use a different approach based on a particular dedicated non-linear operator that enhances local elongated structures. 3. Invertible Orientation Analysis 3.1. General Construction An orientation bundle defined on R d is a function (see also the Appendix) F(x,θ):R d S d 1 R (1) where x indicates all spatial coordinates in R d and θ indicates all angular parameters (coordinates of the (d 1)-dimensional hyper-sphere S d 1 ) defining a local orientation in R d. In addition, we will require that the bundle is a (in general infinitely dimensional) representation of the SO(d) rotation group in R d. When rotating over a set of group parameters (angles) α, the rotation group is acting both on the spatial arguments x R(α)x, where R(α) is a d d matrix from the vector representation of SO(d), and on the angular arguments via its homogeneous action θ T (α)θ on the factor space S d 1 SO(d)/SO(d 1). Here α represents the whole set of SO(d) parameters, but the stationary sub-group SO(d 1) leaves both the radial vector x and the local orientation θ invariant. We can assume therefore, that α SO(d)/SO(d 1) and are of the same nature as θ. We will be particularly interested in linear orientation bundles defined for gray-scale d-dimensional images via the integral transformation F(x,θ)= d d x (x,θ)l(x +x ). (2) R d Here L(x) is the original image and the kernel (x,θ) represents an orientation bundle in its own rights. We will assume that this kernel has a window-type of x-dependence, for example it can be with a finite support or with fast decaying behavior. Typical examples of such kernels can be given by various Gaussian derivatives. We leave the exact x-dependence for the next subsection and concentrate here on the issue of orientation-covariance linking the x and θ behavior of the kernel. Not all orientation bundles provide suitable kernels for local orientation analysis. We want all directions θ in (2) to be taken with the same weight and therefore, the shape of the kernel has to be a rotated version of a single given function: (x,θ)= (R(θ)x,θ 0 ), (3) where θ 0 is any conventional zero-direction on S d 1 (for example we can choose θ 0 ={0,0,...}). An alternative form of the above relation (3) is the covariance constraint: (x,θ)= (R(α)x, T 1 (α)θ). (4) In other words, if we spatially rotate the filter with a set of parameters α we will obtain the same filter-bundle but with a shifted back angular argument.

4 148 Kalitzin, ter Haar Romeny and Viergever Before proceeding further with some particular cases, we define here an invertible orientation filter (x,θ) as a filter for which there exists another covariant orientation filter (x,θ)such that d d x Dθ (x, θ) (x + x, θ) δ(x), (5) R d S d 1 where δ(x) is the Dirac δ-function, Dθ is the covariant spherical volume element (in 3D this is the familiar sin(θ) dθ dφ term), and θ is chosen to be the antipodal point to the one defined by θ on the d 1-dimensional sphere. The last choice is purely conventional and it allows for self-invertible filters where as will be explained later in the paper. From (2) and (3) it is clear that if an orientation filter is invertible then the original image can be reconstructed from its orientation bundle F(x,θ) D Orientation Analysis In this sub-section we consider the two dimensional case. Let L(x i ), i = 1, 2 be a gray scale image defined on R 2, then we can define a general orientation linear bundle with the help of an oriented aperture function (filter) (x,θ) where θ is a single SO(2) angle that parameterizes a circle. Such functions, especially the so-called steerable filters (Perona, 1994; Freeman and Adelson, 1991; Simoncelli and Farid, 1996), have been used widely for local orientation analysis. As in (3) an oriented filter is constructed from a given scalar function φ(x) and its rotations over all angles θ. Explicitly, if we decompose φ(x) into SO(2) irreducible representations φ(x) = 1 φ n (ρ x )e inθ x ; 2π ρ x n x x2 2 ;θ x Arg(x 1, x 2 ), (6) we have for the rotated function (x,θ) (x,θ)= 1 φ n (ρ x )e in(θ θ x ). (7) 2π n In this last construction (7) the orientation covariance constraint (4) is resolved explicitly. Now we can define the linear orientation bundle F(x,θ) derived from the image L(x) as in (2). If we decompose the so constructed bundle F(x,θ)into harmonic modes as (for notations and normalization conventions see the Appendix): F(x,θ)= 1 F n (x)e inθ, (8) 2π then n F n (x) = d 2 x n (x )L(x + x ) (9) where (see also (7)) n (x) = φ n (ρ x )e inθ x (10) are the angular momenta of the oriented filters. In what follows we consider only real filters implying (x,θ) = (x,θ); n (x)= n (x); φ n (ρ) = φ n (ρ). (11) It is clear that all possible linear orientation bundles associated with an image L(x) are parameterized by the infinite set of (in general complex) radial functions φ n (ρ), n Z. Note that the covariance condition (10) implies in complex notations (see the Appendix) the following generic form of the filter harmonic momenta: n (z, z) = z n Ɣ n (z z), n (z, z) = z n Ɣ n (z z), n 0 (12) where Ɣ n (z z) z n φ n ( z ) and reality (11) implies Ɣ n (z z) = Ɣ n (z z). Alternatively, we can consider the integral transformation (2) in the frequency domain. Then for the Fourier amplitudes, using the convolution properties of the Fourier integrals we have (see the Appendix for the definitions and the normalization conventions): F(ω, θ) = 2π φ( ω,θ) L(ω). (13) For the (Fourier transformed) angular momenta, we get: F n (ω) = 2π n ( ω) L(ω). (14) Here n (ω) are the angular momenta of the aperture functions in the frequency representation. It is easy to

5 Orientation Filters in Image Analysis 149 show that the covariance condition (10) holds also in the frequency domain: n (ω) = φ n ( ω )e inθ ω, ω ω 12 +ω 22 ;θ ω Arg(ω 1,ω 2 ), (15) where φ n ( ω ) are arbitrary radial-frequency functions. The last can serve as an alternative set of filtergenerating functions. Note that the reality constraint in the Fourier representation is n (ω) = n ( ω). (16) Invertibility (3) means existence of a transformation of the orientation bundle F(x,θ)to the space of ordinary functions in R 2 : M(x) = d 2 x dθ (x,θ)f(x+x,θ+π) (17) such that it restores the original image M(x) L(x). (18) We can look at Eq. (17) as an inverse transform of (2) and the θ + π θ angular shift in the last term is the antipodal orientation transformation for the 2D case. From the harmonic expansions (7) and (8) we can express the field M(x) alternatively as M(x) = n d 2 x ( 1) n n (x )F n (x + x ), (19) where n (x) ψ n (ρ x ) exp( inθ x ) are the harmonic momenta of the aperture function ψ(x,θ) as in (10). The factor ( 1) n comes from the θ θ + π replacement in (17). Invertibility condition (18) imposes a constraint on the set of functions φ n (ρ) and ψ n (ρ). The simplest way to obtain this constraint is to consider a 2D Fourier transform of all 2D fields. Then formula (19) takes the form: M(ω) = 2π n ( 1) n n ( ω) F n (ω). (20) From (20), (14) and the conventions from the Appendix we obtain the invertibility condition in the frequency domain: (2π) 2 n ( 1) n n ( ω) n ( ω) = 1. (21) This equation must hold for all ω. Equation (21) contains the Fourier spectral components of the aperture harmonic functions n (x) and n (x). In terms of the filter generators φ n ( ω ) and ψ n ( ω ) defined in (15) condition (21) becomes: (2π) 2 n ( 1) n ψ n ( ω ) φ n ( ω )=1. (22) So far we focused our attention mainly on the orientation properties of the filter. Our purpose is to find invertible orientation filters that are localized with some aperture and therefore are suitable for exploring local pixel neighborhoods. For that purpose we consider an important example of an orientation bundle with scaling properties that can be obtained from (12) by taking Ɣ n (z z,σ)= 1 σ ng n ( z z σ 2 ), (23) where the kernels g n ( ρ2 ) give a dilatation-invariant σ 2 form of the orientation filter. The proper factor of σ n ensures that all harmonic momenta are of the same dimensionality and therefore can be summed. The generic formulas (12) and (23) give the most general form of a 2D orientation filter with scaling properties. To construct a concrete example we consider a filter defined as: g n ( z z σ 2 ) = a n g(z z,σ) (24) where constants a n = a n are still arbitrary and g(z z,σ) is the Gaussian kernel at scale σ as defined in (43). The choice (24) provides a local orientation bundle constructed entirely from the components of the Gaussian derivative jet. Indeed, according to the definitions and notations of the Appendix, we have ( ) 1 z n n (x,σ) = a n e z z 2πσ 2 σ σ 2 1 a n 2πσ 2( σ z) n e z z σ 2 (25) for n 0, and the complex conjugates for n < 0 as required by the reality condition (11). We can substitute

6 150 Kalitzin, ter Haar Romeny and Viergever Figure 1. From left to right: plot representation of the harmonic functions in (26) with n = 0, 1, 2, 3. The n = 0 component is real and only the real parts of the n > 0 components are depicted. Their imaginary parts can be obtained simply by a π 2n rotation. (25) in Eq. (9) and then introduce L(z, z,σ)as the convolution of the original image L(x) L(z, z) with the Gaussian kernel g(z, z,σ). After partial integration of the derivatives we get: F n (x,σ)=a n (σ z ) n L(z, z,σ) (26) for n 0, and the complex conjugates for n < 0. The n = 0 term in (26) is the original image blurred with a Gaussian kernel of scale σ. An illustration of the first four harmonic components of (26) is given in Fig. 1. Thus, for any choice of the numerical coefficients a n for which the formal series makes sense, we obtain a local orientation bundle that can be expressed entirely in terms of Gaussian derivatives at some scale σ of the original image. We turn now to the question of reconstructing the original image L(x) from its orientation Gaussian bundle as defined by (26). Let us assume that the candidate for the inverse transform (17) is defined by functions n (x,σ)similar to (25): ( ) 1 z n n (x,σ) = b n e z z 2πσ 2 σ σ 2 1 b n 2πσ 2( σ z) n e z z σ 2 (27) Here b n are complex constants, in general different from a n. Given the definition of the harmonic coefficients (25) and (27) as Gaussian derivatives, we can substitute (45) into the condition (21). Assuming that the forward and inverse transformation are taken with two different scale parameters σ 1 and σ 2 correspondingly, we obtain: ( a 0 b 0 + (a n b n +ā n b n )(σ 1 σ 2 ν ν) )e n ν ν(σ2 1 2) +σ2 =1. n>0 (28) Let us now consider the choice for the parameters a n and b n so that M(x) = L(x) when σ 1 = σ 2. One (symmetric) option is a 0 = b 0 = 1; a n = b n = 2 n 1, n 0. (29) n! Indeed, with the choice (29) for the numbers a n, b n we obtain by summing the powers of ν ν 1 = e 2σ 1σ 2 ν ν e ν ν(σ σ 2 2 ) e (σ 1 σ 2 ) 2 ν ν when σ 1 = σ 2. Note that Eq. (30) implies: (30) M(x,σ 1,σ 2 )=L(x,λ) (31) where λ = σ 1 σ 2. Clearly, for any 0 <σ 2 <2σ 1 a de-blurring effect will be present. This effect is maximal when the two scales are equal and (if all harmonic frequencies are used) the initial image will be reconstructed. The symmetric choice of the coefficients (29) corresponds to a self-invertible bundle. From (31) we also see that λ 0 for any choice of the two scales. Therefore, a de-blurring beyond the original acuity is impossible which is in accord with the well-posedness arguments. In Fig. 2 we present the aperture function φ(x) defined by (24) and (29). The orientation bundle is obtained from a single bright spot (frame 1 in Fig. 2). The resulting field (or the impulse response function ) φ(x,45 ) is shown in the second frame in Fig. 2 for angle θ = 45. In frames 3 and 4 we present the power spectra for the original image and the aperture function. The picture shows clearly the idea of spatial versus angular exchange of information that is realized. In terms of 2D frequency amplitudes, the oriented bundle taken for a fixed angle θ performs a band-pass filtering in the spatial direction aligned with this angle. In other words, if we define longitudinal and

7 Orientation Filters in Image Analysis 151 Figure 2. From left to right: Frame 1: image with a bright spot of size of two pixels. Frame 2: Gaussian orientation filter response to the image of frame 1 and simulated for scale σ = 2.0 and angle θ = 45. Frame 3, 4: 2D power spectra for the images in frames 1 and 2 respectively. Note that the power spectrum of the bright spot is not a constant function due to the finite size of the spot. Although the oriented filter response is assymetrical, its Fourier power spectrum has no directional bias. The directional information is contained in the phases of the Fourier coefficients. The filter was constructed with maximal harmonic momentum of 127. Figure 3. Frame 1: Image of pixels with two vertical bright bars of dimension 30 2 pixels and separated by a gap of 2 pixels. Second row frames (2 5): zero harmonic mode (n = 0) of the Gaussian orientation bundle at scales 0.5, 1.0, 2.0 and 3.0 pixels. The angular resolution (number of sectors taken on the circle) is 64 and the maximal orientation harmonic momentum (n) is 31. Third row of frames (6 9): reconstructed images at the four scales. Note that the reconstruction process has used only a finite number oh harmonic modes and therefore is not exact. transversal coordinates as x θ = x 1 cos(θ) + x 2 sin(θ) and x θ = x 1 sin(θ) + x 2 cos(θ) correspondingly, then for large scales we have xθ F(x θ, x θ,θ,σ) 0. In Fig. 3 the invertibility property is illustrated for four different scales (σ 1 = σ 2 ) taken for the Gaussian invertible bundle. We see that even when the zero-mode blur becomes significant with increasing scale (second row in Fig. 3), the invertibility of the orientation bundle ensures the proper restoration of the original image (third row in the same figure). It is clear that the solution given in (29) is by far not the only solution providing invertibility of the local Gaussian filter. Condition (22) on the other hand can be viewed as a defining equation for the filter amplitudes. We will use this later for finding a steerable version of our construction D Orientation Analysis A 3D orientation filter can be constructed on the same basis as prescribed by the covariance constraint (4). If we assume standard parameterization of the sphere with θ [0,π],φ [ π, π] as spherical angles, then in Fourier representation we have (ω, θ, φ) = l=0 m= l l f l,m (ω)p l, m (θ, φ), (32)

8 152 Kalitzin, ter Haar Romeny and Viergever where f lm (ω), m = l,...,lare functions on only the 3D-vector ω {ω x,ω y,ω z } and possibly some scalar constants. These functions form a functional representation of the 3D rotational group with momentum l. The functions P lm (θ, φ) are the orthogonal spherical harmonic functions. Orientation covariance manifests itself by the absence of tensors other than the vector ω in (32). Because the vector ω is an l = 1 SO(3) irreducible representation, the generic form of the functions f l,m is (compare with (12)): f lm (ω) = ω i1...ω il i 1...i l lm Ɣ l( ω ), (33) where i 1...i l lm is a constant symmetric tensor projecting the l-tensor product of l vectors onto the irreducible representation of SO(3) with highest momentum-l. This tensor can be related to the Clebsch-Gordon coefficients (Conwell, 1990; Miller, 1972), but we have applied a direct construction to extract the irreducible representation of highest momentum from the symmetric vector product. We refer to the Appendix for the explicit construction and here we notice that the scalar function of the radial frequency argument Ɣ l ( ω ) can only depend on the representation label l and not on the component index m. As shown in the Appendix, we can choose a normalization for the tensors lm such that l m= l 1 l! i 1...i l l,m j 1... j l (l, m) = δ (i 1 j 1 δ i l) j l perm(i 1...i l ) δ (i 1 j 1 δ i l) j l, (34) where the rising and lowering of the SO(3) vector indices is always done by the invariant SO(3) unit tensor δ ij =δ j i =δ ij and we assume an index-symmetrization convention for all tensor indices enclosed in () brackets. Taking into account the orthogonality of the spherical harmonics π 0 sin(θ) dθ π π dφ P l,m (θ, φ)p l, m (θ, φ) = δ ll δ mm, (35) it is easy to obtain a self-invertible orientation filter by choosing Ɣ l ( ω ) = σ l 2 l l! e ω2 σ 2, (36) where ω 2 in the exponent is the Euclidean scalar product of the vector ω with itself. As in the 2D case, the power of the scale parameter σ is uniquely prescribed by dimensional reasons. Inserting the kernel defined by (32) (36) in the unitarity formula (5) in the 3D Fourier domain and using the fact that under the spherical antipodal transformation θ π θ; φ φ + π a representation of SO(3) with momentum l multiplies by ( 1) l, we get: ( ) (σ ω) 2l 2 l e 2ω2 σ 2 1. (37) l! l=0 As in the 2D case, the choice for antipodal transformation in (5) provides the possibility for the orientation filter to be self-invertible, or (x,θ) (x,θ). Otherwise we can take of course, the inverse filter to be the antipodal transform of Finite Angular Representations (Steerability) It is clear from both (28) and (37) that the inverse orientation transformation can only compensate the spatial aperture by using the whole infinite set of harmonic amplitudes. If only a finite number of harmonic terms is involved (in practice this is the only possibility), then the inversion relation (5) is fulfilled only approximately. Although in most applied results this approximation turns out to be quite satisfactory, we can easily modify our apertured kernels so that the invertibility relations are preserved in an exact form even with only finite number of harmonic components. To this end we substitute the Gaussian aperture function in the frequencydomain definition of the filter momenta ((25), (29) and (45) in the 2D case and (36) for the 3D filter) with an arbitrary function g(σ 2 ω ω). Keeping the form-factors (powers of ω) the same as before (they are prescribed uniquely by the covariance and scaling arguments) and also taking the same values for the coefficients (a nonunique but convenient choice that provides a direct comparison to the infinite case) we can see that the invertibility condition is satisfied for aperture kernels

9 Orientation Filters in Image Analysis 153 defined in the Fourier domain as: g(σ 2 1 ω ω) =, (38) 1 + (2σω) (2σω)2N N! where N is the label of the highest harmonic representation (highest n in 2D or highest l in the 3D case). The price for this exact invertibility is worse low-pass properties of the local apertured filter. We end the finite-n filter discussion by noticing that taking only a finite dimensional (reducible) representation of the local orientation group is an equivalent formulation of the steerability constraint. The advantage of the present scheme is that it is explicitly covariant and uses the natural description of the group-representation properties of the filter degrees of freedom (harmonic momenta). This feature of our construction is apparent in the 3D case where a steerability constraint would have been rather cumbersome and moreover there is no way to take arbitrary number of symmetrically deployed points on the sphere. Using direct spherical harmonic parametrization avoids all of the above hurdles. We do not treat in this paper the question of spatial discretization. A sub-sampling procedure in the longitudinal direction (at every bundle point) has shown that the amount of redundancy in the filtered image can be significantly reduced. These results will be published soon. In the above general constructions and in the following applications we can perform the orientation analysis at any fixed scale. Therefore the scale participates as an external model parameter. Selecting the value of this parameter lies beyond the scope of this work. Alternatively, we can apply our method for a sequence of scales and using the invertibility of the orientation bundle at any of these scales obtain the scale spectrum of the desired structures. As the constructions for different values of the scale are completely independent from each other, a scale discretization problem does not appear in our construction. 4. Applications 4.1. Detection of Elongated Structures The main motivation for our analysis has been a specific class of visual tasks that involve detection and segmentation of extended, sub-dimensional structures. Such structures are for example curves in two-dimensional images and curves and surfaces in 3D images. In some cases these objects are defined as borders between two (or more) volumes (or areas in 2D cases). Other images may contain thin, elongated structures that do not separate any distinctive volumes. Such a case is shown in Fig. 4, frame 1 where a catheter guide is visible as a thin (on places fuzzy) long structure. To solve the problem of finding extended objects, we must bring in topological knowledge in order to identify Figure 4. Frame 1: Noisy (low dose) fluoroscopy image ( pxls) of a thin catheter guide wire. In the lower left corner a broader vessel can be seen. Frame 2: Orientation analysis with the non-linear invariant (39) (scale σ = 3.0, angular resolution of 32 sectors and maximal harmonic momentum (n) of 15) and summed over all orientations (sectors). Frame 3: Inverse projection of the bundle defined by the invariant. Frame 4: Original image is superimposed with the result from frame 3. In this application, the modified kernel (38) was used.

10 154 Kalitzin, ter Haar Romeny and Viergever the features of these objects. Such feature identification can conveniently be accomplished by a nonlinear local operator over the orientation bundle. The choice of such an operator depends on the definition of the structures that we want to detect. A well constructed operator can even serve as a definition of those structures. If we locally define elongated structures as pixel neighborhoods where the largest eigenvalue of the second derivative tensor (the Hessian) is large in comparison with the second largest eigenvalue, then we have already defined also the measure of elongateness. In our approach we have richer differential structure due to the additional orientation degrees of the bundle. In this subsection we define elongated structures as orientation bundle neighborhoods where the product of the partial Laplacians in the orientation and in the transversal directions takes high values. The reason for this definition is that every point in the orientation bundle defines an orientation over the original image space. Therefore instead of determining the most probable ridge orientation (via the eigenvalues of the Hessian for example), we can estimate the elongateness in all directions. We present here only the simplest bi-linear operator which we use as a modulator for the original orientation bundle. We proceed according to the following scheme. Compute the field F(x,θ,σ) from (12), (24), (29) and for the selected scale σ. Compute an appropriate non-linear local detector over the orientation bundle. For detecting long, string-like structures we used ( I(x,θ,σ)=F(x,θ,σ) sin(θ) x 1 + cos(θ) x 2 F(x,θ,σ) 2 F(x,θ,σ). (39) θ2 The first factor in (39) selects for strong (positive or negative) responses in the original orientation bundle. The two second derivative terms in this product are modulators that take high values near thin, elongated structures as they represent a ridgeness detector in the transversal and the angular directions correspondingly. Project the bundle I (x,θ,σ)back by (17) with functions (27) and the same scale parameter. The back projection is necessary in order to restore the original spatial resolution. In Fig. 4 we compare the results without inverting (second frame) and ) 2 those after inverting (third frame). The non-inverted image is taken as integral of I (x,θ,σ)over all angles. In the last frame of Fig. 4 we superimpose the detected structures on the original image and we verify their exact location as a result of the inverse transform. We compare in Fig. 5 our method with the results obtained from a second order Gaussian derivative steerable filter. In the last case both the sum over all orientations of the filter (effectively measuring the Laplacian operator) and the maximal value projection are considered. Our conclussion is that in both Figs. 4 and 5 the inverse transformation inproves the spatial accuity of the detection results. One of the merits of the above method is that it can be used at any scale. The invertability constraint (21) and (28) is valid for any σ and therefore we can use our scheme at several spatial scales independently. This enables us to detect structures of different thickness selectively as shown in Fig. 6. The essentially non-linear formula (39) can be generalized to any number of spatial dimensions. The second factor will then be the d 1 dimensional Laplacian computed in the direction orthogonal to the current direction (defined by the set of d 1 spherical angles θ). The third factor in (39) extends to the d 1 dimensional spherical Laplacian. We note here that the computational complexity increases rapidly with the dimension of the image. In the important 3D case, for example, the harmonic expansion up to highest spherical momentum L involves L l=0 (2l + 1) (L + 1)2 functions Removal of Elongated Artifacts In this class of applications the objective is to filter the image in such a way that only a particular class of structures is suppressed while the rest of the information must be preserved as accurately as possible. A practical application of this type in geological data is illustrated on Fig. 7, frame 1 where in the borehole acoustic impedance image the vertical scratches are caused by sliding microphones and must be removed. We applied our 2D orientation analysis using the fact that it provides at any scale a complete representation of the original image. In the same way as in the previous application, we first construct the orientation bundle (2) but instead of a non-linear filtering as in (39), we simply remove the vertical sectors of the bundle. Then the inverse transform (17) is performed to restore the original acuity. This procedure is purely linear and

11 Orientation Filters in Image Analysis 155 Figure 5. Comparison between the method described in Section 4.1 and results obtained by using a second order Gaussian orientation steerable filter G(x, y,θ,σ)=( x cos(θ)+ y sin(θ)) 2 e ( x2 y 2 )/σ 2. Upper left frame: Noisy fluoroscopy optical image of human retina, pixels. Upper right frame: Blood vessels are enhanced with the method described in Section 4.1 using scale σ = 2. Bottom frames: The Gaussian steerable filter was applied with the same scale σ = 2. In the left frame the sum ove all angles θ is depicted and in the right frame the maximal value over the angle is shown. Figure 6. Frame 1: Noisy (low dose) fluoroscopy image ( pixels.) of a thin catheter guide wire. Frames 2 and 3: High-lightened elongated structures as in Fig. 2, frame 3 but for scales σ = 0.8 and σ = 3.0 respectively. The angular resolution was 32 sectors and the maximal harmonic momentum was 15. can be applied at any scale or more scales sequentially. The result shown in Fig. 7 frame 2 preserves most of the remaining structures and little trace of the artifacts. In this application we use the knowledge that the artifacts are elongated structures with predominant vertical orientation. 5. Appendix: Technical Issues, Notations and Conventions In this Appendix we give some definitions and conventions concerning the formulas in the previous sections.

12 156 Kalitzin, ter Haar Romeny and Viergever Figure 7. Frame 1: Bore-hole D image containing artifacts (predominantly vertical scratches). Frame 2: Inverse projection of orientation bundle with removed sectors that are ±10 around the vertical axis. The filtering was performed for three scales (σ = 2, 3 and 5 pixels). In all filters the angular resolution was of 32 sectors and the maximal harmonic momentum was 15. The modified kernel (38) was used. Euclidean rotational groups and direction parameters. In a d-dimensional Euclidean space R d the (connected) orthogonal rotational group is SO(d). The unit sphere S d 1 represents the set of all spatial directions and is isomorphic to the factor-space SO(d). Any function on the spatial directions is SO(d 1) therefore a (infinite-dimensional) representation of SO(d) with stationary subgroup SO(d 1).

13 Orientation Filters in Image Analysis 157 Two-dimensional direction functions and harmonic decompositions. In two spatial dimensions, SO(1) 1 and, therefore, a function on the local direction is just a function on the SO(2) U(1) group. As the last is an Abelian group parameterized with a single cyclic parameter, the following decomposition is valid: f (θ) 1 2π n= f n = 1 2π π π f n e inθ, dθf(θ) inθ (40) where f n f n are the complex harmonic coefficients. The sum over n in (40) is a sum over the irreducible U(1) representations contained in the function f (θ). Local orientation bundle. We define local orientation bundle in this paper a fiber bundle where the base manifold is the space where the original image is defined and the standard fiber is the infinitedimensional functional representation of the local rotation group as given in 2D by (40). In higher dimensional cases, the representation is defined as a SO(d) SO(d 1) function on the factor-space. The structural group of the fiber bundle is SO(d). Fourier transformations and complex coordinate conventions. We assume the symmetric normalization conventions for Fourier transforms: 1 f (x) = (2π) d/2 f (ω) = 1 (2π) d/2 dω f(ω)e iωx, dxf(x)e iωx (41) as well as for the complex vector notations: z = 1 2 (x 1 + ix 2 ), ν = 1 2 (ω 1 + iω 2 ), z z = 1 2 ( 1 i 2 ) (42) and their complex conjugates. Gaussian kernel and its derivatives. We assume the following parameterization for the Gaussian kernel in two spatial dimensions: g(x,σ) = 1 2e (x2 1 2) +x2 2σ 2πσ 2 1 2πσ 2e z z σ 2 =g(z, z,σ). (43) with the complex notations from (42). The nth complex Gaussian derivative is thus: ( g n (x,σ) z n g(x,σ)= z ) n g(z, z,σ) (44) σ 2 and an analogous formula holds for its complex conjugate. In the Fourier domain, the notations and conventions (41) and (42) lead to the following form of the Gaussian kernel and its derivatives (44): g(ω, σ ) = 1 2π e 1 2(ω1 2+ω2 2)σ 2 1 2π e ν νσ2 = g(ν, ν,σ) g n (ν, ν,σ) (i ν) n g(ν, ν,σ). (45) SO(3) irreducible representation projection coefficients. Here we present shortly a construction that projects a product of lso(3) vectors onto a SO(3) irreducible representation of momentum l. First we use the SO(3) SU(2) local isomorphism to represent the 3D frequency vector ω = {ω 1,ω 2,ω 3 }in matrix (SU(2)-spinor) notations: ω αβ = ω i σ i αβ (46) where we assume summation on repeated indices and σαβ i, i = 1, 2, 3; α, β = 1, 2 are the (symmetric) Pauli matrices linking SO(3) with SU(2) representations. The highest spherical-momentum representation contained in the SU(2) tensor product ω α1 β 1...ω αl β l is the fully symmetric tensor The coefficients i 1...i l lm i 1...i l lm =a l ω (α1 β 1...ω αl β l ). (47) in (33) are: l! ( σ i 1 (α (l m)!m! 1 β 1...σ (i ) 1 α l β l ) n 1 n 2 =m (48)

14 158 Kalitzin, ter Haar Romeny and Viergever where a full SU(2) index symmetry is assumed and the tensor component with n 1 indices equal 1 and n 2 indices equal 2 is selected so that n 1 n 2 = m; n 1 + n 2 l. Obviously, the numbers l, m uniquely define the pair (n 1, n 2 ). It is clear the the resulting tensor is fully symmetric in its SO(3) indices. The combinatorial coefficient in (48) ensures that the product l m= l i 1...i l lm j 1... j l l( m) (49) is an SO(3) tensor. But the only constant SO(3) tensor is the unit matrix δ ij and therefore, taking into account the symmetries of the SO(3) indices i 1,...,i l and j 1,..., j l, we obtain that the tensor in (49) can only be up to a l-dependent scalar constant the product δ i1 j 1...δ il j l symmetrized over all i (or equivalently j) indices. Now it is clear that by a suitable choice of the free constant multipliers a l in (48) we can obtain the normalization constraint (34). Acknowledgments This work is carried out in the framework of the research program Imaging Science, supported by the industrial companies Philips Medical Systems, KEMA, Shell International Exploration and Production, ADAC Europe and Cordis. References Conwell, J.F Group Theory in Physics, volume II. Academic Press: New York and London. Florack, L.M.J., ter Haar Romeny, B.M., Koenderink, J.J., and Viergever, M.A The Gaussian scale-space paradigm and the multiscale local jet. International Journal of Computer Vision, 18(1): Freeman, W.T. and Adelson, E.H The design and use of steerable filters. IEEE Trans. Pattern Analysis and Machine Intelligence, 13(9): Greenspan, H., Belongie, S., Goodman, R., Perona, P., Rakshit, S., and Anderson, C.H Overcomplete steerable pyramid filters and rotation invariance. In Proc. IEEE Computer Soc. Conf. on Computer Vision and Pattern Recognition, CVPR 94, pp Hel Or, Y. and Teo, P. 1996a. A computational group-theoretic approach to steerable functions. Preprint: STAN-CS-TN Hel Or, Y. and Teo, P. 1996b. A coomon framework for steerability, motion estimation and invariant feature detection. Preprint: STAN-CS-TN Karasaridis, N. and Simoncelli, E.P A filter design technique for steerable pyramid image transforms. In Proceedings ICASSP- 96, Atlanta, GA. Koenderink, J.J The structure of images. Biological Cybernetics, 50: Lee, T.S Image representation using 2D Gabor wavelets. IEEE Trans. Pattern Analysis and Machine Intelligence, 18(10): Lindeberg, T Scale-Space Theory in Computer Vision. The Kluwer International Series in Engineering and Computer Science. Kluwer Academic Publishers: Dordrecht, The Netherlands. Michaelis, M Low Level Image Processing using Steerable Filters. Ph.D. Thesis, Technische Fakultät der Christian-Albrechts- Universität Kiel. Miller, W Symmetry, Groups and Their Representation. Academic Press: New York and London. Perona, P Deformable kernels for early vision. In IEEE CVPR, pp Segman, J. and Zeevi, Y.Y Image analysis by wavelet-type transforms: Group theoretic approach. Journal of Mathematical Imaging and Vision, 3: Simoncelli, E Design of multi-dimensional derivative filters. In First International Conference on Image Processing, Texas. Simoncelli, E A rotation-invariant pattern signature. In International Conference on Image Processing, pp Simoncelli, E.P. and Farid, H Steerable wedge filters for local orientation analysis. In IEEE Transactions on Image Processing. Simoncelli, E., Freeman, W.T., Adelson, E.H., and Heeger, D.J Shiftable multi-scale transforms. IEEE Trans. Information Theory, 32(2): Simoncelli, E. and Freeman, W The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Second Annual IEEE International Conference on Image Processing, Washington. ter Haar Romeny, B.M Front-End Vision and Multiscale Image Analysis: Introduction to Scale-Space Theory. Kluwer Academic Publishers: Dordrecht, The Netherlands, In preparation. Zibulski, M. and Zeevi, Y.Y Analysis of multiwindow Gabortype schemas by frame methods. Applied and Computational Harmonic Analysis, 4: