VARIATIONAL RESTORATION OF NON-FLAT IMAGE FEATURES: MODELS AND ALGORITHMS TONY CHAN AND JIANHONG SHEN COMPUTATIONAL AND APPLIED MATHEMATICS

Transcription

1 VARIATIONAL RESTORATION OF NON-FLAT IMAGE FEATURES: MODELS AND ALGORITHMS TONY CHAN AND JIANHONG SHEN COMPUTATIONAL AND APPLIED MATHEMATICS DEPARTMENT OF MATHEMATICS LOS ANGELES, CA chan, Abstract. We develop both mathematical models and computational algorithms for variational denoising and restoration of non-at image features. Non-at image features are those that live on Riemannian manifolds, instead of Euclidean spaces. Familiar examples include the orientation feature (from optical ows or gradient ows) that lives on the unit circle S, the alignment feature (from ngerprint waves or certain texture images) that lives on the real projective line RP, and the chromaticity feature (from color images) that lives on the unit sphere S 2. In this paper, we apply the variational method to the restoration of non-at image features. General mathematical models for both continuous image domains and discrete domains (or graphs) are constructed. Riemannian objects such as metric, distance and Levi-Civita connection play important roles. We also discuss ecient computational algorithms for the non-linear restoration equations. Numerical results support both the models and algorithms developed in this paper. The mathematical framework can also be applied to the restoration of general non-at data outside the scope of image processing and computer vision. Date: June 0, 999. Find typos? Why wait? \jhshen@math.ucla.edu." 99 Mathematics Subject Classication. Primary: 942; Secondary: 65H0, 49M37. Key words and phrases. Variational model, total variation, denoising and restoration, non-at features, Riemannian manifold, metric and distance, orientation, alignment, chromaticity. Research supported by grants from NSF under grant number DMS and from ONR under N

2 2 CHAN AND SHEN. Introduction In image analysis, restoration and denoising by variational methods are typically applied to gray level image functions or vector-valued (color) image functions (see Mumford and Shah [0], Rudin, Osher and Fatemi [5], and the monographs by Morel and Solimini [9], and Weickert [2]). Such functions are \at" or ane. By atness or anity, we mean a function that takes values in a linear space, such as the real line, or the linear RGB space. There are some important image features that do not live in at spaces, however. One example is the orientation feature, which comes from optical ows or gradient ows (of gray images). One can easily see that in a video that records the rigid motion of an object against a static background, the orientation of the motion stores one half of the whole valuable information (and speed is the other half). The second example is the alignment feature which is commonly found in many texture images. For example, for ngerprints, the alignment pattern of ngerprint \waves" is crucial for identication tasks (see Perona [2]). The chromaticity feature of color images is another example of non-at features (see Tang, Sapiro and Caselles [9, 8]). For a linear RGB color image, the pixel value I is a vector in R 3. The magnitude kik measures the brightness, while its trace on the unit sphere f = I=kIk contains the color saturation information, and is called the chromaticity feature in Tang, Sapiro and Caselles [8]. It lives on the unit sphere S 2 in the linear RGB space. The mathematical object for modeling non-at features is dierential manifolds. Perceptual sensitivity to a particular type of feature is measured by Riemannian metrics. Therefore, our models shall be inevitably based on Riemannian manifolds. The complete picture has been summarized in Figure. The rst step is taken by image processing or video processing Image u or image flow ( gray level or color ) Interesting feature f ( orientation, optical flow, e.t.c. ) Feature manifold M and Riemannian structure Figure. The ow chart of model setup. researchers. Meaningful features are often dened according to our visual system or the type of task we attempt to carry out. The second step leads to mathematical analysis. Once the feature is described, the manifold stands

3 RESTORATION OF NON-FLAT IMAGE FEATURES 3 there. But a truly perceptually meaningful Riemannian metric depends on our visual system. For example, orientations in 2-D images live on the unit circle. It is quite reasonable to assume that generally our visual system does not prefer any single angle (=4, say). Thus we can use the standard Riemannian structure (rotationally invariant) for orientations. Sometimes the converse is true: familiar manifolds usually have their natural (related to certain transform groups on symmetry) Riemannian metrics, and one can directly start with them for convenience. The starting point of this paper is the last box. That is, we assume that a perceptually meaningful Riemannian structure is already dened. As for at features of an image, variational method is one of the most ecient tools for analyzing non-at features. This is primarily due to its denoising and restoration eects. The nonat features obtained initially are often noisy due to common factors like () the noise contaminating original images, or left there even after a denoising pre-processing; (2) regional destruction of raw images (due to the aging of a lm, say), or insuciency of resolution in raw images; (3) ineciency of the algorithm that extracts the non-at features. As in the classical literature, properly designed variational models can (i) attenuate noise, (ii) preserve intrinsic singularities, (iii) enhance the \edges" of non-at features. For general features, an \edge" means a segment where the feature distribution undergoes abrupt changes. For the orientation feature from optical ows, for example, the edges may be identical to the real physical boundaries of moving bodies. Perona has taken the pioneering step in [2] by studying a variational model for orientation diusion with applications to ngerprint analysis. His work denes the right framework for further deeper mathematical and computational study on the subject. Our present work is its followup and extends it both theoretically and computationally. (Note: By the time we completed the initial version of this paper, we found surprisingly (and happily) that Sapiro and his colleagues were also developing the diusion theory for directions (corresponding to the feature manifold S m ) independently (see [8, 9]), almost at the same time. We shamelessly admit that some ideas on chromaticity presented in this nal version are impossible without beneting from this inspiring team.)

4 4 CHAN AND SHEN There are two key mathematical questions upon studying variational restoration of non-at features. First, it is not as obvious as in Euclidean spaces to come up with a feasible energy functional for non-at features. Perona [2] oered some bright physical intuitions for the orientation feature, but a general mathematical framework is still missing, which may delay many potentially important applications for other non-at features. Secondly, as one can easily expect, the associated Euler-Lagrange equations or steepest-descent (diusion) equations are generally non-linear because of the non-atness of feature manifolds and the intrinsic non-linearity of their classical counterparts (the TV model, for instance). To get our mathematical models truly work for applications, it is important to develop ecient (simple, stable, or fast) numerical algorithms for these non-linear equations. Answering these two fundamental questions is the main task of our present paper. We present a systematic way for constructing the energy functions for general non-at data or features. We interpret Perona's original model in this general framework and discuss its generalizations. The mathematical framework also extends the classical total variational (TV) model. Numerical evidence shows that the TV model is more successful than the L 2 model for restoring non-at features which have intrinsic singularities. We also design ecient numerical algorithms to solve the non-linear restoration equations. Numerical results highlight the success of both our mathematical framework and computational algorithms. The organization of the paper is as follows. Section 2 constructs the mathematical foundation for the work of both Perona and ours. We study the restoration of non-at features in a general and abstract way. The starting point is the construction of the energy functional on a Riemannian feature manifold. In section 3, we discuss restoration models specically for orientations and alignments, two non-at image features frequently encountered in gray level image analysis. Construction of energy functionals is through the new concept Locally Riemannian Distance. The latter half of the section is devoted to developing numerical algorithms that can solve eciently the nonlinear equations associated with the orientation and alignment features. Section 4 studies the computational issues for the chromaticity restoration on S 2. A simple two-step ltering algorithm on the sphere is established for the discrete models. Numerical examples are gathered in section 5, where we discuss applications in ngerprint images, optical ows, and color images.

5 RESTORATION OF NON-FLAT IMAGE FEATURES 5 Whereas our work was initially motivated by image processing, we believe that the mathematical framework and algorithms can nd their applications in the restoration of general non-at data in other elds as well. 2. Variational Models and Restoration Equations Since non-at features do not live in a at space, the energy functionals and associated restoration equations depend on the \geometry" of the non-atness. The geometry is described by Riemannian manifolds and Riemannian metrics. These are the basic objects that we work with in this section. We shall establish the most general models for the restoration of non-at features. In the two subsections, we study models for both continuous image domains and discrete image domains (or graphs, more generally). The latter has the advantages of easier implementation in computers and friendly accessibility to the widest readers. 2.. Models for continuous image domains. Let be an image domain in R 2. For the purpose of image analysis, can be a square, a disk, or other regular domains. Suppose that the non-at feature of interest lives on an m-dimensional Riemannian manifold M. We shall call M the feature manifold. For examples, the unit circle S is the feature manifold for orientation (or arrows with unit length), while the real projective line RP is the feature manifold for alignment (or line segments with unit length). An image is a real function u :! R n (n = or 3, practically). The collection of \all" images is denoted by the symbol (! R n ). One can put the class labels such as L p or BV (bounded variation) in front of it. But since we do not study the theoretical sides (i.e., existence and uniqueness) of the model equations in the present paper, the symbol allows more freedom. Moreover, the regularity of most images from natural scenes is usually poor since they belong to the statistical category in a large degree. A map f :! M is called a feature distribution and (! M) denotes the collection of all feature distributions of interest. An M-feature of images is a map F : (! R n )?! (! M); such that for any image u, f = F u is a feature distribution. That is, for any pixel p = (x; y) 2, f(p) = F u (p) is a (feature) point on M.

6 6 CHAN AND SHEN It is often true in applications, as outlined in the introduction section, that the M-feature distribution f = F u associated to a given image u is often noisy. Our major goal is to construct denoising and restoration models using the classical tool of variational method. In practice, a feature distribution f 2 (! M) is typically a map without overall smoothness. However, when we constructing a restortation model, as well practiced in the case of ane features, f(p) is always assumed to have certain degree of regularities. For instance, we can assume that f(p) belongs to the Sobolev space W ;2 (; M). In numerical computation, it is fortunately true that models based on the good regularity assumption often work eectively for irregular images as well. Next section discusses discrete models, for which the regularity of data causes no problem in both theory and computation. Under this x f(p) y f(p) are two tangent vectors in the tangent space T f(p)m. Let h ; i f(p) denote the Riemannian inner product at f(p), and kk f(p) the induced norm in T f(p)m. We shall omit the location subscript if it is clear or insignicant in the context. We now dene the strength function q () e(f; p) = k@ x f(p)k 2 + k@ y f(p)k 2 : To have a better understanding of its meaning, we introduce the correlation matrix C f (p) = " x f i x y f i y x f i y y f i # : Then e(f; p) = q q trace(c f (p)) = ; where and 2 are the singular values of the m (the dimension of M) by 2 \matrix" [@ x y f] (which is well-dened under any local orthonormal frame on M). A similar formulation has also been proposed in Sapiro [7, 6] when f is a vector valued distribution. The strength function dened in this way is rotationally invariant. That is, for any orthogonal transform in the image domain R : R 2! R 2 ; we have e(f(r? ); Rp) = e(f(); p):

7 RESTORATION OF NON-FLAT IMAGE FEATURES 7 The reason is that the corresponding correlation matrices are orthogonally similar to each other. This property is meaningful in real life since we do not want the results to depend on the direction in which an image is displayed before us. The L 2 total energy of a feature distribution f is E(f) = 2 Z e 2 (f; p) dp: Inspired by the ane case where Rudin and Osher [4], Rudin, Osher and Fatemi [5] rst introduced the total variational (TV) model for image restoration, we dene the total variation to be the L total energy E TV (f) = Z e(f; p) dp: Let d denote the metric on M induced by its Riemannian structure. That is, for any two points f and g 2 M, d(f; g) = inf? length(?); allowing? to go over all piecewise smooth paths that link f and g. (We shall also allow other kind of distances later. See the discrete model in the next subsection.) Given a noisy feature distribution f (0) (p), the classical restoration model associated with the L 2 energy is to solve where the tted L 2 energy E(f; ) = E(f) + 2 Z min E(f; ); f Z d 2 (f (0) ; f) dp = e 2 (f; p) dp + d 2 (f (0) ; f) dp: 2 2 As in the ane case, the rst term regularizes the restoration solution and the second one is a tting constraint. For the TV restoration, the associated tted TV energy is Z E TV (f; ) = e(f; p) dp + d 2 (f (0) ; f) dp: 2 The constant above is related to the Lagrange multiplier for the associated constrained optimization problem Minimize E(f) (or E TV (f)) subject to Z Z Z d 2 (f (0) ; f)dp = 2 ; jj where jj is the area of and the standard deviation of the noise. In the at case, the connection between the constrained and unconstrained

8 8 CHAN AND SHEN formulations was discussed theoretically by Chambolle and Lions [2] and numerically by Blomgren and Chan []. In image analysis, is often conveniently xed or chosen a priori, or estimated by the gradient-projection method (Rudin, Osher and Fatemi [5]). To solve the above variational problems, we study the associated Euler- Lagrange equations. For the tted L 2 energy E(f; ), the Euler-Lagrange equation can be shown to be?@ x(@ x y (@ y f)) + 2 grad fd 2 (f (0) ; f) = 0; where, at any pixel p = (x; y) 2, grad f d 2 (f (0) ; f) denotes the gradient vector of the scalar function d 2 (f (0) (p); ) on M; x y denote covariant derivatives acting on vector elds dened on f(). Specically, let r be the Levi-Civita connection (see Helgason [7]) on the feature Riemannian manifold M. Then the covariant x y x = y = : are given explicitly They can legally act on any vector eld X that is dened along f() due to the locality property of covariant derivatives. For instance, if M is a domain in R m, xx X(f(x; Generally, when M is isometrically embedded in Euclidean space R N (N m), a vector eld X on M is also one in R N. Since the at derivative in R X(f(x; usually overshoots from the tangent space of M, one needs an extra step of projection for the covariant derivative (2) where f T f xx = X(f(x; is the orthogonal projection from T f R N onto the tangent space The Euler-Lagrange equation can be solved by the innitesimal steepest descent method (or time-marching method). This leads to the tted x(@ x f) y (@ y f))? 2 grad fd 2 (f (0) ; f);

9 RESTORATION OF NON-FLAT IMAGE FEATURES 9 with the Neumann adiabatic boundary condition plus the initial condition f t=0 = f 0 ; where, f 0 can be the given raw feature distribution f (0), for convenience. All the terms in the diusion equation are in the tangent space T f(p)m. As the marching time increases, f evolves to the equilibrium distribution of the functional. Similarly, for the TV model, the Euler-Lagrange equation f y + e(f; p) e(f; p) 2 grad fd 2 (f (0) ; f) = 0: Steepest descent leads to a forced anisotropic f y? e(f; p) e(f; p) 2 grad fd 2 (f (0) ; f): Remark. Let us understand the rst two terms on the right-hand side of the last equation. Assume the image domain is -dimensional (i.e. a real interval [a; b]) and parameterized by x. Then f(x) is a path on the feature manifold, and minimization of Z b a kf x kdx is to reduce the total path length. This viewpoint connects our problem to the much familiar object geodesic. A geodesic is self-parallel in the well-known sense that if is the unit tangent vector, then r = 0: This condition also means that it is locally a \straight" line on the manifold. Now it is easy to see that = f x kf x k = f x e(f; p) ; and the linearity of the Levi-Civita connection leads to d = kf x kr : dx fx e(f; p) This gives us a better geometric sense about the two \TV-terms" appearing in the last two equations. EXAMPLE. Feature manifold M embedded in R N. When the feature manifold M can be isometrically embedded in Euclidean space R N (such as the unit circle in R 2 and the unit sphere S m in R m+ ), the Levi-Civita connection is simply the Euclidean derivatives followed by a

10 0 CHAN AND SHEN projection as pointed out above. In this case, a feature point f can be seen as an ordinary vector in R N, and the denition of the strength function q e(f; p) = kf x (p)k 2 + kf y (p)k 2 now only involves the ordinary dierentiations of vectors and is exactly krfk = p krf k 2 + krf 2 k krf N k 2 ; if f = (f ; f 2 ; ; f N ) 2 R N. Moreover, the diminishing ow elds F of the energy functionals can be computed explicitly. When M = S m is embedded in R m+, this work has been carried out recently in Tang, Sapiro and Caselles [9], and was also worked out independently by Osher []. For a general embedded feature manifold, for example, the diminishing ow eld F x (@ xf) y (@ yf) for the L 2 energy E is simply given by (3) F = f f; where, is the at Laplacian on, and f is the orthogonal projection from T f R N onto T f M. Spheres S m in R m+ are the most interesting to image processing. For the sphere, it was shown by Tang, Sapiro and Caselles [9] and Osher [] that the diminishing ow for the L 2 energy is (4) F (f) = f + krfk 2 f; and for the TV energy, (5) F TV (f) = r rf + krfk f: krfk Osher [] established these formulae by the method of Lagrange multiplier analysis, and Tang, Sapiro and Caselles [9] proved them by working out explicitly the variational problem. Here we give a simpler geometric proof based on the special form of the Levi-Civita connection (2). Proof of Eq. (4) and (5) (see Figure 2). Eq. (5) for example. For sphere S m, the projection at f is f v = v? (v f)f: Take the TV diminishing ow

11 RESTORATION OF NON-FLAT IMAGE FEATURES S 2 - grad( f ) f grad( f ) div( grad( f ) ) T S 2 f f F TV (f) Figure 2. The TV diminishing ow. Hence for any distribution f :! S m, F TV (f) = f r since f rf = rf 2 =2 = 0. = r = r = r rf krfk rf krfk? r rf krfk f f rf krfk? r f rf krfk? krf k f rf + krfk f; krf k This continuous model provides a universal language for restoring non-at features. However, the model assumes that the readers have some knowledge of modern dierential geometry, concepts such as the Levi-Civita connection. It is unavoidable in this continuous formulation since the variational formulation is global. Fortunately, there does exist a way that one can circumvent this technical diculty. We now develop a simpler model for \digital" image domains, namely graphs, where dierentiation of vector elds (i.e. Levi-Civita connection) can be avoided. Besides, the discrete models have their own computational advantages as discussed later in the paper Models for discrete image domains. Since numerically it is always necessary to lay down a nite discrete grid to approximate the continuous image domain, it is practically convenient and important to study directly

12 2 CHAN AND SHEN non-at features over a discrete grid, or more generally, a graph. This is the main task of this section. Perona took the rst step in [2]. Let G[ n ] be an undirectional graph over a nite set of nodes n (\n" stands for \numerical"), and G is the dictionary of edges. We write if nodes and are linked by an edge. For any node 2 n, denote by N all its neighbors f 2 n : g: We start with the denition of the concept of LRD, which shall play an important role in our models. A locally Riemannian distance (LRD) is a piecewise smooth continuous function d l d l : M M! R + = [0; ); such that () For any f; g 2 M, d l (f; g) = d l (g; f). (2) d l (f; g) = 0, if and only if f = g. (3) d l (f; g) + d l (f; h) d l (g; h). (4) d l (f; g) = d(f; g) + O(d 2 (f; g)) as d(f; g)! 0. Here d is the induced Riemannian distance dened in the previous section. Remark 2. Conditions (){(3) are the basic ingredients of a distance, which authorize d l to \legally" measure the amplitude of noise. Condition (4) restricts our attention to only those distances that are locally Riemannian. The motivation is as follows. Suppose the feature Riemannian manifold is locally at (i.e. locally isometric to Euclidean space R m ). Then condition (4) ensures that the \non-at Laplacian" (see the remarks at the end of this section) is locally exactly the ordinary Laplacian. In this way, many classical analytical results may be transplanted easily. EXAMPLE 2. Perona's distance. Consider the unit circle S as the concerned feature manifold. For any two points f = e i and g = ei 2, dene p d perona (f; g) = 2(? cos(? 2 )): Symmetry is immediate. Now if d perona (f; g) = 0. Then cos(? 2 ) =, which implies that f = g. It is also clear that the leading term of d perona (f; g) is the Riemannian distance when f and g are close. Therefore, to show d perona

13 RESTORATION OF NON-FLAT IMAGE FEATURES 3 is a locally Riemannian distance, it suces to verify the triangular relation. The proof follows. Formula sin( + 2 ) = sin cos 2 + sin 2 cos leads to j sin( + 2 )j j sin( )j + j sin( 2 )j: Noticing that we obtain r? cos 2 j sin j = ; 2 p? cos( ) p? cos 2 + p? cos 2 2 : Now that and 2 are arbitrary, this last inequality holds even without the factor 2, which eventually leads to (by noticing that 3? 2 = (? 2 ) + ( 3? )) p? cos(3? 2 ) p? cos(? 2 ) + p? cos( 3? ): It is the triangular relation for d perona! It is the distance that Perona has proposed in his work [2] (though he did not formulate it in this way). As an example of a non-lrd, consider r? cos(2(? d line (f; g) = 2 )) : 2 It is not a local distance for S since d line (;?) = 0: For the real projective line RP, however, d line is an LRD. For a given LRD d l, we dene the associated strength of a given feature distribution f to be e(f; ) = [ X d 2 l (f ; f )] which generalizes the continuous strength function in Eq. () or krfk when f is a at function. Next, we can dene the l 2 total energy X E(f) = 2 e2 (f; ); 2 n 2 ;

14 4 CHAN AND SHEN and the total variation (TV) E TV (f) = X 2 n e(f; ): For a given feature distribution f (0) and some weight constant, dene the tted l 2 total energy and the tted TV total energy E(f; ) = E(f) + X 2 n 2 d2 l (f (0) ; f ); E TV (f; ) = E TV (f) + X 2 n 2 d2 l (f(0) ; f ): To establish the diusion equations, we rst compute the gradient elds for the total energies. For the l 2 total (6) = d 2 l (f ; for the TV total energy, TV d 2 l (f ; f ) =e(f; ) + =e(f; ) Hence the corresponding tted diusion equations are (8) (9) df dt d 2 l (f ; f )? 2 df dt d 2 l (f ; d 2 l (f (0) ; f ); =e(f; ) + =e(f; ) 2 2 d 2 l (f (0) ; f ): Notice that here our starting point is the locally Riemannian distance d l. In his work on orientation diusion, Perona [2] started with a choice of energy, which is originally inspired by physical considerations. For general non-at feature manifolds, d l (and the special case of d l = d, the Riemannian distance) seems to be more fundamental. As an example, we now apply the discrete models to the restoration of chromaticity. The continuous diusion model without the tting constraint was rst discussed in Tang, Sapiro and Caselles [8]. Our discrete model has the following advantages: ) unlike the continuous model, one does not need to choose a numerical discretization scheme (often complicated) for the spatial derivatives; 2) the discrete model rigorously diminishes the energy function; while in the continuous model, the numerical PDE's only

15 RESTORATION OF NON-FLAT IMAGE FEATURES 5 asymptotically (i.e. as the step size of time or space tends to zero) diminish the continuous energy functionals. EXAMPLE 3. Restoration of the chromaticity feature on S 2. The 2-dimensional sphere S 2 models the chromaticity feature of color images. Let I(p) denote the linear RGB vector value at pixel p. Then I(p) can be perceptually separated into the brightness component B = ki(p)k, and the chromaticity component f(p) = I(p) B : f(p) records the saturation degree of colors. We now apply the discrete models developed above to chromaticity restoration. Take the TV model (9) for example. Embed S 2 in R 3 and take d l to be the embedded Euclidean distance, i.e. Let r f d l (f; g) := kf? gkr 3 = p (f? g) 2 ; for any f; g 2 S 2 : denote the gradient for f 2 R 3 the gradient on S 2. Then for any scalar function G(f) on G(f) = fr f G(f): Therefore, for any xed feature point g 2 d2 l (f; g) = f r f (f? g) 2 = 2 f (f? g) =?2 f (g) =?2(g? (g f)f): The TV model (9) is explicitly given by df X (0) dt = f(f ) e(f; ) + e(f; ) + f (f (0) ): There are no spatial derivatives in the last equation! In computation, one only needs to take care of the time discretization, which is easy to handle with the geodesic marching scheme we propose in Section 4. It is easily seen that the tted diminishing ow on the right-hand side is indeed on the tangent plane at f, which is crucial to prevent f from wandering away from the feature manifold innitesimally. This completes our example. We will return to it later in Section 4. More generally, suppose the feature manifold M is isometrically embedded in R N. Let f denote the orthogonal projection from T f R N onto the tangent

16 6 CHAN AND SHEN plane T f M. If we choose d l to be the straight line distance in R N (but restricted on M), then the TV restoration equation is given by df X () dt = f(f? f ) e(f; ) + + f (f (0)? f ): e(f; ) Especially, if M is the level set of a non-singular (i.e. with non-zero gradient everywhere) function (f) in R m+ : = 0, then f f v = v? (v f ) k f k 2 ; where f denotes the gradient of and v 2 T f R m+. In the continuous case, Osher [] rst proposed to consider the level-set feature manifolds (and indeed, both S and S 2 are level-set manifolds). Some comments are in order now. Remark 3.. Non-at Laplacian. Suppose the feature manifold M is in fact a at domain in R m. Take d l to be the natural Euclidean distance Then and d(f ; f ) = kf? f k; in R m 2 l (f ; f = 2(f? = 2 X (f? f ); which is the at graph Laplacian acting on f (up to a constant). Graph Laplacians unlock many combinatorial secrets of a graph, a topic which has been recently well studied in the spectral graph theory (see Chung [4], for example). Therefore, formula (6) can be viewed as a \non-at" Laplacian. 2. Nonlinear Adaptive Filter. The right-hand side of formula (7) generalizes the curvature term frequently appearing in image analysis (Rudin and Osher [4]; Rudin, Osher and Fatemi [5]) ru r ; jruj which leads to the total variational anisotropic diusion of a gray level image u, and is a special but the most commonly applied case of Perona and Malik's general concept of anisotropic diusion [3]. For a general graph, formula (9) denes a non-linear adaptive diusion. The feature

17 RESTORATION OF NON-FLAT IMAGE FEATURES 7 diuses faster near a node where the strength e is smaller, or equivalently, where the feature distribution is smoother. It adaptively slows down when the distribution gets rougher and the local strength grows. This adaptivity is particularly useful for preserving local characteristic architectures of a given feature distribution, such as the singularities of ngerprint \waves," and feature \edges." 3. The Graph Curvature Term. Suppose that M is isometrically embedded in R N, and d l is the restricted Euclidean distance. Denote by K (f) the at ow eld in R N X (f? f ) e(f; ) + : e(f; ) Then the TV restoration equation = fk (f) + f (f (0)? f ): We now show that K (f) is exactly the graph version for the classical curvature term. Fix a node. Let! e denote any of the directed edges starting from. Dene the directional derivative of a at feature f along! e := f? f ; where is the tail node of! e. Then the strength function is exactly e(f; ) = 2 4 e which generalizes the magnitude of the ordinary gradient. Here! e includes all directed edges starting from. Under this setup, it is easy to see that K (f) e e ; ; which is exactly in the form of graph divergence and gradient! 3. Restoration of Orientation and Alignment: Equations and Algorithms In this section, based on the general mathematical models constructed above, we study the restoration of two classes of non-at features frequently encountered in gray level image analysis, namely, the orientation feature and

18 8 CHAN AND SHEN the alignment feature. For the time being, we shall only discuss the discrete models. 3.. Restoration equations for orientation. Orientation lives on the unit circle S, which can be naturally parameterized by the angle parameter. Dene the sawtooth function on R: D(x) = 8 < : jxj jxj 2 periodic extension for general x: Then the Riemannian distance d is given by d(e i ; e 2 ) = D(? 2 ); or simply, d( ; 2 ) = D(? 2 ), which measures the length of the shortest arc on S that links the two feature points. Recall that in the previous example, we discussed Perona's locally Riemannian distance p d perona ( ; 2 ) = 2(? cos(? 2 )) = 2 sin? 2 2 : The fact that it is an LRD is better seen geometrically through Figure 3, which shows that d perona measures the Euclidean distance of the two feature points e i and ei 2 in R2 (instead of measuring along S ). e i θ sin θ θ 2 2 x e i θ 2 Figure 3. The Perona distance. Remark 4. The Perona distance belongs to a wider class of isometrically embedded distances (IED). An IED, denoted by d ie, is the result of an isometric embedding of S in R 2 (see Figure 4). Precisely, this means to establish a smooth (C 2, say) injective mapping F : S! R 2 ;

19 RESTORATION OF NON-FLAT IMAGE FEATURES 9 which satises the isometry condition kdf kr = kdsk 2 S ; for any innitesimal segment ds on the circle and its image df. The IED associated with F is dened by (2) d ie (f; g) = kf (f)? F (g)kr 2; for any f; g in S. d ie is an LRD because of the isometry condition. Such a distance is biased, or not rotationally invariant. It can be useful in occasions where we do know a priori that certain directions are more robust or important than the others. f x g x F F (f) x d (f, g) ie x F (g) R 2 Figure 4. The isometrically embedded distance (IED). One advantage of d perona over D is the smoothness of d 2 perona, on which we take derivatives in the restoration formulae Eq.(6{9). Remarkably, d perona is also rotationally invariant. distance function. In what follows, we shall only discuss this We now derive the parametric restoration equations for d perona. Parameterize the circle by! e i. Then the tangent (or equivalently, ie i ) is an orthonormal basis for the tangent space at each point. All vectors appearing in the diusion equations are expressible using this basis. Assume f = e @d 2 perona (f ; f ) = cos(? = 2 sin(? 2 perona(f ; f (0) ) = cos(? @ = 2 sin(?

20 CHAN AND SHEN Then the tted l 2 diusion equation (8) becomes d X (3) = 2 sin(? ) + sin( (0)? ); dt which is exactly the equation that Perona rst studied in [2]. Similarly, the tted TV diusion equation is d X (4) = sin(? ) dt e(; ) + + sin( (0)? ); e(; ) where, according to the previous section, the strength is X e(; ) = [ 2(? cos(? ))] 2 Remark 5. Notice that the parametric equations (3) and (4) are equivalent to equations (8) and (9). This is one of the main advantages of the discrete (graph) model it is less sensitive to the discontinuities in the raw data (0). In fact, a graph is \blind" to discontinuities since there is no innitesimal distance. For the continuous model, the parametric equations (i.e. on ) are not equivalent to the corresponding embedded equations (i.e. on f, treated as a vector in R 2 ), when the given raw data (0) has discontinuities. Equations (3) and (4) generalize classical at equations. consider a smooth orientation distribution, that is, To see it,? = O(); for all 2 N and some small characteristic parameter. Then X X sin(? ) = (? ) + O( 3 ): The leading term is exactly the at Laplacian on. This is generally true for any choice of locally Riemannian distances because of the last condition in the denition list. Similarly, the strength function can be shown to be X e(; ) = [ (? ) 2 ] 2 ( + O( 2 )): For example, for a typical internal cross node = (i; j) in image processing, its neighbors are u = (i; j? ); d = (i; j + ); l = (i? ; j); r = (i + ; j):

21 RESTORATION OF NON-FLAT IMAGE FEATURES 2 Thus the leading term of the strength function is simply the nite dierence approximation to the magnitude of the gradient " p 2 2hkrk = 2h where h is the discretization step 2 # Restoration equations for alignment. For alignment, direction of a line segment is unimportant (see Figure 5). This amounts to saying that the feature manifold is the real projective line RP. Energy Increases. Largest Energy Energy Decreases Figure 5. The alignment feature. A line segment in the direction of is denoted by l. Then l and l + denote the same line segment. A one-to-one correspondence between RP and the unit circle S can be established by representing each line segment l by a point e i2. This embedding technique has already been practiced in image processing (see Perona [2], and Granlund and Knuttson [6]). The Riemannian metric on RP is given by whenever j? 2 j < =2. d(l ; l 2 ) = j? 2 j; The line distance mentioned in Section 2 is given by r? cos(2(? 2 )) d line (l ; l 2 ) = = j sin(? 2 )j: 2 d line is in fact the pull-back of d perona under the circle representation of RP. If fact, any LRD d l on the circle (an IED, especially) can be pulled back to RP by dening d RP l (l ; l 2 ) = 2 d l(e i2 ; e i2 2 ): In what follows, we shall only work with d line.

22 22 CHAN AND SHEN Similar to the orientation diusion, we obtain the tted l 2 diusion equations for d line (5) d dt X = sin(2? 2 ) + 2 sin(2(0)? 2 ): The tted TV diusion equation is (6) d dt X = 2 sin(2? 2 ) e(; ) + + e(; ) 2 sin(2(0)? 2 ); where, the local strength is given by e(; ) = 2 4 X? cos(2? 2 ) 2 We shall only discuss these two equations in the following : Our later numerical experiment shows that the alignment restoration model is particularly useful for ngerprint processing (also see Perona [2]) Algorithms. The parametric restoration equations (3{6) are all nonlinear. The good news from the discrete model is that the steady equations are all algebraic. Various existing solvers for non-linear algebraic equations can thus make their contributions, though careful modications should be made. In this section, we discuss three numerical approaches. We rst study the l 2 restoration equations in details, and then point out how to make suitable adjustments for the TV restoration equations. Let n denote the graph image domain. For the purpose of image processing, one can be happy with the standard rectangular grid, where each (internal) node = (i; j) has four neighboring nodes = (i; j ); (i ; j) such that. Considering that our models and algorithms may be applicable even out of the scope of image processing, in the following, we still state the algorithms using the abstract notation for nodes: ; ;. Recall from the previous subsections that we try to solve directly the system of steady restoration equations (7) 2 X sin(? ) + sin(? ) = 0; 2 n :

23 RESTORATION OF NON-FLAT IMAGE FEATURES 23 Here is the given raw (noisy) angle distribution. energy is given by E(; ) = 2 X (? cos(? )) + X where denotes all distinct edges. The associated total 2 n (? cos(? )); The non-linear Gauss-Seidel method. We rst propose a nonlinear Gauss-Seidel method for solving it numerically. It proceeds as follows. Choose an initial guess 0 ( 0 =, say) and assign a linear ordering for all nodes: < < < < : For convenience, for each node, dene N? := f 2 N : < g; N + := f 2 N : > g: Let k denote the global updating clock and a certain node. Suppose that in the beginning of step kj, we have the following data available k : < ; k? ; k? : > : We try to update k? to k by the end of step kj. Set x =. k According to the restoration equation (7), we demand that (8) 2 X sin( k?? x) + 2 X sin( k? x) + sin(? k? ) = 0: Dene + C = 2 X + S = 2 X? X cos( k? ) + 2 sin( k?? ) + 2 X cos( k ); sin( k ): +? Then (9) S cos x? C sin x =? sin(? k? ); Dene r = p C 2 + S 2 ;! = \(C; S): Here \(a; b) is the angle in the direction of (a; b). Then Eq. (9) simplies to sin(!? x) =? r sin(? k? );

24 24 CHAN AND SHEN which gives (mod 2) or x =! + arcsin r sin(? k? ) ; x =!? arcsin r sin(? k? )? : Here we take the principle range of arcsin to be [?=2; =2]. To determine which one we should choose, consider the updated orientation distribution 0 = ( k We require that the total l 2 energy : < ; x; k? : > ): E( 0 ) = 2 X (? cos( 0? 0 )) can be as small as possible. This implies that x should be chosen so that X X 2 (? cos( k? x)) + 2 (? cos( k?? x))? + is the smallest, or equivalently, X X 2 cos( k? x) + 2 cos( k?? x)? + can be the biggest. This amounts to maximizing C cos x + S sin x under the constraint (9). Hence x is given by () k = x =! + arcsin r sin(? k? ) : This completes the kj{step of the nonlinear Gauss-Seidel updating. Remark 6. Careful readers may have noticed that in the updating equation (8), we do not replace k? by x in the tting term. This is one important lesson we learned from numerical experiments. If one permits that replacement, then the algorithm tends to giving a constant solution. Gauss-Seidel is simple, local and reliable, but with a relatively slow convergence compared with the other schemes we propose below.

25 RESTORATION OF NON-FLAT IMAGE FEATURES Linearized iteration method (LIM). Let sinc(x) denote the function sin x=x. Then Eq. (7) can be written as (2) X (? )2 sinc(? ) + (? ) sinc(? ) = 0: The linearized iteration method generates the iteration as follows. Take a guess 0 (taken to be, say) for the initial step 0. Suppose in the beginning of step k, k? is available. We update k? to k by solving a linear algebraic system. Dene (22) (23) a = 2 sinc( k?? k? ); if ; c = sinc(? k? ); for all 2 n: Based on Eq (2), for each, we require that X a ( k? k ) + c (? k ) = 0: This leads to an ordinary linear system A k = b with b = (c : 2 n ) T and A = X (A : ; 2 n ): A = c + a ; A = 8 < :?a ; if 2 N ; 0; else: Under this linearization, the non-linear updating can be achieved using any numerical linear algebra solver. LIM converges geometrically, faster than the nonlinear Gauss-Seidel scheme above The time marching method. In this subsection, we approach the steady distribution by time marching according to the discrete diusion equation. Recall that the tted diusion model using l 2 energy is given by (24) d dt = 2 X sin(? ) + sin(? ); 2 n : Take a marching step size h and set k = (kh): Then the system of dierential equations (24) can be solved using various numerical methods.

26 26 CHAN AND SHEN Introduce the non-at Perona Laplacian symbol 0 0 = X sin(? ): The simplest single-step method that ignores the non-linearity of the diusion equation is the Euler method for each step k. k+ positive free parameters a and b: = k + h2 0 k + h sin(? k ); In practice, this is implemented by choosing two small k+ = k + a 0 k + b sin(? k ): Numerically, Euler method suers from low accuracy and slow convergence. The major advantage is its simplicity and easy implementation, especially for such a non-linear problem. It can also be easily parallelized. Moreover, from the practical point of view, the exact steady distribution is not so unique and critical for human vision. Numerical experiments show that near-steady distributions are usually already good enough for human vision and recognition of characteristic patterns residing in a given raw (noisy) feature distribution. This is the good news Euler method. From the numerical point of view, one can also try other high order singlestep methods such as Runge-Kutta 2. But both methods require a small time step to ensure stability. In order to achieve faster marching by allowing larger step size h, we discuss the implicit trapezoidal scheme (or second-order Adams-Moulton method, see Golub and Ortega [5]) combined with the idea of linearization in the preceding subsection. The major advantage of such a scheme is its stability and permission for a large marching step size. Recall that for a system of equations like d dt = f (); 2 ; the implicit trapezoidal schemes at step k is given by Applying it to our problem, we have or k+ = k + h 2 [f ( k ) + f ( k+ )]: k+ = k + [h 0 k + h 2 sin(? k )] + [h 0 k+ + h 2 sin(? k+ )]; k+? [h 0 k+ + h 2 sin(? k+ )] = k + [h 0 k + h 2 sin(? )]: k

27 RESTORATION OF NON-FLAT IMAGE FEATURES 27 Next, similar to the previous method, we linearize the left hand side by the substitution (25) X k+ sincx k ( sin X k+ ; for any expression X k+ involving the current unknown k+. Dene a = h sinc( k? k ); if ; c = h 2 sinc(? k ); 2 n ; b = k + h0 k + h 2 sin(? k ) + c ; 2 n : Then the above implicit scheme leads to a linear system of equations A k+ = b with b = (b : 2 n ) T and A = (A : ; 2 n ): A = c + X a ; A = 8 < :?a ; if 2 N ; 0; else: This linear system is bandlimited, symmetric and positive denite for suf- ciently small h and can thus be solved easily using the standard numerical linear algebra solvers Algorithms for the TV restoration equations. With the algorithms ready for the tted l 2 energy, it is easy to modify each one separately for the tted TV energy. We summarize them briey below. The non-linear Gauss-Seidel method. The modication is done by computing the local strengths e's explicitly without involving x. Therefore, Eq. (9) holds with a new pair of constants (C; S). The discussion there is also valid. For the at case, the idea appeared in Chan and Vese [3]. The linearized iteration method (LIM). The sin{sinc linearization technique extends to the TV model by modifying the constant a to a = sinc( k?? k? ) e( k? ; ) + e( k? : ; ) This generalizes the algorithm for the classical at case ru 0 =?r + (u? u 0 ); jruj where Vogel and Oman [] proposed the linear iteration scheme ru k 0 =?r + jru k? (u k? u 0 ): j

28 28 CHAN AND SHEN The time marching method. The modication is done by modifying (25) to Remark 7. k k+ sincx X e( k ; ) ( sin X k+ e( k+ ; ) :. In practice, the local strength function e is usually modied to e a = p a 2 + e 2 ; for some small parameter a (a = 0?4, for example). This modication avoids the zero denominator in all the above TV formulae; On the other hand, it produces an intermediate eect between the TV energy and the l 2 energy. If a is large enough, then this modied TV model gets closer to the l 2 energy. This regularization technique is commonly practiced in the classical literature (see Marquina and Osher [8], for example). 2. For the time marching method, as Marquina and Osher [8] recently proposed for the continuous case, instead of solving the direct innitesimal steepest descent = X 2 sin(2? 2 ) e a (; ) + e a (; ) one can solve its = X 2 sin(2? 2 ) + 2 sin(2(0)? 2 ); + e a(; ) + e a (; ) 2 e a(; ) sin(2 (0)? 2 ): Since the \diagonal" multiplier e a (; ) is non-zero, the preconditioned equation still marches to the original steady solution. However, from the numerical point of view, this modication dramatically improves the marching speed and numerical stability. 4. Restoration of Chromaticity: Algorithms and Discussions In this section, we discuss the restoration models for the chromaticity feature living on the unit sphere S 2 and the related computational issues. We shall discuss the computational issues only for the discrete models developed in Section 2. For the work on the continuous pure diusion model (i.e. without the tting term), see Tang, Sapiro and Caselles [8, 9].

29 RESTORATION OF NON-FLAT IMAGE FEATURES The continuous models. Let f :! S 2 be a given raw noisy chromaticity distribution. In Section 2, we have discussed the continuous restoration model corresponding to the L 2 and TV regularization energies: f t = f + krfk 2 f + f (f (0)? f); (L 2 ) rf f t = r + krfkf + f (f (0)? f): (TV) krfk Here we have chosen the distance function for the tting term to be the restricted Euclidean distance in R 3 (not the geodesic distance on S 2 ). The orthogonal projection is given by f : T f R 3! T f S 2 ; f v := v? (v f)f: Since f f = 0, the two equations simplify to (26) (27) f t = f + krfk 2 f + f f (0) ; rf f t = r + krfkf + f f (0) : krfk One can start the evolutions with the initial condition for instance. f t=0 = f (0) ; There are two practical problems with these continuous models. Take the TV model for instance. First, if a numerical scheme is sought for Eq. (27), we must take a special care for the diminishing ow term rf F TV (f) = r + krfk f; krfk to make sure that the numerical F TV (f) does lie on the tangent plane T f S 2. If the distribution f is smooth, this tangency condition causes no serious problem since it is in any case satised to the leading order. But our task is to denoise and restore feature distributions with poor regularities! We have to permit discontinuities and singularities. In this realistic situation, typically, the numerical diminishing ow F TV overshoots the tangent plane T f S 2. This can of course be xed by articially projecting it back onto the tangent plane. But by doing so, we introduce a new degree of uncertainty regarding the diminishing eect of the algorithm. Besides this overshooting problem, another issue is that we lack a rigorous explanation for why the numerical evolution truly minimizes the continuous energy functional E TV (f; ) on S 2. The gap between continuous energy

30 30 CHAN AND SHEN functionals and numerical PDE's cannot be smoothed away easily because of the discretization error. These are the intrinsic problems with the continuous model. They disppear in the discrete formulation, since the latter starts with discrete energies and results in algebraic equations (on S 2 ), which need no numerical discretization for spatial derivatives The discrete models. Let f (0) : n! S 2 be the given raw discrete chromaticity distribution. By taking d l to be the restricted Euclidean distance, we have established in Section 2 the discrete restoration equations corresponding to the l 2 and TV energies: df X dt = 2 f f + f(0) f ; 2 n ; (l 2 ) df X dt = 2 f f e(f; ) + + f f (0) ; 2 n : (TV) e(f; ) Dene the weight w (f) = e(f; ) + e(f; ) ; for any. Then w = w. Rewrite the above equations using the linearity of the projection operator df dt = f df dt = f X 2f + f (0) A =: F (f); 2 n ; X w f + f (0) A =: F TV (f); 2 n : The steady restoration equations are thus given by (28) (29) F (f) = f X F TV (f) = f X 2f + f (0) A = 0; 2 n ; w f + f (0) A = 0; 2 n : Here 0 denotes the zero vector in T f S 2. These are \algebraic" equations on the chromaticity sphere S 2. Now we discuss the numerical implementations of both the marching equations and the steady algebraic equations.

31 RESTORATION OF NON-FLAT IMAGE FEATURES Geodesic marching for the discrete evolution equation. Take the TV model df dt = F TV (f); 2 n; for example. Set f n = f (nt). The classical Euler method is a at marching ~ f n = f n? followed by a projection (see Figure 6) + tf TV (f n? ); n- f α f n = ~ f n k ~ f n k : - n f α F TV α ( f n- ) n f α geodesic marching Figure 6. A at marching followed by a projection, and the geodesic marching. To compress these two steps into a single one, we propose the geodesic marching: (30) f n = cos tkf TV (f n? )k f n? It means that f n? + sin tkf TV (f n? )k F TV (fn? ) kf TV (fn? )k : marches for distance tkf TV (f n? )k along the big circle (i.e. the geodesic) determined by f n? and the ow vector F TV (f n? ). Therefore, the marching always stays on the sphere. It is easy to see that for small t, the above two-step classical scheme is equivalent to the geodesic marching up to the leading order. Precisely, the latter always marches a distance O(t 2 ) farther than the former (see Figure 6). The geodesic marching scheme is intrinsic for the geometry of the feature manifold, and thus has the most general meaning for numerical restorations of non-at features.

32 32 CHAN AND SHEN Fixed-point iteration, ltering, and the Convex Cone Theorem. Again take the system of TV equations F TV (f) = f X w f + f (0) A = 0; 2 n ; for example. It can be interpreted as: for the optimal restoration f, at each node, P w f + f (0) is parallel to f. This explanation inspires the X iteration scheme: ~ f n = w (f n? )f n? + f (0) f n = ~ f n =k ~ f n k: Or, to stabilize the algorithm (and even lead to new insights), dene h (f) = w (f) P w (f) + ; h (f) = We consider h as an adaptive lowpass lter. stabilizes to (3) (32) ~ f n = X h n? f n = ~ f n =k~ f n k: ; fn? + +h n? f(0) ; P w (f) + : Then the above algorithm Here h n? = h (f n? ). This ltering formulation immediately rewards us with one important property. A region? on S 2 is said to be a convex cone if its spanning cone ~? in R 3 is convex. ~? = R +? = fr v : r 0; v 2?g Theorem (Convex Cone Theorem). Suppose the raw data f (0) 's belong to a convex cone?. If we start the above iterations using f (0) as the initial data, then for any step n, all f n 's belong to?. The simple proof is left for our readers. Since RGB values are nonnegative, the chromaticity feature lives on the positive cone (one eighth of the sphere), which is convex. Therefore, the above xed point iteration scheme keeps new feature points from wandering away from the positive cone on the sphere. This another advantage of the discrete model.

33 RESTORATION OF NON-FLAT IMAGE FEATURES 33 The above xed-point iteration is similar to the Gauss-Jacobi method in numerical linear algebra. In the same fashion, we can construct the spherical Gauss-Seidel-like iteration. Assign a linear ordering for all the pixel nodes: < < < < : Let n denote the Gauss-Seidel clock and a pixel under consideration. Suppose in the beginning of step nj, we have available the data (33) (f n : < ; f n? ; f n? : > ): Within step nj, we update f n? X X ~ f n = h 0 fn + to f n by h 0 fn? + h 0 f(0) ;? f n = ~ f n =k~ f n k: + Here the lowpass lter coecients (h 0 ) are dened similar to the previous algorithm, but for the distribution given by (33). The Convex Cone Theorem still holds for this non-linear Gauss-Seidel method. 5. Numerical Examples and Applications 5.. Denoising by the l 2 model: -D example (Figure 7). Figure 7 displays the restoration eect of the l 2 model for a noisy orientation distribution. The rst subplot shows the angle distributions, and the rest three are the corresponding orientation distributions. In the rst subplot, the solid line represents the clean angle distribution. Then uniform random noise is added to produce the dotted dash line. The restoration with the a priori Lagrange multiplier = 0:0 is given by the dash line. Small leads to smooth interiors. However, the resulting weak eect of the tting term may smear orientation \edges," as clearly observable from the gure Restoration by the TV model: -D example (Figure 8). Similar to Figure 7, we apply the TV model (4) to restore the noisy orientation distribution. We have chosen =, and the small parameter a discussed in Section to be 0:000. Compared to the l 2 model, the TV model clearly works better. The restoration of orientation \edges" is more faithful. This is generally true when the underlying clean feature distribution undergoes abrupt changes.

34 34 CHAN AND SHEN Denoising using the fitted L 2 model (λ=0.0) : the angle distributions. 2.5 Noisy Clean Denoised Angle Distribution θ The clean orientation distribution The noisy orientation distribution The denoised orientation distribution Figure 7. Denoising using the l 2 model Eq. (3). The orientation edges are smoothed out. (See Section 5..) 5.3. Restoring wave lines of a ngerprint (Figure 9). For ngerprints, the alignment pattern of wave lines is the most important feature. Therefore, instead of Eq. (3), we use Eq. (5), which yields better results. We rst apply the enhancement and diusion techniques to the original raw ngerprint image to obtain the alignment distribution. This was done by computing the gradient ow of the processed image. We then have the

35 RESTORATION OF NON-FLAT IMAGE FEATURES 35 Denoising and restoring edges by the TV model (λ=, a=.000) : the angle distributions. 2.5 Noisy Clean Restored Angle Distribution θ The clean orientation distribution The noisy orientation distribution The denoised orientation distribution Figure 8. Denoising and restoration of edges through the TV model Eq. (4). The orientation edges are faithfully restored. (See Section 5.2.) \heads of the arrows" removed. The alignment pattern has been superimposed on the ngerprint image in the top subplot Figure 9. Notice that it is indeed a noisy distribution regardless our smoothing eort. In the bottom subplot, we apply the line model Eq. (5) with = 0: to denoise the above distribution. The result is very successful. The alignment

36 36 CHAN AND SHEN pattern coincides very well with the ngerprint except on a couple of local regions where the original image is not clear or damaged. Alignment distribution obtained from the denoised image by the gradient method Denoised alignment distribution (λ=0.) Figure 9. Restoring the alignment feature of ngerprints by the l 2 model Eq. (5). (See Section 5.3.) 5.4. Restoration of optical ows by the l 2 model (Figure 0). In Figure 0, we show an application of the l 2 model for denoising optical ows.

37 RESTORATION OF NON-FLAT IMAGE FEATURES 37 Noisy optical flow: a football hits a soccer ball Restoration by the L 2 model with λ= Angle distribution of the raw optical flow The restored angle distribution Figure 0. Restoration of optical ows by the l 2 Eq. (3). (See Section 5.4.) model The upper left subplot shows the noisy optical ow of an imagined video recording the spinless rigid motion of a football and a soccer ball. The background is noisy. The upper right subplot shows the restored optical ow obtained by the l 2 model. We have chosen = 0:0. The length of each arrow is not changed. The two subplots at bottom plot the corresponding angle distributions. Observe that the motion near the boundaries is smeared due to the smoothing eect of the l 2 energy.

38 38 CHAN AND SHEN Optical flow: a football hits a soccer ball Restoration by the TV model with λ=, a= The angle distribution of the raw optical flow The restored angle distribution Figure. Restoration of optical ows by the TV model Eq. (4). (See Section 5.5.) 5.5. Restoration of optical ows by the TV model (Figure ). Similar to Figure 0, in Figure, we have tested the TV model with = and a = 0:000. TV model restores the boundaries better than the l 2 model Restoration of chromaticity (Figure 2 and 3). In Figure 2, we apply the discrete chromaticity restoration model for a clown image with Gaussian noise (the top image). We rst separate the vector RGB image I into the brightness component B = kik, and the chromaticity component f (0) = kik=b. Then we apply the xed-point TV ltering algorithm (3) to f (0) and get the optimal restoration f. Finally, we use the original brightness

39 RESTORATION OF NON-FLAT IMAGE FEATURES 39 The raw image with Gaussian noise The new image with restored chromaticity. λ= Figure 2. Restoration of chromaticity from Gaussian noise by the discrete TV model and spherical ltering Eq. (3). The brightness component remains unchanged. (See Section 5.6. The raw image is from the courtesey of G. Sapiro.) value B to assemble the new image I new = f B, which is shown in the bottom subplot. (One can also apply the classical at restoration model to

40 CHAN AND SHEN the scalar eld B along with the chromaticity restoration. See Tang, Sapiro and Caselles [8].) Raw R channel Restored R channel Raw G channel Restored G channel Raw B channel Restored B channel Figure 3. TV Restoration of chromaticity from Gaussian noise: the eect on each channel. (See Section 5.6.) In Figure 3, we plot the raw and restored images (from Figure 2) channel by channel.