Image enhancement by non-local reverse heat equation

Transcription

1 Image enhancement by non-local reverse heat equation Antoni Buades 1, Bartomeu Coll 2, and Jean-Michel Morel 3 1 Dpt Matematiques i Informatica Univ. Illes Balears Ctra Valldemossa km 7.5, Palma de Mallorca, Spain {vdmiabc4, tomeu.coll}@uib.es 2 Centre de Mathématiques et Leurs Applications, ENS Cachan 61, Av du Président Wilson, Cachan, France Abstract. In 1955 Kovasznay et al. proposed to enhance an image by reversing the heat equation. This process is highly unstable and blows up the noise. Thus an efficient deblurring depends crucially on accurate denoising. In this paper we investigate the use of neighborhood filters and of a recent variant, NL-means, to stabilize the reverse heat equation. We shall prove that adding to the heat equation a term involving the self-similarities of the image stabilizes the reverse heat equation. By experiments on good quality images, not artificially blurred, we will illustrate the feasibility of the method to reveal image details. In contrast with the various PDE s for image enhancement, the non-local reverse heat equation thus introduced is linear. 1 Introduction The difference between the original u 0 and a blurred version image of it, k u 0, is roughly proportional to its Laplacian. In order to formalize this far ranging remark, we have to assume that k is spatially concentrated, and we must rescale k as k h (x) = 1 h k( x ), with h 0. Let us denote by x = (x, y) a point of the plane h 1 2 and assume that u 0 is C 3 around x. Assume further that k is a positive radial kernel satisfying (1+ x 2 + x 3 )k(x)dx < and x 2 k(x)dx = y 2 k(y)dy = 2. (This last relation can always be imposed, by a rescaling of k.) Under these assumptions and using a Taylor expansion of u 0 around x, it is an easy exercise to prove that k h u 0 (x) u 0 (x) u 0 (x) h as h 0. We can rewrite this relation as k h u 0 (x) u 0 (x) = h u 0 (x) + o(h). (1) Now let u(t, x) denote the solution of the heat equation t = u, u(0, x) = u 0(x).

2 If u 0 is sufficiently smooth, namely C 2 and bounded, then we can write u(t, x) u(0, x) = t u 0 (x) + o(t). (2) The comparison of equations (1) and (2) shows that blurring u 0 with a kernel k h is for small h equivalent to applying the heat equation to u 0 at scale h. These equations have led to an image restoration method: We read in the paper [15] by Lindenbaum, Fischer, and Bruckstein that Kovasznay and Joseph [12] introduced in 1955 the notion that a slightly blurred image could be deblurred by subtracting a small amount of its Laplacian. Numerically, this amounts to subtracting a fraction λ of the Laplacian of the observed image from itself: u restored = u observed λ u observed. Dennis Gabor studied this process and determined that the best value of λ was the one that doubled the steepest slope in the image [15]. In other terms one can to some extent enhance an image by reversing time in the heat equation: t = u, u(0) = u observed. Numerically, this amounts to iterating substraction of its Laplacian from the observed image. This operation can be repeated several times with some small values of h, until it blows up. Indeed, the reverse heat equation is extremely ill-posed. All the same, this enhancement method is efficient and can be applied with some success to most digital images obtained from an optical device. We refer to [15] for a historical review about this image restoration method. Since then one can list several attempts to stabilize the time-reverse heat equation or to emulate it by another more stable partial differential equations The Osher-Rudin shock filter The term of shock filter itself was framed by Rudin in his PhD dissertation [21], inspired from the use of nonlinear filters in shock simulation for P.D.E. s. Osher and Rudin [20] proposed to sharpen a blurred image u 0 by applying the following equation: t = sign( u) Du, with u(0, x) = u 0(x) where Du is the spatial gradient of u at x and Du its euclidean norm u 2 x + u 2 y. This equation can be seen as a pseudo reverse heat equation, where the propagation term Du is tuned by the sign of the laplacian. The Kramer Algorithm. In [13], Kramer defined a filter for sharpening blurred images. The filter replaces the gray level value at a point x by either the minimum or the maximum of the gray level values in a circular neighborhood B(x, h). This choice depends on which is the closest to the current value. This filter is then iterated on the image. Kramer s filter can be interpreted as a partial differential equation, by the same kind of heuristic arguments which Gabor developed to

3 derive the heat equation. It was proved in [23] that the Kramer and the Osher- Rudin filters share the same asymptotic behavior for regular 1D signal. In other terms, they are infinitesimally identical in 1D. However this is not true in the case of images. In that case, Guichard and Morel [8] proved that the PDE underlying the Kramer filter is t = sign(d2 u(du, Du)) Du. Thus, the laplacian in the Rudin-Osher equation is replaced by a directional second derivative of the image, D 2 u(du, Du). There is a direct link between these filters and two very classical edge detectors, namely the Canny filter and the zero-crossings of the laplacian. The study in [8] established the following table. In 2D: Kramer Filter Osher-Rudin Filter t = - Du sgn(d2 u(du, Du)) - Du sgn( u) replace u by min or max of u min or max of u in a neighborhood in a neighborhood depending on which of the two on whether the mean of u is the nearest to u in a neighborhood is above or below u. Associated edge detector Canny edge detector Zero-Crossing of Laplacian D 2 u(du, Du) = 0 u = 0 This table makes clearer the relationships between the various kinds of edge detectors and deblurring filters. It also specifies in the mid column which kind of local underlying operation is linked with each one of these filters. They can in no way pretend to emulate a reverse heat equation. By the same arguments as Osher-Rudin [20] one sees that Kramer s operator simply enhances Canny edges in the same way as the initial shock filter enhances Hildreth-Marr edges. Before entering the core of the subject, namely how to reverse the heat equation itself, let us point out that there have been other more and more sophisticated attempts to perform the inverse heat equation ; see (e.g.o) Hummel et al. [9,10]. And in fact nonlinear local filters for image enhancement were proposed as early as 1951 [28] for television images. The shock filters asymptotic is also linked to classical reaction-diffusion equations. Alvarez and Mazorra [2] or Price at al. [19] proposed to combine in restoration an anisotropic diffusion with a shock filter. Let us also mention [1] for the

4 combined use of shock filter and diffusion of equations in image quantization : in some extent, quantization acts as a sharpening. As a further variant on the initial ideas, Perona and Malik in 1987 [17,18] proposed to smooth what needs to be smoothed, namely, the irrelevant homogeneous regions, and to deblur the boundaries. With this in mind, the diffusion should look like the heat equation when Du is small, but it should act like the inverse heat equation when Du is large. Here is an example of a Perona Malik equation: = div(g( Du )Du), (3) t where g(s) = 1/(1 + λ 2 s 2 ). It is easily checked that we have a diffusion equation when λ Du 1 and an inverse diffusion equation when λ Du > 1. Thus the model mixes the heat equation and the reverse heat equation. Variational deblurring (case where the blurring kernel is known). Another class of attempts covers the so called variational restoration methods. Their principle is to search for a function u restored which, once blurred by the heat equation or another blurring kernel k, gives the original image u observed. Since applying the heat equation is equivalent to the convolution with an appropriate Gaussian function G σ, one searches for u restored satisfying u restored G σ = u observed noise. Because of its instability, this last equation does not yield numerically a unique solution. So, among possible solutions u restored is chosen as the most regular ones. The regularity is then measured by an energy norm, like the Total Variation [22]. The Rudin Osher Fatemi algorithm is efficient when the observed image u 0 is of the form k u + n, where k and the statistics of the noise n are known. Given the observed image u 0, one tries to find a restored version u that minimizes the functional ( E T V := Du(x) + λ(k u(x) u0 (x)) 2) dx, (4) Ω where Ω denotes the image domain and the parameter λ controls the oscillation in the restored version u. If λ is large, the restored version will closely satisfy the equation k u = u 0, but it may be very oscillatory. If instead λ is small, the solution is smooth but inaccurate. This parameter can be computed in principle as a Lagrange multiplier. 2 Neighborhood filters and NL-means Although this may seem paradoxical at first, the deblurring performance is strongly linked to a denoising performance. This is easily understood in frequency terms: a deblurring algorithm attempts to divide in the frequency domain the image by the blurring kernel, which is small on high frequencies. By doing so the high frequencies of the noise, inherent to image formation, blow up. If we manage to cure the image of this new added noise, we will be able to go further

5 into deblurring. However, since most denoising methods also destroy some image structures, this cure might pile denoising artifacts on top of the deblurring artifacts. We shall not discuss all denoising methods but concentrate on those which create the least artifacts. Among those, neighborhood filters seem to be the most adequate as pointed out by several perceptual and structure criteria in [3] and [4]. We shall call neighborhood filters all image filters which reduce the noise by averaging similar pixels. 2.1 Local neighborhood filters In order to denoise a pixel, it is better to average the color of this pixel with the nearby pixels with similar colors and only them. This is exactly the technique of the sigma-filter. This famous algorithm is generally attributed to J.S. Lee [14] in 1983 but can be traced back to L. Yaroslavsky and the Soviet Union image processing school [27]. The idea is to average neighboring pixels which also have a similar color value. The filtered value by this strategy can be written as NF h,ρ u(x) = 1 C(x) B ρ(x) e u(y) u(x) 2 h 2 u(y)dy, (5) where u(x) is the color at x and NF h,ρ u(x) its denoised version. Only pixels inside B ρ (x) are averaged, h controls the color similarity and C(x) is the normalization factor. SUSAN [24] and the bilateral filter [26] make this process more symmetric by involving a bilateral gaussian depending on both space and grey level. This leads to SNF h,ρ u(x) = 1 C(x) 2.2 Non local averaging e x y 2 ρ 2 e u(y) u(x) 2 h 2 u(y)dy The most similar pixels to a given pixel have no reason to be close to it. Think of periodic patterns, or of the elongated edges which appear in most images. In 1999 Efros and Leung [6] used non local self-similarities to synthesize textures and to fill in holes in images. Their algorithm scans a vast portion of the image in search of all the pixels that resemble the pixel in restoration. The resemblance is evaluated by comparing a whole window around each pixel, not just the color of the pixel itself. Applying this idea to neighborhood filters leads to a generalized neighborhood filter, the non-local means (or NL-means) filter [3][4]. NL-means has a formula quite similar to the sigma-filter, NLu(x) = 1 C(x) Ω e (Gρ u(x+.) u(y+.) 2 )(0) h 2 u(y) dy, (6) where G ρ is the Gauss kernel with standard deviation ρ, C(x) is the normalizing factor, h acts as a filtering parameter and (G ρ u(x +.) u(y +.) 2 )(0) = G ρ (t) u(x + t) u(y + t) 2 dt. R 2

6 The formula (6) means that u(x) is replaced by a weighted average of u(y). The weights are significant only if a gaussian window around y looks like the corresponding gaussian window around x. Thus the non-local means algorithm uses image self-similarity to reduce the noise. 3 Non-local deblurring 3.1 A non-local deblurring energy Our proposition for deblurring follows from the discussion of the preceding sections. We have seen that all above mentioned PDE s work as Ersatz for the reverse heat equation but in fact do not hold their promise of performing a reverse heat equation. As pointed out in Table 1 the various pseudo reverse heat equations or shock filters simply perform a very local and simple operation on a neighborhood. They cannot be qualified of more than enhancement operators. The Osher-Rudin-Fatemi method performs better as pointed out in [8]. Now its smoothing part, the total variation, is a texture killer. Indeed it removes small image oscillations and does not take into account image structure. Thus better use an algorithm taking into account fine image structure like NL-means. Now we need a variational form for this operator, as pointed out in [11] and [7]. The translation of NL-means in a variational framework is almost straightforward. The new hypothesis we wish to introduce to justify a nonlocal deblurring method is following: Hypothesis 1 The deblurred image must maintain the same similarities as the blurry image. It is clear that this hypothesis is a strong limitation since it forbids recreating by deblurring new structures. On the other hand it permits to rebuild all structures provided they left a trace, even tiny, in the blurry image. Thus given a slightly blurry image u 0 we want to define a deblurred version u which has the same self-similarities as u 0. In order to do so, define the new operator associated with u, NL 0 u(x) = 1 w 0 (x, y) u(y) dy, (7) C(x) Ω with w 0 (x, y) = e (G ρ u 0 (x+.) u 0 (y+.) 2 )(0) h 2. This means that NL 0 u is obtained by applying the NL-means algorithm, but instead of computing the weights with u they are computed with u 0. We are now in a position to define a non-local deblurring functional. ((u(x) E NL (u) := NL0 u(x)) 2 + λ(k u(x) u 0 (x)) 2) dx, (8) The first term can be viewed as a integral of the Laplacian of u with respect to a new metrics defined on the image by the internal similarities of

7 u 0. Thus it forces u to maintain the same coherence as u 0 had. For this reason it can be viewed as a regularization term. Let us see what link we can make for this term with more classical regularity term. If for instance we had w 0 (x, y) = e x y 2 /2h 2 (up to a normalization constant) and h is large enough, the first term would boil down to the integral (x) 2 dx. (This point of view is developed in [25][5].) The derivation of the NL-means from such a variational principle was also commented in [11]. The second term introduces the hypothesized convolution kernel k. In the case where no particular information is available, k is taken to be a gaussian. The parameter λ is the usual Tikhonov weight parameter between the fidelity to the data term and the regularity term. 3.2 The nonlocal reverse heat equation One can further simplify the process without loss of generality or precision. Indeed, we have mentioned that λ is left to the user s choice. Thus it seems more sound to view, as the models of section 1 did, the deblurring process as an evolution equation where tiny steps are taken at a time. Here the evolution equation writes as a non locally stabilized version of the reverse heat equation, namely t = u + λnl 0u, (9) where λ plays the same role as before and therefore has the same name. This linear equation can be by Chernoff s principle implemented algorithmically by an alternate scheme. One step of reverse heat equation is alternated with one step of nonlocal regularization. Thus the numerical scheme can be described by the discrete iteration u n+1 = NL 0 (u n ) c NL 0 ( u n ) = NL 0 (u n c u n ) (10) However, the NL-means filter is not fully symmetric and one can have w(x, y) w(y, x) because of the normalization factor C(x) in the definition of w(x, y). Thus the consistency of the above scheme with (9) is not guaranteed unless one symmetrizes the filter N L. However, the alternate scheme (10) makes sense by itself and this is why it was used in all experiments. The parameter c must be small enough to maintain stability CFL conditions. Notice finally that the reverse heat equation and its implementation are linear, in contrast with all above mentioned PDE s. 4 Experiments The non local reverse heat equation performs a uniform enhancement. Pixels with a similar configuration and therefore likely to belong to the same image structure are enhanced uniformly as displayed in Figure 2. In this case, an artificial image with geometric features has been slightly blurred and noisy. The

8 Fig. 1. Test images used in the experimentation of this paper. Left: Artificial image slightly convolved with a Gaussian kernel with standard deviation h = 2.5 and with additive white noise (σ = 2.5). Middle: Lena image. Right: Maillot image. Both natural images haven t been retouched, neither convolved or noisy. non local reverse heat equation enhances each figure uniformly and controls the noise at the same time. In order to control the contrast of the processed image and following Gabor remarks we allow the slope of the edges to be modified only by a factor m = 2 or 3. The backward diffusion is stopped at each pixel when the value of the Laplacian u exceeds m u 0. Figure 3 illustrates how controlling the growth of the Laplacian avoids a blow up of the edges. Due to the uniform enhancement applied by the NL-means regularization we can increase the contrast by a factor three without enhancing the noise. Figure 5 compares the non local reverse heat equation with two classical equations, the shock filter and the Perona Malik equation. The shock filter enhances the noise and oscillations of the edges making the result unconvincing. The shock filter can be combined with a mean curvature motion to avoid this effect as proposed in [2]. However, the mean curvature motion filters out details and texture and rounds off corners as the ones on Figure 2. The Perona-Malik enhances the main edges but filters the rest of the image. This makes this filter suitable for edge detection but not for enhancement. 5 Conclusion In this paper we made a quick review of PDE s emulating a reverse heat equation for image enhancement, when the blurring kernel is not known. Such algorithms can be used for enhancing images of relatively good quality by a numerical step by step procedure equivalent to a PDE. In fact these PDE s combine a smoothing term with a reverse equation term but the reverse equation is not a real reverse heat. In continuation we have shown that the smoothing term could be the NLmeans algorithm which preserves and enhances all image self-similarities. Thus alternating this filter with a real reverse heat equation permits to control the noise and yields a enhancement equation. Applying Gabor s stopping principle we have shown that the whole process can be stopped on time, thus yielding

9 Fig. 2. Experiment comparing the reverse heat equation and the modified version alternating with the NL-means algorithm. Top: Application of the reverse heat equation for times 0.5, 0.7. Bottom: Modified NL-means reverse heat equation for the same time and λ = 0.5. The reverse heat equation enhances both the noise and the edges until the image blows up. NL-means performs the same enhancement at edges but controls the noise in the flat regions. The geometry of the image is preserved by alternating with the NL-means even if the contrast at edges quickly increases. an optimal and artifact free image enhancement. Further work will concentrate on the mathematical analysis of the consistency of a Chernoff iteration. The method will also be extended to deal with more specific applications where the blurring kernel and the noise model are known. In that case, as pointed out in the paper, variational methods have to be involved and not just a step by step reverse heat equation. References 1. L. Alvarez and J. Esclarìn, Image quantization using reaction-diffusion equations, SIAM J. Appl. Math., 57 (1997), pp L. Alvarez and L. Mazorra, Signal and image restoration using shock filters and anisotropic diffusion, SIAM J. Numer. Anal., 31 (1994), pp A. Buades, B. Coll and J.M. Morel, A review of image denoising methods, with a new one, Multiscale Modeling and Simulation, vol 4 (2), pp , A. Buades, B. Coll and J.M. Morel, A non-local algorithm for image denoising, IEEE Int. Conf. on Computer Vision and Pattern Recognition, 2005.

10 Fig. 3. Experiment comparing the reverse heat equation and the modified version alternating with the NL-means algorithm. Top: Application of the reverse heat equation for times 0.4, 0.5. Middle: Modified NL-means reverse heat equation for the same time and λ = The combination with NL-means performs a better enhancement than the reverse heat equation since pixels with a similar structure are uniformly enhanced. However, the contrast at edges ends up growing too much. For this reason, following Gabor s ideas the reverse heat equation is stopped at each point where the laplacian value exceeds m times the original laplacian of the point. Bottom: Application of the reverse heat equation with stopped backward diffusion at m = 2, 3.

11 Fig. 4. Comparison of the modified reverse heat equation by regularizing with the neighborhood filter and the NL-means algorithm. From left to right: original image, reverse heat equation with regularization by the neighborhood and the NL-means filters. The NL-means regularization is able to restore the fine texture from the hat while the neighborhood filter blurs it. 5. Coifman, R.R. and Lafon, S. Diffusion maps, Preprint, Jan A. Efros and T. Leung, Texture synthesis by non parametric sampling, Proc. Int. Conf. Computer Vision (ICCV 99), vol. 2, pp G. Gilboa and S. Osher Nonlocal Linear Image Regularization and Supervised Segmentation, UCLA CAM Report 06-47, Sept Guichard, F. and Morel, J.M., A Note on Two Classical Enhancement Filters and Their Associated PDE s, International Journal of Computer Vision, vol. 52, n 2, pp , R. A. Hummel, B. Kimia, and S. W. Zucker, Gaussian blur and the heat equation: Forward and inverse solutions, in IEEE Computer Vision and Pattern Recognition, 1983, pp R. A. Hummel, B. Kimia, and S. W. Zucker, Deblurring gaussian blur, Computer Vision, Graphics, and Image Processing, 38 (1987), pp Kindermann S, Osher S and Jones PW, Deblurring and denoising of images by nonlocal functionals, Multiscale Modeling and simulation, 4 (4), pp , L. S. G. Kovasznay and H. M. Joseph, Image processing, Proc. IRE, 43 (1955), p H. P. Kramer and J. B. Bruckner, Iterations of a non-linear transformation for enhancement of digital images, Pattern Recognition, 7 (1975). 14. J.S. Lee, Digital image smoothing and the sigma filter, Computer Vision, Graphics and Image Processing, vol. 24, pp , M. Lindenbaum, M. Fischer, and A. M. Bruckstein, On gabor s contribution to image enhancement, Pattern Recognition, 27 (1994), pp M. Mahmoudi and G. Sapiro, Fast Image and Video Denoising via Non-Local Means of Similar Neighborhoods, IEEE Signal Processing Letters, vol. 12 (12), P. Perona and J. Malik, Scale space and edge detection using anisotropic diffusion, in Proc. Workshop on Computer Vision, Representation and Control, IEEE Computer Society, IEEE Press, Piscataway, NJ, 1987, pp P. Perona and J. Malik, Scale space and edge detection using anisotropic diffusion, IEEE Trans. Patt. Anal. Mach. Intell., 12 (1990), pp

12 Fig. 5. Comparison of different enhancement algorithms. From top to bottom and left to right: reverse heat equation (t = 0.7), shock filter, the Perona-Malik equation and the non local reverse heat equation with λ = 0.25 and stopping criterion at three times the original laplacian. The shock filter enhances noise and edge irregularities. The Perona-Malik equation enhances the main edges but blurs the rest of the image, thus loosing many details. 19. C.B. Price, P. Wambacq, and A. Oosterlinck. Image enhancement and analysis with reaction-diffusion paradigm. IEE-P, I:137(3), pp , S. Osher and L. I. Rudin, Feature-oriented image enhancement using shock filters, SIAM J. Numer. Anal., 27 (1990), pp L. I. Rudin, Images, Numerical Analysis of Singularities and Shock Filters, PhD thesis, California Institute of Technology, Pasadena, CA, L. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D, 60, pp , Schavemaker, J.G.M., Reinders, M.J.T., Gerbrands, J.J. and Backer, E. Image sharpening by morphological filtering, Pattern Recognition, vol. 33, n 6, pp , S.M. Smith and J.M. Brady, Susan - a new approach to low level image processing, International Journal of Computer Vision, Volume 23 (1), pp , 1997.

13 25. Szlam, A.D., Maggioni, M., Coifman, R.R. and Bremer Jr, J.C. Diffusion-driven multiscale analysis on manifolds and graphs: top-down and bottom-up constructions, Sensor bias estimation from measurements of targets with known deterministic dynamics. Edited by Burns, Paul; Blair, William D. Proceedings of the SPIE, 5914, pp , C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images, Sixth International Conference on Computer Vision, pp L.P. Yaroslavsky, Digital Picture Processing - An Introduction, Springer Verlag, P.C. Goldmark and J.M. Hollywood A new technique for improving the sharpness of television pictures, pp , Proc. IRE, Oct