Laplace-distributed increments, the Laplace prior, and edge-preserving regularization

Transcription

1 J. Inverse Ill-Posed Probl.? (????), 1 15 DOI /JIIP.????.??? de Gruyter???? Laplace-distributed increments, the Laplace prior, and edge-preserving regularization Johnathan M. Bardsley Abstract. For a given two-dimensional image, we define the horizontal and vertical increments at a pixel location to be the difference between the intensity values at that pixel and at the neighboring pixels to the right and above, respectively. For a typical image, it makes intuitive sense that the increments will usually be near zero, corresponding to areas of smooth variation in image intensity, but will often have large magnitude, corresponding to edges where sharp intensity changes occur. In this paper, we explore the use of the Laplace increment model, in which the increments are assumed to be independent and identically distributed Laplace random variables a distribution with heavy tails allowing for large increment values with zero mean. The prior constructed from the Laplace increment model is very similar to the total variation (TV) prior. We perform a theoretical analysis of its properties, which shows that the Laplace prior yields a regularization scheme with regularized solutions contained in the space of bounded variation, just as for the TV prior. Moreover, numerical experiments indicate that the Laplace prior yields reconstructions that are qualitatively very similar to those obtained using TV. Keywords. inverse problems, regularization, Bayesian inference, total variation, Markov random fields. 1 Mathematics Subject Classification. 15A29, 65F22, 65C. 1 Introduction In this paper, we focus on linear inverse problems that can be modeled as a Fredholm integral equation of the first kind, which after numerical discretization yield a system of linear equations of the form b = Ax, where b R N is the observed data, x is the N 1 vector of unknowns, and A is an N N matrix with singular values decaying continuously to zero, and hence is ill-conditioned. In practice, b contains random noise. The most common choice of noise model is independent and identically distributed (iid) Gaussian, i.e. b = Ax + η, (1.1)

2 2 J. M. Bardsley where η is a Gaussian random vector with components satisfying η i N (, λ 1 ) for all pixels i; here the inverse-variance parameter λ is known as the precision. Another common noise model, used in both astronomical and medical imaging, is Poisson: b = Poisson(Ax + γ), (1.2) where γ is the m 1 vector of background counts and is assumed known. The probability density functions for (1.1) and (1.2) are given, respectively, by p(b x) exp ( λ2 ) Ax b 2, (1.3) n 2 p(b x) exp {([Ax] j + γ j ) + b j ln([ax] j + γ j )}, (1.4) j=1 where denotes proportionality. In both cases, due to the properties of the matrix A, the maximum likelihood estimator x ML = arg max p(b x) x is unstable with respect to the noise in the data b. Such instability is a characteristic of inverse problems, and it has to do with the fact that the matrix A is the numerical discretization of a compact operator defined on a function space with singular values that decay continuously to zero. The standard technique for overcoming this instability is regularization. For general discussions of inverse problems and techniques for regularization, from both numerical and functional analytic points-of-view, see one of the many excellent tests on the subject, e.g., [1, 13, 15, 19]. In the context of Bayesian statistics, regularization corresponds to the choice of the prior probability density function. Bayes Theorem states that given p(b x), and an assumed prior probability density function p(x), the posterior probability density function p(x b) can be written p(x b) p(b x)p(x). (1.5) One can then obtain a stable reconstructed image by computing the maximum of the posterior density the so called maximum a posteriori (MAP) estimator via x MAP = arg min { ln p(b x) ln p(x)}. (1.6) x In this paper, we obtain the prior by assuming that the increments (i.e. the differences between neighboring pixels) are iid Laplace distributed. This approach is

3 The Laplace prior for edge-preserving regularization 3 known as conditional autoregression [8] and it defines a Markov random field prior p(x). We begin with the one-dimensional case and assume that the increments are defined x i = x i+1 x i. Then our assumption is that x i Laplace(, δ 1 ), i = 2,..., n, (1.7) where Laplace(µ, δ 1 ) has probability density function p(x µ, δ 1 ) = δ exp ( δ x µ ). 2 Because of our assumption of independence, the joint density for x (i.e. the prior) is given by ( ) n p(x) exp δ x i x i 1 i=2 = exp ( δ Dx 1 ), (1.8) where 1 denotes the l 1 -norm and D is the forward difference matrix with periodic boundary conditions, i.e., D = n n. We note that a Neumann or Dirichlet boundary condition can also be assume here, which corresponds to a modification to the matrix D. The assumption of Laplacian increments is motivated by the fact that in many signals, the increments sizes are typically small, but outliers (large increments) are not uncommon. Due to the fact that the Laplace distribution has heavy tails, large increments (outliers) are much more probable than if a Gaussian increment model (as in [18]) is assumed. We note that the Gaussian increment model leads to the standard regularization function δx T Lx, where L is the discretized negative- Laplacian matrix. To illustrate the difference between the Laplace and Gaussian probability densities, we plot them together in Figure 1. The prior (1.8) yields total variation regularization [19]. The connection between the Laplacian increment model (1.7) and total variation regularization for 1D signals is discussed in some detail in [12]. In this paper, we explore the use

4 4 J. M. Bardsley.5.45 Normal(,1) Laplace(,1) Figure 1. The Laplace and Normal probability density functions with mean and variance 1. of the Laplace increment model for two-dimensional (2D) signals, which yields a regularization that is not 2D total variation, but yields very similar results. Indeed, we will show that viewed in the function space setting, these two regularization functions yield convergent regularization schemes with regularized solutions lying in the space of bounded variation. The paper is organized as follows. In the next section, we present the Laplace increment model for 2D signals, and then present a brief theoretical analysis that shows that the resulting prior yields a convergent regularization scheme with solutions lying in the space of bounded variation. Then in Section 3, we implement the regularization on problems from image deblurring and computed tomography. And finally, we end with conclusions in Section 4. 2 A Laplacian Increment Model for 2D Signals We begin by defining the horizontal and vertical increments, respectively, as h x ij = x i+1,j x ij and v x ij = x i,j+1 x ij, and assume h x ij, v x ij iid Laplace(, δ 1 ), i, j = 1,..., n 1. (2.1)

5 The Laplace prior for edge-preserving regularization 5 Then the probability density function for x has the form (see [18, Chapter 3]) p(x) exp δ n n 1 n n 1 x i+1,j x ij + x i,j+1 x ij 2 j=1 i=1 i=1 j=1 ( = exp δ 2 ( (I D)x 1 + (D I)x 1 ) ( = exp δ ) 2 ( D vx 1 + D h x 1 ), (2.2) where denotes the Kronecker product, D v = D I is the discrete vertical derivative, and D h = I D the discrete horizontal derivative. Note that in contrast, the 2D total variation prior has the form ( p(x) exp δ n ) [D v x] 2 i 2 + [D hx] 2 i. (2.3) i=1 It is well-known that when the total variation prior is used, the resulting reconstructed images have a cartoon texture, or in other words are approximately piecewise constant. In the next section, we present a theoretical analysis that shows that (2.2) can be expected to yield reconstructed images that are qualitatively similar to those obtained using total variation. One possible benefit of (2.2) over (2.3) is that different regularization parameters could be used for the vertical and horizontal increment terms, i.e. δ v and δ h rather than just δ, which would penalize the vertical and horizontal increments differently. Such an approach is discussed in the Gaussian case in [18]. ) 2.1 Theoretical Analysis The theoretical analysis of regularization schemes constitutes a significant portion of the work that s been done in the field of inverse problems; see, e.g., [1] and the references therein. While mathematical inverse problems is an interesting field on its own, it can also yield insights into a computational regularization method that cannot be obtained otherwise. This fact is perhaps best illustrated by the example of total variation (TV) regularization. In the discretized setting, where computations are done, the reconstructions obtained using TV have striking visual qualities that suggest that there is something special about the method (see, e.g., the results in the numerical experiments section). However, what makes TV regularization unique can only be made explicit through a theoretical analysis. In particular, for the Gaussian and

6 6 J. M. Bardsley Poisson likelihoods, (1.3) and (1.4), it is shown in [1] and [6], respectively, that when TV regularization is used, solutions lie in the Banach space of functions of bounded variation [19]. In this section, we will show that the Laplace increment prior (2.2) has the same properties as the TV prior. To do this, we mimic the exposition found in [2]. First, consider the functional analogue of (1.1), namely b = Ax. (2.4) Our application of interest is image processing, so that b L () denotes the image intensity, and x L 2 () the (nonnegative) intensity of the unknown object. Each function is defined on a closed, bounded domain R d. Finally, Ax(s) def = a(s; t)u(t) dt, where a L ( ) is the error free, nonnegative point spread function (PSF). Given these assumptions, A : L 2 () L 2 () is a compact operator, and hence, the problem of solving (2.4) for x is ill-posed [19]. Moreover, Ax whenever x, and hence, assuming that the true image x exact, we have that the error free data b = Ax exact is bounded below by zero. Note that here, and in the remainder of the document, we omit almost everywhere" from the mathematical statements in which its presence is called for. The functional analogue of the MAP estimation problem (1.6) defines the following function { } R α (a, b) def = min T α (Ax; b) def = T (Ax; b) + αj(x), (2.5) where x C C = {x L 2 () x }, and α and J are the regularization parameter and functional, respectively. For the Gaussian likelihood case, T (Ax; b) def = (Ax b) 2 dt (2.6) while for the Poisson likelihood, T (Ax; b) def = ((Ax + γ) b log(ax + γ)) dt. (2.7) In the case of total variation regularization, the regularization functional J has the form J(x) def = sup x y dt, (2.8) y Y

7 The Laplace prior for edge-preserving regularization 7 where is the divergence operator, and Y = {y C 1 (; R d ) : y(t) 1 t }. The functional analogue of the Laplace increment prior (2.2) takes the form J(x) = d i=1 sup y Y x y t i dt, (2.9) with Y as above, t = (t 1,..., t d ), and = 1 d. Note that for d = 1 these two definitions coincide. If x is continuously differentiable on, (2.8) and (2.9) take the less intimidating form [19, Remark 8.1] J(x) = x 2 dt, (2.1) J(x) = d i=1 x t i dt, (2.11) respectively, where is the gradient operator. Note that (2.1) and (2.11) are the functional analogues of (2.3) and (2.2), where there d = 2. We can finally define the space of bounded variation [11]: BV () = {x L 1 () : J(x) < + }, (2.12) with J defined by (2.1), which is a Banach space with norm R α defines a regularization scheme x BV () = x L 1 () + J(x). In this subsection, we define the notion of a regularization scheme, and then prove that R α defined in (2.5) with J given by (2.11) satisfies the conditions of this definition. We also show that the resulting regularized solutions lie in BV (). The regularization operator R α defined in (2.5) has domain B = {â L ( ) â } {ẑ L () ẑ }, which is a closed subset of the Banach space L ( ) L () with induced def norm (â, ẑ) B = max{ â L ( ), ẑ L ()}. To define regularization scheme, we need a sequence of operator equations b n = A n x, (2.13)

8 8 J. M. Bardsley where b n L () is nonnegative, and A n x(s) def = a n (s; t)x(t) dt, with a n L ( ) a nonnegative point spread function (PSF). Note, then, that A n x whenever x C. The functions b n and a n should be viewed as approximations of the true data b and PSF a, respectively. In astronomy and PET, for example, both the data and PSF are estimated and hence contain errors. Thus our definition of regularization scheme, which we present now, accounts for errors in the measurements of both b and a. Definition 2.1. {R α } α> defines a regularization scheme on B provided (i) R α is well-defined and continuous on B for all α > ; (ii) given a sequence {(a n, z n )} n=1 B such that (a n, z n ) (a, z) B, there exists a positive sequence {α n } n=1 such that for some p 1. R αn (a n, z n ) u exact L p () Next, we state and prove the main result of this section. Theorem 2.2. Let R α : B C be defined as in (2.5), with T defined by (2.6) or (2.7) and J defined by (2.8) or (2.11). Then {R α } α> is a regularization scheme provided the null-space of A does not contain the constant functions. Moreover, in both cases, Range(R α ) BV (). Proof. For J defined by (2.1), the result follows from [1] for least squares fit-todata (2.6) and from [6] for negative-log Poisson fit-to-data (2.7). For J defined by (2.11), the result for both fit-to-data functions follows from [1] and [6] together with the fact that 1 d d i=1 x t i dt x 2 dt d i=1 x t i dt.

9 The Laplace prior for edge-preserving regularization 9 3 Numerical Experiments The theoretical results of the previous section suggest that the use of the Laplace increment prior will yield regularized solutions that are qualitatively similar to those obtain when total variation regularization is used. We will see in this section that that is indeed the case. Our reconstruction technique is the extension of the lagged-diffusivity fixed point iteration to the Laplace increment case. First, note that p(x) defined by (2.2) is non-differentiable due to the presence of the absolute value, and hence we use the following differentiable approximation: ln p(x) δ n n 1 n n 1 ψ((x i+1,j x ij ) 2 ) + ψ((x i,j+1 x ij ) 2 ), 2 j=1 i=1 i=1 j=1 where ψ(t) = 2 t + β. This expression has gradient L(x)x where (see [19]) L(x) = D T v diag(ψ (D v x))d v + D T h diag(ψ (D h x))d h, (3.1) which motivates the following lagged-diffusivity-type algorithm. Algorithm 1. Set k =, L = D T v D v + D T h D h, and choose α >. (i) Compute x k+1 α = arg min x { ln p(b x) + α 2 xt L k x }. (ii) Update L k+1 = L(x k+1 α ) via (3.1), set k = k + 1, and return to Step 1. Remark 3.1. In practice, α > can be chosen at the outset (as in Algorithm 1) using a regularization parameter selection method with regularization matrix L. It can also be updated every jth iteration of Algorithm 1. In either case, we advocate the use of generalized cross validation (GCV), which can be found in [19] for the least squares likelihood and [5] for the Poisson likelihood. 3.1 Image Deblurring with Gaussian Noise Now we test the above iteration on a two-dimensional image deblurring problem. The forward model, mapping the unknown x to the observation b, both defined on [, 1] [, 1], has convolution form: b(s 1, s 2 ) = 1 1 a(s 1 t 1, s 2 t 2 )x(t 1, t 2 )dt 1 dt 2. For our experiments, we choose a Gaussian convolution kernel a, and discretize the integral using mid-point quadrature on a uniform computational grid

10 1 J. M. Bardsley Figure 2. On the left is the two-dimensional image used to generate the data, and on the right is the blurred noisy data. over [,1] [,1]. We assume that x extends periodically outside of [,1] [,1], which after discretization yields a linear system of equations b = Ax in which A has block circulant structure. Thus A can be diagonalized by the discrete Fourier transform (DFT) [19]. The data b is generated using (1.1) with the noise variance λ 1 chosen so that the noise strength is 2% that of the signal strength. The image used to generate the data and the data itself are shown in Figure 2. Since Gaussian noise is assumed, the likelihood function is given by (1.3) yielding a quadratic minimization problem in Step 1 of Algorithm 1. However since the matrix L k will not have block circulant structure, the preconditioned conjugate gradient method (PCG) must be used to approximately compute x k+1 α. For the preconditioner, we use M = A T A + α(d T h D h + D T v D v ), which given our assumptions is diagonalizable by the DFT; indeed, ( ( )) M 1 1 r = vec IDFT DFT(R), â s 2 + αˆl s where R is an n n array; r = vec(r) stacks the columns of R to create r; ˆl s is the n n eigenvalue array for D T h D h + D T v D v ; â s is the n n eigenvalue array of A; and IDFT is the inverse discrete Fourier transform. See [19] for more detail on the diagonalization of the block circulant matrices by the DFT. Finally, in order to obtain a value for α, we implement generalized cross validation (GCV) with regularization matrix L. This allows us to exploit circulant structure making the computation very efficient; specifically, we take α to be the

11 The Laplace prior for edge-preserving regularization Figure 3. On the left is the reconstruction obtained after 1 iterations of Algorithm 1, while on the right is the corresponding edge map. minimizer of G(α) = α 2 n i,j=1 [ˆl s ] 2 ˆB ij 2 ij / n 2 ( â s 2 ij + α[ˆl s ] ij ) 2 n i,j=1 â s 2 ij â s 2 ij + α[ˆl s ] ij where ˆB = DFT(B) with B the n n array satisfying b = vec(b). We are now ready to test Algorithm 1 on the above image deblurring test problem. We choose β =.1 in our definition of ψ(t) and show reconstructions after 1 iterations of the algorithm. The reconstruction is given on the upperleft in Figure 3, while the edge map, which is a plot of the diagonal values of (Dv x 1 ) 2 + (D h x 1 ) 2 is plotted on the upper-right. Finally, the relative error for the final reconstruction was x α x true / x true =.19. 2, 3.2 Positron Emission Tomography with Poisson Noise In positron emission tomography (PET), a radioactive tracer element is injected into the body, which exhibits radioactive decay, resulting in photon emission. The emitted photons that leave the body are recorded by a photon detector, which also determines the line of response (LOR) L(ω, y), along which the photon(s) have propagated; given a fixed coordinate system, L(ω, y) is the unique line making an angle ω with an axis (e.g. the vertical) that is a perpendicular distance of y from the origin. We parameterize L(ω, y) by L(ω, y) = {z(s) s S}. In PET, the data b(ω, y) corresponds to the number of detected incidents along

12 12 J. M. Bardsley L(ω, y). The model relating the tracer density x to the data is given by b(ω, y) = A ω,y (z(s))x(z(s))ds, L(ω,y) where the impulse response function A ω,y (z(r)) can be viewed as the probability that an emission event located at z(r) along L(ω, y) is recorded by the detector system. To determine A ω,y, we note that a pair of photons are emitted at a location z(r) along L(ω, y) with detectors located at z() and z(s). Then the probability that both photons reach the detector is A ω,y (z(r)) = e r µ(z(t)) dt e S r µ(z(t)) dt = e L(ω,y) µ(z(t)) dt, which doesn t depend on r and µ(z). Hence we can simplify the model to b(ω, y) = e L(ω,y) µ(z(t)) dt x(z(s))ds. (3.2) L(ω,y) Note that dividing both sides of (3.2) by e L(ω,y) µ(z(t)) dt yields the Radon transform, which is what is solved in the computed tomography inverse problem [16]. After discretization, (3.2) can be written as a system of linear equations of the form (1.1). The discretization occurs both in the spatial domain, where µ and x are defined, as well as in the Radon transform ((ω, y)) domain, where the data b is defined. We use a uniform n n spatial grid, and a uniform grid for the transform domain with n angles and n sensors. In our experiments, n = 128 so that (1) has size To generate the data b, we use the Poisson noise model (1.2) with γ = 1 synthetically generated Poisson noise. The true tracer density, given on the left in Figure 4, is the Shepp-Logan phantom. We take µ = in (3.2) to construct our matrix A, which is standard for PET numerical experiments [17], and scale the true tracer density x so that the percent-noise is approximately 11. The data is shown on the right in Figure 4. Next, we test Algorithm 1 on this synthetic PET example. We again choose β =.1 in our definition of ψ(t) and compute α using the GCV method of [5] for Poisson negative-log likelihood function with regularization matrix L. However in the Poisson case, this choice of α does not work well in later iterations of Algorithm 1, but using the hierarchical approach of [3], it can be motivated that α = α does work well, so this is what we use here. On various synthetic tests for PET data, this method of choosing α is effective, however we have not extensively tested it. Given this choice of α, the reconstruction after 1 iterations of Algorithm 1 is given on the upper-left in Figure 5, while the edge

13 The Laplace prior for edge-preserving regularization Figure 4. On the left is the true image, and on the right is the PET data Figure 5. On the left is the reconstruction obtained after 1 iterations of Algorithm 1, while on the right is the corresponding edge map. map, which is a plot of the diagonal values of (D v x 1 ) 2 + (D h x 1 ) 2, is plotted on the upper-right. Finally, the relative error for the final reconstruction was x α x true / x true = Notice that in both instances, the reconstructions are qualitatively similar to those obtained using total variation. 4 Conclusions In this paper, we focused on the use of the Laplace prior, or regularization function, constructed from the assumption that the increments in the unknown image are independent and identically distrubuted, zero-mean Laplace random variables.

14 14 J. M. Bardsley The Laplace prior is very similar to the total variation (TV) prior indeed, in one-dimension they are the same and yields reconstructed images that are both quantitatively, and qualitatively, very similar. We present a theoretical analysis that shows that just as for TV, the Laplace prior yields regularized solutions that lie in the space of bounded variation; and we present numerical experiments from both image deblurring and positron emission tomography showing that Laplace prior works well and yields TV-like reconstructed images. The benefit of using the Laplace prior, as apposed to TV, is that it follows from concrete distributional assumptions regarding the increments in the unknown image, which can be modified to better fit the specific situation. Bibliography [1] R. ACAR AND C. R. VOGEL, Analysis of bounded variation penalty methods for ill-posed problems, Inverse Problems, 1 (1994), pp [2] J. M. BARDSLEY, A Theoretical Framework for the Regularization of Poisson Likelihood Estimation Problems, Inverse Problems and Imaging, 4(1) (1), p [3] J. M. BARDSLEY, D. CALVETTI, AND E. SOMERSALO, Hierarchical regularization for edge-preserving reconstruction of PET images, Inverse Problems, 26 (1), 351. [4] J. M. BARDSLEY AND J. GOLDES, An Iterative Method for Edge-Preserving MAP Estimation when Data-Noise is Poisson, SIAM Journal on Scientific Computing, 32(1), (1), pp [5] J. M. BARDSLEY AND J. GOLDES, Regularization Parameter Selection Methods for Ill-Posed Poisson Maximum Likelihood Estimation, Inverse Problems, 25 (9) 955. [6] J. M. Bardsley and A. Luttman, Total variation-penalized poisson likelihood estimation for ill-posed problems, Advances in Computational Mathematics, 31 (9), [7] J. M. BARDSLEY AND C. R. VOGEL, A Nonnnegatively Constrained Convex Programming Method for Image Reconstruction, SIAM Journal on Scientific Computing, 25(4), 4, pp [8] J. BESAG, Spatial Interaction and the Statistical Analysis of Lattice Systems, Journal of the Royal Statistical Society, Series B, 36(2) (1974), pp [9] D. CALVETTI AND E. SOMERSALO, Introduction to Bayesian Scientific Computing, Springer 7. [1] H. K. ENGL, M. HANKE, AND A. NEUBAUER, Regularization of Inverse Problems, Kluwer, 1996.

15 The Laplace prior for edge-preserving regularization 15 [11] L. C. Evans and R. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, [12] M. GREEN, Statistics of images, the TV algorithm of Rudin-Osher-Fatemi for image denoising, and an improved denoising algorithm, CAM Report 2-55, UCLA, October 2. [13] P. C. HANSEN, Discrete Inverse Problems: Insight and Algorithms, SIAM, Philadelphia, 1. [14] J. HUANG AND D. MUMFORD, Statistics of Natural Images and Models, IEEE Conf. on Computer Vision and Pattern Recognition, 1999, pp [15] J. KAIPIO AND E. SOMERSALO, Statistical and Computational Inverse Problems, Springer 5. [16] F. NATTERER AND F. WÜBBELING, Mathematical Methods in Image Reconstruction, SIAM 1. [17] J. M. OLLINGER AND J. A. FESSLER, Positron-Emission Tomography, IEEE Signal Processing Magazine, pp , (January 1997). [18] H. RUE AND L. HELD, Gaussian Markov Random Fields: Theory and Applications, Chapman and Hall/CRC, 5. [19] C. R. VOGEL, Computational Methods for Inverse Problems, SIAM, Philadelphia, 2. [] C. R. VOGEL AND M. E. OMAN, A fast, robust algorithm for total variation based reconstruction of noisy, blurred images, IEEE Transactions on Image Processing, 7 (1998), pp Received???. Author information Johnathan M. Bardsley, 32 Campus Drive, Department of Mathematical Sciences, University of Montana, Missoula, MT, 59812, USA. bardsleyj@mso.umt.edu