), σ is a parameter, is the Euclidean norm in R d.

Transcription

1 WEIGHTED GRAPH LAPLACIAN AND IMAGE INPAINTING ZUOQIANG SHI, STANLEY OSHER, AND WEI ZHU Abstract. Inspired by the graph Laplacian and the point integral method, we introdce a novel weighted graph Laplacian method to compte a smooth interpolation fnction on a point clod in high dimensional space. The nmerical reslts in semi-spervised learning and image inpainting show that the weighted graph Laplacian is a reliable and efficient interpolation method. In addition, it is easy to implement and faster than graph Laplacian. 1. Introdction. In this paper, we consider interpolation on a point clod in high dimensional space. This is a fndamental problem in many data analysis problems and machine learning. Let P = {p 1,,p n } be a set of points in R d and S = {s 1,,s m } be a sbset of P. Let be a fnction on the point set P and the vale of on S P is given as a fnction g over S, i.e. (s) = g(s), s S. The goal of the interpolation is to find the fnction on P with the given vales on S. Since the point set P is nstrctred in high dimensional space, traditional interpolation methods do not apply. One model which is widely sed in many applications is to minimize the following energy fnctional, (1.1) J() = 1 w(x,y)((x) (y)) 2, 2 with the constraint x, (1.2) (x) = g(x), x S. Here w(x,y) is a given weight fnction. One often sed weight is the Gassian weight, w(x,y) = exp( x y 2 σ ), σ is a parameter, is the Eclidean norm in R d. 2 It is easy to derive the Eler-Lagrange eqation of the above optimization problem, which is given as follows, (w(x,y)+w(y,x))((x) (y)) = 0, x P\S, (1.3) (x) = g(x), x S. This approach is the well known graph Laplacian [3, 17] which has been sed widely in many problems. Recently, it was observed that the soltion given by the graph Laplacian is not continos at the sample points, S, especially when the sample rate, S / P, is low [13]. Consider a simple 1D example. Let P be the nion of 5000 randomly sampled points over the interval (0,2) and we label 6 points in P. Points 0,1,2 are in the labeled set S and the other 3 points are selected at random. The soltion given by the graph Laplacian,(1.3) is shown in Fig. 1. Clearly, the fnction compted by the Research spported by DOE-SC and NSF DMS Z. Shi was partially spported by NSFC Grant , Ya Mathematical Sciences Center, Tsingha University, Beijing, China, zqshi@mail.tsingha.ed.cn. Department of Mathematics, University of California, Los Angeles, CA 90095, USA, sjo@math.cla.ed Department of Mathematics, University of California, Los Angeles, CA 90095, USA, weizh731@math.cla.ed 1

2 2.5 Graph Laplacian Fig. 1. Soltion given by graph Laplacian in 1D examples. Ble line: interpolation fnction given by graph Laplacian; red circles: given vales at label set S. graph Laplacian is not continos at the labled points. It actally does not interpolate the given vales. It was also shown that the discontinity is de to the fact that one important bondary term is dropped in evalating the graph Laplacian. Consider the harmonic extension in the continos form which is formlated as a Laplace-Beltrami eqation with Dirichlet bondary condition on manifold M, (1.4) { M (x) = 0, x M, (x) = g(x), x M, In the point integral method [6, 7, 11, 12], it is observed that the Laplace-Beltrami eqation M (x) = 0 can be approximated by the following integral eqation. 1 (y) (1.5) ((x) (y))r t (x,y)dy 2 t n R t (x,y)dτ y = 0, M where R t (x,y) = R( x y 2 x y 2 4t ), Rt (x,y) = R( 4t ) and d R(s) ds = R(s). If R(s) = exp( s), R = R and Rt (x,y) becomes a Gassian weight fnction. n is the otwards normal of the bondary M. Comparing the above integral eqation (1.5) and the eqation in the graph Laplacian (1.3), we can clearly see that the bondary term in (1.5) is dropped in the graph Laplacian. However, this bondary term is not small and neglecting it cases troble. To get a continos interpolation, we need to inclde the bondary term. This is the idea of the point integral method. To deal with the bondary term in (1.5), a Robin bondary condition is sed to approximate the Dirichlet bondary condition, (1.6) (x)+µ (x) n M = g(x), x M, 2

3 where 0 < µ 1 is a small parameter. 1 t (x,y)dy t M((x) (y))r 2 µ The corresponding discrete eqations are (1.7) M R t (x,y)(g(y) (y))dτ y = 0. R t (x,y)((x) (y))+ 2 R t (x,y)((y) g(y)) = 0, x P, µ y S The point integral method has been shown to give consistent soltions [13, 12, 6]. The interpolation algorithm based on the point integral method has been applied to image processing problems and gives promising reslts [9]. Eqation (1.7) is not symmetric, which makes the nmerical solver not very efficient. The main contribtion of this paper is to propose a novel interpolation algorithm, the weighted graph Laplacian, which preserves the symmetry of the original Laplace operator. The key observation in the weighted graph Laplacian is that we need to modify the energy fnction (1.1) to add a weight to balance the energy on the labeled and nlabeled sets. min x P\S with the constraint w(x,y)((x) (y)) 2 + P S w(x,y)((x) (y)) 2, x S (x) = g(x), x S. P, S are the nmber of points in P and S, respectively. When the sample rate, S / P, is high, the weighted graph Laplacian becomes the classical graph Laplacian. When the sample rate is low, the large weight in weighted graph Laplacian forces the soltion become continos near the labeled set. We test the weighted graph Laplacian on MNIST dataset and image inpainting problems. The reslts show that the weighted graph Laplacian gives better reslts than the graph Laplacian, especially when the sample rate is low. The weighted graph Laplacian provides a reliable and efficient method to find reasonable interpolation on a point clod in high dimensional space. The rest of the paper is organized as follows. The weighted graph Laplacian is introdced in Section 2. The tests of the weighted graph Laplacian on MNIST and image inpainting are presented in Section 3 and Section 4 respectively. Some conclsions are made in Section Weighted Graph Laplacian. First, we split the objective fnction in (1.1) to two terms, one is over the nlabeled set and the other over the labeled set. min w(x,y)((x) (y)) 2 + w(x,y)((x) (y)) 2, x S x P\S If we sbstitte the optimal soltion into above optimization problem, for instance the soltion in the 1D example (Fig. 1), it is easy to check that the smmation over the labeled set is actally pretty large de to the discontinity on the labeled set. However, when the sample rate is low, the smmation over the nlabeled set actally 3

4 overwhelms the smmation over the labeled set. So the continity on the labeled set is sacrificed. One simple idea to assre the continity on the labeled set is to pt a weight ahead of the smmation over the labeled set. min x P\S w(x,y)((x) (y)) 2 +µ w(x,y)((x) (y)) 2, x S This is the basic idea of or approach. Since this method is obtained by modifying the graph Laplacian to add a weight inside, we call this method the weighted graph Laplacian, WGL for short. To balance these two terms, one natral choice of the weight µ is the inverse of the sample rate, P / S. Based on this observation, we get following optimization problem (2.1) min x P\S with the constraint w(x,y)((x) (y)) 2 + P S w(x,y)((x) (y)) 2, x S (x) = g(x), x S. The optimal soltion of (2.1) can be obtained by solving a linear system (w(x,y)+w(y,x))((x) (y)) ( ) P + S 1 w(y,x)((x) (y)) = 0, x P\S, y S (x) = g(x), x S. This linear system is symmetric and positive definite. Comparing WGL with the graph Laplacian, in WGL, we see that a large positive term is added to the diagonal of the coefficient matrix which makes the linear system easier to solve. After solving the above linear system sing some iterative method, for instance conjgate gradient, we find ot that it converges faster than graph Laplacian. In or tests, the weighted graph Laplacian is abot two times faster than graph Laplacian on average. In addition to the symmetry, the weighted graph Laplacian also preserves the maximm principle of the original Laplace-Beltrami operator, which is important in some applications. Figre 2 shows the comparison between WGL and GL in the 1D example. The reslt of WGL perfectly interpolates the given vales while GL fails. We want to remark that there are other choices of the weight µ in WGL. P / S works in many applications. Based on or experience, P / S seems to be the lower bond of µ. In some applications, we may make µ larger to better fit the sample points. We can prove the consistency of weighted graph Laplacian for the Dirichlet problem for Laplace-Beltrami eqation based on a volme constraint [4, 10]. Intitively, a large weight in WGL makes the soltion close to the given vale near the labeled set. It is likely to confine the soltion in a volme arond the labled set. Using the theory of volme constraints, we can prove the consistency of the weighted graph Laplacian. 4

5 2.5 Graph Laplacian 2.5 Weighted Graph Laplacian Fig. 2. Soltion given by graph Laplacian and weighted graph Laplacian in 1D examples. Ble line: interpolation fnction given by graph Laplacian; red circles: given vales at label set S. The idea of weighted graph Laplacian also applies on general constraints, L = b with L being a measrement operator and b being the observations. In this case, the labeled set S shold be defined as the points involved in the measrement L. For instance, if L is the average operator, i.e. (x) = b, then the labeled set S = P, althogh there is only one measrement. 3. Semi-spervised Learning. In this section, we briefly describe the algorithm of semi-spervised learning based on that proposed by Zh et al. [17]. We plg into the algorithm the aforementioned approach for weighted graph Laplacian, and apply them to the well-known MNIST dataset, and compare their performances. Assme we are given a point set P = {p 1,p 2,,p n } R d, and some labels {1,2,,l}. A sbset S P is labeled, S = x P l S i, i=1 S i is the labeled set with label i. In a typical setting, the size of the labeled set S is mch smaller than the size of the data set P. The prpose of the semi-spervised learning is to extend the label assignment to the entire P, namely, infer the labels for the nlabeled points. The algorithm is smmarized in Algorithm 1. In Algorithm 1, we se both graph Laplacian and weighted graph Laplacian to compte φ i respectively. In weighted graph Laplacian, φ i is obtained by solving the following linear system (w(x,y)+w(y,x))(φ i (x) φ i (y)) ( ) P + S 1 w(y,x)(φ i (x) φ i (y)) = 0, x P\S, y S φ i (x) = 1, x S i, φ i (x) = 0, x S\S i. 5

6 Algorithm 1 Semi-Spervised Learning Reqire: A point set P = {p 1,p 2,,p n } R d and a partial labeled set S = l i=1 S i. Ensre: A complete label assignment L : P {1,2,,l} for i = 1 : l do Compte φ i on P, with the constraint end for for (x P\S) do Lable x as following φ i (x) = 1, x S i, φ i (x) = 0, x S\S i, L(x) = k, where k = arg max 1 i l φ i(x). end for In graph Laplacian, we need to solve (w(x,y)+w(y,x))(φ i (x) φ i (y)) = 0, x P\S, φ i (x) = 1, x S i, φ i (x) = 0, x S\S i. We test Algorithm 1 on MNIST dataset of handwritten digits [2]. MNIST dataset contains 70, gray scale digit images. We view digits 0 9 as ten classes. Each image can be seen as a point in 784-dimensional Eclidean space. The weight Fig. 3. Some images in MNIST dataset. fnction w(x, y) is constrcted as following ) (3.1) w(x,y) = exp ( x y 2 σ(x) 2 σ(x) is chosen to be the distance between x and its 20th nearest neighbor, To make the weight matrix sparse, the weight is trncated to the 50 nearest neighbors. 6

7 100/70,000 70/70,000 50/70,000 WGL GL WGL GL WGL GL 91.83% 74.56% 84.31% 29.67% 79.20% 33.93% 89.20% 58.16% 87.74% 42.29% 78.66% 21.12% 93.13% 62.98% 83.19% 37.02% 71.29% 24.50% 90.43% 41.27% 91.08% 12.57% 71.92% 29.79% 92.27% 44.57% 84.74% 35.15% 80.92% 37.50% Table 1 Accracy of weighted graph Laplacian and graph Laplacian in the test of MNIST. In or test, we label 100, 70 and 50 images respectively. The labeled images are selected at random in 70,000 images. For each case, we do 5 independent tests and the reslts are shown in Table 1. It is qite clear that WGL has a better performance than GL. The average accracy of WGL is mch higher than that of GL. In addition, WGL is more stable than GL. The flctation of GL in different tests is mch higher. De to the inconsistency, in GL, the vales in the labeled points are not well spread on to the nlabeled points. On many nlabeled points, the fnction φ i is actally close to 1/2. This makes the classification sensitive to the distribtion of the labeled points. As for the comptational time, in or tests, WGL takes abot half the time of GL on average, 15s vs 29s (not inclding the time to constrct the weight), with matlab code in a laptop eqiped with CPU intel i GHz. In WGL, a positive term is added to the diagonal of the coefficient matrix which makes conjgate gradient converge faster. As a remark, in this paper, we do not intend to give a state-of-the-art method in semi-spervised learning. We jst se this example to test the weighted graph Laplacian. 4. Image Inpainting. In this section, we apply the weighted graph Laplacian to the reconstrction of sbsampled images. To apply the weighted graph Laplacian, first, we constrct a point clod from a given image by taking patches. We consider a discrete image f R m n. For any (i,j) {1,2,...,m} {1,2,...,n}, we define a patch p ij (f) as a 2D piece of size s 1 s 2 of the original image f, and the pixel (i,j) is the center of the rectangle of size s 1 s 2. The patch set P(f) is defined as the collection of all patches: P(f) = {p ij (f) : (i,j) {1,2,...,m} {1,2,...,n}} R d, d = s 1 s 2. For a given image f, the patch set P(f) gives a point clod in R d with d = s 1 s 2. We also define a fnction on P(f). At each patch, the vale of is defined to be the intensity of image f at the central pixel of the patch, i.e. (p ij (f)) = f(i,j), where f(i,j) is the intensity of image f at pixel (i,j). Now, we sbsample the image f in the sbsample domain Ω {(i,j) : 1 i m, 1 j n}. The problem is to recover the original image f from the sbsamples f Ω. This problem can be transfered to interpolation of fnction in the patch set P(f) with is given in S P(f), S = {p ij (f) : (i,j) Ω}. Notice that the patch set P(f) is not known, we need to pdate the patch set iteratively from the recovered 7

8 Algorithm 2 Sbsample image restoration Reqire: A sbsample image f Ω. Ensre: A recovered image. Generate initial image 0. while not converge do 1. GeneratepatchsetP( n )fromcrrentimage n andgetcorrespondinglabeled set S n P( n ). 2. Compte the weight fnction w n (x,y) for x,y P( n ). 3. Update the image by compting n+1 on P( n ), with the constraint n+1 (x) = f(x), x S n. 4. n n+1. end while = n. image. Smmarizing this idea, we get an algorithm to reconstrct the sbsampled image which is stated in Algorithm 2. There are different methods to compte n+1 on P( n ) in Algorithm 2. In this paper, we se weighted graph Laplacian and graph Laplacian to compte n+1. In the weighted graph Laplacian, we need to solve the following linear system (w n (x,y)+w n (y,x)) ( n+1 (x) n+1 (y) ) ( n ) ( ) mn + Ω 1 w n (y,x)( n+1 (x) n+1 (y)) = 0, x P( n )\S n, y S n n+1 (x) = f(x), x S n. While in the graph Laplacian, we solve the other linear system, (w n (x,y)+w n (y,x)) ( n+1 (x) n+1 (y) ) = 0, x P( n )\S n, ( n ) n+1 (x) = f(x), x S n. Actally, Algorithm 2 with graph Laplacian is jst a nonlocal method in image processing[1, 5]. In nonlocal methods, people try to minimize energy fnctions sch as with the constraint min m i=1 j=1 n w (i,j) 2, (i,j) = f(i,j), (i,j) Ω. w (i,j) is the nonlocal gradient which is defined as w (i,j) = w(i,j;i,j )((i,j ) (i,j)), 1 i,i m, 1 j,j n. 8

9 w(i,j;i,j ) is the weight from pixel (i,j) to pixel (i,j ), ( d(f(i,j),f(i w(i,j;i,j,j ) )) ) = exp d(f(i,j),f(i,j )) is the patch distance in image f, d(f(i,j),f(i,j )) = h 1 k= h 1 h 2 σ 2 l= h 2 χ(k,l) f(i+k,j +l) f(i +k,j +l) 2 χ is often chosen to be 1 or a Gassian, h 1,h 2 are half sizes of the patch. It is easy to check that the above nonlocal method is the same as solving the graph Laplacian on the patch set. If the weight w is pdated iteratively, we get Algorithm 2 with the graph Laplacian. Next, we will show that this method has inconsistent reslts, and we can se the weighted graph Laplacian to address this isse. In or calclations below, we take the weight w(x, y) as following: ) w(x,y) = exp ( x y 2 σ(x) 2. σ(x) is chosen to be the distance between x and its 20th nearest neighbor, To make the weight matrix sparse, the weight is trncated to the 50 nearest neighbors. The patch size is For each patch, the nearest neighbors are obtained by sing an approximate nearest neighbor (ANN) search algorithm. We se a k-d tree approach as well as an ANN search algorithm to redce the comptational cost. The linear system in weighted graph Laplacian and graph Laplacian is solved by the conjgate gradient method. PSNR defined as following is sed to measre the accracy of the reslts (4.1) PSNR(f,f ) = 20log 10 ( f f /255) where f is the grond trth. First, we rn a simple test to see the performance of weighted graph Laplacian and graph Laplacian. In this test, the patch set is constrcted sing the original image, Figre 4(a). The original image is sbsampled at random, only keeping 1% pixels. Since the patch set is exact, we do not pdate the patch set and only rn WGL and GL once to get the recovered image. The reslts of WGL and GL are shown in Figre 4(c),(d) respectively. Obviosly, the reslt of WGL is mch better. To have a closer look at of the recovery, Figre 5 shows the zoomed in image enclosed by the boxes in Figre 4(a). In Figre 5(d), there are many pixels which are not consistent with their neighbors. Compared with the sbsample image 5(b), it is easy to check that these pixels are actally the retained pixels. The reason is that in graph Laplacian a non-negligible bondary term is dropped [13, 12]. On the contrary, in the reslt of WGL, the inconsistency disappears and the resltant recovery is mch better and smoother as shown in Figre 4(d) and 5(d). At the end of this section, we apply Algorithm 2 to recover the sbsampled image. In this test, we modify the patch to add the local coordinate p ij (f) = [p ij (f),λ 1 i,λ 2 j] 9

10 (a) (b) (c) (d) Fig. 4. Test of image restoration on image of Barbara. (a): original image; (b): 1% sbsample; (c): reslt of GL; (d): reslt of WGL. with λ 1 = 3 (f Ω) m, λ 2 = 3 (f Ω). n This semi-local patch cold accelerate the iteration and give better reconstrction. The nmber of iterations in or comptation is fixed to be 10. The initial image is obtained by filling the missing pixels with random nmbers which satisfy a Gassian distribtion, where µ 0 is the mean of f Ω and σ 0 is the standard deviation of f Ω. The reslts are shown in Figre 6. As we can see, WGL gives mch better reslts than GL both visally and nmerically in PSNR. The reslts are comparable with those in LDMM [9] while WGL is mch faster. For the image of Barbara ( ), WGL needs abot 1 mintes and LDMM needs abot 20 mintes in a laptop eqiped with CPU intel i GHz with matlab code. In WGL and GL, the weight pdate is the most comptationally expensive part in the algorithm by taking more than 90% of the entire comptational time. This part is the same in WGL and GL. So the total time of WGL and GL are almost same, althogh the time of solving the linear system is abot half in WGL, 1.7s vs 3.1s. 10

11 (a) (b) (c) (d) Fig. 5. Zoomed in image in the test of image restoration on image of Barbara. (a): original image; (b): 1% sbsample; (c): reslt of GL; (d): reslt of WGL. 5. Conclsion and Ftre Work. In this paper, we introdce a novel weighted graph Laplacian method. The nmerical reslts show that weighted graph Laplacian provides a reliable and efficient method to find reasonable interpolation on a point clod in high dimensional space. On the other hand, it was fond that with extremely low sample rate, formlation of the harmonic extension may fail [8, 15]. In this case, we are considering sing the -Laplacian to compte the interpolation on point clod, i.e. solving the following optimization problem with the constraint min max x P 1/2 w(x,y)((x) (y)) 2, (x) = g(x), x S. This work will be reported in or sbseqent paper soon. Another interesting problem is the semi-spervised learning stdied in Section 3. In semi-spervised learning, ideally, the fnctions, φ i, shold be either 0 or 1, so 11

12 original image 10% sbsample GL WGL db db db db db db db db Fig. 6. Reslts of sbsample image restoration. they are piecewise constant. In this sense, minimizing the total variation shold give better reslts [5, 14, 16]. Based on the weighted graph Laplacian, we shold also add a weight to correctly enforce the constraints on the labeled points. This idea implies the following weighted nonlocal TV method, min x P\S with the constraint w(x,y)((x) (y)) 2 1/2 + P S w(x,y)((x) (y)) 2 x S (x) = g(x), x S. 12 1/2,

13 This seems to be a better approach than the weighted graph Laplacian in the semispervised learning. We will explore this approach and report its performance in or ftre work. REFERENCES [1] A. Bades, B. Coll, and J.-M. Morel. A review of image denoising algorithms, with a new one. Mltiscale Model. Siml., 4: , [2] C. J. Brges, Y. LeCn, and C. Cortes. Mnist database. [3] F. R. K. Chng. Spectral Graph Theory. American Mathematical Society, [4] Q. D, M. Gnzbrger, R. B. Lehocq, and K. Zho. Analysis and approximation of nonlocal diffsion problems with volme constraints. SIAM Review, 54: , [5] G. Gilboa and S. Osher. Nonlocal operators with applications to image processing. Mltiscale Model. Siml., 7: , [6] Z. Li and Z. Shi. A convergent point integral method for isotropic elliptic eqations on point clod. SIAM: Mltiscale Modeling Simlation, 14: , [7] Z. Li, Z. Shi, and J. Sn. Point integral method for solving poisson-type eqations on manifolds from point clods with convergence garantees. arxiv: [8] B. Nadler, N. Srebro, and X. Zho. Semi-spervised learning with the graph laplacian: The limit of infinite nlabelled data. NIPS, [9] S. Osher, Z. Shi, and W. Zh. Low dimensional manifold model for image processing. Technical Report, CAM report 16-04, UCLA, [10] Z. Shi. Enforce the dirichlet bondary condition by volme constraint in point integral method. arxiv: [11] Z. Shi and J. Sn. Convergence of the point integral method for poisson eqation on point clod. arxiv: [12] Z. Shi and J. Sn. Convergence of the point integral method for the poisson eqation with dirichlet bondary on point clod. arxiv: [13] Z. Shi, J. Sn, and M. Tian. Harmonic extension on point clod. arxiv: [14] K. Yin, X.-C. Tai, and S. Osher. An effective region force for some variational models for learning and clstering. Technical Report, CAM report 16-18, UCLA, [15] X. Zho and M. Belkin. Semi-spervised learning by higher order reglarization. NIPS, [16] W. Zh, V. Chayes, A. Tiard, S. Sanchez, D. Dahlberg, D. Kang, A. Bertozzi, S. Osher, and D. Zosso. Nonlocal total variation with primal dal algorithm and stable simplex clstering in nspervised hyperspectral imagery analysis. Technical Report, CAM report 15-44, UCLA, [17] X. Zh, Z. Ghahramani, and J. D. Lafferty. Semi-spervised learning sing gassian fields and harmonic fnctions. In Machine Learning, Proceedings of the Twentieth International Conference ICML 2003), Agst 21-24, 2003, Washington, DC, USA, pages ,