A Non-Local Low-Rank Approach to Enforce Integrability

Transcription

1 A Non-Local Low-Rank Approach to Enforce Interability Hicham Badri, Hussein Yahia To cite this version: Hicham Badri, Hussein Yahia. A Non-Local Low-Rank Approach to Enforce Interability. IEEE Transactions on Imae Processin, 6. <hal-037> HAL Id: hal Submitted on 8 May 6 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teachin and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseinement et de recherche français ou étraners, des laboratoires publics ou privés.

2 IEEE TRANSACTIONS ON IMAGE PROCESSING A Non-Local Low-Rank Approach to Enforce Interability Hicham Badri and Hussein Yahia Abstract We propose a new approach to enforce interability usin recent advances in non-local methods. Our formulation consists in a sparse radient data-fittin term to handle outliers toether with a radient-domain non-local low-rank prior. This reularization has two main advantaes : ) the low-rank prior ensures similarity between non-local radient patches, which helps recoverin hih-quality clean patches from severe outliers corruption, 2) the low-rank prior efficiently reduces dense noise as it has been shown in recent imae restoration works. We propose an efficient solver for the resultin optimization formulation usin alternate minimization. Experiments show that the new method leads to an important improvement compared to previous optimization methods and is able to efficiently handle both outliers and dense noise mixed toether. Index Terms Interability, non-local methods, low-rank, surface reconstruction. I. INTRODUCTION Reconstruction from radients is an important step in various computer vision and raphics applications such as shapefrom-shadin (SfS) [] and photometric stereo (PS) [2]. PS and SfS methods first compute the surface normals. An estimated radient is then calculated from the normals and used to reconstruct the depth map. However, due to perturbations in the input imaes such as noise, outliers, shadows and other sources, the estimated radient field is subect to hihmanitude corruption (outliers) and dense corruption (noise). This downrades the radient field to a non-interable vector field. As a result, a straihtforward interation approach results in a deformed surface with various artifacts. The problem of interation is not limited only to surface reconstruction from a non-interable field (SfG). For instance, in Adaptive Optics (AO) [3], the wavefront phase reconstruction consists in recoverin the phase from low-resolution corrupted radient measurements. Various low-level radient-domain processin applications directly manipulate radient fields via transformations or mixin various radients [4], [5], [6]. In all these cases, the resultin transformed radient field is no loner interable and direct interation results in various disturbin artifacts. This paper proposes a new optimization formulation for robust interation to handle dense and sparse hih-manitude radient-domain corruptions. Motivated by recent advances in imae restoration [7], [8], we propose to use a sparse radient residual formulation coupled with a non-local reularization. The non-local reularization is substantially different from H. Badri and H. Yahia are with INRIA Bordeaux Sud-Ouest. hicham.badri@inria.fr, hussein.yahia@inria.fr. previous interation works as it manipulates non-local patches of the vector field instead of independent radient points. By iteratively atherin non-local radient patches, stackin them in a matrix and forcin this matrix to be low-rank, our method is able to sinificantly correct imperfections in the vector field. We propose an efficient way to solve the correspondin problem via alternate minimization. To our knowlede, this is the first approach that uses a non-local prior in the context of interability enforcement/surface-fromradients. Experiments on synthetic and real data demonstrate the excellent performance of the proposed method even in extreme mixed sparse/dense corruptions. A. Interability Enforcement II. RELATED WORK The problem of interability enforcement has received important attention in the vision community since early works on SfS and PS applications. The most popular and straihtforward approach consists in formulatin the problem in terms of a quadratic enery, which results in the Poisson equation as a solution [9]. Frankot and Chellappa propose to perform a proection of the non-interable radient field onto the set of Fourier basis. Kovesi [0] performed the proection on shapelets. The work in [] uses loopy belief propaation scheme to deal with dense noise in the vector field. A eneralization of the Poisson interation framework was proposed by Arawal et al. [2], which uses various methods such as M- estimation, α-surface and diffusion. An alebraic approach to deal with sparse corruptions only in the vector field based on the zero-curl constraint was proposed in [3]. Spectral and Tikhonov reularization were proposed in [4] to improve robustness of the least-squares fittin. Sparsity-based methods have been proposed recently to deal with both sparse and dense corruptions. The work in [] proposes to use the l -norm instead of the l 2 -norm in the interability formulation. Similarly, the method in [6] uses the l -norm to model the sparsity of the residual radient and a minimum curl constraint, which improves robustness to hih-manitude sparse corruptions. The method in [7] uses notions from Compressed Sensin to improve surface recovery. The method in [8] proposes to use a sparse radient reularization toether with a sparse radient residual formulation, which has shown to sinificantly improve robustness to both outliers and noise corruptions. Another set of methods known as kernel methods [9] follow a different direction to deal with radient-domain perturbations by usin kernel basis functions for hih-dimensional fittin. Finally, the approach proposed by Xie et al. in [] formulates the surface reconstruction problem usin discrete eometry and performs

3 IEEE TRANSACTIONS ON IMAGE PROCESSING 2 processin on the normals not on the vector field to reconstruct a 3D mesh. Contrary to this approach, we study in this paper the problem of reconstruction from corrupted radient fields in the eneral case. Our solution is thus not limited only to surface-from-radients problems. B. Non-Local Methods Non-local methods have emered as a powerful approach to various imae restoration tasks. The idea consists in atherin similar patches via block-matchin (called non-local patches), stack them in a matrix such as each column is a vectorized version of a patch, and then process this matrix usin a transformation. The standard block-matchin approach widely used in low-level vision consists in clusterin patches that minimize the l 2 -distance in a neihborhood. More sophisticated and faster block-matchin approaches were proposed as well [2]. Contrary to local methods that manipulate each pixel separately, non-local methods take into account the selfsimilarities within an imae, which leads to hih-quality recovery even in the case of hih-level corruptions. The first works on non-local methods that were proposed tackle the problem of imae denoisin. Various non-local transformations have been proposed : weihted averain [22], 3D collaborative filterin [23], sparse codin/dictionary learnin [24] and lowrank estimation [7], [8]. Some of the methods have been extended to imae deconvolution [] and more recently to optical flow estimation [26] and imae editin [27]. In this paper, we propose to use this powerful prior for the first time in the context of interability enforcement. C. Low-Rank Estimation Low-rank estimation has led to many important improvements in various applications as it can efficiently recover missin data and deal with outliers [28], [29], []. The approach consists mainly in reducin the manitude of the sinular values of a matrix via a shrinkae operator, which forces the columns of the matrix to be more similar, and thus et rid of corruptions. Particularly, it has shown to produce interestin results in the context of imae restoration via non-local denoisin [7], [8]. Another work worth mentionin uses a low-rank formulation for Photometric Stereo normals estimation [3]. Note however that this work addresses a completely different problem than robust interation. Moreover, the work that we present in this paper uses low-rank estimation as a non-convex reularization that results in a new mathematical formulation, while the method in [3] uses a robust PCA formulation for matrix separation. In a nutshell, we propose to use a new optimization approach to directly recover the surface/imae from a severely corrupted vector field, motivated by recent advances in nonlocal methods/low-rank estimation. Contrary to previous work, we use a non-local reularization that manipulates patches of the vector field and imposes low-rankness on the correspondin non-local patch matrices. This approach has many advantaes : ) it can efficiently reduce dense corruptions such as noise, 2) it can efficiently deal with hih-manitude corruptions such as outliers, 3) it can recover missin data. The correspondin optimization problem can be efficiently solved via alternate minimization. We evaluate the new technique on synthetic and real data and compare with previous works. Our method shows excellent performance even in challenin situations when the vector field contains a hih amount of sparse and dense corruptions. III. PROBLEM FORMULATION Let S(x, y) be the desired discrete surface to recover. We denote by s its vectorized form (which corresponds to concatenatin each column in one vector). Let v = (v x, v y ) be the iven corrupted radient field in the vectorized form. Typically, the iven vector field v is an estimate of the true radient of the surface that we denote by s = (s x, s y ), which is subect to dense noise and hih-manitude outliers. Due to the present of hih-manitude outliers in v, the residual radient ( s v ) tends to be sparse. This observation was made in previous works [6], [], [8]. Non-convex sparsity via the l p< -norm has shown to produce better results than the l -norm in various situations as it promotes more sparsity [32], [33], [8]. We use in this paper the non-convex l p< -norm to model sparsity of the residual radient as it reflects better the statistics of the residual. Fiure plots the empirical distribution of residual radients comin from a Photometric Stereo experiment. As can be seen, the kurtotic hyper-laplacian distribution with a low p value (in red) reflects better the empirical distribution (in black). This suests that the l p< -norm is a better choice than the l -norm to model residual radients. lo2 probability Empirical Lap (p=.0) H Lap (p=0.) Residual radient Fi. : lo distribution of the residual radient error in a Photometric Stereo experiment. A kurtotic distribution is needed to better model the empirical distribution of the residual. Due to the hih amount of corruptions that may affect the estimated radient field, a reularization is necessary. We propose to use a new reularization via non-local low-rank estimation to recover a ood quality surface even in severe corruption situations. This prior consists in atherin patches

4 IEEE TRANSACTIONS ON IMAGE PROCESSING 3 (a) Ground-Truth (b) Initial solution (c) 70 iterations (d) 00 iterations Fi. 2: Proressive non-local radient patch recovery example. Our method starts with an initial solution calculated via the Poisson equation and iteratively improves the result by forcin low-rankness of block-matched patches. In this experiment, the corrupted radient field contains 0% of outliers and the surface used is the popular Mozart dataset. of the estimated vector field via block-matchin, stack them as columns in a matrix and force this matrix to be low-rank. Reducin the rank of this matrix efficiently attenuates dense corruptions in the estimated vector field, and also helps to correct more outliers or missin data. We use a standard blockmatchin that clusters patches in a neihborhood based on their l 2 -distance similar to what is used in [7], [8]. We start with an initial solution iven by the Poisson equation and iteratively improve recovery. Fiure 2 shows an example of a non-local radient patch cluster and how the proposed method iteratively improves the result. As can be seen, the clustered patches tend to be very similar. Low-rankness is thus very effective to et rid of corruptions. Let X be a matrix. Its Sinular Values Decomposition (SVD) is iven by X = Udia(σ)V T, where σ are the sinular values of X and dia(.) is a diaonal operator. Lowrank estimation consists in forcin σ to be sparse typically by applyin a shrinkae operator. The most popular way to promote sparsity on the sinular values is known as the socalled nuclear norm [34], which comes to use the l -norm on σ ( X = σ ). Considerin the l p< -norm that promotes more sparsity than the l case, we propose to use the l p - nuclear norm that we denote by X,p = σ p p as a prior in our problem. Puttin it all toether, the proposed optimization formulation is iven as follows armin s v p p + λ s { R x D x s,p2 + RyD } y s,p2, = () where λ is a positive reularization term, is the number of clusters of non-local patches, D x, D y and are discrete differential operators and Rx, Ry are binary matrices that select block-matched patches for cluster and p,2. Optimization Problem () is hard to evaluate directly. We use alternate minimization to estimate the latent surface/imae s. This consists in introducin additional variables w s, w x, w y and split the problem into sub-problems that are easier to solve. We use a Half-Quadratic (HQ) solver [] that is widely used in the computer vision community. Applyin this approach to problem () results in the followin armin w s p p + β 2 ( s v ) w s 2 2 s,w s,wx,w y + λ { } = wx,p2 + β 2 R xd x s wx λ { } = wy,p2 + β 2 R yd y s wy 2 2, where β is a new positive reularization parameters. The correspondin subproblems for alternate minimization are iven by considerin one variable a time as follows (p ) : w (k+) s (p 2 ) : w,(k+) x armin w s armin w x (p 3 ) : wy,(k+) armin wy (p 4 ) : s (k+) armin λ = { R xd x s w,(k+) x s w s p p + β 2 ( s(k) v ) w s 2 2 w x,p2 + β 2 R xd x s (k) w x 2 2 w y,p2 + β 2 R yd y s (k) w y 2 2 ( s v ) w (k+) s R yd y s w,(k+) y 2 2 } (2). (3) As problems (p ), (p 2 ) and (p 3 ) are in the proximal form, they can be efficiently solved via pixel-wise shrinkae. Problem (p 4 ) is quadratic and admits a closed-form. Solvin problem (p ) Problem (p ) takes the followin proximal form armin x β x p p + 2 x y 2 2. (4) This problem admits a closed-form solution in the convex case p = via soft-thresholdin. The non-convex case p < admits a first-order approximation [32], [8] (see Appendix) x = shrink lp (y, ( β ) = max 0, y ) β y + ɛ p sin(y). (5) Applyin this result to problem (p ) ives the followin solution ) w (k+) s = w (k+) s,x = shrink lp (D x s (k) v x, β) (6) w (k+) s,y = shrink lp (D y s (k) v y, β

5 IEEE TRANSACTIONS ON IMAGE PROCESSING 4 Note that this first-order proximal approach is an approximation which is as fast as the standard nuclear norm minimization. The method in [36] presents a more advanced optimization approach for more accurate results. However, this approach is slower as it requires solvin a weihted nuclear norm minimization problem for various iterations. Solvin problems (p 2 ) and (p 3 ) Problems (p 2 ) and (p 3 ) take the followin eneral form armin X β X,p + 2 X Y 2 2. (7) Consider the SVD of X = Udia(σ)V T, where σ are the sinular values. The l p - nuclear norm is expressed in terms of the sinular values only X,p = σ p p. As a result, a first-order approximation of (7) can be expressed usin the l p -thresholdin operator (5) as follows X = shrink,p (y, β ) = Udia(shrink l p (σ, β ))V T. (8) Applyin this result to problems (p 2 ) and (p 3 ) ives the followin solutions Solvin problem (p 4 ) wx,(k+) = shrink,p2 (RxD x s (k), β ) wy,(k+) = shrink,p2 (RyD y s (k), β ). (9) Problem (p 4 ) is quadratic, hence it admits a closed-form. However, it is not straihtforward to et the closed-form due to the presence of the non-local matrices Rx and Ry. For this reason, we need a reformulation of this form R Ds w 2 2 Ds w 2 2. (0) = Once this formulation established, the solution s (k+) can be efficiently calculated via Fourier transform by considerin periodic boundary conditions. Consider the eneral problem armin A x b 2 x 2, () which admits the followin closed-form A T A x = A T b. (2) Applyin this result to problem () results in the followin solution R T R Dx = R T w. (3) As the non-local roups are not overlappin, we have RT R = I. The equivalence is thus iven as follows R Ds w 2 2 Ds R T w 2 2. (4) = The R T w simply consists in placin the patches of w in their correspondin position. As each pixel can have estimates, division by permits to areate all the patches. Applyin this result to problem (p 4 ) results in the followin subproblem (p 4 ) : s (k+) armin ( s v ) w s (k+) 2 2 s +λ D x s = RT x +λ D y s = RT y w,(k+) x 2 2 w,(k+) y 2 2. () Considerin periodic boundary conditions, the differential operator can be replaced with convolutions. The problem in this case can be efficiently solved via Fourier Transform F as follows { } s (k+) = F F(div(d x,d y)) d x = (v x + w s,x (k+) ) + λ d y = (v y + w s,y (k+) ) + λ (+λ) lap ˆ = RT x = RT y w,(k+) x w,(k+) y, (6) where div is the discrete diverence operator and lap ˆ is the optical transfer function (OTF) of the discrete Laplacian filter. IV. ANALYSIS l vs. l p< As explained before, we use non-convex sparsity via the l p< penalty instead of the popular l -norm. The reasons why we do so is that the l p< penalty reflects better the empirical residual radient distribution that is hihly kurtotic. For the low-rank reularization, the l p2< penalty would respect better the nature of the sinular values. As the sinular values are sorted in deceasin order and that the first sinular values contain more enery than the rest, the shrinkae operator should act accordinly. As one can observe, the shrinkae operator iven by the l -norm consists in shrinkin all the values with the same value shrink l (σ, β ) = max ( 0, σ β ). (7) As a result, all the sinular values would be shrunk the same way and this would severely affect the first sinular values. In contrast, the shrinkae operator iven by the l p< penalty performs a weihted shrinkae of this form shrink lp (σ, ( β ) = max 0, σ ) β (σ + ɛ)p, (8) where the weihts (σ + ɛ) p uide the shrinkae operator. Note that, for p 0 and ɛ 0, we have lim σ + (σ + ɛ)p = 0. (9) As a result, the amount of penalization decreases as the manitude of the elements increases, which is exactly the nature of sinular values. To demonstrate this in our case, we run surface reconstruction experiments on three popular datasets with various values of p =, 0.5, 0.. The results are presented in Table I. As can be seen, the non-convex The sin is bein omitted as the sinular values are positive.

6 IEEE TRANSACTIONS ON IMAGE PROCESSING 5 5% Outliers % Outliers % Outliers 5% Outliers + Noise % Outliers + Noise % Outliers + Noise Ramp Peaks p =, p 2 = db.67 db db.59 db.27 db 3.88 db p = 0.5, p 2 = db 65.0 db db db 38. db.4 db p = 0., p 2 = db 07.0 db db db.7db 37.4 db Vase p =, p 2 = db 43.4 db 3.78 db db 33.2 db.09 db p = 0.5, p 2 = db 62. db db db.4 db db p = 0., p 2 = db db db. db db db Mozart p =, p 2 = 62.6 db 42. db 34.5 db.6 db.24 db 33.0 db p = 0.5, p 2 = db 44.9 db.4 db db 4.03 db db p = 0., p 2 = db db db db 4. db.50 db TABLE I: Mean PSNR comparison between the convex case (p, p 2 = ) and the non-convex case (p, p 2 < ). As can be seen, non-convexity improves the reconstruction results as it promotes more sparsity. 5% Outliers % Outliers % Outliers 5% Outliers + Noise % Outliers + Noise % Outliers + Noise Ramp Peaks Ground-Truth Block-Matchin 3.72 db 09. db db db 42.9 db 4.99 db Iterative Block-Matchin 3.53 db 07.0 db db db.7db 37.4 db Vase Ground-Truth Block-Matchin 44.4 db db db 4. db 38.7 db.88 db Iterative Block-Matchin 42.7 db db db. db db db Mozart Ground-Truth Block-Matchin 7.70 db 53. db db db 4.99 db 36.3 db Iterative Block-Matchin 7. db db db db 4. db.50 db TABLE II: Mean PSNR comparison between round-truth block-matchin calculated on clean round-truth radients and iterative block-matchin in our method. As can be seen, a simple l 2 -distance clusterin block-matchin produces ood quality results even in the presence of both outliers and noise. (a) % outliers (b) % outliers (c) % outliers + 7% noise (d) % outliers + 7% noise Fi. 3: Proposed reconstruction method in the convex case (p = p 2 =.0) (top) and non-convex case (p = p 2 = 0.) (bottom). The non-convex approach leads to better results in both outliers only case (a-b) and the mixed corruption case (c-d).

7 IEEE TRANSACTIONS ON IMAGE PROCESSING 6 PNSR (db) L2 Diffusion M estimation L 3xlp Proposed Outliers corruption (%) 5 0 Outliers corruption (%) medium noise level 5 0 Outliers corruption (%) hih noise level 00 L2 Diffusion M estimation L 3xlp Proposed Outliers corruption (%) Outliers corruption (%) medium noise level Outliers corruption (%) hih noise level L2 Diffusion M estimation L 3xlp Proposed PNSR (db) Outliers corruption (%) 5 0 Outliers corruption (%) medium noise level 5 0 Outliers corruption (%) hih noise level Fi. 4: Performance of the proposed method under various levels of outliers corruption when there is no noise, medium amount of noise and hih amount of noise. The proposed method outperforms previous optimization methods in all the cases (visual results are reported in 6). penalty with p < outperforms the l -norm, for both the residual and the low-rank term. For visual comparison, we show reconstruction results of the Ramp-Peaks dataset in the convex and non-convex cases under hih outliers corruption (%, % of the radient points are corrupted with hihmanitude), and hih mixed corruption (%, % of the radient points are corrupted with hih-manitude + hih Gaussian noise corruption with standard deviation 7% of the maximum intensity). As can be seen in Fiure 3, the nonconvex approach leads to better results. It is able to better smooth the surface in both the outliers case only (a-b) and the mixed noise situation (c-d). Throuhout the paper, we consider only the non-convex case. Block-Matchin Due to the hih-amount of corruption that may occur in the estimated radient field, we analyze the robustness of blockmatchin. We use a standard approach (same code as [7]) that consists in clusterin patches in a neihborhood that minimize the l 2 -distance. In all the experiments, each non-local cluster contains patches of size 6 6. This may not be the optimal settin as one should increase the patch size with respect to the amount of noise, but this setup already works very well in our experiments. We start with an initial solution calculated via the Poisson equation, calculate the radients of the solution and iteratively improve the result. Block-matchin is performed on each radient component separately. As this operation is quite slow, we do not perform it at each iteration. We rather run block-matchin each iterations to offer a ood balance between processin time and quality of reconstruction. We compare with a round-truth matchin calculated on clean round-truth radients. The results are presented in Table II. As can be seen, it turns out that a simple l 2 -distance matchin works very well even in severe corruption situations. Parameters The parameter β typically takes a small value, it is fixed to β = 0 4 for all the experiments. β is increased at each iteration, we do this by multiplyin its value by. at each iteration β (k+).β (k). The remainin parameters are : the reularization parameter λ, the patch size s s for the non-local reularization and the number of similar patches per cluster b and the choice of the sparsity p,2. Ase mentioned before, we fix parameters to s = 6, b = for all experiments. Parameter λ is proportional to the amount of corruption as in any reularization (low λ for low corruption and hiher for more corruption). We fix the penalty parameters to around p = p 2 = 0., which seems to work well in practice as mentioned before.

8 IEEE TRANSACTIONS ON IMAGE PROCESSING 7 Complexity The method consists in shrinkae operations that are fast to calculate as they correspond to pixel-wise operations, a Fourier-based reconstruction that can be performed efficiently and a non-local sparsity part. The block-matchin operation in our case is a straihtforward MATLAB code. This operation can be accelerated usin fast dedicated methods such as PatchMatch [2] or parallel processin. For the non-local sparsity part, as processin each roup is independent (because the roups are non-overlappin), the low-rank processin step can be performed in parallel as well. Patch manipulation (extraction and reconstruction) are easy operations that are efficiently implemented in C (MEX). On a laptop with a i7-2670qm CPU usin 4 cores, the block-matchin reduces to 0. second and the low-rank estimation subproblem to 0. second per iteration for a heavily corrupted radient field. V. EXPERIMENTS We run various experiments to demonstrate the effectiveness of the proposed method on synthetic and real data. Quantitative Experiments The first part of the experiments consists in quantitative experiments on standard synthetic surfaces. Given a roundtruth radient field for each surface, we enerate sparse outliers of density τ% and manitude 5 times the maximum radient intensity value. Dense Gaussian noise is then added in three cases : no noise, medium noise and hih noise 2. We then evaluate how the proposed method is able to recover the true surface for various corruption situations. We compare with previous optimization-based methods l 2 [9], M-estimation [2], l [6], [], 3 l p [8]. The results are iven in Fiure 4. As can be seen, the proposed method outperforms the other methods for all outliers corruption levels (from medium (5%) to hih (%)) for the noise free case, as well as in the presence of medium and hih levels of dense noise. To compare the visual quality, we present the results in Fiure 6 in the case of mixed outliers/noise (hih corruption % with a hih amount of noise). For this comparison, we consider other methods such as [0], [4], [7]. As can be seen, our method preserves better important structures and correctly smooths the surface. While the sparse reularization method [8] leads to an important improvement compared to the other methods, it tends to oversmooth the surface. This is because the local prior prefers a piecewise constant solution. In contrast, the proposed nonlocal solution promotes non-local smoothness of the patches, which does not produce over-smoothin. It is worth nothin also that the method in [8] needs various parameters to set in the case of mixed outliers/noise. The proposed technique requires settin only the reularization parameter λ, it is thus easier to use. 2 Because the surfaces do not have the same radient dynamic rane, the medium noise level is determined visually so it matches the three surfaces. The hih noise level is simply twice the medium noise level. Real Photometric Stereo We apply the proposed method to a real Photometric Stereo (PS) problem 3. In this case, the corruptions in the estimated radient field are caused by imperfections in the input imaes of the PS method. To make the experiment more challenin, we use a simple least-squares solution to estimate the normals. This leads to much more corruption in the estimated radient field. We add two types of corruption to the input imaes : additive Gaussian dense noise with standard deviation equals to 5% of the maximum intensity, and random sparse outliers with density % and manitude three times the maximum intensity. Note that this is a very challenin reconstruction case as the input imaes are heavily corrupted. The results are iven in Fiure 5. As can be seen, the proposed method is able to reconstruct a hih-quality smooth surface even thouh the input imaes for the PS method are severely corrupted. VI. CONCLUSION We present a new approach to surface-from-radients usin a non-local low-rank reularization. The proposed method iteratively athers non-local patches of the corrupted vector field and applies low-rank estimation to reduce perturbations such as outliers and dense noise. This reularization is used alon with a sparse data-fittin term that imposes non-convex sparsity on the residual radient and ensures better outliers handlin. We propose an efficient alternate minimization solution to the problem usin a Half-Quadratic approach. Experiments conducted on synthetic and real data show that the proposed method is able to recover a hih-quality surface from severely corrupted vector fields in mixed dense/sparse corruption situations. REFERENCES [] B. K. P. Horn, Heiht and radient from shadin, Int. J. Comput. Vision, vol. 5, no., pp , Sep [2] R. J. Woodham, Shape from shadin, B. K. P. Horn and M. J. Brooks, Eds. Cambride, MA, USA: MIT Press, 989, ch. Photometric method for determinin surface orientation from multiple imaes, pp [3] F. Roddier, Adaptive Optics in Astronomy. Cambride University Press, 999. [4] P. Pérez, M. Gannet, and A. Blake, Poisson imae editin, in ACM SIGGRAPH 03 Papers, ser. SIGGRAPH 03. New York, NY, USA: ACM, 03, pp [5] R. Fattal, D. Lischinski, and M. Werman, Gradient domain hih dynamic rane compression, ACM Trans. Graph., vol. 2, no. 3, pp , Jul. 02. [6] A. Levin, A. Zomet, S. Pele, and Y. Weiss, Seamless imae stitchin in the radient domain, in In Proceedins of ECCV, 06. [7] W. Don, G. Shi, and X. Li, Nonlocal imae restoration with bilateral variance estimation: A low-rank approach, IEEE Trans. Imae Processin, vol. 22, no. 2, 3. [8] S. Gu, L. Zhan, W. Zuo, and X. Fen, Weihted nuclear norm minimization with application to imae denoisin, in In Proceedins of CVPR, 4. [9] T. Simchony, R. Chellappa, and M. Shao, Direct analytical methods for solvin poisson equations in computer vision problems, IEEE Trans. Pattern Anal. Mach. Intell., vol. 2, no. 5, pp , May 990. [0] P. Kovesi, Shapelets correlated with surface normals produce surfaces, in In Proceedins of CVPR, The dataset is publicly available on Neil Alldrin s homepae : vision.ucsd. edu/ nalldrin/research/

9 IEEE TRANSACTIONS ON IMAGE PROCESSING 8 (a) l2 -minimization [9], [37] (b) Shapelets [0] (c) l -minimization [6], [] (d) Sparse Gradient [8] (e) Non-Local (Proposed) (a) l2 -minimization [9], [37] (b) Shapelets [0] (c) l -minimization [6], [] (d) Sparse Gradient [8] (e) Non-Local (Proposed) Fi. 5: Photometric stereo on real corrupted imaes. The proposed method is able to recover a hih-quality surface even when the input PS imaes are heavily corrupted. [] N. Petrovic, I. Cohen, B. J. Frey, R. Koetter, and T. S. Huan, Enforcin interability for surface reconstruction alorithms usin belief propaation in raphical models. in In Proceedins of CVPR, 0. [2] A. Arawal, R. Raskar, and R. Chellappa, What is the rane of surface reconstructions from a radient field? in In Proceedins of ECCV, 06. [3] A. Arawal, R. Chellappa, and R. Raskar, An alebraic approach to surface reconstruction from radient fields, in In Proceedins of ICCV, 05. [4] M. Harker and P. O Leary, Least squares surface reconstruction from radients: Direct alebraic methods with spectral, tikhonov, and constrained reularization. in In Proceedins of CVPR,. [] Z. Du, A. Robles-Kelly, and F. Lu, Robust surface reconstruction from radient field usin the l norm, ser. DICTA 07, 07, pp [6] D. Reddy, A. K. Arawal, and R. Chellappa, Enforcin interability by error correction usin l -minimization, in In Proceedins of CVPR, 09. [7] M. Rostami, O. V. Michailovich, and Z. Wan, Surface reconstruction in radient-field domain usin compressed sensin, IEEE Trans. Imae Processin, vol. 24, no. 5, pp ,. [8] H. Badri, H. Yahia, and D. Aboutadine, Robust surface reconstruction via triple sparsity, in In Proceedins of CVPR, 4. [9] T.-P. Wu, J. Sun, C.-K. Tan, and H.-Y. Shum, Interactive normal reconstruction from a sinle imae, in ACM SIGGRAPH Asia 08 Papers, ser. SIGGRAPH Asia 08, 08, pp. 9: 9:9. [] W. Xie, Y. Zhan, C. C. L. Whan, and R. C.-K. Chun, Surface-fromradients: An approach based on discrete eometry processin, in In Proceedins of CVPR, 4. [2] C. Barnes, E. Shechtman, D. B. Goldman, and A. Finkelstein, The eneralized patchmatch correspondence alorithm, in In Proceedins of ECCV, 0. [22] A. Buades, B. Coll, and J.-M. Morel, A non-local alorithm for imae denoisin, in In Proceedins of CVPR, 05. [23] K. Dabov, A. Foi, V. Katkovnik, and K. Eiazarian, Imae denoisin by sparse 3d transform-domain collaborative filterin, IEEE Trans. Imae

10 IEEE TRANSACTIONS ON IMAGE PROCESSING 9 (a) Ground-Truth (b) l 2-minimization [9], [37] (c) l -minimization [6], [] (d) Compressed Sensin [7] (e) Shapelets [0] (f) Tikhonov [4] () Sparse Gradient [8] (h) Non-Local (Proposed) (a) Ground-Truth (b) l 2-minimization [9], [37] (c) l -minimization [6], [] (d) Compressed Sensin [7] (e) Shapelets [0] (f) Tikhonov [4] () Sparse Gradient [8] (h) Non-Local (Proposed) (a) Ground-Truth (b) l 2-minimization [9], [37] (c) l -minimization [6], [] (d) Compressed Sensin [7] (e) Shapelets [0] (f) Tikhonov [4] () Sparse Gradient [8] (h) Non-Local (Proposed) Fi. 6: Reconstruction quality with various methods in the case of hih outliers corruption (%) mixed with a hih level noise. The proposed method leads to a better reconstruction quality even in extreme corruption situations.

11 IEEE TRANSACTIONS ON IMAGE PROCESSING 0 Processin, vol. 6, no. 8, 07. [24] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, Non-local sparse models for imae restoration, in proceedins of ICCV, 09. [] A. Danielyan, V. Katkovnik, and K. Eiazarian, Bm3d frames and variational imae deblurrin, IEEE Trans. Imae Processin, vol. 2, no. 4, 2. [26] X. H. Weishen Don, Guanmin Shi and Y. Ma, Nonlocal sparse and low-rank reularization for optical flow estimation, IEEE Trans. Imae Processin, vol. 23, no. 0, 4. [27] H. Talebi and P. Milanfar, Nonlocal imae editin, IEEE Trans. Imae Processin, vol. 23, no. 0, 4. [28] Z. Zhan, A. Ganesh, X. Lian, and Y. Ma, Tilt: Transform invariant low-rank textures, International Journal of Computer Vision, vol. 99, no., 2. [29] D. Zhan, Y. Hu, J. Ye, and X. Li, Matrix completion by truncated nuclear norm reularization, in In Proceedins of CVPR, 2. [] H. Ji, C. Liu, Z. Shen, and Y. Xu, Robust video denoisin usin low rank matrix completion, in In Proceedins of CVPR, 0. [3] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wan, and Y. Ma, Robust photometric stereo via low-rank matrix completion and recovery, in In Proceedins of Asian Conference on Computer Vision (ACCV). [32] R. Chartrand, Fast alorithms for nonconvex compressive sensin: Mri reconstruction from very few data, in Int. Symp. Biomedical Imain, 09. [33] D. Krishnan and R. Ferus, Fast imae deconvolution usin hyperlaplacian priors, in In Proceedins of the Neural Information Processin Systems Conference (NIPS), 09. [34] J. Wriht, A. Ganesh, S. Rao, and Y. Ma, Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization, in In Proceedins of NIPS, 09. [] D. Geman and G. Reynolds, Constrained restoration and the recovery of discontinuities, IEEE Trans. Pattern Anal. Mach. Intell., vol. 4, no. 3, pp , 992. [36] C. Lu, J. Tan, S. Yan, and Z. Lin, Generalized nonconvex nonsmooth low-rank minimization, in In Proceedins of the IEEE Conference on Computer Vision and Pattern Reconition (CVPR), 4. [37] R. T. Frankot and R. Chellappa, A method for enforcin interability in shape from shadin alorithms, IEEE Trans. Pattern Anal. Mach. Intell., vol. 0, no. 4, pp. 439, Jul [38] N. Parikh and S. Boyd, Proximal alorithms, Foundations and Trends in Optimization, 3. [39] H. Badri, H. Yahia, and D. Aboutadine, Fast ede-aware processin via first order proximal approximation, IEEE Trans. Visualization and Computer Graphics, vol. 2, no. 6,. l p -Shrinkae Solution APPENDIX The proof of the approximation of the proximal operator associated to the l p -norm is iven here. First, consider the followin proximal operator [38] prox λf (x) = armin λf(y) + y 2 x y 2 2. () This problem admits an exact solution via an inverse function prox λf (x) = (I + λ f) (x). (2) Unfortunately, this inverse function cannot be evaluated directly for the l p -norm. However, a solution can be approximated via a first-order Taylor expansion [39] : prox λf (x) x λ f(x). (22) As the solution is a shrinkae operation, the solution does not depend on the sin prox λf (x) = max {0, x λ f( x )} sin(x). (23) The shrinkae operator is also pixel-wise, we thus have prox λf (x i ) = max {0, x i λ f( x i )} sin(x i ). (24) For the l p -norm, by considerin f(x) = p x p p, we have f( x i ) x i + ɛ p, where ɛ is a small value to prevent division by zero. The solution is thus iven as follows prox λf (x i ) = max { 0, x i λ x i + ɛ p } sin(x i ). () This solution is similar to the one proposed by Chartrand [32] calculated usin Leendre-Fenchel transform. Hicham Badri received the M.S. deree in computer science and telecommunications from Mohammed V-Adal University and the Ph.D. derees in imae processin from both University of Bordeaux (INRIA) and Mohammed V-Adal University. His research interests include imae processin, sparse methods and machine learnin. Hussein Yahia is the head of the INRIA research team GEOSTAT (Geometry and statistics in acquisition data). Dr. H. Yahia received the Doctorat ode 3eme cycle from University Paris (Orsay) and the HDR (Habilitation Dirier des Recherches) form University Paris 3. He specializes in non-linear sinal processin and the analysis of complex sinals and systems usin advanced nonlinear physics. Dr. H. Yahia has also made substantial contributions to computer raphics and imae processin. Dr. H. Yahia is developin a stron collaboration with the LEGOS Laboratory in Toulouse (UMR CNRS 55 66), ICM-CSIC in Barcelona, and IIT Roorkee in India (Associated team OPTIC with Prof. Dharmendra Sinh). He is also involved in many contracts with the French spatial aency CNES and the European Spatial Aency (ESA). Dr. H. Yahia is the author or co-author of about 80 publications in international peerreviewed ournals and conferences includin top-ranked conferences such as ACM SIGGRAPH, CVPR and ECCV. He is a member of the editorial board of ournals in sinal processin or complex systems and has supervised more than 0 PhD students.