An Empirical Bayesian interpretation and generalization of NL-means

Transcription

1 Computer Science Tecnical Report TR , October 2010 Courant Institute of Matematical Sciences, New York University ttp://cs.nyu.edu/web/researc/tecreports/reports.tml An Empirical Bayesian interpretation and generalization of NL-means Martin Rapan Courant Inst. of Matematical Sciences New York University Eero P. Simoncelli Howard Huges Medical Institute, Center for Neural Science, and Courant Inst. of Matematical Sciences New York University Abstract A number of recent algoritms in signal and image processing are based on te empirical distribution of localized patces. Here, we develop a nonparametric empirical Bayesian estimator for recovering an image corrupted by additive Gaussian noise, based on fitting te density over image patces wit a local exponential model. Te resulting solution is in te form of an adaptively weigted average of te observed patc wit te mean of a set of similar patces, and tus bot justifies and generalizes te recently proposed nonlocalmeans (NL-means) metod for image denoising. Unlike NL-means, our estimator includes a dependency on te size of te patc similarity neigborood, and we sow tat tis neigborood size can be cosen in suc a way tat te estimator converges to te optimal Bayes least squares estimator as te amount of data grows. We demonstrate te increase in performance of our metod compared to NL-means on a set of simulated examples. 1 Introduction Local patces witin a potograpic image are often igly structured. Moreover, it is quite common to find many patces witin te same image tat ave similar or even identical content. Tis fact as been exploited in developing new forms of signal representation [10], and in applications suc as denoising [5], clustering [3], inpainting [4], and texture syntesis [15, 7, 8]. Most of tese metods are based on assumptions (sometimes implicit) about te probability distribution over te intensity vectors associated wit te patces. A recent example is a patced-based inference metodology, known as nonlocal means (NL-means), in wic eac pixel in a noisy image is replaced by a weigted average over oter pixels in te image tat occur in similar image patces [1, 2]. Te weigting is determined by te similarity of te patces, as measured wit a Gaussian kernel. Tis metod is intuitively appealing, and gives good (altoug less tan state-of-te-art) denoising performance on natural images. More recent variants ave been developed tat acieve state-ofte-art [5]. But as we sow ere, te NL-means as te drawback tat te estimate depends crucially on te Gaussian kernel widt, and even in te limit of infinite data, tere is no way to coose tis widt so as to guarantee convergence to te optimal answer. In tis paper, we develop an alternative patc-based denoising algoritm as an empirical Bayes () estimator, and sow tat it generalizes and improves on te NL-means solution. We use an exponential density model for local patces of a noisy image, and derive a maximum likeliood metod for estimating te parameters from data. We ten combine tis wit a nonparametric form of te optimal least-squares estimator for additive Gaussian noise [13]. Tis simple, but often overlooked form expresses te usual Bayes least squares estimator directly in terms of te density of noisy observations, witout reference to te (clean) prior density. Te resulting denoising metod resembles NL-means, in tat te denoised value is a function of te

2 weigted average over oter pixels in te same image tat come from similar patces. But te function is nonlinear, and is scaled according to te size of te neigborood over wic intensities are averaged. As a result, we prove tat te estimator converges to te Bayes least squares optimum as te amount of data increases, a feature not present in te standard NL-means approac. 2 NL-means estimation Suppose Y is a noise-corrupted measurement of X, were eiter or bot variables can be finite-dimensional vectors. Given tis measurement, te estimate of X tat minimizes te expected squared error is te Bayes Least Squared (BLS) estimator, wic is te expected value of X conditioned on Y, E{X Y }. In particular, for an observation y 0, te estimate is ˆx BLS (y 0 )=E{X Y = y 0 }. In te NL-means approac, a vector observation Y = y 0 is denoised by averaging it togeter wit patces tat are similar, i.e., tat are in some neigborood, N, of size about te vector y 0. As te number of data points available increases to infinity, te NL-means estimator will converge to ˆx NL (y 0 ) E{Y Y N }. (1) Notice tat tis is a conditional expectation of Y, not X (as required for te BLS estimate). Te original presentation of NL-means [1] justifies tis by considering te case of estimating te center pixel, X c of te patc based on te surrounding pixels. In tis case, and assuming zero-mean noise, we can write E{Y c Y c = y c } = E{E{Y c X c,y c = y c } Y c = y c } = E{X c Y c = y c }, (2) were Y c indicates te patc witout te center pixel. Tus, by setting N to be a neigborood wic does not depend on te central coefficient of te vector, Eq. (1) will tend to te optimal estimator of te central coefficient as a function of te surrounding coefficients, Eq. (2). In practice, NL-means is implemented to denoise te central coefficient in te patc, but includes te central coefficient in determining similarity of patces. Under tese conditions, te solution does not, in general, converge to te correct BLS estimator, regardless of te size of te neigborood. To see tis, consider te simplified situation were te neigborood is defined as N = {y : y y 0 }. (3) Clearly, ˆx NL (y 0 ) will depend on te value of. For a fixed value of, we end up wit a conditional expectation of Y instead of X, as mentioned above. And, in te limit as 0, te estimate will converge to te identity, i.e. ˆx NL (y 0 ) y 0. Intuitively, tis is because te mean of data witin a neigborood must also lie witin tat neigborood. Tus, te best we can ope for is to find some binwidt,, tat works reasonably well asymptotically, but tis coice will be igly dependent on te (unknown) signal prior. Tis is illustrated for a simple scalar example in Fig. 1. Te prior ere is made up of delta functions wic are well separated, relative to te standard deviation of te noise. Wit te rigt coice of binwidt (about twice te widt of te noise distribution), for any amount of data, NL-means provides a good approximation to te optimal denoiser. But note tat te NL-means beavior depends critically on te coice of binwidt: Fig. 1 also sows a result for a smaller binwidt, wic is clearly suboptimal. In general, tere is no simple way to coose an optimal binwidt, and to do so in a way tat makes te NL-means estimator converge to te BLS estimator. 3 Patc-based empirical Bayes least squares estimator We wis to develop a patc-based estimator tat makes explicit te dependence on neigborood size,, and for wic we can demonstrate proper convergence in te limit as 0. For te particular case of additive Gaussian noise, Miyasawa [13] developed an nonparametric empirical form of te Bayes least squares estimate tat is written directly in terms of te measurement (noisy) density P Y (Y ), wit no explicit reference to te prior: E{X Y = y 0 } = y 0 +Λ y ln(p Y (y 0 )), (4) were Λ is te covariance matrix of te noise (assumed known). Note tat tis expression is not an approximation: te equality is exact, assuming te noise is additive and Gaussian (wit zero mean and covariance Λ), and assuming te measurement density is known.

3 P Y (y) y E{X Y=y} 0 y E{Y Y y } 0 y Fig. 1: Observed density and optimal estimator for prior made up of tree isolated delta functions. Top panel: observed density (arrows represent prior). Middle panel: BLS estimator (assuming true prior is known). Bottom: NL-means estimates calculated for two different binsizes (tick solid and dased lines). Note also tat te NL-means estimator approaces te identity function as te binsize goes to zero. 3.1 Learning te estimator function from data Te formulation of Eq. (4) offers a means of computing te BLS estimator from noisy samples, if one can construct an approximation of te noisy distribution, P Y (Y ). Simple istograms will not suffice for tis approximation, because Eq.(4) requires us to compute te logaritmic derivative of te estimated density. Instead, we use an exponential model for te local density centered on an observation Y k = y 0 witin some neigborood, N of size, centered on y 0 : P Y Y N (y) = ea(y y0) Z(a,) 1 N (5) were 1 N denotes te indicator function of N, and Z(a,) normalizes te density over te interval: Z(a,)= e a (y y0) dy. (6) N Tis type of local exponential approximation is similar to tat used in [11] for density estimation. Te sape of te neigborood, N, is unspecified, but te linear size sould scale wit. For example, it could be a sperical ball of radius around y 0 or te ypercube centered on y 0 of lengt 2 on eac side. In any case, te logaritmic gradient of tis density evaluated at y 0 is simply a. We can ten fit te parameter a by maximizing te likeliood of te conditional density over te neigborood: â = arg max ln(p Y Y N (Y n )) (7) a {n:y n N } Substituting Eq. (5), taking te gradient of te sum, and setting it equal to zero gives a ln(z(â,)) = Ȳ y 0 (8) were Ȳ is te average of te data tat lie in N. Note tat substituting Eq. (6)inforZ(â, ) and adding y 0 to bot sides gives E{Y Y N } = Ȳ. Tus, te ML solution for â matces te local mean of te fitted density to te local empirical mean. Equation (8) provides an implicit definition of â wic involves te function Z(a,). Noting tat Z scales according to Z(a,)= d Z(a, 1), we can solve explicitly for â: â = 1 F 1 (Ȳ y 0 ) (9)

4 were F (a) = a ln(z(a, 1)), (10) does not depend on. Finally, replacing y ln(p Y (y)) in Eq. (4) byâ gives te empirical Bayes estimator: ˆx (y 0 )=y 0 +Λ 1 F 1 (Ȳ y 0 ). (11) Note tat it may be difficult to calculate F 1 for a general neigborood. We ll return to tis issue in section Convergence wit proper coice of binwidt Te empirical Bayes estimator of Eq. (11) depends an te coice of binwidt,, wic not only appears explicitly in te expression, but also implicitly affects te coice of wic data vectors are included in calculating te local mean, Ȳ. Tis binwidt can vary for eac data point, but in tis paper, we will consider only te simpler case of a constant binwidt for all data. In order for our estimator to converge to te BLS estimator, we need â, asdefined in Eq. (9), to converge to te logaritmic derivative of te true density, and tis means te binwidt must srink to zero as te total amount of data grows. Te rate of srinkage sould be set in suc a way tat te amount of data witin eac neigborood goes to infinity. In tis subsection, we calculate te rate at wic te binwidts sould srink, in order to ensure convergence. First, we note tat te factor of 1 in Eq. (9) implies tat, for â to converge at all, it is necessary tat (Ȳ ) F 1 y0 0. (12) Since it can be sown tat 0 is te only zero of F, we need only sow tat Ȳ y 0 0. (13) Looking again at Eq. (9), and considering a Taylor approximation of F about zero, we see tat for for â to converge, it is furter necessary tat Ȳ y0 ave some finite limit 2 Ȳ y 0 2 v, (14) for some value v. In tis case, a bit of calculation sows tat te Taylor approximation of Eq. (9) may be written â V 1 C1 1 v, were C 1 /V 1 is te Jacobian matrix of F evaluated at zero, wit V 1 denoting te volume of te neigborood N 1 (N, wit size =1), and C 1 = (ỹ y 0 )(ỹ y 0 ) T dỹ. N 1 Finally, combining te Taylor approximation wit te definition of â tells us tat our estimator converges to te optimal BLS estimator if and only if we can find a coice of binwidt so tat 1 2 V 1C1 1 (Ȳ y 0) ln(p Y (y 0 )). (15) We now discuss ow te binwidt sould srink as te amount of data increases in order to ensure tis condition is met. First, we decompose te asymptotic mean squared error into variance and squared bias terms: E { (Ȳ y0 2 C 1 {Ȳ y0 = Var 2 ) 2 } ln(p Y (y 0 )) V 1 } + ( E {Ȳ y0 2 } C ) 2 1 ln(p Y (y 0 )), (16) V 1

5 We expect (and will sow) tat te variance term sould decrease as grows (since more data will fall into eac bin), wereas te squared bias sould increase as te neigborood grows. Tis is illustrated in Fig. 2. Te trick is to find a way to srink te binwidts as te amount of data increases, so tat te bias goes to zero, but slowly enoug to ensure tat te variance still goes to zero as well. For large amounts of data, we expect to be small, and so we may use small approximations for te bias and variance. Since te local average, Ȳ is calculated using only data wic fall in N, we may write te expected value as E{Ȳ } = E{Y Y N N } = ỹp Y (ỹ)dỹ (17) N P Y (ỹ)dỹ To get an idea of ow tis beaves asymptotically, we can take te local Taylor approximation of te density and substitute into Eq. (17). Since te neigboroods are centered on y 0, te integrals of terms wit odd degree will vanis, leaving E{Ȳ } y 0 C P Y (y 0 )+O( d+4 ) P Y (y 0 )V + O( d+2 (18) ) were V denotes te volume of te neigborood and C = (ỹ y 0 )(ỹ y 0 ) T dỹ N For symmetric neigboroods, C will be a diagonal matrix. If te N as te same sape for all just rescaled by te factor, a simple cange of variables sows tat V = d V 1 and C = d+2 C 1 so tat E{Ȳ y 0 2 } C 1 ln(p Y (y 0 )) + O( 2 ) (19) V 1 so tat te bias term in Eq. (16) will srink to zero as long as te binwidt srinks to zero. Now consider te variance term in Eq. (16). We need to coose binwidts tat srink to zero slowly enoug to allow {Ȳ } y0 Var 0. 2 First note tat E{ Y y 0 2 Y N } = O( 2 ) (20) Combining tis wit Eq. (19) gives Var(Ȳ y ) = n 4 Var(Y y 0 Y N ) = O( 1 n 2 )+O( 1 n ) (21) were n is te number of data points wic fall in te bin. Tus, we see tat as long as n 2 (22) te variance of our approximation will also tend to zero. If tere are N total data points, we can approximate te number falling in N as n P Y (y 0 )NV 1 d (23) and, inserting tis into Eq. (22) gives For example, we migt use N d+2, 0 (24) N 1 m+d+2,m>0 (25)

6 variance increasing N bias Fig. 2: Bias-variance tradeoff, as a function of binwidt,. Solid blue line indicates squared bias. Dased lines indicate variance, for different values of N (number of data points). Solid black line (wit black points) sows te variance resulting wen is cosen as a function of N, as indicated in Eq. (25) wit m =2. Under tese conditions, te variance falls to zero as srinks. Te variance resulting from tis coice (for m =2) is illustrated in Fig. 2, and can be seen to approac zero as decreases. Wit tis coice of binwidt, it now follows tat te estimator in Eq. (11) converges asymptotically to te optimal BLS solution. Note tat a consequence of our asymptotic analysis is tat, wen te estimator in Eq. (11) converges, we can use te Taylor series approximation of te estimator, so tat for te same coice of binwidts ˆx (y 0 ) y 0 +ΛV 1 C1 1 (Ȳ y 0 2 ) (26) will also converge asymptotically to te BLS estimator. In te case of sperical neigboroods we get an estimate of te logaritmic derivative very similar to tat derived for te mean-sift metod of clustering[3]. Note, toug, tat te derivation used for tat approac uses te derivative of te Epanecnikov kernel, wic would only seem to work for sperical neigboroods. Our metod, on te oter and, works for general neigboroods. Note also tat mean-sift is meant to be used as an iterative gradient ascent, moving data towards peaks in te density. For our purposes, owever, we ave sown tat wen te logaritmic gradient is scaled by te noise variance, we acieve BLS denoising in one step. 4 Empirical Bayes versus NL means We ave now introduced bot te Empirical Bayes metod for approximating te BLS estimator, Eq. (11), and te NL-means estimator Eq. (1). For pedagogical reasons, we will compare and contrast te two using low dimensional examples wic allow for bot easier visualization and easier computation. As discussed in te last section, we expect our estimator to converge to te ideal solution as te amount of data goes to infinity. NL-means, on te oter and does not ave tis asymptotic property in general, since if te binwidts were to srink te NL-means metod would do no denoising. Terefore, we expect te optimal optimal binwidt to asymptote to some finite, nonzero optimal value tat depends eavily on bot te prior and te amount of noise. For tis reason, tis metod may be able to give us improved performance for low to moderate amounts of data, but will in general be asymptotically biased. To illustrate te convergence beavior as a function of te amount of data, we ave simulated a scalar estimation problem. Signal values are drawn from a generalized Gaussian density, p(x) exp( x β ), wit exponent β =0.5. For eac N, we draw N samples from tis distribution, and add Gaussian wite noise to eac. Te noise was cosen so tat te noisy Signal to Noise Ratio (SNR) (10 log of squared error divided by signal variance) was 7.8 db. We ten apply our estimator, wit binwidt cosen according to Eq. (25). As a reference, we also sow te performance of te ideal BLS estimator (wic knows te true prior). We also sow te result obtained using NL-means (Eq. (1)), wit binwidt cosen to optimize performance (i.e., by comparing te denoising result to te clean signal). For eac N, tis simulation is performed 1000 times, so as to obtain estimates of te average performance, as well as te variability. Te SNR performance is plotted, as a function of N, in Fig. 3. Note tat te average performance of NL-means is reasonable for small N

7 SNR improvement(db) NL-means 1.3 BLS # samples (a) SNR improvement(db) NL-means 1.6 BLS # samples (b) Fig. 3: Performance of Empirical Bayes metod compared wit NL-means, and te optimal BLS estimator, for (a) a generalized Gaussian prior; (b) te sum of two generalized Gaussians. Lines (solid, dased, and dotted) indicate mean performance of eac estimator over 1000 simulations. Gray regions (for and NLmeans) indicate plus/minus one standard deviation. See text. True Observation NL means True Observation NL means True Observation NL means Fig. 4: Beavior of metod compared wit NL-means, for two-dimensional data, wit tree different neigborood coices. Prior is a delta at te origin (indicated by cross). Only a subsample of te data points are sown, for visual clarity. Boxes indicate neigboroods. Note tat in te tird example, te neigborood extends beyond te display. See text. (altoug te variability is quite ig), but saturates as N grows, and tus does not converge to te optimal performance of te BLS estimator. By comparison, our estimator starts out wit a lower SNR at small N, but asymptotically converges (in bot mean and variance) to te optimal solution. Figure 3 sows anoter example, for a signal drawn from a mixture of two generalized Gaussian variables, wit mean values ±5. Here, we see a more substantial asymptotic bias for te NL-means estimator. Figure 4 sows a comparison of our metod to NL-means in two dimensions. Te signal samples in tis example are drawn from a delta function at te origin. We see tat te NL-means solution is igly dependent on te size and sape of te neigborood. It is possible to acieve good results in tis particular case of a delta function prior by using a very large neigborood (3rd panel). But for te smaller neigboroods, te NL-means estimate is restricted to lie witin te neigborood, and is tus eavily biased, wile te metod is not. 5 Discussion We ve derived a patc-based metod for signal denoising by assuming a local exponential density model for te noisy data. We ve proven tat, despite te fact tat we do not include any assumption about te

8 overall form of te signal prior, our estimator converges to te optimal (Bayes) least squares estimator as te amount of data increases, assuming te binsize is reduced appropriately. Te estimator is a nonlinear function of te average of similar patces, and tus may be viewed as a generalization of te NL-means and mean-sift algoritms. Te main drawback of te metod (as wit NL-means) is te ig computational cost associated wit gatering te data patces tat are witin an -neigborood of te patc being denoised. We believe tat some of te metods tat ave been developed for accelerating NL-means [18, 14, 6] (or te oter patc-based applications mentioned in te introduction) may be utilized in our estimator as well, and we are currently working on suc an implementation so as to examine empirical performance of our metod on image denoising. We envision a number of ways in wic our metod may be extended. First, te binsizes need not be uniform for all patces, but can be adapted according to te local sample density. Specifically, for eac patc tat is to be denoised, a rule suc as tat of Eq. (25) could be used to adjust te binsize to as to acieve te best bias/variance tradeoff (i.e., so as to best approximate te BLS estimator). Tis extension necessarily increases te computational cost (since te binsize must now be optimized for eac patc) but may lead to substantial increases in performance. Second, te NL-means metod computes a weigted average of patces, were te weigting is a function of bot te patc similarity, and te patc proximity witin te signal/image (for example, bilateral filtering may be viewed as an NL-means metod, wit one-pixel patces). We are exploring means by wic our metod can be generalized to include bot of tese forms of weigting. Tird, estimators ave been developed for observation processes oter tan additive Gaussian noise [17, 12, 9, 16], and eac of tese could be developed into a patc-based estimator (altoug most of tese would not be written in terms of local means). And finally, we believe our local exponential density estimate will prove useful as a substrate for generalizing oter patc-based empirical metods tat ave become popular in computer vision, grapics, and image processing. References [1] A. Buades, B. Coll, and J. M. Morel. A review of image denoising algoritms, wit a new one. Multiscale Modeling and Simulation, 4(2): , July , 2 [2] A. Buades, B. Coll, and J. M. Morel. Nonlocal image and movie denoising. Intl Journal of Computer Vision, 76(2): , [3] D. Comaniciu and P. Meer. Mean sift: A robust approac toward feature space analysis. IEEE Pat. Anal. Mac. Intell., 24: , , 6 [4] A. Criminisi, P. Perez, and K. Toyama. Region filling and object removal by exemplar-based inpainting. IEEE Trand Image Proc, 13: , [5] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Trans. Image Proc., 16(8), August [6] A. Dauwe, B. Goossens, H. Luong, and W. Pilips. A fast non-local image denoising algoritm. In Proc SPIE Electronic Imaging, volume 6812, [7] J. S. De Bonet. Multiresolution sampling procedure for analysis and syntesis of texture images. In Computer Grapics. ACM SIGGRAPH, [8] A. A. Efros and T. K. Leung. Texture syntesis by non-parameteric sampling. In Proc. Int l Conference on Computer Vision, Corfu, [9] Y. C. Eldar. Generalized SURE for exponential families: Applications to regularization. IEEE Trans. on Signal Processing, 57(2): , Feb [10] N. Jojic, B.Frey, and A. Kannan. Epitomic analysis of appearance and sape. In ICCV, [11] C. R. Loader. Local likeliood density estimation. Annals of Statistics, 24(4): , [12] J. S. Maritz and T. Lwin. Empirical Bayes Metods. Capman & Hall, 2nd edition, [13] K. Miyasawa. An empirical Bayes estimator of te mean of a normal population. Bull. Inst. Internat. Statist., 38: , , 2 [14] J. Orcarda, M. Ebraimi, and A. Wong. Efcient nonlocal-means denoising using te SVD. In Proc IEEE Intl Conf on Image Processing, [15] K. Popat and R. W. Picard. Cluster-based probability model and its application to image and texture processing. IEEE Trans Im Proc, 6(2): , [16] M. Rapan and E. P. Simoncelli. Least squares estimation witout priors or supervision. Neural Computation, In Press. 8 [17] H. Robbins. An empirical Bayes approac to statistics. Proc. Tird Berkley Symposium on Matematcal Statistics, 1: , [18] J. Wang, Y. Guo, Y. Ying, Y. Liu, and Q. Peng. Fast non-local algoritm for image denoising. In Proc IEEE Intl Conf on Image Processing,