SPARSE SIGNAL RECOVERY USING A BERNOULLI GENERALIZED GAUSSIAN PRIOR

Transcription

1 3rd Euroean Signal Processing Conference EUSIPCO SPARSE SIGNAL RECOVERY USING A BERNOULLI GENERALIZED GAUSSIAN PRIOR Lotfi Chaari 1, Jean-Yves Tourneret 1 and Caroline Chaux 1 University of Toulouse, IRIT - INP-ENSEEIHT, France firstname.lastname@enseeiht.fr Aix-Marseille University, CNRS, Centrale Marseille, IM, UMR 7373, France caroline.chaux@univ-amu.fr ABSTRACT Bayesian sarse signal recovery has been widely investigated during the last decade due to its ability to automatically estimate regularization arameters. Prior based on mixtures of Bernoulli and continuous distributions have recently been used in a number of recent works to model the target signals, often leading to comlicated osteriors. Inference is therefore usually erformed using Markov chain Monte Carlo algorithms. In this aer, a Bernoulli-generalized Gaussian distribution is used in a sarse Bayesian regularization framework to romote a two-level flexible sarsity. Since the resulting conditional osterior has a non-differentiable energy function, the inference is conducted using the recently roosed non-smooth Hamiltonian Monte Carlo algorithm. Promising results obtained with synthetic data show the efficiency of the roosed regularization scheme. Index Terms Sarse Bayesian regularization, MCMC, ns-hmc, restoration 1. INTRODUCTION Sarse signal and image restoration has been of increasing interest during the last decades. This issue arises esecially in large data alications where regularization is essential to recover a stable solution of the related inverse roblem. These alications include remote sensing and medical image reconstruction. Since observation systems are mostly ill-osed, regularization is usually needed to imrove the quality of the reconstructed signals. The underlying idea of regularization is to constrain the search sace through some rior information added to the model in order to stabilize the inverse roblem. This rior information usually involves additional arameters that have to be tuned. Fixing these arameters can be a roblem since they can have a dee imact on the quality of the target solution. However, they can be estimated using Bayesian methods that have already shown their efficiency in a number of recent works 1, ]. As regards regularization, Bayesian techniques have also been widely investigated in a number of recent works using hierarchical Bayesian models, This work was suorted by the CNRS Imag In roject under grant OP- TIMISME e.g., for medical 3] and hyersectral imaging 4]. In the recent literature, riors constructed from mixtures of Bernoulli and continuous distributions have gained a significant interest due to their ability to aroach the l 0 enalization 3, 5]. These riors are generally combined with the likelihood to derive the osterior distribution of the arameters of interest. The resulting osterior is then used to derive estimators for the model arameters such as the maximum a osteriori MAP estimator. Since the target osterior is difficult to handle in most cases, Markov chain Monte Carlo MCMC samling schemes can be used to draw samles according to the osterior. The Metroolis-Hastings MH algorithm 6] is one of the most used samling techniques, even if it requires to design efficient roosal distributions esecially for highdimensional signals or images. This task is not always easy to erform, which has led to the emergence of the Hamiltonian Monte Carlo HMC technique 7]. In this aer, we roose a sarse Bayesian regularization scheme involving a Bernoulli-generalized Gaussian BGG rior. This rior has the advantage to romote a two-level structured sarsity. The first level is due to the Bernoulli random variable, while the second one is due to the l -like energy function of the GG distribution. For the same reasons aforementioned, samling techniques are required to build the inference and derive the MAP estimator. However, the target distribution involves a non-differentiable energy function, which makes the use of the HMC technique imossible. We therefore resort to the recently roosed non-smooth HMC technique ns-hmc which handles such kind of osterior distributions 8] see also 9] for similar techniques. This technique resents the advantage of fast convergence seed and reduced correlation between the generated samles. The rest of this aer is organized as follows. The addressed roblem is formulated in Section. The roosed hierarchical Bayesian model is detailed in Section 3 before introducing the inference scheme in Section 4. Exerimental results are resented in Section 5. Finally, some conclusions and ersectives are drawn in Section 6.. PROBLEM FORMULATION Let x R M be the target signal measured by y R N through a linear distortion oerator K. Accounting for a os /15/$ IEEE 1741

2 3rd Euroean Signal Processing Conference EUSIPCO sible additive noise n, the observation model writes y = Kx + n. 1 When the above inverse roblem is ill-osed, its direct inversion yields distorted solutions. We roose here a framework for sarse Bayesian regularization involving suitable riors for estimating high-dimensional signals x from the observed vector y. Without loss of generality, we focus in this aer on linear observation roblems with an additive Gaussian noise of covariance matrix σ ni N, where I N is the identity matrix. 3. HIERARCHICAL BAYESIAN MODEL By adoting a robabilistic aroach, y and x in 1 are assumed to be realizations of random vectors Y and X. In this context, our goal is to characterize the robability distribution of X Y, by considering a arametric robabilistic model and by estimating the associated hyerarameters Likelihood Assuming that the observation noise n is additive and Gaussian with variance σ n, the likelihood can be exressed as N/ 1 fy x, σn = πσn ex where. denotes the Euclidean norm. 3.. Priors y Kx In our model, the vector of unknown arameters is denoted by θ = {x, σ n}. As regards the noise variance, the only available knowledge is the ositivity of this arameter. Therefore, and as motivated in 10], we use a Jeffrey s rior defined as fσn 1 σn 1 R +σn 3 where 1 R + is the indicator function on R +, i.e., 1 R +ξ = 1 if ξ R + and 0 otherwise. For the signal x, we define a rior denoted as fx Φ, where Φ is a hyerarameter vector. Assuming that the signal comonents x i are a riori indeendent, the resulting rior distribution for x writes fx Φ = fx i Φ. 4 Promoting the sarsity of the target signal, one can use a Bernoulli-Gaussian BG rior for every x i i = 1,..., M. To better cature the zero signal coefficients and well regularize the non-zero art, we use here a Bernoulli-Generalized Gaussian rior BGG for x i which is defined as fx i Φ = 1 ωδx i + ωggx i, 5 with Φ = {ω,, } and GGx i, = Γ1/ ex x i where > 0 and > 0 are the scale and shae arameters, and Γ. denotes the gamma function. In 5, δ. is the Dirac delta function and ω is a weight belonging to 0, 1]. The adoted BGG model romotes a two-level structured sarsity. The first level is guaranteed due to the Bernoulli model and the Dirac delta function. The second level is ensured by the GG distribution, esecially when 1. This distribution has been widely used in sarse signal recovery 11 13]. When varies between 0 and, the GG distribution gives more flexibility to reresent signals with different sarsity levels. BGG regularization therefore aroaches the l 0 + l one. Using a BGG model for every x i, and assuming indeendence between the different riors, the rior in 4 reduces to fx Φ = 1 ωδx i + ω Γ1/ ex x i ] Hyerarameter riors Searable non-informative riors are used for each variable of the hyerarameter vector Φ = {ω,, }. A uniform distribution on the simlex 0, 1] may be used for ω, i.e., ω U 0,1]. As regards R +, a suitable rior is the conjugate inverse-gamma IG distribution denoted as IG α, β f α, β = βα Γα α 1 ex β R + 8 where α and β are fixed hyerarameters in our exeriments these hyerarameters were emirically set to α = β = For the shae arameter, a uniform rior on the simlex 0, ] is considered. Setting U 0,] guarantees a high level of sarsity for the target solution while using a non-informative rior at the same time. 4. INFERENCE SCHEME Adoting a MAP strategy, the joint osterior distribution of {θ, Φ} can be exressed as fθ, Φ y fy θfθ ΦfΦ 9 fy x, σ nfx ω,, fσ nfωff α, β. Based on the hierarchical Bayesian model described in Section 3, the joint osterior can be written N/ 1 fθ, Φ y πσn ex y Kx 1 ωδx i + ω Γ1/ ex x i ] 1 σn 1 R +σnu 0,1] ω βα Γα α 1 ex β U 0,]

3 3rd Euroean Signal Processing Conference EUSIPCO Unfortunately, no closed-form exression for the Bayesian estimators associated with 10 can be obtained. Thus, we roose to design a Gibbs samler GS that generates samles asymtotically distributed according to 10. The GS iteratively generates samles distributed according to the conditional distributions associated with the target distribution. More recisely, the GS iteratively samles according to fσn y, x, fω x, f x, ω,, f x,, α, β and fx y, Φ as detailed below Gibbs samler Following the strategy detailed above, the main stes of the roosed samling algorithm are summarized in Algorithm 1. The adoted samling strategy generates samles Algorithm 1: Gibbs samler for Sarse Bayesian regularization. - Initialize with some θ 0 and Φ 0 and set r = 0; for r = 1... S do ➀ Samle σ nr according to its osterior fσ n x, y; ➁ Samle ω according to its osterior fω x; ➂ Samle according to its osterior f x; ➃ Samle according to its osterior f x, α, β; ➄ Samle x r according to its osterior fx y, ω,, σ n, ; ➅ Set r r + 1 ; end {x r } r=1,...,s that will be used to estimate the target signal/image, where S is the total number of generated samles. Using the samled chain {x r } r=1,...,s, x can therefore be aroximated by calculating the MAP or the MMSE estimate not over R M, but only over a art of the samled values which theoretically form a reresentative samle of the sace in which x is living. The retained art must exclude, from the samled chain, vectors generated during the burn-in eriod. Results below are given using the MAP estimator since it hels retreiving more recise estimates for the zero coefficients with our model. In addition to x, and using the same strategy, the roosed algorithm will also allow the estimation of σ n, and ω based on the samled chains {σ nr }r=1,...,s, { r } r=1,...,s and {ω r } r=1,...,s, resectively. The conditional distributions used in the GS are detailed below. 4.. Conditional distributions of fθ, Φ y Samling according to fσn y, x From the osterior in 10, straightforward calculations lead to the following conditional distribution for the noise variance σn x, y IG σn N/, y Kx / 11 which is easy to samle Samling according to fω x Starting from 10, we can show that the osterior of ω is the following beta distribution ω B1 + x 0, 1 + M x 0 1 which is also easy to samle Samling according to f x, ω, The conditional distribution associated with the hyerarameter can be written as f x 1 ωδx i + ω Γ1/ ex x i ] U 0,]. 13 An MH ste with a ositive truncated Gaussian roosal can be used to samle according to Samling according to f x,, α, β The conditional distribution of is ω f x, α, β 1 ωδx i + Γ1/ ex x i ] βα Γα α 1 ex β. 14 We roose to use a standard MH move with a ositive truncated Gaussian roosal to samle according to f x, α, β Samling according to fx y, ω,, σn, Because of the sohisticated rior used for x, samling according to fx y, ω,, σn, is not straightforward. However, we can derive the conditional distribution of each element x i given all the other arameters. Precisely, we decomose x onto the orthonormal basis B = {e 1,..., e M } such that x = x i + x i e i. We denote by x i the vector x whose i th element is set to 0 denoted as v i = y Kx i, and k i = Ke i. Straightforward calculations lead to fx i y, x i, ω,, 1 ω ex v i ω Γ1/ ex x i v i k i x i σ n δx i + 1 ω ex v i δx i + x i vt i v i + ki Tk ix i vt i k ix i ω Γ1/ ex 1 ω i δx i + σ n ω i Z i, ex U ix i ]

4 3rd Euroean Signal Processing Conference EUSIPCO where U i x i = x i + kt i k ix i vt i k ix i 16 u i and ω i = u i + 1 ω, u i = ωz i, Γ1/. 17 In 15 and 17, Z i, is a normalizing constant such that Dx i, = Z i, ex U i x i ] defines a robability density function. Samling according to the conditional distribution in 15 can be erformed in two stes 1- Generate z i according to a Bernoulli distribution, i.e., set z i = 0 with robability 1 ω i, and z i = 1 with robability ω i. { x i = 0 if z i = 0 - set x i Dx i, if z i = 1. For efficiency reasons, we resort to ns-hmc to samle from Dx i,. Indeed, ns-hmc has recently been roosed 8] to make feasible the use Hamiltonian dynamics for samling from non-differentiable log-concave distributions such as Dx i,. The resulting schemes are therefore more efficient than the standard MH algorithm both in terms of convergence seed and decorrelation roerties 8]. As detailed in 8], to samle according to Dx i,, we need to calculate the roximity oerator 14] of Ux i in 16. If 1 so that U remains convex, following 15, Lemma.5i], the roximity oerator of U can be obtained as follows s R, rox U s = rox ϕ/κ+1 s u/κ + 1] 18 where κ = k i /σ n, u = v T i k i/σ n and ϕ = /. The roximity oerator for the function ϕ is given by x R, rox ϕ x = signxρ 19 where ρ is the unique minimizer of ρ + ρ 1. Note that rox ϕ reduces to the soft thresholding oerator for = 1, i.e. rox ϕ x = signx max { x 1 }, 0. 0 Other forms for different values of can be found in 15]. 5. EXPERIMENTAL VALIDATION This section validates the roosed Bayesian sarse regularization model both for 1D and D signal recovery roblems D sarse signal recovery In this exeriment, a 1D synthetic sarse signal x of size 100 is recovered from its distorted version y observed according to 1, where the observation oerator K is the second order difference oerator 1 and the additive noise n is Gaussian with diagonal covariance matrix Σ = σni M, with σn = 1. In 1 One can also cosider other linear oerators such as uniform blur addition to the reference signal, Fig. 1 shows the recovered signal obtained with the roosed algorithm ns-hmc. For the sake of comarison, a Bernoulli-Gaussian model has been used in a Bayesian aroach similar to the one in 5]. This model aroaches the variational l 0 + l regularization recently investigated in 16] Reference ns-hmc Fig. 1. Original and restored signals using the roosed model ns-hmc and the BG regularization. Fig. 1 clearly shows that the roosed model gives very close results to the reference, esecially where the original signal is exactly zero. From a quantitative oint of view, results are evaluated in terms of signal to noise ratio SNR. SNR values are given in Table 1 and confirm the good erformance of the roosed method. To further evaluate the robustness of the roosed method, sarsity cardinalities are also given in Table 1 through the seudo-norm x 0. The reorted values indicate that the rox-mc method allows the suort of the reference to be recovered very accurately. Note that the shae arameter has been fixed to 1 in the above simulations. Table 1. SNR values and sarsity measures for the reference 1D signal, ns-hmc and BG. SNR db x 0 Reference - 5 Observation ns-hmc BG Regarding the comutational cost, using a Matlab imlementation on a 64-bit 3.GHz Xeon architecture, the ns-hmc results have been obtained in 17 seconds, while the BG algorithm required only 4 seconds. The same burn-in eriod of 300 iterations has been considered for both algorithms. After the burn-in, 300 more iterations are collected to calculate the MAP estimators S = D sarse image recovery In this exeriment, a D sarse image of size 6 6 is recovered from its distorted version y observed according to the model 1, with the same linear oerator as in the revious ex- BG 1744

5 3rd Euroean Signal Processing Conference EUSIPCO Table. SNR values and sarsity measures for the reference image, ns-hmc and BG. SNR db x 0 Reference Observation ns-hmc BG eriment and σ n = 10. Under these assumtions, the original and distorted images obtained with SNR=-8.68 db are dislayed in Fig.. This figure also shows restored images using ns-hmc and the BG regularization. From a visual viewoint, both methods rovide similar erformance excet for some non-zero ixels where the dynamic is better recovered with the roosed model. This is better exlained through the quantitative evaluation in Table which rovides the SNR and suort cardinalities of the recovered signals. Ground truth ns-hmc Observed Fig.. Ground truth, observed and restored images using the roosed method ns-hmc and the BG regularization. Indeed, our model and the BG regularization recover the same number of non-zero coefficients, while the SNR is higher using our aroach. Regarding comutational burden, and keeing the same burn-in eriod and hysical architecture, the roosed algorithm takes a reasonable time with 115 seconds while the BG regularization required 48 seconds. 6. CONCLUSION This aer roosed a new method for Bayesian sarse regularization using a Bernoulli generalized Gaussian rior romoting a two-level structured sarsity. The roosed method is based on a hierarchical Bayesian model and the recent ns- HMC technique is used for samling the osterior of this model and to build estimators for its unknown arameters and BG hyerarameters. Promising results obtained with signal and image restoration exeriments showed the good erformance of the roosed technique. Future work will be focused on the alication of this technique to large-scale real data for medical image reconstruction. 7. REFERENCES 1] M. Vega, J. Mateos, R. Molina, and A. K. Katsaggelos, Bayesian arameter estimation in image reconstruction from subsamled blurred observations, in IEEE Int. Conf. on Image Process. ICIP, Brussels, Belgium, Se , ] L. Chaari, J.-C. Pesquet, J.-Y. Tourneret, Ph. Ciuciu, and A. Benazza- Benyahia, A hierarchical Bayesian model for frame reresentation, IEEE Trans. on Signal Process., vol. 18, no. 11, , Nov ] N. Dobigeon, A. O. Hero, and J.-Y. Tourneret, Hierarchical Bayesian sarse image reconstruction with alication to MRFM, IEEE Trans. Image Process., vol. 18, no. 9, , Set ] Y. Altmann, N. Dobigeon, S. McLaughlin, and J.-Y. Tourneret, Nonlinear sectral unmixing of hyersectral images using Gaussian rocesses, IEEE Trans. Signal Process., vol. 61, no. 10, , May ] L. Chaari, J.-Y. Tourneret, and H. Batatia, Sarse Bayesian regularization using Bernoulli-Lalacian riors, in Proc. Euro. Signal Process. Conf. EUSIPCO, Marrakech, Morocco, Setember, , ] W. K. Hastings, Monte Carlo samling methods using markov chains and their alications, Biometrika, vol. 57, , ] R. M. Neal, MCMC using Hamiltonian dynamics, in Handbook of Markov Chain Monte Carlo, G. Jones X. L. Meng S. Brooks, A. Gelman, Ed., chater 5. Chaman and Hall/CRC Press, ] L. Chaari, J.-Y. tourneret, C. Chaux, and H. Batatia, A Hamiltonian Monte Carlo method for non-smooth energy samling, 015, htt://arxiv.org/abs/ ] M. Girolami and B. Calderhead, Riemann manifold Langevin and Hamiltonian Monte Carlo methods, J. R. Statist. Soc. B, vol. 73, , ] C. Robert and G. Castella, Monte Carlo statistical methods, Sringer, New York, ] M. N. Do and M. Vetterli, Wavelet-based texture retrieval using generalized Gaussian density and Kullback-Leibler distance, IEEE Trans.on Image Process., vol. 11, no., , Feb ] M. W. Seeger and H. Nickisch, Comressed sensing and Bayesian exerimental design, in Int. Conf. on Machine Learn., Helsinki, Finland, July , ] L. Chaari, P. Ciuciu, S. Mériaux, and J.-Ch. Pesquet, Satio-temoral wavelet regularization for arallel MRI reconstruction: alication to functional MRI, Magn. Reson. Mater. in Phys., Bio. Med., vol. 7, no. 6, , ] J.-J. Moreau, Proximité et dualité dans un esace hilbertien, Bulletin de la Société Mathématique de France, vol. 93, , ] C. Chaux, P. Combettes, J.-C. Pesquet, and V.R Wajs, A variational formulation for frame-based inverse roblems, Inv. Prob., vol. 3, no. 4, , Aug ] E. Chouzenoux, J.-C. Pesquet, H. Talbot, and A. Jezierska, A memory gradient algorithm for l l 0 regularization with alication to image restoration, in IEEE Int. Conf. on Image Process. ICIP, Brussels, Belgium, Se ,