Expectation-Maximization Bernoulli-Gaussian Approximate Message Passing

Transcription

1 Expectation-Maximization Bernoulli-Gaussian Approximate Message Passing Jeremy Vila an Philip Schniter Dept. of ECE, The Ohio State University, Columbus, OH Abstract The approximate message passing AMP algorithm originally propose by Donoho, Maleki, an Montanari yiels a computationally attractive solution to the usual l -regularize least-squares problem face in compresse sensing, whose solution is known to be robust to the signal istribution. When the signal is rawn i.i. from a marginal istribution that is not least-favorable, better performance can be attaine using a Bayesian variation of AMP. The latter, however, assumes that the istribution is perfectly known. In this paper, we navigate the space between these two extremes by moeling the signal as i.i. Bernoulli-Gaussian BG with unknown prior sparsity, mean, an variance, an the noise as zero-mean Gaussian with unknown variance, an we simultaneously reconstruct the signal while learning the prior signal an noise parameters. To accomplish this task, we embe the BG-AMP algorithm within an expectationmaximization EM framework. umerical experiments confirm the excellent performance of our propose EM-BG-AMP on a range of signal types. 2 I. ITRODUCTIO We consier the problem of recovering a signal x R from noisy linear measurements y Ax + w R M in the unersample regime where M <. Roughly speaking, when x is sufficiently sparse or compressible an when the matrix A is sufficiently well-conitione, accurate signal recovery is possible with polynomial-complexity algorithms. One of the best-known approaches to this problem is known as Lasso [], which minimizes the convex criterion ˆx lasso argmin ˆx y Aˆx 2 2 +λ lasso ˆx, with λ lasso a tuning parameter. When A is constructe from i.i. Gaussian entries, the so-calle phase transition curve PTC gives a sharp characterization of Lasso performance for K-sparse x in the large system limit, i.e., as K,M, with fixe unersampling ratio an sparsity ratio K/M [2]. The Lasso PTC is illustrate in Figs. -3. For noiseless observations, the PTC partitions the -versus-k/m plane into two regions: one where Lasso reconstructs the signal perfectly with high probability, an one where it oes not. For noisy observations, the same curve inicates whether the noise sensitivity i.e., the ratio of estimation-error power to measurement-noise power uner the worst-case signal istribution of Lasso remains boune [3]. One remarkable feature of the noiseless Lasso PTC is that it is invariant to signal istribution. In other wors, if we aopt a This work has been supporte in part by SF-I/UCRC grant IIP , by SF grant CCF-08368, an by DARPA/OR grant Portions of this work were presente in a poster at the Duke Workshop on Sensing an Analysis of High-Dimensional Data, July 20. probabilistic viewpoint where the elements of x are rawn i.i. from the marginal pf p X x λfx+ λδx, for Dirac elta δx, active-coefficient pf fx, an λ K/, then the Lasso PTC is not affecte by f. This PTC invariance implies that Lasso is robust, but that it cannot benefit from the restriction of x to an easier signal class. For example, if the coefficients in x are known to be non-negative, then there exists a polynomial-complexity algorithm whose PTC is better than that of Lasso [2]. Although, in some applications, robustness to worst-case signals may be the ominant concern, in many other applications the goal is to maximize average-case performance over an anticipate signal class. When the signal x is rawn i.i. from an arbitrary known marginal istribution p X an the noise w is i.i. Gaussian with known variance, there exist very-low-complexity iterative Bayesian algorithms to generate approximately maximum a posteriori MAP an minimum mean-square error MMSE signal estimates, notably the Bayesian version of Donoho, Maleki, an Montanari s approximate message passing AMP algorithm [4]. AMP is formulate from a loopy-belief-propagation perspective, leveraging central-limit-theorem approximations that hol in the large system limit for suitably ense A, an amits a rigorous analysis in the large-system limit [5]. Meanwhile, AMP s complexity is remarkably low, ominate by one application of A an A T per iteration which is especially cheap if A is an FFT or other fast operation, with typically < 50 iterations to convergence. More recently, a generalize AMP GAMP algorithm [6] was propose that relaxes the requirements on the noise istribution an on the sensing matrixa. See Table I for a summary. Given that it is rare to know the signal an noise istributions perfectly, we take the approach of assuming signal an noise istributions that are known up to some statistical parameters, an then learning those unknown parameters while simultaneously recovering the signal. Examples of this empirical Bayesian approach inclue several algorithms base on Tipping s relevance vector machine [7] [9]. Although the average-case performance of those algorithms is often quite goo epening on the signal class, of course, their complexities are generally much larger than that of GAMP. In this paper, we propose a GAMP-base empirical- Bayesian algorithm. In particular, we treat the signal as Bernoulli-Gaussian BG signal with unknown sparsity, mean, an variance, an the noise as Gaussian with unknown variance, an then we then learn these statistical parameters using

2 an expectation-maximization EM approach [0] that calls BG-GAMP once per EM-iteration. II. BEROULLI-GAUSSIA GAMP A core component of our propose metho is the Bernoulli- Gaussian BG GAMP algorithm, which we now review. For BG-GAMP, the signal x [x,...,x ] T is assume to be i.i. BG, i.e., to have marginal pf p X x;λ,θ,φ λδx+λx;θ,φ, 2 where δ enotes the Dirac elta, λ the sparsity rate, θ the active-coefficient mean, an φ the active-coefficient variance. The noise w is assume to be inepenent of x an i.i. zeromean Gaussian with variance ψ: p W w;ψ w;0,ψ 3 In our approach, the parameters q [λ,θ,φ,ψ] that efine these prior istributions are treate as eterministic unknowns, an learne through the EM algorithm, as etaile in Section III. Although above an in the sequel we assume realvalue Gaussians, all expressions can be converte to the circular-complex-gaussian case by replacing all with C an removing all 2 s. GAMP can hanle an arbitrary probabilistic relationship p Y Z y m z m between the observe output y m an the noiseless output z m a T mx, where a T m is the m th row of A. Our aitive Gaussian noise assumption implies p Y Z y z y; z, ψ. To complete our escription, we nee only to specify g in, g in, g out, an g out in Table I. Using straightforwar manipulations, our p Y Z yiels [6] g out y,ẑ,µ z ;q y ẑ µ z +ψ 4 g outy,ẑ,µ z ;q µ z +ψ, 5 an our BG signal prior 2 yiels g in ˆr,µ r ;q πˆr,µ r ;qγˆr,µ r ;q 6 µ r g in ˆr,µr ;q πˆr,µ r ;q νˆr,µ r ;q+ γˆr,µ r ;q 2 where πˆr,µ r ;q πˆr,µ r ;q 2 γˆr,µ r ;q 2, 7 + λ λ γˆr,µ r ;q ˆr/µr +θ/φ νˆr,µ r ;q ˆr;θ,φ+µ r ˆr;0,µ r 8 9 /µ r +/φ /µ r +/φ. 0 Table I implies that BG-GAMP s marginal posteriors are p y;q C n p X ;q ;ˆr n,µ r n C n λδxn +λ ;θ,φ ;ˆr n,µ r n 2 efinitions: p Z Y z y;ẑ,µ z p Y Z y zz;ẑ,µ z z p Y Z y z z ;ẑ,µ z D g out y,ẑ,µ z µ z EZ Y {z y;ẑ,µ z } ẑ D2 g out y,ẑ,µz varz Y {z y;ẑ,µ z } µ z µ z D3 p X Y x y;ˆr,µ r p X xx;ˆr,µ r x p Xx x ;ˆr,µ r D4 g in ˆr,µ r x xp X Yx y;ˆr,µ r D5 g in ˆr,µr µ x r x g inˆr,µ r 2 p X Y x y;ˆr,µ r D6 initialize: n : ˆ x xp Xx I n : µ x x ˆxn 2 p X x I2 m : û m0 0 I3 for t,2,3,... m : ẑ mt Amnˆxnt R m : µ z mt Amn 2 µ t R2 m : ˆp mt ẑ mt µ z m tûmt R3 m : û mt g out y m, ˆp mt,µ z mt R4 m : µ u mt g out ym, ˆpmt,µz mt R5 n : µ r nt Amn 2 µ u mt R6 n : ˆr nt ˆt+µ r n t M ma mnûmt R7 n : µ t+ µ r ntg in ˆrnt,µr nt R8 n : ˆt+ g in ˆr nt,µ r nt R9 en TABLE I THE GAMP ALGORITHM [6] for scaling factor C n p X ;q ;ˆr n,µ r n. From 2, it is straightforwar to show that BG-GAMP yiels the following posterior support probabilities: Pr{ 0 y;q} πˆr n,µ r n;q. 3 III. EM LEARIG OF THE PRIOR PARAMETERS q We use the expectation-maximization EM algorithm [0] to learn the statistical parameters q [λ,θ,φ,ψ]. The EM algorithm is an iterative technique that increases the likelihoo at each iteration, guaranteeing convergence to a local maximum of the likelihoo py;q. In our case, we choose the hien ata to be {x,w}, which yiels the EM upate q i+ argmaxe { lnpx,w;q y;q i }, 4 q where i enotes EM iteration an E{ y;q i } enotes expectation conitione on the observations y uner the parameter hypothesis q i. Moreover, we use the well-establishe incremental upating scheule [], where q is upate one element at a time while keeping the other elements fixe. A. EM upate for λ We now erive the EM upate for λ given previous parameters q i [λ i,θ i,φ i,ψ i ]. Because x is apriori inepenent of w an i.i., the joint pf px,w;q ecouples into C p X ;λ,θ,φ for a λ-invariant constant C, an so λ i+ argmax λ 0, E { lnp X ;λ,θ i,φ i y;q i }. 5 The maximizing value of λ in 5 is necessarily a value of λ that zeroes the erivative, i.e., that satisfies p y;q i λ lnp X ;λ,θ i,φ i 0. 6

3 For the p X ;λ,θ,φ given in 2, it is reaily seen that λ lnp X ;λ,θ i,φ i ;θ i,φ i δ p X ;λ,θ i,φ i { λ 0 λ 0. 7 Plugging 7 an 2 into 6, it becomes evient that the neighborhoo aroun the point 0 shoul be treate ifferently than the remainer of R. Thus, we efine the close ball B ǫ [ ǫ,ǫ] an B ǫ R\B ǫ, an note that, in the limit ǫ 0, the following is equivalent to 6: p y;q i p y;q i. λ x } n B ǫ λ {{} x } n B ǫ {{} ǫ 0 πˆr n,µ r n;q i ǫ 0 πˆr n,µ r n;q i 8 To verify that the left integral converges to the πˆr n,µ r n;q i efine in 8, it suffices to plug 2 into 8 an apply the Gaussian-pf multiplication rule; 3 meanwhile, for any ǫ, the right integral must equal one minus the left. Finally, the EM upate for λ is the unique value satisfying 8 as ǫ 0, i.e., λ i+ πˆr n,µ r n;q i. 9 Conveniently, {πˆr n,µ r n;q i } are GAMP outputs. B. EM upate for θ Similar to 5, the EM upate 4 for θ can be written as θ i+ argmax θ R E { lnp X ;λ i,θ,φ i y;q i }, 20 The maximizing value of θ in 20 is necessarily a value of θ that zeroes the erivative, i.e., that satisfies p y;q i θ lnp X ;λ i,θ,φ i 0. 2 For the p X ;λ,θ,φ given in 2, it is reaily seen that θ lnp X ;λ i,θ,φ i θ φ i λ i ;θ,φ i p X ;λ i,θ,φ i { xn θ φ i Splitting the omain of integration in 2 into B ǫ an B ǫ, an then plugging in 22, we fin that the following is equivalent to 2 in the limit of ǫ 0: θp y;q i x;a,ax;b,bx; a/a+b/b /A+/B, /A+/B 0;a b,a+b. 4 If the user has goo reason to believe that the true signal pf is symmetric aroun zero, then they may consier fixingθ0 an avoiing this EM upate. The unique value of θ satisfying 23 as ǫ 0 is then θ i+ ǫ 0 p y;q i lim ǫ 0 p y;q i 24 λ i+ πˆr n,µ r n;q i γˆr n,µ r n;q i 25 for the GAMP outputs {γˆr n,µ r n;q i } efine in 9. The equality in 25 can be verifie by plugging the GAMP posterior expression from 2 into 24 an simplifying via the Gaussian-pf multiplication rule. C. EM upate for φ Similar to 5, the EM upate for φ can be written as ˆφ i+ argmax φ>0 E { lnp X ;λ i,θ i,φ y;q i }. 26 The maximizing value of φ in 26 is necessarily a value of φ that zeroes the erivative, i.e., that satisfies p y;q i φ lnp X ;λ i,θ i,φ For the p X ;λ,θ,φ given in 2, it is reaily seen that φ lnp X ;λ i,θ i,φ xn θ i 2 2 φ 2 λ i ;θ i,φ φ p X ;λ i,θ i 28,φ { xn θ i 2 2 φ 2 φ Splitting the omain of integration in 27 into B ǫ an B ǫ, an then plugging in 29, we fin that the following is equivalent to 27 in the limit of ǫ 0: xn θ i 2 φ p y;q i The unique value of φ satisfying 30 as ǫ 0 is then φ i+ lim ǫ 0 θ i 2 p y;q i lim 3 ǫ 0 p y;q i θ λ i+ πˆr n,µ r n,q i i γˆr n,µ r n;q i 2 +νˆr n,µ r n;q i 32 for the GAMP outputs {νˆr n,µ r n;q i } efine in 0. The equality in 32 can be verifie by plugging the GAMP posterior expression from 2 into 3 an simplifying using the Gaussian-pf multiplication rule.

4 K/M Fig.. Empirical noiseless PTCs for Bernoulli- Gaussian signals an theoretical PTC for Lasso. K/M Fig. 2. Empirical noiseless PTCs for Bernoulli- Raemacher case an theoretical PTC for Lasso. K/M Fig. 3. Empirical noiseless PTCs for Bernoulli signals an theoretical PTC for Lasso. D. EM upate for ψ Finally, we erive the EM upate for ψ given previous parameters q i. Because w is apriori inepenent ofxan i.i., the joint pf px,w;q ecouples into C M m p Ww m ;ψ for a ψ-invariant constant C, an so ψ i+ argmax ψ>0 E { lnp W w m ;ψ y;q i }. 33 m The maximizing value of ψ in 33 is necessarily a value of ψ that zeroes the erivative, i.e., that satisfies pw m y;q i w m ψ lnp Ww m ;ψ m Because p W w m ;ψ w m ;0,ψ, it is reaily seen that ψ lnp Ww m ;ψ wm 2 2 ψ 2, 35 ψ which, when plugge into 34, yiels the unique solution ψ i+ w m 2 pw m y;q i. 36 M w m m Since w m y m z m for z m a T mx, we can also write 5 ψ i+ M M y m z m 2 pz m y;q i z m 37 ym ẑ m 2 +µ z m 38 m m where ẑ m an µ z m, the posterior mean an variance of z m, are available from GAMP see steps R-R2 in Table I. E. EM Initialization Since the EM algorithm converges only to a local maximum of the likelihoo function, proper initialization is essential. We initialize the sparsity as λ 0 M ρ SE M, where ρ SE M is the sparsity ratio K M achieve by the Lasso PTC [2] ρ SE M max a 0 2 M [+a2 Φa aφa] a 2 2[+a 2 Φa aφa] Empirically, we have observe that the EM upate for ψ works better with the µ z m term in 38 weighte by M an suppresse until later EM iterations. We conjecture that this is ue to bias in the GAMP variance estimates µ z m. We initialize the active mean as θ 0 0, which effectively assumes that the active pff is symmetric. Finally, noting that E{ y 2 2} SR+Mψ for SR tra T Aλφ/Mψ, we see that the variances, φ an ψ, can be initialize base on y 2 2 an a given hypothesis SR 0 0. In particular, ψ 0 y 2 2 SR 0 +M, φ0 y 2 2 Mψ 0 tra T Aλ, 40 0 where, without other knowlege, we suggest SR IV. UMERICAL RESULTS A. oiseless Phase Transitions First, we escribe the results of experiments that compute noiseless empirical phase transition curves PTCs uner various sparse-signal istributions. To compute each empirical PTC, we constructe a gri of oversampling ratio M [0.05,5] an sparsity ratio K M [0.05,5] for fixe signal length 000. At each gri point, we generate R 00 inepenent realizations of K-sparse signal x an M i.i.-gaussian measurement matrix A. From the measurements y Ax, we attempte to reconstruct the signal x using various algorithms. A recovery ˆx from realization r {,...,R} was consiere a success i.e., S r if MSE x ˆx 2 2/ x 2 2 < 0 4, where the average success rate is efine as S R R r S r. The empirical PTC was then plotte, using Matlab s contour comman, as the S contour over the sparsity-unersampling gri. Figures 3 show the empirical PTCs for three recovery algorithms: the propose algorithm, 6 a genieaie BG-GAMP that knew the true [λ, θ, φ, ψ], an the from [2]. For comparison, Figs. 3 also isplay the PTC 39. The signal was generate as BG with zero mean θ 0 an unit variance φ in Fig., as Bernoulli-Raemacher BR in Fig. 2 i.e., non-zero coefficients chosen uniformly from {, }, an as Bernoulli in Fig. 3 i.e., all non-zero coefficients set equal to or, equivalently, BG with θ an φ 0. Figures 3 emonstrate that, for all three signal types, the empirical PTC of improves on that for as well as the PTC. The latter two are known to converge in the large system limit 6 Matlab coe available at schniter/emturbogamp

5 MSE [B] MSE [B] MSE [B] Fig. 4. MSE for noisy recovery of a Bernoulli- Gaussian signal. Fig. 5. MSE for noisy recovery of a Bernoulli- Raemacher signal. Fig. 6. signal. MSE for noisy recovery of a Bernoulli [2]. The smallest gains over Lasso appear when the signal is BR i.e., the least-favorable istribution [3], whereas the largest gains appear when the signal is Bernoulli. Amazingly, EM-BG-AMP perfectly recovere almost every Bernoulli realization when M 5. The figures suggest that the EM algorithm oes a ecent job of learning the parameters λ,θ,φ. In fact, slightly outperforms genie-bg-gamp in Figs. 2 ue to realization-specific ata fitting. B. oisy Signal Recovery Figures 4 6 show MSE for noisy recovery of the same three sparse signal types consiere in Figs. 3. To construct these new plots, we fixe 000, K 00, SR 25B, an varie M. Each ata point represents MSE average over R 500 realizations. For comparison, we show the performance of the propose, Bayesian Compressive Sensing [9], Sparse Bayesian Learning [8] via, ebiase genie-aie 7 Lasso via SPGL [2], an Smoothe-l 0 [3]. All algorithms were run uner the suggeste efaults, with noise small in. In Fig. 4 an Fig. 6, we see outperforming all other algorithms for all meaningful values of unersampling ratio M. In fact, for Bernoulli signals Fig. 6, we see a significant improvement, especially when M [,8]. We have verifie in simulations not shown here that similar behavior persists at lower SRs. In Fig. 5, we see EM-BG- GAMP outperforming all algorithms except, which oes B better for large values of M. Apparently, the prior assume by is a better fit to BR than the BG prior. Amittely, the near-ominant performance observe for perfectly sparse signals in Figs. 6 oes not hol for all signal classes. As an example, Fig. 7 shows noisy recovery MSE for a Stuent s-t signal with pf p X x;q Γq+/2 2πΓq/2 +x 2 q+/2 4 uner the non-compressible parameter choice q.67 [4]. There, we see outperforme by an genie-aie Lasso, although not by an. In fact, among the competing algorithms, those that performe best for exactly sparse signals seem to o worst for this non-compressible signal, an vice versa. We attribute these 7 We ran SPGL in BPD moe: minˆx x s.t. y Ax 2 σ, for tolerances σ 2 {0.,,...,.5} Mψ, an reporte the lowest MSE. behaviors to a poor fit between the assume an actual signal priors, motivating future work on an EM Gaussian-Mixture GAMP with automatic selection of the mixture orer. REFERECES [] R. Tibshirani, Regression shrinkage an selection via the lasso, J. Roy. Statist. Soc. B, vol. 58, no., pp , 996. [2] D. L. Donoho, A. Maleki, an A. Montanari, Message passing algorithms for compresse sensing, Proc. ational Acaemy of Sciences, vol. 06, pp , ov [3] D. L. Donoho, A. Maleki, an A. Montanari, The noise-sensitivity phase transition in compresse sensing, arxiv:004.28, Apr [4] D. L. Donoho, A. Maleki, an A. Montanari, Message passing algorithms for compresse sensing: I. Motivation an construction, in Proc. Inform. Theory Workshop, Cairo, Egypt, Jan [5] M. Bayati an A. Montanari, The ynamics of message passing on ense graphs, with applications to compresse sensing, IEEE Trans. Inform. Theory, vol. 57, pp , Feb. 20. [6] S. Rangan, Generalize approximate message passing for estimation with ranom linear mixing, arxiv:04, Oct [7] M. E. Tipping, Sparse Bayesian learning an the relevance vector machine, J. Machine Learning Research, vol., pp , 200. [8] D. Wipf an B. Rao, Sparse Bayesian learning for basis selection, IEEE Trans. Signal Process., vol. 52, pp , Aug [9] S. Ji, Y. Xue, an L. Carin, Bayesian compressive sensing, IEEE Trans. Signal Process., vol. 56, pp , June [0] A. Dempster,. M. Lair, an D. B. Rubin, Maximum-likelihoo from incomplete ata via the EM algorithm, J. Roy. Statist. Soc., vol. 39, pp. 7, 977. [] R. eal an G. Hinton, A view of the EM algorithm that justifies incremental, sparse, an other variants, in Learning in Graphical Moels M. I. Joran, e., pp , MIT Press, 999. [2] E. van en Berg an M. P. Frielaner, Probing the Pareto frontier for basis pursuit solutions, SIAM J. Scientific Comput., vol. 3, no. 2, pp , [3] H. Mohimani, M. Babaie-Zaeh, an C. Jutten, A fast approach for overcomplete sparse ecomposition base on smoothe norm, IEEE Trans. Signal Process., vol. 57, pp , Jan [4] V. Cevher, Learning with compressible priors, in Proc. eural Inform. Process. Syst. Conf., Vancouver, B.C., Dec Fig. 7. MSE [B] MSE for noisy recovery of a non-compressible Stuent s-t signal.