Sparse Bayesian Learning Using Approximate Message Passing

Transcription

1 Sparse Bayesia Learig Usig Approximate essage Passig aher Al-Shoukairi ad Bhaskar Rao Dept. of ECE Uiversity of Califoria, Sa Diego, La Jolla, CA. Abstract We use the approximate message passig framework AP [] to address the problem of recoverig a sparse vector from udersampled oisy measuremets. We propose a algorithm based o Sparse Bayesia learig SBL []. Ulike the origial E based SBL that requires matrix iversios, the proposed algorithm has liear complexity, which makes it well suited for large scale problems. Compared to other message passig techiques, the algorithm requires fewer approximatios, due to the coditioal Gaussia prior assumptio o the origial vector. umerical results show that the proposed algorithm has comparable ad i may cases better performace tha existig algorithms despite sigificat reductio i complexity. I. ITRODUCTIO I the basic sigle measuremet vector SV sparse sigal recovery problem, we wish to recostruct the origial sparse sigal x R from oisy liear measuremets y R : y = Ax + e, where A R is a kow measuremet matrix ad e R is the additive oise modeled by e 0, σ I. If x is sufficietly sparse ad A is well desiged, accurate recovery of x is possible. While this SV model has a wide rage of applicatios, such as magetic resoace imagig RI, directio of arrival DOA estimatio ad electroecephalography EEG/ magetoecephalography EG. The model ca be exteded to address the case whe multiple measuremet vectors are available. The multiple measuremet vector V model ca be used for a umber of applicatios, such as DOA estimatio ad EEG/EG source localizatio. The V model is described by: Y = AX + E, Y. where Y [Y Y.,...,Y Y.T ] R T is the measuremet matrix cotaiig T measuremet vectors, X [X X.,...,X X.T ] R T is the ukow source matrix. Ad E is the ukow oise matrix. I this model we assume a commo sparsity profile, which meas that the support, which is the locatio of o zero elemets, i X is idetical across all colums. I practice it was also foud that the differet measuremet vectors experiece temporal correlatio. It was foud i [] that exploitig this temporal correlatio will yield superior performace whe compared to solvig the V problem while assumig idepedet Y.T vectors. Compared to other techiques such as greedy algorithms, mixed orm optimizatio ad reweighted algorithms, usig Bayesia techiques for sparse sigal recovery geerally achieves the best performace. Amog these Bayesia algorithms is the sparse Bayesia learig algorithm SBL which was first itroduced by Tippig [4] ad the used for the first time for sparse sigal recovery by Wipf ad Rao []. SBL has some appealig properties sice its global miimum was prove to be at the sparsest solutio [], ulike l based algorithms. Ad it was also prove to have less umber of local miima whe compared to classical recovery algorithms such as the FOCUSS algorithm [5]. However, the SBLs complexity level makes it usuitable for the use o large scale problems such as imagig for example. I this work, we develop a algorithm based o the same assumptios used i SBL, but use the belief propagatio method proposed i [6] to recover the sparse sigal. Compared to the origial E based SBL which requires matrix iversios, the proposed approach results i liear complexity, which makes it suitable for large scale problems. The approximate message passig algorithm was first itroduced by Dooho, aleki ad oteari i []. The algorithm uses loopy belief propagatio to solve the l miimizatio problem. It uses the cetral limit theorem to approximate all messages o the factor graph by Gaussia messages, which requires keepig track of the mea ad variace of these messages istead of calculatig the actual message at each step. The techique also uses Taylor series approximatio to reduce the umber of messages eve further. The authors also provided a framework to apply the algorithm to Bayesia techiques. The use of a Bayesia framework with belief propagatio was proposed previously i [7] [8]. However, because of the Beroulli-Gaussia or Gaussia mixture priors imposed o the origial vector x, several approximatios had to be made i order to make all the messages o the factor graph Gaussia. O the other had, the SBL coditioal Gaussia prior employed i our approach, results i Gaussia messages all over the graph, without the eed for approximatios. I additio to that, the Gaussia prior results i a much simpler model, ad easier message schedulig especially whe we cosider the case of multiple measuremets with time correlatio. Our umerical results show that the proposed algorithm has

2 comparable performace to other algorithms ad i may cases performs slightly better, while reducig the complexity sigificatly ad simplifyig the model. II. SV AP-SBL As metioed before, i this work we will follow the assumptios of the origial SBL algorithm. SBL assumes the Gaussia liklihood model: P y x; σ = πσ exp y σ y Ax Usig a hierarchical Bayesia framework, it was show i [9] that a umber of sparsity promotig priors ca be represeted usig a Gaussia scale mixture. A coditioal Gaussia prior is imposed o P x; γ, ad the desired prior o P x ca be obtaied by chagig the prior o P γ. I the SBL case, a o-iformative prior is assumed o γ. The coditioal prior o x becomes a parametrized Gaussia 4, where the parameter vector γ ca be leared from the data. P x; γ = πγ x exp 4 γ = This prior was show i [4] to be a sparsity promotig prior o x. Ad sice it is Gaussia whe coditioed o γ, it is appealig to use with the AP framework. Based o the previous assumptios ad usig 5, we ca costruct the factor graph i figure. P x y = P y xp x; γ 5 Where g m = P y m x ad f = P x ; γ. x f Where x deotes the itegratio over all x i, i. Whe V xq g m x x q ; µ qm, v qm, we use the followig fact: x; µ q, v q x; q q µ qvq, q v q q v q 8 Ad the mea ad variace of a message from the factor ode g m to the variable ode x becomes: V gm x A m x ; z m, c m ; 9 Where z m = y m q A mq µ qm 0 c m = σ + q A mq v qm ext we compute the message from a variable ode x to a factor ode g m. Ad by usig 8 agai, this message correspods to a Gaussia pdf, ad we oly eed to evaluate its mea ad variace. V x g m V f x V gl x V gl x x ; A lz l /c l A lm, I the large system limit we approximate c l by: c l c A lm c m 4 m= Ad the mea ad variace of V x g m are give by: g x f γ µ m = A l z l 5 c + γ g x f v m = c γ c + γ 6 Fially, we ca estimate the posterior P x y usig: g x f Fig. : AP SBL Factor Graph Sice all the fuctios o the factor graph are Gaussia pdfs, it will be show that all the messages passed by the sum product algorithm are also Gaussia, ad we oly eed to keep track of the mea ad variace of these messages. V f x x f x ; γ 6 V gm x x g m x V xq g m x 7 x q P x y V f x l= V gl x x ; µ, v 7 γ µ = A l z l c + γ l= 8 v = c γ c + γ 9 The aalysis above ca be cosidered a simplified versio of the aalysis i [] which assumes a Beroulli-Gaussia prior o x. We the use the same Taylor series approximatio give by the origial AP algorithm, ad summarize oe iteratio of the AP-SBL algorithm i Table. This process is repeated, util we reach a stable solutio, or util a predetermied maximum umber of iteratios are completed.

3 As we discussed earlier, we will eed to lear γ from the data, ad to do this we use the E algorithm which is also used i the origial SBL. The E updates ca be foud by: γ i+ = argmax γ E x y;γ i,σ [P y, x; γ, σ ] 0 = argmax γ E x y;γ i,σ [P y x; σ P x; γ] = E x y;γ i,σ [x ] = µ + v The E algorithm will also be used to fid the oise variace σ i the case that it is ukow. σ i+ = argmax E i[p y, x; γ, x y;γ,σ σ ] 4 σ 5 The derivatio is very similar to the origial SBL oe [] ad yields a very similar result: σ i+ = y Ax σ = γ v Defiitios F K, c = K γ c+γ G K, c = c.γ c+γ F γ K, c = c+γ essage updates K = m= A mzm + µ µ = F K, c v = G K, c c = σ + = v z m t = y m = Amµ + zm = F µ, c Parameter updates γ = V + µ σ i+ y Ax σ = = γ v TABLE I: AP SBL Algorithm III. V AP-TSBL 6 The origial sigle vector SBL ca be exteded for the case of multiple measuremet vectors V. I [0] the case of idepedet measuremets was studied. However, sice the extesio to the idepedet measuremet case is straight forward, ad it ca be cosidered a special case of the time correlated multiple measuremets that share a commo support, we will oly discuss the time correlated case i this sectio. The above factor graph ca be exteded i accordace to the followig model: y t = Ax t + e t, t =,...T 7 x t C, y t C 8 The model ca be restated as: ȳ = DA x + ē 9 Where ȳȳȳ [y y, y.., y T ], x x x [x x, x.., x T ], ēēē [e e, e.., e T ] ad DA is a diagoal matrix costructed from T replicas of A. P x ȳ [ T t= m= P y t m x t = P x t We ca ow costruct the factor graph i figure. g T g T g T Where: x T x T x T x T f T f T f T f T T g g g x x x x f f f f g g g x x x x Fig. : AP TSBL Factor Graph g m t x = P y m t x t = y t x t x t f f f f ] 0 m ; a H mx x t, σ ad we model the temporal correlatio betwee measuremet vectors by a AR process, with correlatio coefficiet β: f t x = P x t x t = x t ; βx t, β γ The derivatio of messages withi each measuremet vector is very similar to the SV case ad will ot be show here due to space cosideratios. We will oly show the derivatio of messages passed betwee differet measuremet vectors. Agai i this factor graph, all of the messages are Gaussia, ad we oly eed to keep track of the mea ad variace of each message. We start by fidig the messages from factor odes to eighborig variable ode i the forward directio: l= V f x x ; 0, γ V t f x t V t g x t l x t ; η t, ψ t 4 V f t x t P x t x t dx t 5

4 l= xt ; µ t, v t x t ; η t, ψ t Where: µ t = x t l= ; βx t, β γ dx t 6 A lz t l, vt = c t = m= c t m 7 Usig rules for Gaussia pdf multiplicatio ad covolutio we get: η t ψ t = β = β µ t v t v t + ηt ψ t v t ψ t + ψ t v t ψ t + ψ t v t 8 + β γ 9 We ow fid the message updates betwee vectors i the backward directio: V t+ f x t x t ; θ t, φ t 40 Defiitios F K t, c t = G K t, c t = F K t, c t = essage Updates η = 0 ψ = γ fort = : T η t ψ t = β Kt c t = β ψt K t c t + η t ψ t c t + ψ t c t + ψ t c t c t + ψ t + ηt ψ t c t ψ t + θt φ t + φ t + φ t + φ t ψt c t ψ t +c t + +c t β γ K t = m= A m zt m + µ t µ t = F K t, c t v t = G K t, c t c t = σ + = vt z m t = y m fort = T : θ t = β Kt+ c t+ φ t = = Amµt + θt+ φ t+ β φt+ c t+ φ t+ + zt m φt+ c t+ θ t+ +c t+ = F µt, c t + +c t+ β γ Parameter updates γ = T [ T t= V t + µ t β T t= V t σ = T T t= [ yt y t Ax t σ = TABLE II: AP TSBL Algorithm + µ t ] t V ] γ l= V t+ g x t+ l l= xt+ x t+ V f t+ x t+ ; µ t+, v t+ P x t+ x t+ x t dx t+ 4 ; θ t+, φ t+ ; βx t, β γ dx t+ 4 Usig rules for Gaussia pdf multiplicatio ad covolutio agai we get: θ t = β φ t = β µ t+ v t+ v t+ + θt+ φ t+ v t+ φ t+ + φ t+ v t+ φ t+ + φ t+ v t+ + β γ 4 44 Followig the same framework i [6] Taylor series approximatios are made to reduce the umber of updates. Fially, we use the E algorithm to update the values of γ, σ ad β. Table II summarizes oe iteratio of the AP- TSBL algorithm. While the formula for the E updates for β is ot show i the table, it follows stadard E derivatio, ad requires the solutio of a cubic equatio. IV. UERICAL RESULTS We coduct umerical experimets to compare the performace of the proposed algorithms to the origial SBL [], TSBL [] ad AP-V [8] algorithms. The elemets of A are geerated accordig to A m 0,. The source vector x o was geerated with K ozero elemets, ad the source matrix X o was geerated with K ozero rows. While the locatios of the ozero elemets/rows were radomly chose. We use two performace measures. The first measure is the ormalized SE. I the SV case SE E{ ˆxˆxˆx x o }/E{ x o } ad i the V case SE E{ ˆXˆXˆX Xo F }/E{ X o F }. We also use the rutime of each algorithm as a measure of complexity. I each experimet we ru 00 iteratios, with = 00, = 00 ad λ = K =.. Figure shows a compariso betwee SE of the origial SBL ad the proposed AP-SBL over a rage of SR values. We ca see that AP-SBL performs slightly better tha SBL. However, checkig the complexity rutime i figure 4, we ca see that a sigificat reductio i complexity is offered by AP-SBL. The complexity differece will be eve more sigificat at larger problem sizes, sice for each iteratio AP-SBL requires O + liear operatios, while the origial SBL requires a iversio of a matrix. I figure 5 we ru a compariso betwee AP-TSBL, TSBL ad AP-V over a rage of SR values ad correlatio coefficiet β values. The AP-V algorithm uses a Beroulli-Gaussia prior ad requires several approximatios

5 to make all messages o the factor graph Gaussia. Ad it also requires more complicated message schedulig compared to AP-TSBL because the factor graph it costructs cotais more loops. From figure 5 we ca see that the performace of AP-TSBL is ofte comparable ad sometimes better tha the other two algorithms. However, comparig rutimes figure 6 we ca see a sigificat reductio i the case o AP-TSBL compared to the other two algorithms. It is worth otig that the problem size we are usig here is ot very large, due to rutime limitatios of the origial TSBL. As the problem size grows both the AP-TSBL ad AP-V complexities will grow liearly, which is ot the case for TSBL that requires matrix iversios. Fig. : SE Compariso for SV Algorithms Fig. 6: Rutime Compariso for V Algorithms V. COCLUSIO I this paper we cosidered the problem of sparse sigal recovery i the sigle measuremet case ad i the multiple measuremet case with temporal correlatio betwee measuremets. We preseted two algorithms that use the origial assumptios of the SBL ad TSBL algorithms. The proposed algorithms apply the approximate messaeg passig framework with these assumptios to sigificatly reduce the complexity of the origial algorithms. We compared the proposed algorithms to the origial SBL ad TSBL algorithms, ad to the AP-V algorithm which also uses the AP framework. We showed that while SE performace is comparable with previous algorithms, the ew approach offers the lowest complexity level. The complexity reductio ca be attributed to usig the AP framework with a Gaussia prior o x whe coditioed o γ. I additio to this complexity reductio, the proposed techique offers much simpler implemetatio whe compared to other previously used Bayesia techiques. ACKOWLEDGET This research was supported by the atioal Sciece Foudatio grat CCF-4458 ad Ericsso edowed chair fuds. REFERECES Fig. 4: Rutime Compariso for SV Algorithms Fig. 5: SE Compariso for V Algorithms [] D. L. Dooho, A. aleki ad A. otaari, essage passig algorithms for compressed sesig, Proceedigs of the atioal Academy of Scieces, vol. 06, o. 45, 009. [] D.P. Wipf ad B.D. Rao, Sparse Bayesia Learig for Basis Selectio, IEEE Trasactios o Sigal Processig, vol. 5, o. 8, August 004 [] Z. Zhag ad B. D. Rao, Sparse Sigal Recovery with Temporally Correlated Source Vectors Usig Sparse Bayesia Learig, IEEE Joural of Selected Topics i Sigal Processig, vol.5, o.5, 0. [4].E. Tippig, Sparse Bayesia learig ad the relevace vector machie Joural of achie Learig, vol., 00. [5] I. Goroditsky ad B. D. Rao Sparse sigal recostructio from limited data usig FOCUSS: a recursive weighted orm miimizatio algorithm, IEEE Trasactio o Sigal Processig, vol. 45, o., 997. [6] D. Dooho, A. aleki, ad A. otaari, How to Desig essage Passig Algorithms for Compressed Sesig [Olie]. Available: 0. [7] J. P. Vila ad P. Schiter, Expectatio-aximizatio Gaussia-ixture Approximate essage Passig, Proc. Cof. o Iformatio Scieces ad Systems, 0 [8] J. Ziiel ad P. Schiter, Efficiet High-Dimesioal Iferece i the ultiple easuremet Vector Problem, IEEE Trasactios o Sigal Processig, vol. 6, o., Ja. 0. [9] J.A. Palmer, K. Kreutz-Delgado, D.P. Wipf ad B.D. Rao, Variatioal E algorithms for o- gaussia latet variable models, Advaces i eural Iformatio Processig Systems. IT Press, Cambridge, 006. [0] D. P. Wipf ad B. D. Rao, A empirical Bayesia strategy for solvig the simultaeous sparse approximatio problem, IEEE Trasactios o Sigal Processig, vol. 55, o. 7, 007. [] P. Schiter, Turbo recostructio of structured sparse sigals, Proc. Cof. o Iformatio Scieces ad Systems, 00