Conditional Approximate Message Passing with Side Information

Transcription

1 Condiional Approximae Message Passing wih Side Informaion Dror Baron Dp. Elecrical & Comp. Eng. NC Sae Universiy Anna Ma Ins. Mah. Sciences Claremon Grad. U. Deanna Needell Dp. Mahemaics UCLA Cynhia Rush Dp. Saisics Columbia U. Tina Woolf Ins. Mah. Sciences Claremon Grad. U. Absrac In informaion heory, side informaion (SI) is ofen used o increase he efficiency of communicaion sysems. This work lays he framework for a class of Bayes-opimal signal recovery algorihms referred o as condiional approximae message passing (CAMP) ha make use of available SI. CAMP involves a linear inverse problem, where noisy, linear measuremens acquire an unknown inpu vecor using a measuremen marix wih independen and idenically disribued enries, and he SI vecor obeys a symbol-wise dependence wih he inpu. Despie having a simple and sraighforward derivaion, our CAMP algorihm obains lower mean squared error han oher signal recovery algorihms ha have been proposed o incorporae SI. The good performance of CAMP is due is Bayes-opimaliy properies, which are no presen in previous approaches o SI-aided signal recovery. I. INTRODUCTION The core focus of research in many disciplines is on accuraely recovering a high-dimensional, unknown inpu signal from a limied number of noisy linear measuremens by exploiing probabilisic characerisics and srucure of he inpu. We consider he following model for his ask. For an unknown inpu signal x R N, y Ax + z, (1) where y R M are noisy measuremens, A R M N is he measuremen marix, and z N (0, zi) is measuremen noise. The objecive of signal recovery is o recover or esimae x from knowledge of only y and A, and possible saisical knowledge abou x and z. In informaion heory [4], when separae communicaion sysems hemselves share side informaion (SI) in he form of addiional informaion abu he signal, overall communicaion ofen becomes more efficien. For example, hree dimensional (3D) video acquisiion could be performed by acquiring each frame of video, which is a D image, independenly of oher frames using a single pixel camera [13]. While recovering he curren frame, one is likely simulaneously recovering adjacen frames, which can be used as SI. Anoher example is channel esimaion in wireless sysems, where he channel srucure ofen varies slowly and he esimae from he previous ime bach can be used as SI for he curren bach. Our recovery algorihm has access o SI, allowing for improvemens in recovery qualiy. Approximae message passing or AMP [6], [7], [10] is a low-complexiy algorihmic framework for efficienly solving high-dimensional regression asks (1). AMP algorihms are derived as Gaussian or quadraic approximaions of loopy belief propagaion algorihms (e.g., min-sum, sum-produc) on he dense facor graph corresponding o (1). AMP has he aracive feaure ha under suiable condiions on A and x is performance can be racked accuraely wih a scalar ieraion called sae evoluion (SE) [1], [1]. In paricular, performance measures such as he l 1 - or l -error in he algorihm s ieraions concenrae o consans prediced by SE. AMP algorihms work by ieraively updaing esimaes of he unknown signal. These AMP updaes rely on appropriaely-chosen denoisers, which reduce he noise in each ieraion. Assuming ha he measuremen marix A has independen and idenically disribued (i.i.d.) N (0, 1/M) enries and he enries of he signal x are i.i.d. f(x), where f(x) is he probabiliy densiy funcion (pdf) of he signal, one useful feaure of AMP is ha he inpu o he denoiser a any ieraion,, is approximaely equal o he rue signal x plus i.i.d. Gaussian noise wih variance λ, a consan value given by he SE equaions. Owing o hese favorable properies, η in ieraion is ofen he minimum mean squared error (MMSE) denoiser based on he pdf of x: η (a) E[X X + λ Z a], () where Z N (0, 1), and X f(x) is a random variable (RV) whose pdf is he same as ha of x. In his paper, we se he foundaions for a novel Bayes-opimal algorihmic framework ha uses AMP for signal recovery from noisy linear measuremens where SI is available. In our problem seing, he marix A is i.i.d. Gaussian, and he SI akes he form of a sequence x R N, wih each symbol of x saisically dependen on he corresponding symbol of x hrough some join pdf f(x, X). While oher ypes of dependence beween x and x are also possible, we leave hese for fuure

2 research. For such symbol-by-symbol dependencies, we propose a condiional denoiser, η (a, b) E[X X + λ Z 1 a, X b], (3) which provides he MMSE esimae of he signal when SI is available o he denoiser. Applying his condiional denoiser (3) wihin AMP, we obain our condiional approximae message passing (CAMP) approach. Alhough our condiional denoiser (3) can be applied o various linear regression asks, in his paper we focus on underdeermined sysems, i.e., M < N, where x is approximaely K-sparse (meaning ha x has K nonzero enries on average) for K N. This framework has been exensively sudied in he compressed sensing lieraure [], [5]. We show ha despie having a simple and sraighforward derivaion, our CAMP algorihm ouperforms oher signal recovery algorihms ha have been proposed o incorporae SI. This improved recovery performance is likely due o he Bayes-opimal naure of CAMP, which is discussed in Secion II-B. Paper Ouline: In Secion II we inroduce he CAMP algorihm and lay he framework for is heoreical analysis in he case of Gaussian measuremen noise and SI wih addiive whie Gaussian noise (AWGN). While our analysis in his paper focuses on he case where he SI is he inpu wih AWGN for wo specific signal priors, CAMP can be used in any seing where a condiional denoiser (3) can be employed wihin AMP ieraions. In Secion III we sudy wo example pdfs for our inpu signal x, he nearly leas favorable signal and he Bernoulli-Gaussian signal. For hese examples we demonsrae how one derives he condiional denoiser (3) for SI wih AWGN. Finally, Secion IV presens numerical resuls demonsraing he good performance of CAMP for he wo examples compared o oher signal recovery algorihms making use of SI. Comparison o previous work. While inegraing SI ino signal recovery algorihms is no new [3], [8], [9], [11], [15], CAMP proposes a unified framework wihin AMP for ackling his problem ha suppors any per-symbol dependence beween he SI and signal pair. Prior work using SI has been eiher heurisic, limied in focus o specific applicaions, or ouside he AMP framework. For example, Wang and Liang [15] inegrae SI ino AMP for a specific signal prior densiy, bu he mehod lacks Bayes opimaliy properies and is difficul o generalize. The algorihmic framework we lay ou in Secion II overcomes hese limiaions. II. CONDITIONAL AMP FRAMEWORK A. Condiional AMP inroducion AMP algorihm: The sandard AMP algorihm [6] ieraively updaes esimaes of he unknown inpu signal wih x R N being he esimae a ieraion. The algorihm is given by he following se of updaes. Assume x 0 0 he all-zero vecor and updae for 0: r y Ax + r 1 η δ 1 (x 1 + A T r 1 ), (4) x +1 η (x + A T r ). (5) Noe ha η : R R is he denoising funcion a he -h ieraion and δ M N is he measuremen rae. The denoising funcion acs elemen-wise on is inpu and has derivaive η (w) w η (w). Moreover, w 1 N N i1 w i is he empirical mean, where w R N. Sae evoluion (SE): As noed earlier, i has been shown [1], [1] ha he pseudo-daa, x +A T r, which is he inpu o he denoiser, η ( ), is approximaely equal in disribuion o he rue inpu x plus independen Gaussian noise, i.e., x + λ Z, where Z N (0, I) R N, and he noise variance λ can be prediced by SE, inroduced below. These favorable saisical properies of he pseudo-daa are due o he presence of he Onsager erm, r 1 δ η 1 (x 1 + A T r 1 ), used in he residual sep (4) of he AMP updaes. Le λ 0 z + E[X ]/δ and for 0, λ z + 1 δ E [(η 1 (X + λ 1 Z) X) ], (6) where X f(x) is independen of Z N (0, 1). CAMP, inroduced below, inegraes SI ino AMP. CAMP for SI wih AWGN: Condiional AMP, or CAMP, uses he same updaes as in (4) - (5), bu incorporaes SI using he condiional denoiser given in (3). In he following, we discuss he proposed condiional denoiser (3) for wo sparse pdfs, where i is assumed ha he signal, X, and SI, X, are relaed by X X + N (0, I), (7) which follows he model presened in [15]. While his model of SI is quie limied, i is pracical for an iniial invesigaion of he CAMP framework. I should be noed ha CAMP easily suppors oher SI models. In wha follows, we use lower case leers o denoe realizaions of he RVs denoed by capial leers. Under he assumpion (7), he CAMP denoiser akes he form η (a, b) E[X X + λ Z 1 a, X + Z b], (8) for independen sandard Gaussian RVs Z 1 and Z, where λ is given by he SE equaions for CAMP: le λ 0 z + E[X ]/δ and for 0, λ z + 1 δ E [(η 1 (X + λ 1 Z 1, X + Z ) X) ], (9) where X f(x) is independen of Z 1 and Z.

3 B. Bayes-opimaliy properies of CAMP When he SI is he rue inpu wih AWGN, as in (7), we will show ha he fixed poins of he CAMP SE (9) coincide wih he fixed poins of sandard AMP SE (6) wih effecive measuremen rae δ eff αδ and effecive measuremen noise variance eff z/α where he enhancemen facor α 1 depends on he pdf of he signal and he SI noise variance given in (7). The enhancemen facor implies ha for SI wih AWGN, he CAMP algorihm emulaes signal recovery for sandard (no SI) wih more measuremens and reduced measuremen noise variance in he linear regression problem (1). Before demonsraing he aforemenioned Bayesopimaliy propery of CAMP, we use mached filer argumens o provide a simplified represenaion of he condiional denoiser of (3) for SI wih AWGN. In calculaing he CAMP denoiser (3), we wan o esimae X from wo noisy observaions, he pseudo daa, X + λ Z 1 a, and SI, X + Z b, where Z 1 and Z are sandard Gaussian RVs. We define signal and noise vecors as s [1 1] T and v [λ Z 1 Z ] T, respecively, where [ ] T is he ranspose operaor. The mached filer esimaes he unknown X by compuing he inner produc beween [ ] a b [ ] X + λ Z 1 sx + v, X + Z and a mached filer h R. An opimal h ha maximizes he signal o noise raio while having uni norm is compued by invering R v E[vv T ], he auocovariance marix of v, and is given by h (R v) 1 s (R v ) 1 s. I can be shown ha h [ λ ] T /( + λ ), and he inner produc is defined as µ (a, b) : µ (a, b) [a b] T, h a + bλ + λ Noe ha µ (X + λ Z 1, X + Z ) equals (X + λ Z 1 ) + (X + Z )λ + λ. (10) d X + Z, d where Z is sandard Gaussian, denoes equaliy in disribuion, and he variance erm, ( ), is ( ) (λ ) + (λ ) ( + λ ) λ + λ. (11) The above provides us wih he following simplificaion of he CAMP denoiser (8) for SI wih AWGN, η (a, b) E[X X + Z µ ], (1) where µ (a, b) and are defined in (10) and (11). We noe ha µ is a funcion of (a, b), bu for breviy we drop his dependence in he following. Using he represenaion in (1), we analyze he SE equaions in (9). Considering (9) and (1), noe ha η (X + λ Z 1, X + Z ) E[X X + Z]. (13) We can herefore simplify he SE equaions given in (9) using (13) and he definiion of in (11). Le λ 0 z + E[X ]/δ and for 0, ( [ ] ) λ z + 1 δ E E X X + λ 1 Z X. (14) + λ 1 The resuls in (1) and (14) provide a simplified way o calculae he condiional denoiser of (3) and he SE when he signal and he SI are relaed hrough Gaussian noise. Moreover, a he saionary poin of (14) we have [ λ z + [(E 1 ] ) ] δ E X X + λ +λ Z X, (15) where λ is he scalar channel variance. Comparing (6) (SE wihou SI) and (15), we denoe he variance in he condiional expecaion by λ λ +λ. Noe ha λ λ 0, because λ, and we can rewrie he λ above as λ ( λ )z + 1 [ ( λz] X) ] E E[X X +. δ λ (16) We herefore see ha he CAMP SE (9) has fixed poins coinciding wih he fixed poins of sandard AMP ( SE (6) ) wih effecive measuremen rae δ eff δ +λ and ( effecive ) measuremen noise variance eff +λ z where is he noise in he SI given in (7), λ is he saionary poin of (14), and he enhancemen facor is α ( + λ )/. This effecive change in δ and implies ha he incorporaion of SI wih AWGN via he CAMP algorihm gives us signal recovery for a sandard (no SI) linear regression problem (1) wih more measuremens and/or reduced measuremen noise variance han our own, and he effec becomes more pronounced, i.e., he enhancemen facor α increases, as he noise variance in he SI,, ges small. The above analysis relies on he fac ha for he condiional expecaion denoiser in sandard (no SI) AMP (4)- (5), he corresponding SE equaion (6) in is convergen saes coincides wih Tanaka s fixed poin equaion [14], ensuring ha if AMP runs unil i converges, he resul provides he bes possible mean square error (MSE) achieved by any algorihm under cerain condiions. (These condiions on δ and ɛ, while ouside he scope of his paper, ensure ha here is a single soluion o Tanaka s fixed poin equaion, since muliple soluions may creae a dispariy beween he MSE of AMP and 3

4 he MMSE [7], [16], implying ha CAMP migh be sub-opimal in such cases.) However, he above analysis relies heavily on he Gaussianiy of he SI noise and is generalizaion is lef for fuure work. III. TWO EXAMPLE CAMP DERIVATIONS Having discussed our CAMP esimaion framework, we derive he CAMP denoiser funcions of (8) for wo pdfs for he signal. A. Nearly leas favorable signal The firs is he nearly leas favorable (NLF) signal, as sudied in [15]. The NLF signal x is enry-wise i.i.d., where each individual enry x i of x obeys X i (1 ɛ)δ 0 + ɛ δ h + ɛ δ h, (17) where δ is a Dirac dela funcion a, h is a consan ha can be compued as in [15], and ɛ K N is he expeced sparsiy rae. Wih such a prior, he elemens of x ake only hree values: 0 wih probabiliy 1 ɛ and ±h, each wih probabiliy ɛ/. Considering (1), we can compue he denoiser easily via Bayes rule using he values µ and ( ) defined in (10) and (11), η (a, b) E[X X + Z µ ] hp (X h X + Z µ ) hp (X h X + Z µ ) hɛ f Z( µ h ) hɛ f Z( µ +h ) ɛ f Z( µ +h ) + ɛ f Z( µ h ) + (1 ɛ)f Z ( µ ), where f Z represens ( he) sandard Gaussian pdf, i.e., f Z (z) 1 π exp z. Then for he NLF signal, using he above derivaion for he denoiser, we can easily find he SE using he resul of (14). B. Bernoulli-Gaussian signal Nex, we derive he CAMP denoiser (8) for a Bernoulli-Gaussian (BG) signal pdf, 1 x X i ɛ exp i + (1 ɛ)δ 0, (18) π meaning ha each elemen of he signal akes he value 0 wih probabiliy 1 ɛ, and oherwise is sampled from a sandard Gaussian pdf. Again considering (1), we can compue he denoiser easily via Bayes rule, η (a, b) E[X X + Z µ ] P E[X X + Z µ, X 0]. (19) In he above, P P (X 0 X + Z µ ) is he probabiliy ha he individual enry being denoised is nonzero and i can be compued using Bayes rule: µ ɛf Z 1+ P. (1 ɛ)f Z ( µ µ ) + ɛf Z 1+ ρ MSE (GENP-AMP) MSE (Proposed) TABLE I: MSE of GENP-AMP and he proposed mehod for NLF signals, δ 0.1. We can simplify he expecaion in (19) as follows: E[X X + Z µ, X 0] x xf Z( µ x )f X X 0 (x)dx x f Z( µ x )f X X 0 (x)dx E Z where Z N (µ 1 ( ) ) 1 + ( ), 1 + ( ). Plugging he above ino (19), using he definiions in (10) and (11), we see ha ( a + bλ ) η (a, b) P + λ + λ. (0) The second erm in (0) can be inerpreed as a Wiener filer based on a and b. Then for he BG signal, using (0) we can easily find he SE using he resul of (14). IV. NUMERICAL RESULTS We now presen simulaion resuls for CAMP using our proposed condiional denoiser, and compare o he generalized elasic ne prior (GENP) AMP algorihm of Wang and Liang [15], a proposed AMP algorihm for incorporaing SI. The algorihm developed by Wang and Liang uses a (sub-opimal) sof-hreshold denoiser. In our firs experimen, we uilize he same seup as in [15]. The signal, generaed using he NLF disribuion (17), has dimension N, 000. The enries of he measuremen marix A are i.i.d. sandard Gaussian, wih uni norm columns, and δ M N 0.1. We se ρ ɛ δ K M as in [15]. The measuremen noise z in (1) has sandard Gaussian enries, and he Gaussian noise in x is N (0, ), where {1,, 4}. For each of 0 rials, we calculae he MSE as 1 N x x, where x is he oupu of he corresponding algorihm; he repored resul is he average MSE over all 0 rials. The resuls in Table I show ha he MSE of he proposed approach is lower han ha of GENP-AMP in each case. 4

5 MSE CAMP SE Ieraion Fig. 1: Empirical MSE performance of CAMP vs. SE predicion. (N 5, 000, M, 500, K 750, 0.1, and z 0.01.) Nex, we repea he same experimen for he BG signal defined in (18). Resuls for GENP-AMP and he proposed approach are given in Table II. Again, CAMP ouperforms GENP-AMP in each case. Finally, Figure 1 compares he empirical MSE performance of CAMP and he performance prediced using SE. To do so, we averaged over 0 rials of a BG recovery problem where N 5, 000, M, 500, K 750, 0.1, and z I can be seen ha he empirical MSE racks ha prediced by SE well. ρ MSE (GENP-AMP) MSE (Proposed) TABLE II: MSE of GENP-AMP and he proposed mehod for BG signals, δ 0.1. REFERENCES [1] M. Bayai and A. Monanari. The dynamics of message passing on dense graphs, wih applicaions o compressed sensing. IEEE Trans. Inf. Theory, 57(): , Feb [] E. Candès. Compressive sampling. In Proc. In. Congress of Mah., Madrid, Spain, Augus 006. [3] M. Chen, F. Renna, and M. Rodrigues. On he design of linear projecions for compressive sensing wih side informaion. In Proc. IEEE In. Symp. Inf. Theory (ISIT), pages , Barcelona, Spain, July 016. [4] T. Cover and J.Thomas. Elemens of Informaion Theory. New York, NY, USA: Wiley-Inerscience, 006. [5] D. Donoho. Compressed sensing. IEEE Trans. Inf. Theory, 5(4): , 006. [6] D. Donoho, A. Maleki, and A. Monanari. Message passing algorihms for compressed sensing. Proc. Nal. Acad. Sci., 106(45): , 009. [7] F. Krzakala, M. Mézard, F. Sausse, Y. Sun, and L. Zdeborová. Probabilisic reconsrucion in compressed sensing: Algorihms, phase diagrams, and hreshold achieving marices. J. Sa. Mech. Theory E., 01(08):P08009, Aug. 01. [8] H. Van Luong, J. Seiler, A. Kaup, S. Forchhammer, and N. Deligiannis. Measuremen bounds for sparse signal reconsrucion wih muliple side informaion. Arxiv preprin arxiv: , Jan [9] J. Moa, N. Deligiannis, and M. Rodrigues. Compressed sensing wih prior informaion: sraegies, geomery, and bounds. IEEE Trans. Inf. Theory, 63(7): , 017. [10] S. Rangan. Generalized approximae message passing for esimaion wih random linear mixing. Arxiv preprin arxiv: , Oc [11] F. Renna, L. Wang, X. Yuan, J. Yang, G. Reeves, A. Calderbank, L.Carin, and M. Rodrigues. Classificaion and reconsrucion of high-dimensional signals from low-dimensional feaures in he presence of side informaion. IEEE Trans. Inf. Theory, 6(11): , 016. [1] C. Rush and R. Venkaaramanan. Finie sample analysis of approximae message passing. Proc. IEEE In. Symp. Inf. Theory, June 015. Full version: hps://arxiv.org/abs/ [13] D. Takhar, J. Laska, M. Wakin, M. Duare, D. Baron, S. Sarvoham, K. Kelly, and R. Baraniuk. A new compressive imaging camera archiecure using opical-domain compression. In Elecronic Imaging 006, pages Inernaional Sociey for Opics and Phoonics, 006. [14] T. Tanaka. A saisical-mechanics approach o large-sysem analysis of CDMA muliuser deecors. IEEE Trans. Inf. Theory, 48(11): , Nov. 00. [15] X. Wang and J. Liang. Approximae message passing-based compressed sensing reconsrucion wih generalized elasic ne prior. Signal Processing: Image Communicaion, 37:19 33, 015. [16] J. Zhu and D. Baron. Performance regions in compressed sensing from noisy measuremens. In Conf. Inf. Sciences Sysems, pages 1 6. IEEE, 013. ACKNOWLEDGMENTS The auhors hank Junan Zhu for insighful conversaion, Yaning Ma for having he insigh ha led o he represenaion of he CAMP denoiser given in Secion II-B, specifically ha of (16), and Joe Zhou for helping us improve our manuscrip. In addiion, Needell acknowledges suppor from NSF CAREER #134871, and NSF BIGDATA #174035; and Baron acknowledges suppor from NSF EECS #