An Analysis of State Evolution for Approximate Message Passing with Side Information

Transcription

1 A Aalysis of Sae Evoluio for Approxiae Message Passig wih Side Iforaio Hagji Liu NC Sae Uiversiy Eail: Cyhia Rush Colubia Uiversiy Eail: Dror Baro NC Sae Uiversiy Eail: Absrac A coo goal i ay research areas is o recosruc a ukow sigal x fro oisy liear easurees. Approxiae essage passig AMP is a class of low-coplexiy algorihs for efficiely solvig such high-diesioal regressio asks. Ofe, i is he case ha side iforaio SI is available durig recosrucio. For his reaso a ovel algorihic fraework ha icorporaes SI io AMP, referred o as approxiae essage passig wih side iforaio AMP-SI, has bee recely iroduced. A aracive feaure of AMP is ha whe he elees of he sigal are exchageable, he eries of he easuree arix are idepede ad ideically disribued i.i.d. Gaussia, ad he deoiser applies he sae o-lieariy a each ery, he perforace of AMP ca be prediced accuraely by a scalar ieraio referred o as sae evoluio SE. However, he AMP-SI fraework uses differe ery-wise scalar deoisers, based o he ery-wise level of he SI, ad herefore is o suppored by he sadard AMP heory. I his work, we provide rigorous perforace guaraees for AMP-SI whe he ipu sigal ad SI are draw i.i.d. accordig o soe joi disribuio subjec o fiie oe cosrais. Moreover, we provide uerical exaples o suppor he heory which deosrae epirically ha he SE ca predic he AMP- SI ea square error accuraely. I. INTRODUCTION High-diesioal liear regressio is a well-sudied odel ha has bee used i ay applicaios icludig copressed sesig, iagig, ad achie learig ad saisics 3. The ukow sigal x R is viewed hrough he liear odel: y Ax + w, where y R are he easurees, A R is a kow easuree arix, ad w R is easuree oise. The goal is o esiae he ukow sigal x havig kowledge oly of he oisy easurees y ad he easuree arix A. Whe he proble is uder-deeried i.e., <, i order for recosrucio o be successful, i is ecessary o exploi srucural or probabilisic characerisics of he ipu sigal x. Ofe a prior disribuio o he ipu sigal x is assued, ad i his case approxiae essage passig AMP algorihs ca be used for he recosrucio ask. AMP, 4 is a class of low-coplexiy algorihs for efficiely solvig high-diesioal regressio asks. AMP works by ieraively geeraig esiaes of he ukow ipu vecor, x, usig a possibly o-liear deoiser fucio ailored o ay prior kowledge abou x. Oe favorable feaure of AMP is ha uder soe echical codiios o he easuree arix A ad x, he observaios a each ieraio of he algorih are alos surely equal i disribuio o x plus idepede ad ideically disribued i.i.d. Gaussia oise i he large syse i. AMP wih Side Iforaio AMP-SI: I iforaio heory 5, whe differe couicaio syses share side iforaio SI, overall couicaio ca becoe ore efficie. Recely 6, 7, a ovel algorihic fraework, referred o as AMP-SI, has bee iroduced for icorporaig SI io AMP for high-diesioal regressio asks. AMP- SI has bee epirically deosraed o have good recosrucio qualiy ad is easy o use. For exaple, we have proposed o use AMP-SI for chael esiaio i eergig ileer wave couicaio syses 8, where he ie dyaics of he chael srucure allow previous chael esiaes o be used as SI whe esiaig he curre chael srucure 7. We odel each ery of he observed SI, deoed by x R, as depedig saisically o he correspodig ery of he ukow sigal x hrough soe joi probabiliy desiy fucio pdf, fx, X. AMP-SI uses a codiioal deoiser, η : R R, o icorporae SI, η a, b EX X + λ Z a, X b. The AMP-SI algorih ieraively updaes esiaes of he ipu sigal x: le x 0 0, he all-zeros vecor, he r y Ax + r η δ x + A T r, x, 3 x + η x + A T r, x, 4 where x R is he esiae of ipu sigal a ieraio, ad he deoiser i is applied ery-wise o vecor ipus. The derivaive η s,. s η s,. is wih respec o he firs ipu, ad v is he epirical average of a vecor v, i.e., v v i. Usig he deoiser i, he AMP-SI algorih 3-4 provides he iiu ea squared error MMSE esiae of he sigal whe SI X is available 6. Sae Evoluio SE: I has bee prove ha he perforace of AMP, as easured, for exaple, by he oralized squared l -error x x bewee he esiae x ad rue sigal x, ca be accuraely prediced by a scalar recursio referred as SE 9, 0 whe he easuree arix A is i.i.d. Gaussia uder various assupios o he elees of he sigal. The SE equaio for AMP-SI is as follows. Assue

2 he eries of he oise w are i.i.d. fw wih σw EW, ad le λ 0 σw + EX /δ. The for 0, λ σw + δ E η X + λ Z, X X, 5 where X, X fx, X are idepede of Z N 0,, where we use N µ, σ o deoe a Gaussia disribuio wih ea µ ad variace σ. Cosiderig AMP-SI 3-4, however, we cao direcly apply he exisig AMP heoreical resuls 9, 0, as he codiioal deoiser depeds o he idex i hrough he SI, eaig ha differe scalar deoisers will be used a differe idices wihi he AMP-SI ieraios. Rece resuls, however, exed he asypoic SE aalysis o a larger class of possible deoisers, allowig, for exaple, each elee of he ipu o use a differe o-liear deoiser as is he case i AMP-SI. We eploy hese resuls o rigorously relae he SE preseed i 5 o he AMP-SI algorih i 3-4. Relaed Work: While iegraig SI io recosrucio algorihs is o ew, AMP-SI iroduces a uified fraework wihi AMP supporig arbirary sigal ad SI depedecies. Prior work usig SI has bee eiher heurisic, ied o specific applicaios, or ouside he AMP fraework. For exaple, Wag ad Liag iegrae SI io AMP for a specific sigal prior desiy, bu he ehod is difficul o apply o oher sigal odels. Ziiel ad Schier 3 develop a AMP-based recosrucio algorih for a ie-varyig sigal odel based o Markov processes for he suppor ad apliude. This sigal odel is easily icorporaed io he AMP-SI fraework as discussed i he aalysis of he birhdeah-drif odel of 6, 7. Maoel e al. iplee a AMP-based algorih i which he ipu sigal is repeaedly recosruced i a sreaig fashio, ad iforaio fro pas recosrucio aeps is aggregaed io a prior, hus iprovig ogoig recosrucio resuls 4. This recosrucio schee resebles ha of AMP-SI, i paricular whe he Beroulli-Gaussia odel is used see Secio II-B. Coribuio ad Oulie: Ma e al. use uerical experies o show ha SE 5 accuraely racks he perforace of AMP-SI 3-4 7, as was show rigorously for sadard AMP. Ma e al. cojecure ha rigorous heoreical guaraees ca be give for AMP-SI as well 7. I his work, we aalyze AMP-SI perforace whe he ipu sigal ad SI are draw i.i.d. accordig o a geeral pdf fx, X obeyig soe fiie oe codiios, he AMP-SI deoiser is Lipschiz, ad he easuree arix A is i.i.d. Gaussia. I Secio II, we give he ai resuls, exaples for various sigal ad SI odels, ad uerical experies coparig he epirical perforace of AMP-SI ad he SE predicios. The proof of our ai heore is provided i Secio III. A. Mai Theore II. MAIN RESULTS Our ai resul provides AMP-SI perforace guaraees whe cosiderig pseudo-lipschiz loss fucios, which we defie i he followig. Defiiio II.. Pseudo-Lipschiz fucios 9: For k N >0 ad ay N >0, a fucio φ : R R is pseudo-lipschiz of order k if here exiss a cosa L, referred o as he pseudo-lipschiz cosa of φ, such ha for ay x, y R, φx φy L + x k + y k x y. 6 For k, his defiiio coicides wih he sadard defiiio of a Lipschiz fucio. Throughou his work, deoes he Euclidea or. We are ow ready o sae our ai resul. Throughou he paper we le p deoe covergece i probabiliy. Theore II.. For ay order k pseudo-lipschiz fucios φ : R R ad ψ : R 3 R, assue he followig. A The easuree arix A has i.i.d. Gaussia eries wih ea 0 ad variace /. A The oise w is i.i.d. fw wih fiie E W k. A3 The sigal ad SI X, X are sapled i.i.d. fro fx, X wih fiie E X k, fiie E X k, ad fiie E X X. A4 For 0, he deoisers η, defied i are Lipschiz coiuous, eaig for scalars a, a, b, b, ad cosa L > 0, The, η a, b η a, b L a, b a, b. φri, w i p E φw + λ σw Z, W, ψx i + A T r i, x i, x i p E ψx + λ Z, X, X, 7 where Z, Z are sadard Gaussias, idepede of W fw ad X, X fx, X. I he above, x ad r are defied i he AMP-SI recursio 3-4, ad λ i he SE 5. Secio III coais he proof of Theore II.. The proof follows fro Berhier e al., Theore 4 ad he srog law of large ubers. The ai echical deails ivolve showig ha our assupios A A4 are eough o saisfy he assupios eeded for, Theore 4. The deails are give i Secio III. As a cocree exaple of how Theore II. provides perforace guaraees for AMP-SI, le us cosider a few ieresig pseudo-lipschiz loss fucios. Corollary II... Uder assupios A A4, leig ψ : R 3 R be he l loss, ψ x, y, z x y, he by Theore II., x + A T r x p λ, where λ is defied i 5. Siilarly if ψ : R 3 R is defied as ψ x, y, z η x, z y, he by Theore II. x+ x p δλ σ w.

3 B. Exaples Nex, we cosider a few sigal ad SI odels o show how oe ca derive he deoiser i, use his o cosruc he AMP-SI algorih ad he SE, ad apply Theore II.. Before we ge o he exaples we sae a lea ha allows us kow abou how fucios wih bouded derivaive are Lipschiz. Lea II.. A fucio φ : R derivaives, 0 < D, D <, x φx, y D ad R havig bouded y φx, y D is Lipschiz coiuous wih Lipschiz cosa D + D. Proof. The resul follows usig he Triagle Iequaliy ad Cauchy-Schwarz, φx, y φx, y φx, y φx, y + φx, y φx, y φx, y φx, y + φx, y φx, y D y y +D x x D + D y y + x x D + D x, y x, y. Gaussia-Gaussia Sigal ad SI: I his odel, referred o as he GG odel heceforh, he sigal has i.i.d. Gaussia eries wih zero ea ad fiie variace ad we have access o SI i he for of he sigal wih addiive whie Gaussia oise AWGN. The sigal, X, ad SI, X, are relaed by X X + N 0, σ I. 9 I his case, he AMP-SI deoiser equals 7 η a, b E XX + λ Z a, X b σ xσ a + σxλ b σxσ + λ + σ λ. 8 0 The he SE 5 ca be copued as λ σw + σxσ λ δ σxσ + λ + σ λ. We oe ha he deoiser i 0 is Lipschiz coiuous as a resul of Lea II. because η σ a, b xσ σxσ + λ + σ λ, ad b η a, b σ xλ σxσ + λ + σ λ, ad herefore he assupios A A5 are saisfied i he GG case ad we ca apply Thoere II.. Beroulli-Gaussia Sigal ad SI: The Beroulli- Gaussia BG odel reflecs sceario i which oe wishes o recover a sparse sigal ad has access o SI i he for of he sigal wih AWGN as i 9. I his odel, each ery of he sigal is idepedely geeraed accordig o x i ɛn 0, + ɛδ 0, where δ 0 is he Dirac dela fucio a 0. I words, he eries of he sigal idepedely ake he value 0 wih probabiliy ɛ ad are N 0, wih probabiliy ɛ. I his case, he AMP-SI deoiser equals 7 η a, b E XX + λ Z a, X b Pr X 0 a, b E X a, b, X 0 σ a + λ b Pr X 0 a, b σ + λ + σ λ, where, leig ρ τ x be he zero-ea Gaussia desiy wih variace τ evaluaed a x, ad defiig ν : σ λ σ +λ + σ λ, where we deoe PrX 0 a, b + T a,b, 3 ɛρ λ T a,b : aρ σ b b ɛρ +σ bρ σ +σ +λ +σ a ɛ σ + λ + σ λ ɛ λ σ { σ a + λ exp b } σ λ σ + λ + σ λ ɛ ν π ɛ λ σ ρ ν σ a + λ b, 4 The he SE 5 ca be copued as λ σw + Ta,b σ + λ + σ λ δ + T a,b σ + λ + σ λ. 5 We agai use Lea II. o show ha he deoiser defied i ad 3 is Lipschiz coiuous so ha he assupios A A5 are saisfied i he BG case ad we ca apply Thoere II.. We sudy he parial derivaives. Deoe Cobiig 3 ad 4 ad 6, σ a + λ b f a,b : σ + λ + σ λ. 6 η a, b + T a,b f a,b. The, η a, b fa,b Ta,b + T a,b + T a,b f a,b + T a,b f a,b T a,b f a,b + T a,b + + T a,b. 7

4 Now we show upperbouds for he wo ers of 7 separaely. For he firs er, we see ha f a,b, so + T a,b f a,b + T a,b. Now we cosider he secod er of Cosider he secod er of 7. Firs we oe ha Ta,b + T a,b f a,b + T. a,b f a,b T a,b f a,b The fro 4 ad 6, ɛ T a,b f a,b πσ a + λ bρ ν σ a + λ b, ɛ he usig ha x ρ τ x x τ ρ τ x, we have σ π ρ ν σ a + λ b σ σ a + λ b ρ ν σ a + λ b ν ɛ πσ ρ ν σ a + λ b ɛ ν ν σ a + λ b. T a,b f a,b ɛ ɛ 8 To upper boud he above, we use exp{ x} +x whe x 0, ad so ρ τ x { x } exp πτ τ π τ τ + x. Usig his i 8, we fid σ T ɛ ν σ a + λ a,b f a,b b ν ɛ ν + σ a + λ b σ ɛ ɛ, ν ɛ σ w ɛ where i he fial iequaliy we use λ σ w by 5, ad σ σ ν λ σ + λ + σ λ λ. + λ σ + λ λ Usig he above i 7, we have η a, b + As i 7 we ca show b η a, b + T a,b + T a,b b f a,b The, + ɛ. σ w ɛ + T a,b Ta,b b f a,b + T a,b + T a,b b f a,b, 9 0, + b T a,b f a,b ad a boud as i 8-9 gives + b T a,b + T a,b Ta,b b f a,b C. Nuerical Exaples f a,b b λ ɛ ν σ a + λ b ν ɛ ν + σ a + λ b λ ɛ ɛ. ν ɛ σɛ T a,b f a,b Fially, we provide uerical resuls o copare he epirical ea square error MSE perforace of AMP-SI ad he perforace prediced by SE. Fig. shows he MSE achieved by AMP-SI i he GG sceario ad he SE predicio of is perforace. I his exaple, he sigal variace σx, he easuree oise variace σw 0.0, he variace of AWGN i SI σ We averaged over 0 rials of a GG recovery proble for epirical resuls of AMP-SI. The copariso i Fig. a, Fig. b ad Fig. c give by hree differe sigal legh. For saller here is soe gap bewee he epirical MSE ad he SE predicio, as show i Fig. for 00, bu he gap shriks as is icreased. The resuls show he epirical MSE racks he SE predicio icely. Fig. shows he MSE achieved by AMP-SI i he BG sceario, ad he SE predicio of is perforace. We agai averaged over 0 rials of a BG recovery proble for epirical resuls of AMP-SI. The sigal legh 0000, 3000, he easuree oise variace σw 0.0, ad ɛ 0., where 0% of he eries i he sigal are ozero. We vary he variace of AWGN i SI fro σ 0.04, σ 0.5, ad σ. The resuls show ha SE ca predic he MSE achieved by AMP-SI a every ieraio. III. PROOF OF THEOREM II. Proof of The proof Theore II. coais wo seps. I he firs sep we use Berhier e al., Theore 4 ad i he secod sep we ake a appeal o he srog law of large ubers SLLN. We reid he reader of he srog law: Defiiio III.. Srog Law of Large Nubers 5: Le X, X,... be a sequece of i.i.d. rado variables wih fiie ea µ. The Pr X + X X µ, I words, he parial averages X +X +...+X coverge alos surely o µ <. A. Sep We will ake use of Berhier e al., Theore 4, resaed here for coveiece. Before proceedig we iclude a defiiio of uiforly pseudo-lipschiz fucios, ha geeralizes he ideal of pseudo-lipschiz fucios give i Defiiio II.. Defiiio III.. Uiforly pseudo-lipschiz fucios : A sequece i of pseudo-lipschiz fucios {φ } N>0

5 5 0 AMP-SI Epirical wih 0.04 AMP-SI Epirical wih 0.5 AMP-SI Epirical wih SE Predicio AMP-SI Epirical wih 00 SE Predicio wih 00 MSE db MSE db MSE db MSE db Ieraio a 00 AMP-SI Epirical wih 000 SE Predicio wih Ieraio b 000 AMP-SI Epirical wih 0000 SE Predicio wih Ieraio c 0000 Fig. : Epirical MSE perforace of AMP-SI ad SE predicio. GG odel, δ 0.3, σ x, σ w 0., ad σ Ieraio Fig. : Epirical MSE perforace of AMP-SI ad SE predicio. BG odel, 0000, 3000, ɛ 0., σ w 0.. is called uiforly pseudo-lipschiz of order k if, deoig by L is he pseudo-lipschiz cosa of order k of φ, we have L < for each ad sup L <. Berhier e al., Theore 4 requires he followig assupios: C The easuree arix A has Gaussia eries wih i.i.d. ea 0 ad variace /. C Defie a sequece of deoisers η : R R o be hose ha apply he deoiser η defied i elee-wise o vecor ipu: η X : η X, X. For each, η are uiforly Lipschiz. A fucio is uiforly Lipschiz i if he Lipschiz cosa does o deped o. C3 x / coverges o a cosa as. C4 The i σ w w / is fiie. C5 For ay ieraios s, N ad for ay covariace arix Σ, he followig is exis. E Z x i η x i + Z i, x i <, E Z,Z η x i + Z i, x i η s x i + Z i, x i <, where Z, Z N0, Σ I, wih deoig he esor produc ad I he ideiy arix. Theore III.. Uder he assupios C C5, for ay sequeces of uiforly pseudo-lipschiz fucios φ : R R R ad ψ : R R R, φ r, w E Z φ w + λ σ w Z, w ψ x + A T r, x E Z ψ x + λ Z, x p 0, p 0, where Z N 0, I, Z N 0, I, x ad r are defied i he AMP-SI recursio 3-4, ad λ i he SE 5. Now we deosrae ha our assupios A A4 saed i Secio II are eough o saisfy he assupios C C6 eeded o apply Theore III..

6 Assupios A ad C are ideical. We will show ha C follows fro A4, C4 follows fro A, ad C3 follows fro A3. Fially we show C5 ad C6 follow fro A3 ad A4. Firs cosider assupio C. The o-separable deoiser η X η X, X applies he AMP-SI deoiser defied i erywise o is vecor ipus. Fro A4, {η, } 0 are Lipschiz coiuous. Thus, for legh- vecors x, x, ad fixed SI x, η x η x η x i, x i η x i, x i ad so L x i x i L x x, η x η x L x x. The Lipschiz cosa does o deped o, so η is uiforly Lipschiz. Now cosider assupio C4. Fro A, he easuree oise w i has i.i.d. eries wih zero-ea ad fiie E W k. The applyig Defiiio III., w wi σw <, where we have used ha σw < follows fro E W k < for k. The proof of C3 siilarly follows usig he SLLN ad he fiieess of E X k give i assupio A3. We ow show ha C5 is e. Recall Z N 0, σzi. Defie y i : x i E Z η x i + Z i, x i for i,,...,. By assupio A3, he sigal ad side iforaio X, X are sapled i.i.d. fro he joi desiy fx, X. I follows ha y, y,..., y are also i.i.d., so by Defiiio III. if EXη X + Z, X < where Z N 0, σz idepede of X, X fx, X, he x i E Z η x i + Z i, x i EXη X + Z, X, We will ow show ha EXη X + Z, X <. Firs oe ha A4 assues η, is Lipschiz, eaig for scalars a, a, b, b ad soe cosa L > 0, η a, b η a, b L a, b a, b Therefore leig a b 0 we have L a a + L b b. η a, b η 0, 0 η a, b η 0, 0 L a +L b, givig he follows upper boud for cosa L > 0, η a, b L + a + b. Now usig ad he riagle iequaliy, EXη X + Z, X L E X + X + Z + X L E X + EX + E X E Z + E X X. 3 Fially, by assupio A3 we have ha E X k, E X k ad E X X are all fiie. The oig ha for ay rado variable, Y, we have Y r + Y k for r k, eaig E Y r < + E Y k he boudedes of EXη X + Z, X follows fro 3 wih assupio A3. The proof of C6 follows siilarly o he proof of C5. Recall Z, Z N0, Σ I. Defie y i : E Z,Z η x i + Z i, x i η s x i +Z i, x i for i,,...,. By assupio A3, he sigal ad side iforaio X, X are sapled i.i.d. fro he joi desiy fx, X. I follows ha y, y,..., y are also i.i.d., so by Defiiio III. if Eη X + Z, Xη s X + Z, X < where Z N 0, σ z ad Z N 0, σ z, idepede of X, X fx, X, he E Z,Z η x i + Z i, x i η s x i + Z i, x i Eη X + Z, Xη s X + Z, X. We will ow show ha Eη X + Z, Xη s X + Z, X <. Usig he boud, Eη X + Z, Xη s X + Z, X E η X + Z, X η s X + Z, X L E + X + Z + X + X + Z + X. The usig he riagle iequaliy, E + X + Z + X + X + Z + X E + X + Z + X + X + Z + X + E X + E X + E X X + EX + E X + E X Z + E X Z + E X Z + E X Z + E Z + E Z + E Z Z + E X + E X + E X X + EX + E X + E Z + E X + E X + E Z + E X + E X + E ZZ. 4 Fially we apply Theore III. o he sequeces of uiforly pseudo-lipschiz fucios φ : R R R ad ψ : R R R defied as follows: for vecors a, b R ad x, y, x R, ad φ a, b : φa i, b i, ψ x, y : ψx i, y i, x i, 5 where φ ad ψ are he pseudo-lipschiz fucios defied i Theore II.. This he gives he resul

7 ad φri, w i p E Z φw i + λ σw Z, w i, ψ x i + A T r i, x i, x i p E Z ψ x i + λ Z, x i, x i. 6 7 As a fial sep, we show ha he fucios defied i 5 are uiforly pseudo-lipschiz as eeded o apply Theore III.. Usig ha φ : R R is pseudo-lipschiz of order k, eaig for vecors a, a, b, b R, ay idex i,,..., φa i, ã i φb i, b i L + ai, ã i k + bi, b i k ai, ã i b i, b i, where L > 0 is soe cosa. The usig he above, Nex, oe ha for vecors v, w R we have v i + w i k v i + w i k. The, a i, ã i k ai + ã i k a i + ã i k a + ã k a, ã k, 30 where he fial sep follows sice by a, ã k we ea o hik of he objec a, ã R R as sigle legh vecor, so a, ã k a i + ã i k a + ã k 3 Now puig 9 ad 30 io 8 we have he desired resul, φ a, ã φ b, b L + a, ã k + b, b k a, ã b, b. 3 B. Sep φ a, ã φ b, b φa i, ã i φb i, b i L + ai, ã i k + bi, b i k ai, ã i b i, b i L + a i, ã i k + b i, b i k a i, ã i b i, b i 8 where he fial sep follows by Cauchy-Schwarz. Firs oe a i, ã i b i, b i a i b i + ã i b i a, ã b, b 9 where a, ã b, b cosiders he objec a, ã R R o be a sigle legh vecor, so a, ã b, b a b + ã b. Now cosider he sep resul 6, our fial goal is do deosrae a.s. E E Z φw i + λ σw Z, w i φw + λ σw Z, W, E Z ψx i + λ Z, x i, x i a.s. E ψx + λ Z, X, X. The resul follows by he SLLN Defiiio III. so log as EφW + λ σ w Z, W ad EψX + λ Z, X, X are boh fiie. Recall, if φ : R R, is a pseudo-lipschiz fucio of order k, he here is a cosa L > 0 such ha for all x R : φx L + x k 9. Usig his, φ W + λ σw Z, W L + W + λ σw Z, W k/ L + 6k/ W k + 4k/ λ σw k/ Z k, 33

8 where we have used ha W + λ σw k Z, W k W + λ σw Z + W 3W + λ σwz k 6 k W k + k λ σ w k Z k. The, usig 33, ad he boudedess of E W k by A, EφW + λ σw Z, W L + 3k X k + 6k/ λk Z k + 3k/ 3 3 X k, 34 where we have used ha X + λ Z, X, X k X + λ Z + X + X k/ 3X + λ Z + X k/ 3 k X k + 6k/ 3 λk Z k + 3k/ 3 X k. Therefore, usig 34, ad he boudedess of E X k ad E X k assued i A3, EψX + λ Z, X, X L + 3 k E X k + 6k/ 3 λk E Z k + 3 3k/ E X k <. ACKNOWLEDGMENT We hak You Joe Zhou for isighful coversaios ad valuable advice. I addiio, Liu ad Baro ackowledge suppor fro NSF EECS #6. REFERENCES D. L. Dooho, A. Maleki, ad A. Moaari. Message passig algorihs for copressed sesig. Proc. Na. Acadey Sci., 0645: , Nov H. Arguello ad G. Arce. Code aperure opiizaio for specrally agile copressive iagig. J. Op. Soc. A., 8:400 43, Nov T. Hasie, R. Tibshirai, ad J. H. Frieda. The Elees of Saisical Learig. Spriger, Aug S. Raga. Geeralized approxiae essage passig for esiaio wih rado liear ixig. Arxiv prepri arxiv:00.54, Oc T. M. Cover ad J. A. Thoas. Elees of Iforaio Theory. New York, NY, USA: Wiley-Iersciece, D. Baro, A. Ma, D. Needell, C. Rush, ad T. Woolf. Codiioal approxiae essage passig wih side iforaio. I Proc. IEEE Asiloar Cof. Sigals, Sys. Copu., A. Ma, Y. Zhou, C. Rush, D. Baro, ad D. Needell. A approxiae essage passig fraework for side iforaio. arxiv: , Jul A. Saleh ad R. Valezuela. A saisical odel for idoor ulipah propagaio. IEEE J. Selec. Areas Cou., 5:8 37, Feb M. Bayai ad A. Moaari. The dyaics of essage passig o dese graphs, wih applicaios o copressed sesig. IEEE Tras. If. Theory, 57: , Feb C. Rush ad R. Vekaaraaa. Fiie saple aalysis of approxiae L + 6 k E W k + 4 k essage passig algorihs. IEEE Tras. If. Theory, 64:764 λ σw k E Z k 786, Nov. 08. <. R. Berhier, A. Moaari, ad P. M. Nguye. Sae evoluio for approxiae essage passig wih o-separable fucios. arxiv: , Siilarly, we have he upper boud, ψx+λ Z, X, X L + X + λ Z, X, X / Aug. 07. X. Wag ad J. Liag. Approxiae essage passig-based copressed k 3 sesig recosrucio wih geeralized elasic e prior. Sigal Process. Iage, 37:9 33, Sep J. Ziiel ad P. Schier. Dyaic copressive sesig of ie-varyig sigals via approxiae essage passig. IEEE Tras. Sigal Process., 6: , Nov Adre Maoel, Flore Krzakala, Eric W Trael, ad Leka Zdeborová. Sreaig bayesia iferece: heoreical is ad ii-bach approxiae essage-passig. I Couicaio, Corol, ad Copuig Allero, 07 55h Aual Allero Coferece o, pages IEEE, J. S. Rosehal. A Firs Look a Rigorous Probabiliy Theory. World Scieific Publishig Co. Pe. Ld., secod ediio, 006.