Least Favorable Compressed Sensing Problems for First Order Methods

Transcription

1 eas Favorable Copressed Sensing Probles for Firs Order Mehods Arian Maleki Deparen of Elecrical and Copuer Engineering Rice Universiy Richard Baraniuk Deparen of Elecrical and Copuer Engineering Rice Universiy Absrac Copressed sensing (CS) ais o exploi he copressibiliy of naural signals o reduce he nuber of saples needed for he accurae reconsrucion. Unlike he radiional Shannon-Nyquis sapling, he recovery algorihs in CS, such as he popular l iniizaion, are copuaionally expensive. Therefore, firs order ehods, such as ieraive sof hresholding (IST) and fas ieraive sof hresholding algorih (FISTA) have been explored exensively as cheap ehods for solving he l iniizaion. However, he heoreical aspecs of hese algorihs have been ainly sudied in he sandard fraework of convex opiizaion, called he deerinisic fraework here. In his paper, we firs show ha he deerinisic approach resuls in very pessiisic conclusions ha are no indicaive of he perforance of hese algorihs in CS. Several papers have considered he saisical convergence rae as an alernaive o he deerinisic view. However, he heoreical aspecs of his convergence rae have reained unexplored in he sparse recovery proble. We sudy his convergence rae for several sandard firs order ehods boh heoreically and epirically. We derive several hallark properies of he saisical convergence, including universaliy over he arix enseble and he leas favorable coefficien disribuion. These resuls ay help he researchers for choosing ore inforaive proble insances in heir siulaions. I. FIRST ORDER METHODS FOR ASSO A. The ASSO proble Consider a siple unconsrained convex opiizaion proble in f(x), () x R N where f is a differeniable funcion. Ieraive approaches are used o approxiae he opial poin when he exac soluion canno be calculaed explicily. Suppose ha he algorih is ieraive and our esiae of he opial poin x a ieraion is called x. Firs-order ehods use he gradien of he funcion a x 0, x,..., x o obain a new esiae a ie +. These ehods are very popular for very large scale probles, due o heir inexpensive ieraions. In his paper we are ineresed in solving he following proble: (P λ ) in x 2 Ax y λ x, where he arix A is drawn fro a rando enseble, e.g., iid aussian or Bernoulli. Wih a sligh abuse of noaion an algorih is called a firs order ehod for P λ if i only uses he gradien of he funcion 2 Ax y 2 2 a he previous ieraions. Using firs order ehods for solving he l -iniizaion proble has been an acive area of research in he las decade, wih new proposals appearing regularly [] [7]. In his paper we focus on hree sandard firs order algorihs and our goal is o characerize he difficul CS probles for hese hree algorihs. B. Ieraive sof hresholding (IST) Wihou loss of generaliy we assue ha all he algorihs sar fro x 0 = 0. The IST ieraion for solving P λ is given by x + = η(x + α A (y Ax ); α λ), where η is he sof hresholding funcion applied coponenwise o he eleens of he vecor. α is called he sep size and ensures he sabiliy of he algorih. For ore inforaion on he hisory of his algorih see [9]. C. Fas ieraive sof hresholding algorih (FISTA) Anoher firs order ehod ha has recenly drawn aenion is FISTA [0], [5]. This algorih uses he following ieraion: x + = η(z + α A (y Az ); λα ), z = x (x x ). Noe ha unlike IST, in addiion o x, x is used in obaining x +. D. Approxiae Message Passing (AMP) Now consider he linear easureen odel y = Ax o + w, where w is he noise of he syse and x o is he coprressible vecor. The ieraions of AMP for his proble are given by x + = η(x + A z ; τ ˆσ ), z = y Ax + I n z, where ˆσ is an esiae of he sandard deviaion of he he vecor x + A T z x o and I is he acive se of he vecor x a ie [3]. For ore inforaion on paraeer τ and is connecion o λ refer o Secion IV-A. Unlike he oher wo ehods AMP is derived fro he saisical fraework and herefore is ore adaped o CS probles. The sof hresholding funcion is defined as η(x; λ) = (x λ) + sign(x).

2 II. DRAWBACK OF THE DETERMINISTIC FRAMEWORK FOR CS To copare he perforance of differen firs order ehods for solving P λ, researchers have considered several differen approaches. The siples and os popular approach is he deerinisic fraework. In his fraework, we consider he following class of funcions, P n,n = { 2 y Ax 2 2+λ x : A R n N, A A 2,2 }. We use he noaion x a,f for he esiae of algorih a on he funcion f P n,n a ie. Furher, assue ha Xf is he se of all he poins ha achieve he iniu of f. Define d(x a,f, X f ) = inf x Xf x a,f x 2 2. In he deerinisic fraework we are ineresed in he perforance of an algorih on he leas favorable funcion in he class, i.e.,, a) = ax f F d(x a,f, Xf ),, a) = ax f F [f(x a,f ) f(x f )], where Xf is he se of iniizers of f and x f X f. In his analysis, we are ainly ineresed in he decay rae, a) and P E (P n,n, a) as grows. To see he ain drawback of he deerinisic fraework we consider he class A of all he firs order ehods and define he following wo iniax errors:, A) = in a A MSE (P n,n, a),, A) = in P a A E (P n,n, a). Theore II.. Consider he class of funcions P n,n. There exiss a consan C such ha for any given, n < N, and for any n 2,, A) C x 2 ( + ) 2,, A) 25 x 2 2. The proof of he above heore is siilar o he proof of Theore 2..7 [8] and herefore for he sake of breviy is skipped here. See [9] for deails. Since in CS we are ainly ineresed in very high diensional probles and we are ofen ineresed in he convergence of x o x, he deerinisic fraework is no indicaive of wha we observe in pracice; x of several firs order ehods, such as AMP, converges o x exreely fas. Due o his drawback, average case analysis and siulaions have played an iporan role in coparisons. In his approach we exploi he randoness of A and insead of calculaing x x o 2 2 on he wors possible case we work wih average case quaniy E( x x o 2 2). Majoriy of he papers published on he ASSO solvers use Mone Carlo siulaions o esiae and copare his quaniy. However, despie he exensive nuber of siulaions in he lieraure, our undersanding of he average convergence rae is very liied. This has led o ad-hoc proble selecion ehods and ad-hoc coparisons. Our goal in his paper is o sudy he properies of he saisical convergence rae on hree sandard firs order ehods FISTA, IST and AMP enioned in Secion I. We show which arix ensebles and which coefficien ensebles are ore difficul for hese algorihs. This provides a guideline for choosing difficul proble insances for each algorih. We will also see he ineresing properies of he consan apliude coefficien enseble ha akes i a difficul coefficien enseble for hese algorihs. III. MAIN CONTRIBUTIONS In his paper we consider he sparse recovery proble in he presence of noise, i.e., le y = Ax o +z, where z is iid aussian noise wih variance ν and x o is he signal o be recovered. We denoe a proble insance by Θ = (D A,, ɛ;, ν). D A represens he disribuion fro which he arix is drawn. is he probabiliy densiy funcion of non-zero eleens of x 2 o. ɛ is he probabiliy ha an eleen of x o is non-zero. In oher words, x oi ( ɛ) 0 (x oi ) + ɛ(x oi ), where x o,i is he i h eleen of he vecor x o. Finally = n/n where n is he nuber of easureens and N is he diension of he vecor x o. Suppose ha x is a sequence resuling fro one of he algorihs. We define he ean-ie-o-converge (α) = inf{ 0 : li N E x x o 2 2/E x o 2 2 α > 0 }. (α) depends on boh he proble insance and he algorih. I is also worh enioning ha if x does no converge o x o in he ean square sense, hen (α) will be infinie for α < li li N E x x o 2 2/E x o 2 2. Also according o [3] li N E x x o 2 2 for hese proble insances can converge o is final value exponenially fas. Definiion III.. The proble insance Θ is called less favorable han he proble insance Θ for IST, FISTA, and AMP algorih if and only if for every λ Θ (or τ Θ ) here exiss λ Θ (or τ Θ ) such ha for every α > 0, λθ (α) λθ (α). Definiion III.2. Two proble insances Θ and Θ are called equivalen if and only if Θ is less favorable han Θ and vice versa. Marix universaliy hypohesis: Suppose ha he eleens of n N easureen arix are chosen iid a rando fro a well-behaved probabiliy disribuion. 3 Furherore, he non-zero eleens of he vecor x o are sapled randoly fro a given disribuion. The observed behavior of E x x o 2 2/N for he FISTA algorih (or IST or AMP) will exhibi he sae behavior as he aussian enseble wih large N. In oher words, under he above assupions he 2 I can be replaced wih he disribuion funcion as well. However, for he sipliciy of noaion we assue ha probabiliy densiy funcion exiss. 3 We assue ha E(A ij ) = 0 and E(A 2 ij ) = n.

3 Fig.. Checking he arix universaliy hypohesis. Top-ef: logarih of he ean square error of FISTA for wo differen arix ensebles defined in Table I a N = 2000, =.5, λ =.00. Boo-ef: The p-values of he Z-es on he null hypohesis of equaliy of ean square errors. Top-Righ: logarih of he ean square error of IST for wo differen arix ensebles defined in Table I a N = 2000, =.5, λ =.. Boo-righ: The p-values. The large p-values confir ha we canno rejec he null hypohesis. proble insances (D A,, ɛ;, ν) and (N(0, /n),, ɛ;, ν) are equivalen. We used a vague saeen of well-behaved since he exac specificaions of he universaliy class are no known ye. Proving he above universaliy hypohesis is clearly ore difficul han he oher universaliy conjecures ha are sill open in CS [20]. Therefore, following [20] we use an saisical analysis o es he above hypohesis. We esed his hypohesis on several arix ensebles defined in Table I. For ore inforaion on our experienal se up and he saisical ess refer o [9]. Figure shows he resul of one of our experiens. TABE I MATRIX ENSEMBES CONSIDERED IN THE MATRIX UNIVERSAITY HYPOTHESIS TESTS. Nae Specificaion RSE iid eleens equally likely o be ± n USE iid eleens N(0, /n) TERN iid eleens equally likely o be 0, 3/2n, 3/2n TERN0P6 iid eleens aken values 0, 5/2n, 5/2n wih P (0) =.6 The oher iporan facor on he perforance of he algorihs is he disribuion of he inpu vecor. Firs assue ha we are considering a class of proble insances in which D A, ɛ,, ν are fixed and he only hing ha can change is. Call his class I(D A, ɛ;, ν). ea III.3. Fix D A,, ɛ;, ν and suppose E (X 2 ) all he proble insances of he for (D A, α (αµ), ɛ;, ν) resuling fro varying α 0 are equivalen for FISTA, IST, and AMP. The proof is very siple and skipped. Based on he above lea define he class I N (D A, ɛ;, ν) = {(D A,, ɛ;, ν) I(D A, ɛ;, ν) : E (X 2 ) = }. Theore III.4. = resuls in he leas favorable proble insance for AMP algorih on I N (D A, ɛ;, ν). This heore which is proved in Secion IV-B saes ha as far as he rae of convergence of he ean square error is concerned he consan apliude disribuion is he leas favorable. Theore III.5. If he leas favorable proble exiss for IST or FISTA, i will necessarily be = The above heore confirs ha he consan apliude enseble resuls in he larges values of (α) for a leas soe values of α. Bu i is no as srong as Theore III.4 ha we proved for AMP; i does no prove ha his is he leas favorable disribuion. However, he epirical observaions confir ha he leas favorable disribuion exiss for hese wo algorihs and herefore according o he above heore i is equal o Epirical Finding 2: Under he above assupions = is less favorable proble insance for FISTA and IST han he oher ensebles defined in able II. We believe ha he above finding holds on a very wide range of disribuions, alhough we have only checked i for a liied nuber of disribuions. TABE II COEFFICIENT ENSEMBES CONSIDERED IN COEFFICIENT ENSEMBE ANAYSIS EXPERIMENTS. Nae 3P 5P 5P2 Specificaion iid eleens aking value 0,, wih P (0) = ɛ iid eleens aking values 0, ±, ±5 wih P () = P ( ) =.3ɛ and P (5) = P ( 5) =.2ɛ iid eleens aking values 0, ±, ±20 wih P () = P ( ) =.3ɛ and P (20) = P ( 20) =.2ɛ U iid U[0, 4] iid N(0, 4) The final iporan facor ha we wan o consider is he sparsiy level ɛ. Fix all he oher paraeers and vary ɛ. Call he resuling se of proble insances I(D A, ;, ν). Theore III.6. If ɛ < ɛ, ɛ resuls in a less favorable proble insance for AMP algorih. The proof of his heore is given in Secion IV-C. The above heore foralizes he inuiion ha as we relax he sparsiy, he probles becoe ore difficul. Theore III.7. For a fixed value of, if ɛ < ɛ, ɛ can no be less favorable for FISTA and IST han ɛ. Clearly, he above heore does no prove ha ɛ is less favorable han ɛ for IST or FISTA. However, epirical observaions confir his.

4 Epirical Finding 3. For a fixed value of if ɛ < ɛ, F ɛ, is ore favorable for FISTA and IST han F ɛ,. For he sake of breviy we reove he experienal resuls ha led o he above epirical findings fro he paper and refer he reader o [9]. IV. PROOFS OF THE MAIN RESUTS A. AMP-ASSO calibraion Here we briefly suarize soe of he recen advances in predicing he asypoic perforance of he ASSO algorih ha helps us in our analysis. In [3] he auhors proposed he sae evoluion fraework as a heoreical fraework o predic he perforance of AMP algorih. According o his fraework, if is he ean square error a ieraion, he ean square error a ieraion + is + = Ψ( ) where Ψ( ) E(η(X + + νz; τ + ν) X)2. X ( ɛ) 0 (µ) + ɛ(µ) and Z N(0, ) are wo independen rando variables. Clearly SE predics he perforance of he algorih in he asypoic seing N. The final ean square error of he AMP algorih also corresponds o he sable fixed poins of he Ψ funcion. I is proved ha he Ψ funcion is concave and herefore i has jus one sable fixed poin. In addiion o MSE, oher observables of AMP algorih can be calculaed hrough he sae evoluion fraework such as, equilibriu hreshold (θ = τ + ν) and equilibriu deecion rae (EqDR = P{ η(y ; θ 0}, where Y = X+ + νz). represens he fixed poin of Ψ funcion. The final coponen ha will be used in our arguens is he equivalence beween ASSO and AMP soluions. The following finding aken fro [2], which has been recenly proved in case of aussian easureen arices in [22], explains he equivalence. For ore inforaion on he deails of he condiions necessary for hese heores refer o [2] and [22]. Theore IV.. [2], [22] For each λ [0, ) we find ha AMP(τ(λ)) and ASSO(λ) have saisically equivalen observables. In paricular he MSE have he sae value when τ and λ saisfy he following relaion: λ = θ (τ)( EqDR(τ)/). ea IV.2. [2] Define τ 0, so ha EqDR(τ) when τ > τ 0. For each λ here is a unique value of τ(λ) [τ 0, ) such ha λ = θ (τ)( EqDR(τ)/). The above discussion is ainly used in he calibraion of λ for wo differen disribuions. B. Coefficien disribuion Proof of Theore III.4: Suppose ha an arbirary disribuion is given and le E (X 2 ) =. Define ɛ (µ) ɛ(µ) + ( ɛ) 0 (µ) and F ɛ, ɛ 2 (µ) + ɛ 2 (µ) + ( ɛ) 0 (µ). Our goal is o prove ha F ɛ, is less favorable han ɛ. For a given τ in AMP for F ɛ,, we choose τ = τ. e be he ean square error on ɛ proble insance a ie and F represen he sae hing for F ɛ,. Suppose ha MSE MSE F a ie, our goal is o firs prove ha + + F. We have + = E ɛ E X (η(x + + νz; τ Also le us define, + F = E Fɛ, E X (η(x + Our clai is ha, + + νz; τ + F + F. The second inequaliy is a siple resul of he fac ha he ean square error is non-decreasing funcion of sandard deviaion of he noise. The firs inequaliy is he resul of Jensen in- MSE equaliy, since E X (η(x + MSE + νz; τ + ν) X) is a concave funcion of X 2. The proof of he ain heore is now a very siple inducion. Since he ean square error for he wo disribuions are he sae a ieraion one. The following lea plays an iporan role in our discussion of Theore III.5. ea IV.3. Consider he disribuion wih E (X 2 ) =. For every λ 0 and for any ν > 0 if li li N N E F ɛ, E xo ˆx λ x o 2 2 ɛ, hen here exiss a corresponding λ such ha, li li N = li N E E xo ˆx λ x o 2 2 li N N E F ɛ, E xo ˆx λ x o 2 2, Proof: According o ea IV.2, here exis a value of τ for which he asypoic ean square error of he sae evoluion fraeworks is he sae as he asypoic ean square error of ASSO(λ). Here is our approach for solving his lea. The firs hing ha we wan o prove is ha here exiss a value of τ for which he asypoic ean square error of AMP(τ) on is below he MSE of AMP(τ) on F ɛ,. Then we prove here exiss a value of τ for which he asypoic ean square error of AMP(τ) is larger han he ean square error of AMP(τ) on F ɛ,. Finally we will use iplici funcion heore o prove ha here exiss a value of τ for which he asypoic ean square error of AMP(τ ) on is exacly he sae as he MSE of AMP(τ) on F ɛ,. Choice τ = is clear. Therefore we focus on consrucing a choice for τ. The clai is ha τ = τ. This is due o he concaviy of E X (η(x + /Z; β) X) 2 in ers of X 2 and he Jensen inequaliy. Therefore for he sake of breviy we skip o wrie he coplee arguen here. The ineresing fac abou he above proof is ha i is also consrucive. For a given value of λ on he proble insance (D A, 2 (µ) + 2 µ, ɛ;, ν) we can calculae λ ha gives

5 us he sae ean square error. This is he value ha has been used for deriving he epirical findings. Noe: There igh be ore han one value of λ ha generaes he sae ean square error on F ɛ, and here ay be ore han one value of λ wih he sae ean square error. In hese cases we copare he fases achievable raes for each case. Proof of Theore III.5: The proof of his heore is also clear fro our discussion of he las wo heores. To prove his heore we focus on he final ean square error of he algorihs and using he equivalence fraework we can prove ha he ean square error of he algorih is highes when he disribuion is 2 (µ) + 2 (µ). For ore deails on he proof, refer o [9]. C. Sparsiy level To siplify he noaion in his secion we ainly focus on = 2 (µ) + 2 (µ). However he resuls can be easily exended o oher disribuions as well. ea IV.4. For he risk of he sof hresholding funcion defined as r s (µ, τ; σ) = E(η(µ + σz; τσ) µ) 2, we have d dµ r s(µ, τ; σ) = 2µ τ+µ/σ τ µ/σ φ Z (z)dz. ea IV.5. Consider he following risk funcion, R(ɛ) = E X Fɛ,/ ɛ E X (η(x + σz; τσ) X) 2 ; R(ɛ) is an increasing funcion of ɛ. See [9] for he proof of he above wo leas. Proof of Theore III.6 Proof: According o ea III.3, F ɛ, is equivalen o F ɛ,/ ɛ and F ɛ, is equivalen o F ɛ,/ ɛ. Therefore we copare hese wo disribuions. We use he aheaical inducion o prove he above heore. e ɛ be he ean squared error of AMP(τ) a ieraion on ɛ. Suppose ha ɛ > ɛ and he goal is o prove ha + ɛ > +. Define + ɛ = E Fɛ, ɛ E X (η(x + ɛ + νz; τ ɛ We clai + ɛ > + ɛ > + ɛ. The firs inequaliy is due o he fac ha he ean square error is a non-decreasing funcion of he sandard deviaion of he noise. The second inequaliy however is he resul of ea IV.5. The base of he inducion is correc since boh algorihs sar wih he sae ean square error and herefore he proof is coplee. V. CONCUSION We sudied he saisical convergence rae of hree sandard firs order ehods. By epirical sudy of hese raes we showed ha a class of easureen arices are equivalen. Also, we showed ha he consan apliude disribuion is he ɛ leas favorable disribuion for FISTA and IST. Our heoreical resuls also confired ha he leas favorable disribuion exiss for he AMP algorih and is equal o he consan apliude. I also showed ha less sparse signals under he sae non-zero disribuion are less favorable. ACKNOWEDMENT The auhors would like o hank David. Donoho, Andrea Monanari, and Rahul Mazuder for helpful discussions. REFERENCES [] I. Daubechies, M. Defrise, and C. De Mol. An ieraive hresholding algorih for linear inverse probles wih a sparsiy consrain. Co. on Pure and Applied Mah., 75:42 457, [2] P.. Cobees and V. R. Wajs. Signal recovery by proxial forwardbackward spliing. Muliscale Model. and Si., 4(4):68 200, [3] W. Yin, S. Osher, D. oldfarb, and J. Darbon. Bregan ieraive algorihs for l -iniizaion wih applicaions o copressed sensing. SIAM J. on Iag. Sci., ():43 68, [4] M. Figueiredo, R. Nowak, and S. Wrigh. radien projecion for sparse reconsrucion: Applicaion o copressed sensing and oher inverse probles. IEEE J. of Sel. Topics of Sig. Proc., (4): , [5] M. Figueiredo and R. Nowak. An EM algorih for wavele-based iage resoraion. IEEE Trans. on Iag. Proc., 2(8):906 96, [6] M. Figueiredo, J. Bioucas-Dias, and R. Nowak. Majorizaioniniizaion algorihs for wavele-based iage resoraion. IEEE Trans. on Iag. Proc., 6(2): , [7] S. Wrigh and M. Figueiredo R. Nowak. Sparse reconsrucion by separable approxiaion. IEEE Trans. on Sig. Proc., 57(7): , [8] M. Elad, B. Maalon, J. Shok, and M. Zibulevsky. A wide-angle view a ieraed shrinkage algorihs. Proc. SPIE (Wavele XII), Augus [9] E. Hale, W. Yin, and Y. Zhang. Fixed poin coninuaion ehod for l iniizaion wih applicaion o copressed sensing. Rice Universiy Technial Repor TR07-07, [0] A. Beck and M. Teboulle. A fas ieraive shrinkage hresholding algorih for linear inverse probles. SIAM J. on Iag. Sci., 2():83 202, [] J. Bioucas-Dias and M. Figueiredo. A new TwIST: Two-sep ieraive shrinkage/hresholding algorihs for iage resoraion. IEEE Trans. on Iag. Proc., 6: , [2] A. Maleki. Coherence analysis of ieraive hresholding algorihs. Preprin, 200. [3] D.. Donoho, A. Maleki, and A. Monanari. Message passing algorihs for copressed sensing. Proc. of Na. Acad. of Sci., 06(45): , [4] A. Maleki and D.. Donoho. Opially uned ieraive hresholding algorih for copressed sensing. IEEE J. of Sel. Areas in Sig. Proc., April 200. [5] Y. Neserov. radien ehods for iniizing coposie objecive funcion. CORE Repor, [6] S. Becker, J. Bobin, and E. Candès. Nesa: a fas and accurae firs-order ehod for sparse recovery subied for publicaion. [7] K. Bredies and D. orenz. inear convergence of ieraive sofhresholding. J. of Four. Anal. and App., 4:83 837, [8] Y. Neserov. Inroducory lecures on convex opiizaion, volue 87. Kluwer Acadeic Publishers, [9] A. Maleki and R. Barraniuk. Difficul copressed sensing proble for firs order ehods. Preprein. hp:// arian/isi.pdf. [20] D.. Donoho and J. Tanner. Observed universaliy of phase ransiions in high-diensional geoery, wih applicaions in odern signal processing and daa analysis. Phil. Trans. of Roy. Soc. A, 367(906): , [2] D.. Donoho, A. Maleki, and A. Monanari. Noise sensiiviy phase ransiion. IEEE Transacions on Inforaion Theory, 200. subied. [22] M. Bayai and A. Monanri. The dynaics of essage passing on dense graphs, wih applicaions o copressed sensing. Preprin, 200.