Recovery Guarantees for Rank Aware Pursuits

Size: px

Start display at page:

Download "Recovery Guarantees for Rank Aware Pursuits"

Ashlynn Oliver
5 years ago
Views:

1 BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS 1 Recovey Guaantees fo Rank Awae Pusuits Jeffey D. Blanchad and Mike E. Davies Abstact This pape consides sufficient conditions fo spase ecovey in the spase multiple measuement vecto (MMV) poblem fo some ecently poposed ank awae geedy algoithms. Specifically we conside the compessed sensing famewok with Gaussian andom measuement matices and show that the ank of the measuement matix in the noiseless spase MMV poblem allows such algoithms to educe the effect of the log n tem that is pesent in taditional OMP ecovey. Index Tems Multiple Measuement Vectos, Geedy Algoithm, Othogonal Matching Pusuit, ank. I. INTRODUCTION Spase signal epesentations povide a geneal signal model that make it possible to solve many ill-posed poblems such as souce sepaation, denoising and most ecently compessed sensing [1] by exploiting the additional spasity constaint. While the geneal poblem of finding the spasest x R n given an obsevation vecto y = Φx, y R m is known to be NP-had [] a numbe of suboptimal stategies have been shown to be able to ecove k-spase signals, x, when m Ck log(n/k) fo some constant C, if Φ is chosen judiciously. An inteesting extension of the spase ecovey poblem is the spase multiple measuement vecto (MMV) poblem, Y = ΦX, Y R m l, X R n l, which has also eceived much attention, e.g. [3], [], [5]. Initially the algoithms poposed fo this poblem wee staightfowad extensions of existing single measuement vecto (SMV) solutions. Howeve, most of these ae unable to exploit the additional infomation available though the ank of Y. In contast, some new geedy algoithms fo joint spase ecovey have been poposed [], [7], [] based aound the MUSIC (MUltiple SIgnal Classification) algoithm [9] fom aay signal pocessing which povides optimal ecovey in the maximal ank scenaio = k []. The aim of this pape is to analyse the ecovey pefomance of two Rank Awae algoithms when the obsevation matix does not have maximal ank, ank(y) < k. Ou appoach follows the ecovey analysis of [11] whee it was shown that, in the noiseless case, Othogonal Matching Pusuit (OMP) can ecove k-spase vectos fom m Ck log n Gaussian measuements with high pobability. We extend this analysis to the MMV spase ecovey poblem and show joint k-spase matices, X, can be ecoveed fom m Ck( 1 log n + 1) MMVs using a ank awae algoithm. 1 This implies that the log n penalty tem obseved fo OMP ecovey can be essentially emoved with vey modest values of ank, log n. II. NOTATION AND PROBLEM FORMULATION We define the suppot of a collection of vectos X = [x 1,..., x l ] as the union ove all the individual suppots: supp(x) := i supp(x i). A matix X is called k joint spase if supp(x) k. We make use of the subscipt notation Φ Ω to denote a submatix composed of the columns of Φ that ae indexed in the set Ω, while the notation X Ω,: denotes a ow-wise submatix composed of the ows of X indexed by Ω. Thus denoting by Ω the cadinality of Ω, the matix X Ω,: is Ω -spase. Copyight (c) 01 IEEE. Pesonal use of this mateial is pemitted. Howeve, pemission to use this mateial fo any othe puposes must be obtained fom the IEEE by sending a equest to pubs-pemissions@ieee.og. J.D. Blanchad is with the Depatment of Mathematics and Statistics, Ginnell College, Ginnell, Iowa USA. T: , F: (jeff@math.ginnell.edu). JDB was suppoted as an NSF Intenational Reseach Fellow (NSF OISE 05991) and by NSF DMS M.E. Davies is with IDCOM, the Univesity of Edinbugh, Edinbugh, UK, and the Joint Reseach Institute fo Signal and Image Pocessing, pat of the Edinbugh Reseach Patneship in Engineeing and Mathematics. (m.davies@ed.ac.uk). This wok is was suppoted in pat by EPSRC gant EP/F03997/1 and the Euopean Commission though the SMALL poject unde FET-Open, gant numbe These esults wee peviously announced at the SMALL wokshop, London, Januay 011 [1].

2 BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS We can now fomally define the spase MMV poblem. Conside the obsevation matix Y = ΦX, Y R m l whee Φ R m n with m < n is the dictionay matix and X R n l is assumed to be jointly k-spase. The task is then to ecove X fom Y given Φ. We will futhe assume that ank(y) = and without loss of geneality that = l. III. GREEDY MMV ALGORITHMS Despite the fact that the ank of the obsevation matix Y can be exploited to impove ecovey pefomance, to date most popula techniques have ignoed this fact and have been shown to be ank-blind []. In contast, a discete vesion of the MUSIC algoithm [], [13] is able to ecove X fom Y unde mild conditions on Φ wheneve m k + 1 if we ae in the maximal ank case, i.e. ank(y) = k. While MUSIC povides guaanteed ecovey fo the MMV poblem in the maximal ank case thee ae no pefomance guaantees fo when ank(x) < k and empiically MUSIC does not pefom well in this scenaio. This motivated a numbe of woks [], [7], [] to investigate the possibility of an algoithm that in some way intepolates between a classical geedy algoithm fo the SMV poblem and MUSIC when ank(x) = k. The appoach poposed in [7], [] was to use a geedy selection algoithm to find the fist t = k coefficients. The emaining components can then be found by applying MUSIC to an augmented data matix [Y, Φ (t) Ω ] which unde identifiability assumptions will span the ange of Φ Λ. In [] two ank awae (RA) algoithms wee pesented. In RA-OMP the geedy selection step was modified to measue the distance of the columns of Φ fom the subspace spanned by the esidual matix at iteation t by measuing the coelation of columns of Φ with an othonomal basis of the esidual matix: U (j 1) = otho(r (j 1) ). Howeve, the ecovey pefomance was shown to deteioate even in the maximal ank scenaio as the algoithm selected moe coefficients. To compensate fo this, the column nomalization used in Ode Recusive Matching Pusuit (ORMP) was included. Specifically at the stat of the tth iteation, if we have a selected suppot set Ω (t), a new column Φ i is then chosen based upon the following selection ule: i (t) = ag max i ϕ T i U(t) P Ω (t) ϕ i, (1) whee P Ω denotes the othogonal pojection onto the null space of Φ (t) Ω (t). The ighthand side of (1) measues the distance of the nomalized vecto P Ω ϕ (t) i / P Ω ϕ (t) i fom the subspace spanned by U (t). This ensues that coect selection is maintained at each iteation in the maximal ank scenaio. The full desciption of the RA-OMP and RA-ORMP ae summaized in Algoithm 1. In the next section the ecovey guaantees fo RA-ORMP and RA-OMP+SA-MUSIC (using RA-OMP to select the fist k indices followed by the subspace augmented SA-MUSIC of [7], []) ae examined and shown to exploit the ank of Y vey effectively. RA-OMP+SA-MUSIC is vey simila to the appoach consideed in [7]. Howeve in [7] only a single othogonalization of Y was pefomed followed by simultaneous OMP (SOMP). [] consideed SOMP+SA-MUSIC but without an initial othogonalization. 3 Both theoetical and empiical ecovey pefomance of SOMP+SA-MUSIC is limited due to the ank-blind popety of SOMP []. IV. SPARSE MMV RECOVERY BY RA-OMP Coect selection by the RA-OMP algoithm at the jth iteation is chaacteized by the following quantity. Definition 1 (Geedy Selection Ratio fo RA-OMP). In iteation j of RA-OMP, let Λ = supp(x) and define the geedy selection atio fo RA-OMP as ρ = max i Λ c ϕt i U max i Λ ϕ T i U. () In pactice, significant complications aise in the noisy case whee the estimation of the ank and signal subspace is non-tivial. Fo an analysis of elated algoithms in the noisy case, see [1]. 3 While witing up this wok we became awae of an updated vesion of [] whee the authos have switched to consideing the RA- OMP+SA-MUSIC poposed hee instead of SOMP+SA-MUSIC. The updated vesion also analyses the pesence of noise in an asympotic setting allowing the poblem size to tend to.

3 BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS 3 Algoithm 1 RA-OMP / RA-ORMP 1: initialization: R (0) = Y, X (0) = 0, Ω 0 = : fo j = 1; j := j + 1 until stopping citeion do 3: Calculate U (j 1) = otho(r (j 1) ), : if (RA-OMP) then 5: i = ag max i ϕ T i U(j 1) : else if (RA-ORMP) then 7: i = ag max i Ω (j 1) ϕ T i U(j 1) / P Ω ϕ (j 1) i : end if 9: Ω = Ω (j 1) i : X Ω,: = Φ Y Ω 11: R = Y ΦX 1: end fo The following obsevation is obvious. Lemma 1. In iteation j, RA-OMP will coectly select an atom ϕ j, j Λ = supp(x) if and only if ρ < 1. To bound ρ we intoduce the following lemmas. Lemma. Suppose U R m with columns U i ange(φ Λ ) and U i = 1 fo all i. Let α = σ min (Φ Λ ) be the smallest singula value of Φ Λ with Λ = k < m. Then max i Λ ϕt i U α k. (3) Poof: We can wite ϕ T i U = e T i Φ T ΛU () whee e i is the standad (diac) basis in R k. Define Ũ = ΦT Λ U. Since U i is in the ange of Φ Λ we have the following bound: Hence Ũi σ min (Φ Λ ) U i = α (5) max i Λ ϕt i U = max i Λ et i Ũ = max i Λ mean i = 1 k [Ũ] i,l = 1 k k Ũl k α l=1 [Ũ] i,l l=1 i=1 l=1 [Ũ] i,l whee we have bounded the maximum by the mean and then swapped the ode of summation. Lemma 3. If Φ R m n with enties daw i.i.d. fom N (0, m 1 ), Λ 1,..., n} is an index set with Λ = k, and U R m is a matix with othonomal columns, ank(u) = and with span(u) span(φ Λ ), then P max i Λ c ϕ T i U < µ } 1 (n k)e (mµ )/. (7) Poof: Let z = ϕ T i U then z R and fo i Λ the enties in z follow the nomal distibution N (0, m 1 ). We can now use the Laplace tansfom method [15] to bound z. P z µ } e λµ E } e λ z ( ) () = e λµ + ln m m λ ()

4 BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS fo any λ > 0. Selecting λ = m/ gives P( z µ ) e (mµ )/. (9) Applying the union bound completes the esult. We will also equie that the esidual matix, R, etains geneic ank (equivalent to the ow non-degeneancy condition in [7]) which is given by: Lemma. Let Φ R m n with enties daw i.i.d. fom N (0, m 1 ) and X be a joint spase matix with suppot Λ 1,..., n}, Λ = k, such that X Λ,: is in geneal postion. If Ω Λ fo j k, then ank(r ) =. Poof: Note that: R = P Ω Y = P Ω Φ Λ Ω X Λ Ω,:. () Since X Λ,: is in geneal position ank(x Λ Ω,:) = min, k j} =, and since Φ is i.i.d. Gaussian then P Ω Φ Λ Ω will have maximal ank with pobability 1. Theefoe ank(r ) =. We can combine the above lemmata to give: Lemma 5. Suppose that afte j < k iteations of RA-OMP, Ω Λ = supp(x) with Λ = k. Define R = Y ΦX and assume X is in geneal position and Φ R m n with enties dawn i.i.d. fom N (0, m 1 ). Then, fo α = σ min (Φ Λ ), P Ω (j+1) Λ } 1 (n k)e (mα k )/. (11) Poof: Let U = otho(r ) then fom Lem. we have ank(u ) = ank(r ) = ank(x) =. Now, Lem. 1 and the assumption that Ω Λ allow us to ewite the pobability statement as P Ω (j+1) Λ } = P ρ(j + 1) < 1 }. Lemmas and 3 with µ = α /k combine to show that P ρ(j + 1) < 1 } = P max j Λ c ϕ T j U < max j Λ ϕt j U } P max ϕ T j j Λ U < α } c k 1 (n k)e (mα k )/. We can now state ou main theoem fo RA-OMP. Theoem (RA-OMP + SA-MUSIC ecovey). Assume X R n, supp(x) = Λ, Λ = k > with X Λ in geneal position and let Φ be a andom matix, independent of X, with i.i.d. enties Φ i,j N (0, m 1 ). Then, fo some C and with pobability geate than 1 δ, RA-OMP + SA-MUSIC will ecove X fom Y = ΦX if: m Ck ( 1 log(n/ δ) + 1 ). (1) Poof: It is sufficient to bound the pobability of making q k successive coect selections afte which SA-MUSIC [7] is guaanteed to ecove the emaining coefficients [7], []. Suppose σ min (Φ Λ ) = α, then Pmax ρ(t) < 1} Pρ(t) < 1} t q t q [1 ] q (n k)e (mα /k )/ (13) 1 q(n k)e (mα /k )/.

5 BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS 5 Now ecall [11] that Pσ min (Φ Λ ) > 1/ } 1 e cm fo some c > 0. Hence: ( Pmax ρ(t) < 1} 1 n e (m )/) ( k 1 e cm) t q 1 n e (m/k )/ e cm 1 n e C(m/k ) (1) whee we have used the fact that q(n k) < n. Now choosing δ n e C(m/k ) and eaanging gives (1). V. SPARSE MMV RECOVERY BY RA-ORMP As RA-OMP and RA-ORMP only diffe in the selection step we can use simila aguments to above, with additional contol of the nomalization tem P Ω ϕ i given by the following. Lemma 7. If Φ R m n with i.i.d. enties Φ i,j N (0, m 1 ), Λ 1,..., n}, Λ = k, and Ω Λ, Ω = j, then fo i Ω and m j we have: P PΩ ϕ i 1 } 1 e m/3 (15) Poof: Since PΩ and ϕ i ae independent, z := PΩ ϕ i is a Gaussian andom vecto within the m j dimensional subspace Null(Φ T Ω ) whose enties have vaiance m 1. Hence we can use the following concentation of measue bound [15]: P z (1 ɛ) m j } 1 e ɛ (m j)/ (1) m Selecting ɛ = 1/ and noting that by assumption m j m/ gives the equied esult. Lemma. If Φ R m n with i.i.d. enties Φ i,j N (0, m 1 ), Λ 1,..., n}, Λ = k, and Ω Λ, Ω = j, then fo i Ω and m j we have: } P PΩ ϕ i 1 e m/1 (17) Poof: Standad Gaussian bounds [15] fo ϕ i give: P ϕ i (1 ɛ) 1} 1 e ɛ m/ Selecting ɛ = 1/ and using P Ω ϕ i < ϕ i gives the equied esult. Theoem 9 (RA-ORMP ecovey). Assume X R n, supp(x) = Λ, Λ = k > with X Λ in geneal position and let Φ be a andom matix, independent of X, with i.i.d. enties Φ i,j N (0, m 1 ). Then, fo some C and with pobability geate than 1 δ, RA-ORMP will ecove X fom Y = ΦX if m satisfies (1). Poof: We fist note that if Ω Λ, j < k and α = σ min (Φ Λ ), then } P Ω (j+1) Λ ϕ T i = P max U i Λ c P ϕ Ω i max i Λ ϕ T i U } P ϕ Ω i. Now using the bounds fom Lemma 3 with µ = α /k and Lemmas 7 and along with the union bound gives: P Ω (j+1) Λ } ( 1 (n k)e (mα k )/) ( 1 (n k)e m/3)( 1 ke m/1) ( 1 (n k) e (mα )/ k + e m/3 + e m/1) 1 (n k)e C(mα k ) fo an appopiately choosen C. To complete the poof fo the coect selection of Ω fo all j < k we can again apply the union bound and emove the dependence on α as in (13) and (1) above. We leave the details to (1) (19) (0)

6 BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS the eade. Fo the selection of the emaining coefficients we note that fo j k the oiginal RA-ORMP task is equivalent to solving R = [P Ω Φ Λ Ω ]X Λ Ω,: using RA-ORMP. Howeve by ou assumptions on X, ank(x Λ Ω,:) = = Λ Ω theefoe we ae in the maximal ank scenaio and fom [] we have guaanteed ecovey by RA- ORMP. VI. NUMERICAL RESULTS Hee we demonstate empiically that the (log n)/ tem in ou ecovey esult appeas to accuately captue the effect of ank on the ecovey pefomance. To this end we pefomed a numbe of expeiments using Gaussian andom matices fo both Φ and X Λ. The paametes m and k wee held fixed, fist at m = 3k/ = 30 and then with m = k = 0. We then vaied the numbe of channels of X fom = 1,..., 15 and n in powes of fom to 09. Fo each set of k,, m, n} we pefomed 0 tials and calculated the empiical pobability of ecovey. The included simulations ae esticted the noiseless case; the algoithms demonstate obustness to noise simila to that obseved in [], [7], []. Figue 1 shows the ecovey plots fo the ecovey algoithms RA-OMP + SA-MUSIC and RA-ORMP. In each case the phase tansition appeas to exhibit an appoximate linea dependency between and log n as highlighted by the ed lines (note the lines ae constained to pass though the oigin). RA OMP+MUSIC Recovey RA ORMP Recovey log n log n RA OMP+MUSIC Recovey RA ORMP Recovey log n log n (a) Fig. 1. Spase MMV ecovey plots showing the phase tansitions fo RA-OMP+SA-MUSIC (a) and RA-ORMP (b) with m = 3k/ = 30 (top) and m = k = 0 (bottom) while vaying the size of the dictionay n =, 1,..., 09 and numbe of channels, = 1,,..., 15. The ed line indicates a linea elation between and log n. (b) VII. CONCLUSION Ou theoetical esults pedict that the ank of the coefficient matix in the noiseless spase MMV ecovey poblem can be successfully exploited in RA-OMP+SA-MUSIC and RA-ORMP to enable joint spase ecovey when m Ck((log n)/ + 1) using a Gaussian measuement matix, although no attempt has been made to optimize the constant C; the analysis in the poof of Thm. 9 leads to a moe pessimistic constant than in Thm.. This emoves the log n penalty tem that is obseved in OMP when thee is only a modest numbe of mutliple measuement vectos log n. Numeical expeiments suggest that this fom may easonably chaacteize the ecovey behaviou in pactice. Empiically the RA-ORMP algoithm appeas to pefom slightly bette than RA- OMP+SA-MUSIC. Howeve this comes with an additional computational expense.

7 BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS 7 REFERENCES [1] D. Donoho, Compessed sensing, IEEE Tans. on Infomation Theoy, vol. 5, no., pp , 00. [] B. K. Nataajan, Spase appoximate solutions to linea systems, SIAM Jounal on Computing, vol., no., pp. 7 3, Ap [3] S. F. Cotte, B. D. Rao, K. Engan, and K. K-Delgado, Spase solutions to linea invese poblems with multiple measuement vectos, IEEE Tansactions on Signal Pocessing, vol. 53, no. 7, pp. 77, 005. [] J. Chen and X. Huo, Theoetical esults on spase epesentations of multiple-measuement vectos, IEEE Tansactions on Signal Pocessing, vol. 5, no. 1, pp. 3 3, 00. [5] J. A. Topp, A. C. Gilbet, and M. J. Stauss, Algoithms fo simultaneous spase appoximation. Pat I: Geedy pusuit, Signal Pocessing, vol., pp. 57 5, 00. [] M. E. Davies and Y. C. Elda, Rank awaeness and in joint spase ecovey, IEEE Tans. Infomation Theoy, vol. to appea, 011. [7] K. Lee and Y. Besle, imusic: Iteative music algoithm fo joint spase ecovey with any ank, 0, axiv pepint: axiv:0.3071v1. [] J. Kim, O. Lee, and J. C. Ye, Compessive music: A missing link between compessive sensing and aay signal pocessing, axiv pepint, axiv:0.39v1. [9] R. O. Schmidt, Multiple emitte location and signal paamete estimation, Poceedings of RADC Spectal Estimation Wokshop, pp. 3 5, [] P. Feng, Univesal minimum-ate sampling and spectum-blind econstuction fo multiband signals, Ph.D. dissetation, Univesity of Illinois, 199. [11] J. A. Topp and A. C. Gilbet, Signal ecovey fom andom measuements via othogonal matching pusuit, IEEE Tansactions on Infomation Theoy, vol. 53(1), no. 1, pp. 55, 007. [1] M. E. Davies. (011, Jan.) Rank awae algoithms fo joint spase ecovey. SMALL Wokshop on spase dictionay leaning. [Online]. Available: [13] P. Feng and Y. Besle, Spectum-blind minimum-ate sampling and econstuction of multiband signals, in Poceedings of IEEE Intenational Confeence on the Acoustics, Speech, and Signal Pocessing, 199, pp [1] K. Lee, Y. Besle, and M. Junge, Subspace methods fo joint spase ecovey, 011, axiv pepint: axiv:0.3071v. [15] A. Bavinok. (005) Measue concentation lectue notes. [Online]. Available: bavinok/total7.pdf

Recovery Guarantees for Rank Aware Pursuits

BLANCHARD AND DAVIES: RECOVERY GUARANTEES FOR RANK AWARE PURSUITS 1 Recovery Guarantees for Rank Aware Pursuits Jeffrey D. Blanchard and Mike E. Davies Abstract This paper considers sufficient conditions