New Restricted Isometry Results for Noisy Low-rank Recovery

Transcription

1 New Resticted Isomety Results fo Noisy Low-ank Recovey Kathik Mohan and Mayam Fazel Abstact he poblem of ecoveing a low-ank matix consistent with noisy linea measuements is a fundamental poblem with applications in machine leaning, statistics, and contol. Reweighted tace minimization, which extends and impoves upon the popula nuclea nom heuistic, has been used as an iteative heuistic fo this poblem. In this pape, we pesent theoetical guaantees fo the eweighted tace heuistic. We quantify its impovement ove nuclea nom minimization by poving tighte bounds on the ecovey eo fo low-ank matices with noisy measuements. Ou analysis is based on the Resticted Isomety Popety (RIP) and extends some ecent esults fom Compessed Sensing. As a second contibution, we impove the existing RIP ecovey esults fo the nuclea nom heuistic, and show that ecovey happens unde a weake assumption on the RIP constants. I. INRODUCION he noisy affine ank minimization poblem aims to find the lowest ank matix consistent with noisy linea measuements, minimize ank(x) () subject to A(X) b ɛ, whee X R m n is the vaiable, A : R m n R p is a linea map, b = A(X 0 )+e denotes the noisy measuements with e ɛ. X 0 R m n is the matix we aim to ecove. We denote its singula value decomposition by X 0 = UΣV = X 0, +(X 0 X 0, ), whee X 0, is fomed by tuncating the SVD afte tems. his poblem (and its vaiations) have many applications including collaboative filteing, quantum tomogaphy, system identification, and Euclidean embedding (see e.g. [9], [6] and efeences theein). When X is a diagonal matix, poblem () educes to the classical poblem of compessed sensing, whee the goal is to ecove a spase vecto. Many appoaches fo poblem () have been poposed using this analogy, including the nuclea nom heuistic [0] (analogous to l minimization), the eweighted tace heuistic [] (analogous to eweighted l ), and SV [], as well as altenative methods not based on nom minimization such as ADMiRA [4], and SVP [5]. he Nuclea Nom Heuistic (NNH) has been paticulaly popula and has been extensively studied fom both theoetical and algoithmic pespectives. his heuistic eplaces ank in the objective of () with the nuclea nom (also known as the Schatten -nom o tace nom) of the matix, denoted by X = min{m,n} i σ i (X) whee σ i (X) ae the singula values. NNH solves the convex poblem minimize X subject to A(X) b ɛ. Electical Engineeing Depatment, Univesity of Washington, Seattle. kana@u.washington.edu, mfazel@u.washington.edu. Reseach funded in pat by NSF CAREER gant ECCS () his heuistic is impoved upon by the Reweighted ace Heuistic (RH) [], discussed in section III, which uses a weighted objective with iteative weight updates. RH can also be intepeted as locally minimizing a smooth concave function, logaithm of the deteminant of the matix, instead of its ank. Both RH and its vecto analog, eweighted l minimization [8], empiically show bette ecovey popeties than NNH o l minimization (see [8], [], [7]). Recently, analytical esults fo the eweighted l heuistic wee given by Needell in [8]. Howeve, no theoetical guaantees on the pefomance of the RH have been available. In this pape, we pesent the fist theoetical guaantees fo the RH, in the case whee the matix vaiable is positive semidefinite and the map A satisfies the Resticted Isomety Popety (RIP). Extending the appoach in [8], we quantify the impovement of RH ove nuclea nom minimization by poving tighte bounds on the ecovey eo fo low-ank matices with noisy measuements. As anothe contibution, we extend ecent RIP esults fo l minimization [], [4], [] to the NNH, and show that ecovey happens unde a weake assumption on the RIP constants. he weake condition is useful in ou analysis of the RH, and may also be of independent inteest. Recent esults [6] show andom maps satisfying RIP guaantee ecovey with the least possible numbe of measuements, O(max{m, n}). hese maps also yield vey tight (in an oacle sense) eo bounds fo noisy ecovey. Futhemoe, even in applications whee the RIP is not satisfied such as Matix Completion, esticted vesions of it (e.g.,[5]) have poven useful. hese esults encouage the use of RIP as an analytical tool in cetain contexts such as analysis of noisy ecovey. It may also lead to moe applications that use andom ensembles. he pape is oganized as follows. he impoved RIP conditions ae deived in Section II, and ae used in the analysis of the RH given in Section III. In Section IV, we put ou contibutions in pespective and discuss some possible extensions of this wok. II. RIP RESUL FOR HE NUCLEAR NORM HEURISIC In this section, we give an RIP-based ecovey esult fo the Nuclea Nom Heuistic (NNH), extending the impovements fom compessed sensing ([],[4]). We begin with a few definitions. Definition II.. he -esticted isomety constant δ of a linea opeato, A : R m n R p is the smallest constant fo which ( δ ) X F A(X) ( + δ ) X F () holds fo all matices X of ank at most. U.S. Govenment Wok Not Potected by U.S. Copyight 57 ISI 00

2 Definition II.. Let t =min(m, n). If + t, the, - esticted othogonality constant θ, of a linea opeato A : R m n R p is the smallest constant fo which A(X), A(X ) θ, X F X F (4) holds fo all matices X of ank at most and all matices X of ank at most, whee X, [ X ae][ such that] if the Σ 0 V SVD of X is X = [U U m ] 0 0 Vn then X = U m Z + Z Vn fo some Z R (m ) n,z R m (n ). We note that in the above definition of θ,, X, X satisfy X, X =0. It can be shown that (simila to the vecto case in [7]) θ, δ +,θ, θ, 4, 4 s.t. + t, + 4 t. (5) Ou ecovey esult fo the nuclea nom heuistic is as follows: heoem II.. Let A have a ( + α)-esticted isomety constant δ +α and a ( + α, )-esticted othogonality constant θ +α,, whee α 4α. Let X be the solution obtained though nuclea nom nom minimization and X 0 be as defined befoe. If δ +α + θ +α, <, then H = X X 0 satisfies H F Cɛ + B X 0 X 0,, whee the constants B,C ae a function of the isomety and othoganality constants. he following nuclea nom inequality poves useful in getting shape bounds and can be shown though the fact that the nuclea nom is the dual nom of the spectal nom. X X Lemma II.4. Fo any block matix, X =, X X 4 X X + X 4. Now, we come to the poof of the main theoem. heoem II. Poof: he poof goes along simila lines to that in [5],[4] whee the idea of suppot splitting is used to show that a spase vecto can be ecoveed though (l ) minimization if the matix A defining the constaint set satisfies RIP. Let SVD of X 0 = Σ U U 0 V m 0 Σ Vn = X 0, +(X 0 X 0, ), with X 0, = U Σ V. Define P U = U U, P U = U m Um, P V = V V, P V = V n Vn. Let H = P (H)+P (H), with P (H) = P U HP V + P U HP V + P U HP V P (H) = P U HP V (6) We note that denotes the set {Z : Z = P U HP V + P U HP V + P U HP V fo some H R m n }. And denotes the set {Z : Z = P U HP V fo some H R m n }, i.e. the set of all matices whose ow and column spaces ae othogonal to the ow and column space of X 0,. Also note that ank(p (H)) and that X0,P (H) = X 0, P (H) =0. U We define U HV = HV U HV n Um HV Um HV = [ n H H ]. heefoe, X H 0 X 0 +H = U (X 0 + H4 [ H)V = Σ + H ] H Σ H Σ+ + H + H4 Σ + H 4 X 0, P U HP V X 0 X 0, + P U HP V, whee the second inequality follows fom Lemma II.4. hus, P (H) P U HP V + X 0 X 0, (7) Let P (H) =[U U U U U..]Σ H [V V V V V..] be the SVD of P (H), with the singula values in Σ H deceasing fom top to the bottom. Σ H is made up of diagonal blocks Σ, Σ, Σ, Σ, Σ,... with Σ and Σ i of size α and Σ i of size ( α) (α 4α) i. Denote, H = U Σ V and H i = U i Σ i Vi,H i = U i Σ i Vi, H i = H i + H i i. At this point, we note that ou goal is to have a good bound on H F = P (H)+H F + i H i F. hus two key steps follow. he fist is to get a good inequality between i H i F and P (H)+H F. We combine (7) and the shifting inequality(see e.g. [4]) towads this end. he second step is to get a bound on P (H) +H F using RIP and esticted othogonality. We note that the shifting inequality can be used to get tighte l,l inequalities between pais of vectos. his idea is easily extended to matices by applying the shifting inequality to the singula values as below. H F = H + H F H + H H i F = H i + H i F H (i ) + H i, i (8) Using the inequalities in (8), (7) and Lemma II.4 it is easy to see that, H i F H + H i i i P (H) F + X 0 X 0, (9) We uppe and lowe bound A(H), A(P (H) +H ) to deive a bound fo P (H) +H and thus a bound fo H. Denoting, S = A(H), A(P (H) +H ) it can be shown(analogous to the deivation in Section. [4]) that, 574

3 and that, S P (H)+H F (( δ +α θ +α,) P (H)+H F θ +α, X 0 X 0, ) (0) S ɛ +δ +α P (H)+H F () Combining (0) and (), we get, P (H)+H F ɛ δ +α + δ +α θ +α, + θ +α, X 0 X 0 () δ +α θ +α, Combining, () and (9), we get a bound on H F, H F = P (H)+H F + i H i F whee, C = ( + θ +α, δ +α θ +α, P (H)+H F +( H i F ) i ( Cɛ + B ) X 0 X 0 () +δ +α +, B = ) +. Note that fo () to δ +α θ +α, hold, we need that the denominato to be positive, i.e. δ +α + θ +α, < (4) Let =4α. Assume that =0k + ω, whee 0 ω 9 and k Z +. hen, ( + α) +(4α) <ζ(ζ Z + ) if α < ζ ζ 5 =k(ζ ) + ω 5. Choose α such that, { k(ζ ) if 0 ω 4 α = (5) k(ζ ) + ζ othewise hen, α, = 4α ae integes. Using (5), we get that δ +α + θ +α, <δ ζ ( + ) < if δ ζ < +. In paticula if ω = 0, we get that fo ζ =, 4, 5, δ < 5 4 = 0.47, δ 4 < 8 40 = and δ 5 < 60 7 =0.607 ae sufficient fo (4) to hold and thus fo () to hold. Even if ω 0, fo easonably lage k, the uppe bound on the RIP constants(δ,δ 4,δ 5 ) that ae sufficient fo ecovey vey quickly tend to the uppe bounds above. It can also be shown by extending the analysis in [] that δ < 0.07 is sufficient fo (4) to hold and thus fo () to hold. Also obseve that with zeo measuement eo,(ɛ =0) and ank(x 0 ), the nuclea nom minimization exactly ecoves X 0 (if at least one of the above conditions on δ, δ, δ 4, δ 5 ae satisfied). Ou δ < 0.07 esult compaes well with the ecent SVP esult [5], whee if δ <, the SVP algoithm guaantees ecovey. he SVP algoithm, though efficient equies apioi knowledge of the ank of X 0. Ou esults also impoves geatly on the pevious esult of [9] and [9], whee ecovey is shown using nuclea nom minimization if δ 5 < 0., the esult of δ < +4 =0.0 in [] and also on the RIP esult of δ 4 < given in [6] fo ecovey using NNH. We also note heoem. in [6] mentions that if p>o(n), then ecovey can be guaanteed with high pobability if the map A is chosen fom cetain andom distibutions. Ou δ esult educes the constant in O(n) by a facto of aound as compaed to the δ 4 esult given in [6]. III. RIP RESUL FOR REWEIGHED RACE HEURISIC In this section we use the guaantee esult in the pevious section to give a fist guaantee esult fo the Reweighted ace Heuistic(RH). he RH iteatively minimizes the lineaization to a concave suogate fo ank(x), the suogate being log det(x+γi), whee γ>0. he (k +) th iteation of the RH [] when X is esticted to be positive semidefinite is given by: X k+ = subject to agmin X (X k + γ k+ I) X X 0, A(X) b ɛ (6) whee, γ k+ > 0 is a constant to ensue invetibility, A : R m m R p and b = A(X 0 )+e, e ɛ. Inteestingly, ou analysis shows that γ k plays an impotant ole in bounding the eo of ecovey. We make an additional assumption that X 0 be[ of ank ][ at most ] with SVD, X 0 = UΣV = Σ 0 V [U U m ] 0 0 Vm. Let W k = X k + γ k+ I and let, X k+ = X 0 + H k+. Let the smallest non-zeo singula value of X 0 be μ. Also, we assume that X k X 0 F M k. We then have the following theoem that gives conditions fo the eweighted tace heuistic to have a bette ecovey eo bound than nuclea nom minimization. heoem III.. Let A have the RIP constant δ, obeying δ < 5 4. Let X be the solution obtained though nuclea nom minimization ().hen X k X 0 F E(k) = C,k ɛ +δ k, whee C δ 0 (+C,k 8 ),k is a constant that depends on E(k ),γ k,μ.ifc,k < k, then the sequence, {E(k)} conveges to a limit E<E(). In paticula, if μ>e() and γ k = E(k )(μ+e(k )) μ E(k ) k then the sequence, {E(k)} conveges to a limit E<E(). Befoe we poceed with the poof, we list some useful inequalities fo eigenvalues and singula values. Lemma III.. [6] Let A, B R n n be hemitian matices. Let λ (C) λ (C)...λ n (C) λ n (C) denote the odeed 575

4 eigen values of any matix C. Let the singula values be odeed similaly and be denoted by σ i (C). hen, λ i (A)+λ n (B) λ i (A + B) λ i (A)+λ (B) σ i (A + B) σ i (A)+σ (B) σ i (A)σ n (B) σ i (AB) σ i (A)σ (B) (7) If both A, B ae positive semidefinite, then the following inequality also holds. λ i (A)λ n (B) λ i (AB) λ i (A)λ (B) (8) heoem III. poof: he poof is inspied by the analysis in [8] and is shown in two pats. In the fist pat, we assume a bound on the eo, H k F M k fom the pevious iteation and deive a bound on the eo H k+ F fo the next iteation of the RH. In the second pat of the poof, we use the ecusive eo bound expession deived in fist pat to show that the eo bounds convege to a limiting eo bound of RH unde the assumptions of the theoem. Poof Pat o simplify the notation, we dop the supescipts on γ k+,h k+,m k,x k,w k and efe to them as γ,h,m,x,w espectively in this pat of the poof. It follows fom (6) that W X 0 W (X 0 + H) = W X 0 +(W k ) H. hus, W H 0 (9) [ We can decompose ] the matix U X k U[ as U X k ] U = X X and U X WU as = = X4 4 X U + γi X WU =. Note that U X X4 + γi W U = ( ) is given by = 4 whee, = ( X + γi X ( X 4 + γi) X ) 4 = ( X 4 + γi X ( X + γi) X ) = ( X + γi) X = ( X 4 + γi) X 4 (0) ae obtained though the fomula fo block matix invesion. Multiplying on left side of U X k U by U and ight hand side by U, we get fou tems which sum to X k. his esults in an additive decomposition: X = X + X + X + X 4. A simila additive decomposition fo W,H gives W = W + W + W + W4,H = H + H + H + H 4. (9) can be decomposed as: W H 4 W (H + H + H ) () It holds that W H 4 = W4 H 4, since the othe tems in the additive[ decomposition of (W k ) cancel out. H Let H = U H HU = ]. H H4 Note that both, W4 and H 4 ae positive semidefinite (X 0 + H = X k+ 0 = H 4 0 = H 4 0). Also, W4 H 4 =U W4 UU H 4 U = 4 H 4 σ min ( 4 )( H 4 ) = σ min ( 4 )(H 4 ), whee the last inequality follows fom (7). σ min ( 4 ) M+γ and thus, W 4 H 4 M + γ (H 4)= M + γ H 4 () We note at this point that P (H) = H + H + H and P (H) = H 4 (whee P,P ae as defined in (6)). We now uppe bound W P (H) in tems of P (H). Note that, W P (H) = (W + W + W )P (H). We can bound the above quantity by obseving that A B A B fo any two A, B. hus, W P (H) (W + W + W ) P (H) = U (W + W + W )U P (H) ( + ) P (H) whee the last inequality uses the fact that = max(, ). We now uppe bound each of M+γ,.Define G(M,γ) = γ(μ M)+γ M. hen, M γ G(M,γ)M M + γ hus, + G(M,γ). he above inequalities can be checked using the definitions in (0) and the inequalities in (7),(8). One key step is bounding σ min ( X + γi) = λ min ( X +γi) =λ min ( X )+γ. Since, X X 0 M, we have that X Σ M. Since, X Σ is symmetic, X Σ λ i ( X Σ) and thus λ min ( X Σ) M. Fom (7), we have that λ min ( X Σ) λ min ( X ) λ min (Σ). Hence, λ min ( X + γi) μ M + γ. he inequalities now follow by using the fact that 0 and though successive applications of the inequalities in (7). heefoe, W P (H) G(M,γ) P (H) () Combining (),(), and (), we have that, P (H) (M + γ)g(m,γ) P (H) (4) hus we have bounded the P (H) in tems of P (H). We can now poceed using a simila analysis as in section II to obtain a bound fo H F. We get that, + C (M,γ) +δ +α H F ɛ δ +α C (M,γ) θ +α,. (5) whee, C (M,γ) =(M + γ)g(m,γ) and α, ae as defined ealie. Poof Pat Note that, the ecovey eo using nuclea nom minimization () can be obtained by setting C (M,γ) =in (5). o simplify ou analysis, we let =4α with α chosen as in (5). hen,δ +α <δ,θ +α, <δ. heefoe, a weake uppe bound can be obtained fom (5) as H F D(M,γ)ɛ, (6) 576

5 whee, D(M,γ) = ɛ +δ + C(M,γ) δ Let (+C (M,γ) ). E(k +) = D(E(k),γ k+ ) k. Also denote C,k+ = C (E(k),γ k+ ) k. hus,e(k +) denotes an uppe bound on the eo at the end of iteation k +. Since the weight is chosen to be identity in the fist iteation,we have that E() = D ɛ, whee D = +δ + δ (+. ) Note that E() E() iff C,. his gives us a bound on μ (the minimum non-zeo singula value in X 0 ): μ E() + γ E() (7) Assuming that E(k) E(k ) fo all pevious k,it holds tue that C,k+ C,k and thus E(k +) E(k) fo all k. Assuming δ < 5 4 (see esults fom section II), E(k) is always positive. hus,e(k) conveges to a limit E. he limit can be obtained by solving the equation, E = D(E,γ). Given a μ, the optimal, γ (μ) = E()(μ+E()) (μ E()) minimizes E(). We also equie that μ E()+ γ E(), i.e. γ > E() <E(). his is ensued if μ E() (since γ > 0). Hence, if γ k is chosen so that E(k) is minimized, i.e. γ k =, then the limiting eo bound, E<E(). E(k )(μ+e(k )) (μ E(k )) IV. DISCUSSION AND FUURE WORK E() μ E() fo We gave an RIP-basesd deteministic ecovey esult fo nuclea nom minimization with RIP constants that ae bette than in existing esults([9],[],[9]). We then used this esult to give a guaantee of ecovey fo RH. o undestand how RH compaes with the NNH, we vay δ,ν(whee we let μ νe()) and using the ecusive expessions fo E(k), we compute how E(k)/E() vaies fo k =,, 4, 5.We choose γ k optimally at each iteation as defined in the pevious section. he esults in able I show that E(k)/E() deceases and conveges as k inceases which is consistent with the statements in heoem III.. A supising phenomenon is that as δ inceases fom 0. to 0.45, E(k)/E() educes dastically. his can be explained by the fact that as δ inceases and appoaches 0.47, E() becomes vey lage but since C,k <, E(k) doesn t gow as lage and hence the small atio. he second to last column shows the apid gowth of E() with incease in δ k. he last column shows that if μ>e() and if γ k is chosen optimally at each iteation, then the uppe bound on the eo, E(k) is consistently small fo lage k. We also obseve that if γ k is fixed,(e.g. =0E()), then E(5)/ɛ is much lage than if γ k wee chosen geedily at each iteation. So it is natual to ask if choosing γ k adaptively at each iteation as a function of the eo bound in the pevious iteation would lead to impoved numeical esults. We have given esults fo RH when the constaint set is esticted to be positive semidefinite. Futhe wok could include extending this esult when the constaint set is convex but not esticted to be positve semidefinite. δ E() E() E(4) E(5) E() E(5) ν E() E() E() E() ɛ ɛ ABLE I COMPARING UPPER BOUNDS ON RECOVERY ERROR A DIFFEREN IERAIONS OF REWEIGHED NUCLEAR NORM MINIMIZAION WIH HE RECOVERY ERROR OF NUCLEAR NORM MINIMIZAION, E() REFERENCES [] J.F. Cai, E.J. Candes, and Z. Shen. A singula value thesholding algoithm fo matix completion Available at Submitted on 8 Oct 008. [].. Cai, G. Xu, and J. Zhang. New bounds fo esticted isomety constants echnical Repot, Available Online. [].. Cai, G. Xu, and J. Zhang. On ecovey of spase signals via l minimization. In Poceedings of the 009 IEEE ansactions on Infomation heoy, pages 88 97, 009. [4].. Cai, G. Xu, and J. Zhang. Shifting inequality and ecovey of spase signals o appea in IEEE ansactions on Signal Pocessing. [5] E.J. Candes. he esticted isomety popety and its implications fo compessed sensing. Academie des Sciences, 008. [6] E.J. Candes and Y. Plan. ight oacle bounds fo low-ank matix ecovey fom a minimal numbe of andom measuements Available at [7] E.J. Candes and. ao. Decoding by linea pogamming. IEEE ansactions on Infomation heoy, 004. [8] E.J. Candes, M.B. Wakin, and S. Boyd. Enhancing spasity by eweighted l minimization. Jounal of Fouie Analysis and Applications, 4: , 008. [9] M. Fazel, E.J. Candes, B. Recht, and P. Pailo. Compessed sensing and obust ecovey of low ank matices. In Poc. Asiloma Confeence, 009. [0] M. Fazel, H. Hindi, and S. Boyd. A ank minimization heuistic with application to minimum ode system appoximation. In Poc. Ameican Contol Confeence, 00. [] M. Fazel, H. Hindi, and S.Boyd. Log-det heuistic fo matix ank minimization with applications to hankel and euclidean distance matices. In Poc. Ameican Contol Confeence, 00. [] S. Foucat and M.J. Lai. Spasest solutions of undedetemined linea systems via l q minimization fo 0 <q<, 009. [] K. Lee and Y. Besle. Guaanteed minimum ank appoximation fom linea obsevations by nuclea nom minimization with an ellipsoidal constaint. Available online at Submitted on 7 Ma 009. [4] K. Lee and Y. Besle. Admia:atomic decomposition fo minimum ank appoximation Available at [5] R. Meka, P. Jain, and I.S. Dhillon. Guaanteed ank minimization via singula value pojection. Available at Submitted on 0 Sep, 009. [6] J.K. Meikoski and R. Kuma. Inequalities fo speads of matix sums and poducts. Applied Mathematics E-Notes, 4:50 59, 004. [7] K. Mohan and M. Fazel. Reweighted nuclea nom minimization with application to system identification. In Poc. ACC 00. [8] D. Needell. Noisy signal ecovey via iteative eweighted l- minimization. In Poc. Asiloma confeence on Signals, Systems and Computes, 009. Available at [9] B. Recht, M. Fazel, and P. A. Pailo. Guaanteed minimum ank solutions to linea matix equations via nuclea nom minimization. Accepted fo publication, SIAM Review.,