Randomized Low-Memory Singular Value Projection

Transcription

1 Randomized Low-Memoy Singula Value Pojection Stephen Becke Volkan Cevhe Anastasios Kyillidis May 17, 2013 Abstact Affine ank minimization algoithms typically ely on calculating the gadient of a data eo followed by a singula value decomposition at evey iteation. Because these two steps ae expensive, heuistic appoximations ae often used to educe computational buden. To this end, we popose a ecovey scheme that meges the two steps with andomized appoximations, and as a esult, opeates on space popotional to the degees of feedom in the poblem. We theoetically establish the estimation guaantees of the algoithm as a function of appoximation toleance. While the theoetical appoximation equiements ae ovely pessimistic, we demonstate that in pactice the algoithm pefoms well on the quantum tomogaphy ecovey poblem. 1 Intoduction In many signal pocessing and machine leaning applications, we ae given a set of obsevations y R p of a ank- matix X R m n as y = AX + ε via the linea opeato A : R m n R p, whee min{m, n} and ε R p is additive noise. As a esult, we ae inteested in the solution of minimize X R m n fx) subject to ankx), 1) whee fx) := y AX 2 2 is the data eo. While the optimization poblem in 1) is non-convex, it is possible to obtain obust ecovey with povable guaantees via iteative geedy algoithms SVP) [MJD10, KC12] o convex elaxations [RFP10, CR09] fom measuements as few as p = Om + n )). Cuently, thee is a geat inteest in designing algoithms to handle lage scale vesions of 1) and its vaiants. As a concete example, conside quantum tomogaphy QT), whee we need to ecove low-ank density matices fom dimensionality educing Pauli measuements [FGLE12]. In this poblem, the size of these density matices gows exponentially with the numbe of quantum bits. Othe collaboative filteing poblems, such as the Netflix challenge, also equie huge dimensional optimization. Without caeful implementations o non-conventional algoithmic designs, existing algoithms quickly un into time and memoy bottlenecks. These computational difficulties typically evolve aound two citical issues. Fist, vitually all ecovey algoithms equie calculating the gadient fx) = 2A AX) y) at an intemediate iteate X, whee A is the adjoint of A. When the ange of A contains dense matices, this foces algoithms to use memoy popotional to Omn). Second, afte the iteate is updated with the gadient, pojecting onto the low-ank space equies a patial singula value decomposition SVD). This is usually poblematic fo the initial iteations of convex algoithms, whee they may have to pefom full SVD s. In contast, geedy algoithms [KC12] fend off the complexity of full SVD s, since they need fixed ank pojections, which can be appoximated via Lanczos o andomized SVD s [HMT11]. Algoithms that avoid these two issues do exist, such as [WYZ10, RR13, LRS + 11, Lau12], and ae typically based on the Bue-Monteio splitting [BM03]. The main idea in Bue-Monteio splitting is to emove the nonconvex ank constaint by diectly embedding it into the objective: as opposed to optimizing X, splitting algoithms diectly wok with its fixed factos UV T = X in an altenating fashion, whee U R m ˆ and V R n ˆ fo some stephen.becke@upmc.f, Laboatoie JLL, UPMC Pais 6, Pais {volkan.cevhe, anastasios.kyillidis}@epfl.ch, LIONS, École polytechnique Fédéale de Lausanne Authos ae listed in alphabetical ode 1

2 ˆ. Unfotunately, igoous guaantees ae difficult. 1 The wok [JNS12] has shown appoximation guaantees if A satisfies a esticted isomety popety with constant δ 2 κ 2 /100) in the noiseless case), whee κ = σ 1 X )/σ X ), o δ 2 1/ ) fo a bound independent of κ. The authos suggest that these bounds may be tightened, and that pactical pefomance is bette than the bound suggests. In this pape, we mege the gadient calculation and the singula value pojection steps into one and show that this not only emoves a huge computational buden, but suffes only a mino convegence speed dawback in pactice. Ou contibution is a natual but non-tivial fusion of the Singula Value Pojection SVP) algoithm in [MJD10] and the appoximate pojection ideas in [KC12]. The SVP algoithm is an iteative had-thesholding algoithm that has been consideed in [MJD10, GM11]. Inexact steps in SVP have been consideed as a heuistic [GM11] but have not been incopoated into an oveall convegence esult. 2 A non-convex famewok fo affine ank minimization including vaiants of the SVP algoithm) that utilizes inexact pojection opeations with povable signal appoximation and convegence guaantees is poposed in [KC12]. Neithe [MJD10, KC12] consides splitting techniques in the poposed schemes. This wok, depating fom [MJD10, KC12], enginees the SVP algoithm to opeate like splitting algoithms that diectly wok with the factos; this added twist deceases the pe iteation equiements in tems of stoage and computational complexity. Using this new fomulation, each iteation is nealy as fast as in the splitting method, hence emoving a dawback to SVP in elation to splitting methods. Futhemoe, we pove that, unde some conditions, it is still possible to obtain pefect ecovey even if the pojections ae inexact. In paticula, ou assumption is that the linea map A satisfies the ank esticted isomety popety, and in section 5.1 we give an application that satisfies this assumption, allowing pefect ecovey in the noiseless case) o stable ecovey in the pesence of noise) fom measuements p mn. This appoach has been used fo convex [RFP10] and non-convex [MJD10, KC12] algoithms to obtain appoximation guaantees. 2 Peliminay mateial Notation: we wite P Ω to be an othogonal pojection onto the closed set Ω when it exists. Fo shothand we wite P to mean P {X:ankX) } which does exist by the Eckat-Young theoem). Compute outine names ae typeset with a typewite font. 2.1 R-RIP The Rank Resticted Isomety Popety R-RIP) is a common tool used in matix ecovey [RFP10, MJD10, KC12]: Definition 1 R-RIP fo linea opeatos on matices [RFP10]). A linea opeato A : R m n R p satisfies the R-RIP with constant δ A) 0, 1) if, X R m n with ankx), We wite δ to mean δ A). 1 δ A)) X 2 F AX δ A)) X 2 F, 2) 2.2 Additional convex constaints Conside the vaiant minimize X R m n fx) subject to ankx), X C, fo a convex set C. Ou main inteests ae C + = {X : X 0} and the matix simplex C = {X : X 0, tacex) = 1}. In both cases the constaints ae unitaily invaiant and the pojection onto these sets can be done by taking 1 If ˆ p, then [BM03] shows thei method obtains a global solution, but this is impactical fo lage p. Moeove, it is shown that the explicit ank ˆ splitting method solves a non-convex poblem that has the same local minima as 1) if ˆ = ). Howeve, the non-convex poblems ae not equivalent e.g. U = 0, V = 0 is a stationay point fo the splitting poblem wheeas X = 0 is geneally not a stationay point fo 1)). Futhemoe, ecovey bounds fo non-convex algoithms, as in [GK09] and the pesent pape, ae statements about a sequence of iteates of the algoithm, and say nothing about the local minima. 2 Inexact steps ae often incopoated into analysis of algoithms fo convex poblems. Of paticula note, [Lau12] allows inexact eigenvalue computations in a modified Fank-Wolfe algoithm that has applications to 1). 3) 2

3 Algoithm 1 RandomizedSVD Finds Q such that X P Q X whee P Q = QQ. Requie: Function h : Z X Z Requie: Function h : Q X Q Requie: N // Rank of output Requie: q N // Numbe of powe iteations to pefom 1: l = + ρ // Typical value of ρ is 5 2: Ω a n l standad Gaussian matix 3: W hω) 4: Q QRW ) // The QR algoithm to othogonalize W 5: fo j = 1, 2,..., q do 6: Z QRh Q)) 7: Q QRhZ)) 8: end fo 9: Z h Q) 10: U, Σ, V ) FactoedSVDQ, I l, Z) // X i+1 = UΣV in the appendix 11: Let Σ be the best ank appoximation of Σ 12: etun U, Σ, V ) // X i+1 = UΣ V in the appendix Algoithm 2 FactoedSVDŨ, D, Ṽ ) Computes the SVD UΣV of the matix X implicitly given by X = Ũ DṼ 1: U, R U ) QRŨ) 2: V, R V ) QRṼ ) 3: u, Σ, v) DenseSVDR U DR V ) 4: etun U, Σ, V ) Uu, Σ, V v) the eigenvalue decomposition and pojecting the eigenvalues. Futhemoe, fo these specific C, P {X:ankX) } C = P C P this is not obvious; see [BCKK13]). 3 In geneal, any convex set C satisfying the above popety is compatible with ou algoithm, as long as X C. We oveload notation to use P C to denote both the pojection of X onto the set as well as the pojection of its eigenvalues onto the analogous set. 2.3 Appoximate singula value computations The standad method to compute a patial SVD is the Lanczos method. By itself it is not numeically stable and equies e-othogonalization and implicit estats. Excellent implementations ae available, but it is a sequential algoithm that calls matix-vecto poducts. This makes it moe difficult to paallelize, which is an issue on moden multi-pocesso computes. The matix-vecto multiplies ae also slowe than gouping into matix-matix multiplies since it is hade to pedict memoy usage and this will lead to cache misses; it also pecludes the use of theoetically faste algoithms such as Stassen s. Theoetically, thee ae no known elative eo bounds in nom à la Theoem 1). As an altenative, we tun to andomized linea algeba. On this font, we estict ouselves to algoithms that equie only multiplications, as opposed to sub-sampling enties/ows/columns, as sub-sampling is not efficient fo the application we pesent. The andomized appoach pesented in Algoithm 1 has been ediscoveed many times, but has seen a ecent esugence of inteest due to theoetical analysis [HMT11]: Theoem 1 Aveage Fobenius eo). Suppose X R m n, and choose a taget ank and ovesampling paamete ρ 2 whee l := + ρ min{m, n}. Calculate Q and P Q via RandomizedSVD using q = 0 and set X = P Q X which is ank l). Then E X X 2 F 1 + ɛ) X X 2 F 3 This fomula is liteally tue fo C + and {X : X 0, tacex) 1}. Fo C = {X : X 0, tacex) = 1} constaints, P C can incease the ank, so fomally we must wok on a esticted subspace and then embed back in the lage space, but this poses no theoetical issues. 3

4 Algoithm 3 Efficient implementation of SVP, K = {R, C} Requie: step-size µ > 0, measuements y, initial points u 0 K m, v 0 K n, d 0 K Requie: optional) unitaily invaiant convex set C Requie: Function A : u, d, v) Au diagd)v ) Requie: Function At : z, w) A z)w Requie: Function At : z, w) A z)) w 1: v 1 0, u 1 0, d 1 0 2: fo i = 0, 1,... do 3: Compute β i // See text 4: u y [u i, u i 1 ], v y [v i, v i 1 ] 5: d y [1 + β i )d i, β i d i 1 ] 6: z Au y, d y, v y ) y // Compute the esidual 7: Define the functions h : w u y diagd y )vyw µatz, w) h : w v y diagd y )u yw µat z, w) 8: u i+1, d i+1, v i+1 ) RandomizedSVDh, h, ) o u i+1, d i+1, u i+1 ) RandomizedEIGh, h, ) 9: d i+1 P C d i+1 ) // Optional 10: end fo 11: etun X u i d i vi // If desied whee X is the best ank appoximation in the Fobenius nom of X and ɛ = The theoem follows fom the poof of Thm in [HMT11] note that Thm is stated in tems of E X X F which is not the same as E X X 2 F ). The expectation is with espect to the Gaussian.v. in RandomizedSVD. Fo the sake of ou analysis, we cannot immediately tuncate X to ank since then the eo bound in [HMT11] is not tight enough. Thus, since X is ank l, in pactice we even obseve that X X 2 F < X X 2 F, especially fo small, as shown in Figue 3. The figue also shows that using q > 0 powe iteations is extemely helpful, though this is not taken into account in ou analysis since thee ae no useful theoetical bounds in the Fobenius nom). Note that vaiants fo eigenvalues also exist; we efe to the equivalent of RandomizedSVD as RandomizedEIG, which has the popety that U = V and Σ need not be positive cf., [HMT11,?]) ρ 1. 3 Algoithm 3.1 Pojected gadient descent Ou appoach is based on the pojected gadient descent algoithm: X i+1 = P ɛ X i+1 µ i fx i )), 4) whee X i is the i-th iteate, f ) is the gadient of the loss function, µ i is a step-size, and P ɛ ) is the appoximate pojecto onto ank matices given by RandomizedSVD. If we include a convex constaint C, then the iteation is In pactice, Nesteov acceleation impoves pefomance: X i+1 = P C P ɛ X i+1 µ i fx i ))). 5) Y i+1 = 1 + β i )X i β i X i 1 6) X i+1 = PY i µ i fy i )), 7) whee β i is chosen β i = α i 1 1)/α i and α 0 = 1, 2α i+1 = 1 + 4αi [Nes83] see [KC12]). Theoem 2 holds fo a stepsize µ i based on the RIP constant, which is unknown. In pactice, the algoithm consistently conveges as long as µ i 2 A. 2 Algoithm 3 shows implementation details that ae impotant fo keeping low-memoy equiements. The implementation of maps like A and At depends on the stuctue of A; see section 5.1 fo explicit examples. 4

5 4 Convegence We assume the obsevations ae geneated by y = AX + ε whee ε is a noise tem, not to be confused with the appoximation eo ɛ. In the following theoem, we will assume that A 2 mn/p, which is tue fo the quantum tomogaphy example [Liu11]; if A is a nomalized Gaussian, then this assumption holds in expectation. Theoem 2. Iteation invaiant) Pick an accuacy ɛ = ρ 1, whee ρ is defined as in Theoem 1. Define l = + ρ 1 and let c be an intege such that l = c 1). Let µ i = 21+δ in 4) and assume c) A 2 mn/p and fx i ) > C 2 ε 2, whee C 4 is a constant. Then the descent scheme 4) o 5) has the following iteation invaiant EfX i+1 ) θfx i ) + τ ε 2, 8) in expectation, whee and θ δ 2 ɛ mn 1 δ c 1 + δ c p ɛ) 3δ ) c, τ 1 + δ ɛ) 1 + 2δ ) ) c δ c The expectation is taken with espect to Gaussian andom designs in RandomizedSVD. If θ θ iteations, then lim i EfX i ) max{c 2 τ, 1 θ } ε 2. < 1 fo all Each call to RandomizedSVD daws a new Gaussian.v., so the expected value does not depend on pevious iteations. By Coollay 3.4 in [NT09], δ c c δ 2, which allows us to put θ and τ in tems of δ 2 if desied, at a slight expense in shapness. The expected value of the function conveges linealy at ate θ to within a constant of the noise level, and in paticula, it conveges to zeo when thee is no noise since C and τ ae finite. Note that convegence of the iteates follows fom convegence of the function f: Coollay 1. If fx i ) γ, then X i X 2 F γ+ ε 2) 2 1 δ 2. Poof. By the R-RIP and the tiangle inequality, 1 + δ2 A) X i X F AX i ) AX ) 2 = AX i ) y) AX ) y) 2 AX i ) y) 2 + ε 2 γ + ε 2 Coollay 2 Exact computation). If ɛ = 0 and thee is no additional convex constaint C, then θ = 2δ2 1 δ C ) and τ = 1 + 2δ2 1 δ 2, hence θ < 1 if δ 2 < 1 3+4/C. Coollay 2 shows that without the appoximate SVD, the R-RIP constants ae quite easonable. Fo example, with exact computation and no noise, any value of δ 2 < 1/3 implies that lim i X i = X. With noise, choosing C = 4 gives δ 2 = 1/5 and θ = 3/4, τ = 3/2 and thus lim i fx i ) max{16, 6} ε 2. Note that the theoem gives pessimistic values fo ɛ. We want the bound on θ to be less than 1 in ode to have a contaction, so we need δ 2 ɛ mn ɛ) 1 + δ 2 3δ c < 1 1 δ c 1 + δ c p 1 δ }{{} c 1 δ }{{ 2 } I II Fo a ough analysis, we will give appoximate conditions so that each of the I and II tems is less than 0.5. It is clea that the tems blow up if δ c 1, so we will assume δ c 1 and hence δ 2 1). Then setting 1 + δ 2 1 in the numeato of I, we equie that 12 1 δc 2 ɛ mn < 1 9) p 2 5

6 p which means that we need ɛ 24mn. Fo quantum tomogaphy, m = n and p = On), so we equie ɛ O/n). Fom Theoem 1, ou bound on ɛ is /ρ 1), so we equie ρ n, which defeats the pupose of the andomized algoithm in this case, one would just do a dense SVD). Numeical examples in the next section will show that ρ can be nealy a small constant, so the theoy is not shap. Fo the II tem, again appoximate 1 + δ 2 1 and then multiply the denominatos and ignoe the δ c δ 2 tem to get 72δ c 1 + ɛ) δ c. 10) Since cetainly ɛ 0.5 and δ 2 + δ c 0.5, a sufficient condition is δ c < 1/216, which is easonable cf. [JNS12]). 5 Numeical expeiments 5.1 Application: quantum tomogaphy As a concete example, we apply the algoithm to the quantum tomogaphy poblem, which is a paticula instance of 1). Fo details, we efe to [GLF + 10,FGLE12]. The salient featues ae that the vaiable X C n n is constained to be Hemitian positive-definite, and that, unlike many low-ank ecovey poblems, the linea opeato A satisfies the R-RIP: [Liu11] establishes that Pauli measuements which compise A) have R-RIP with ovewhelming pobability when p = On log 6 n). In the ideal case, X is exactly ank 1, but it may have lage ank due to some non- Gaussian) noise pocesses, in addition to AWGN ε. Futhemoe, it is known that the tue solution X has tace 1, which is also possible to exploit in ou algoithmic famewok. Since X is Hemitian, the u and v tems in the algoithm ae identical. Seveal computations can be simplified and thee is a vesion of Algoithm 1 which exploits the positive-definiteness to incopoate a Nystöm appoximation and also foces the appoximation to be positive-definite); see [HMT11,?]. Hee, we focus on showing how the functions A and At can be computed due to the complex symmety, At = At). In quantum tomogaphy, the linea opeato has the fom AX)) j = E j, X whee E j = E j is the Konecke poduct of 2 2 Pauli matices. Thee ae fou possible Pauli matices σ x,y,z if we define σ I to be the 2 2 identity matix. Fo a q b -qubit system, E j = σ j1 σ j2... σ jqb. Fo oughly 12 qubits and fewe, it is simple to calculate AX) by explicitly foming E j and then ceating a spase matix A with the j th ow of A equal to vece j ) so that AX) = A vecx). Fo lage systems, stoing this spase matix is impactical since thee ae p n ows and each ow has exactly n non-zeo enties, so thee ae ove n 2 enties in A. To keep memoy low, we exploit the Konecke-poduct natue of E j and stoe it with only q b numbes. When X = xx, we compute E j, X = tacee j xx ) = tacex E j x), and E j x can be computed in Oq b n) time. This gives us A. The output of A is eal even when X is complex. To compute Atz, w) when the dimensions ae small, we just explicitly fom the matix M = Az) and then multiply Mw. To fom M, we use the same spase matix A as above and eshape the n 2 vecto A z into a n n matix. Fo lage dimensions, when it is impactical to stoe A, we implicitly epesent M = p j=1 z je j and thus Mw = p j=1 z je j w. In geneal, the output is complex. Howeve, if it is known a pioi that X is eal-valued, this can be exploited by taking the eal pat of M. This leads to a consideable time savings 2 to 4 ), and all expeiments shown below make this assumption. In ou numeical implementation, we code both A and At in C and paallelize the code since this is the most computationally expensive calculation. Ou paallelization implementation uses both ptheads on local coes as well as message passing among diffeent computes. Thee ae two appoaches to paallelization: divide the indices j = 1,..., p among diffeent coes, o, when x o w has seveal columns, send diffeent columns to the diffeent coes. Both appoaches ae efficient in tems of message passing since A is paameteized and static. The latte appoach only woks when x o w has a significant numbe of columns, and so it does not apply to Lanczos methods that pefom only matix-vecto multiplies. Recoding eo metics can be costly if not done coectly. Let X = xx and Y = yy be ank- factoizations. Fo the Fobenius nom eo X Y F which equies n 2 opeations naively, we expand the tem and use the cyclic invaiance of tace to get X Y 2 F = tacex xx x) + tacey yy y) 2 tacex yy x), which equies only On 2 ) flops. In quantum infomation, anothe common metic is the tace distance [NC10] X Y, whee is the nuclea nom. This calculation equies On 3 ) flops if calculated diectly but can also be calculated cheaply via FactoedSVD on U = V = [x, y] and D = [I, 0; 0, I]. The thid common metic is the fidelity [NC10] given by X 1/2 Y 1/2. If eithe X o Y is ank-1, this can be calculated cheaply as well. 6

7 5.2 Results Eo Lanczos ρ=0, q=1 ρ=0, q=1, no Nystom ite ρ=20, q=0 ρ=5, q=0 time seconds) Full eigenvalue decomposition Top 2 eigenvalues via Lanczos Top 2 eigenvalues via andomized One day One hou One minute Iteations n Figue 1: Left) Convegence ate as a function of paametes to RandomizedSVD/RandomizedEIG. Right) Compaison of just eigenvalue computation times via thee methods. Avg. time pe iteation s) Full eigenvalue Lanczos full memoy) Lanczos low memoy) Randomized full memoy) Randomized low memoy) Avg. time pe iteation s) Standad implementation laptop) Low memoy implementation cluste, 10 nodes x 8 coes) time = n Dimension n Dimension n Figue 2: Mean time of 10 iteations: this includes the matix multiplications as well as eigenvalue computations. Left) shows times fo a complete iteation of ou method on a single compute using spase matix multiplies full memoy ) and, above 11 qubits, the custom low-memoy implementation as well not multi-theaded) on the same compute. Right) shows times fo just the RandomizedSVD/RandomizedEIG. Figue 1 left) plots convegence and accuacy esults fo a quantum tomogaphy poblem with 8 qubits and p = 4n with = 1. The SVP algoithm woks well on noisy poblems but we focus hee on a noiseless and tuly low-ank) poblem in ode to examine the effects of appoximate SVD/eigenvalue computations. The figue shows that the powe method with q 1 is extemely effective even though it lacks theoetical guaantees; without the powe method, take ρ 20 and we see convegence, albeit slowe. When p is smalle and the R-RIP is not satisfied, taking ρ o q too small can lead to non-convegence. Figue 1 ight) is a diect compaison of RandomizedEIG with ρ = 5 and q = 3) and the Lanczos method fo multiplies of the type encounteed in the algoithm. The RandomizedEIG has the same asymptotic complexity but much bette constants. Figue 2 shows that because the eigenvalue decomposition is a significant potion of the computational cost, using RandomizedEIG instead of Lanczos makes a diffeence. The diffeence is not ponounced in the small-scale full-memoy implementation because the vaiable X is explicitly fomed and matix multiplies ae elatively cheap compaed to othe opeations in the code. Fo lage dimensions with the low-memoy code, X is neve explicitly fomed and multiplying with the gadient is quite costly. The andomized method equies fewe multiplies, explaining its benefit. Fo 12 qubits, the Lanczos method aveages 98.4 seconds/iteation, wheeas the andomized 7

8 ρ ρ ρ ρ Figue 3: Top ow: ɛ fo left) q = 0 and ight) q = 1 powe iteations. Bottom ow: ɛ fo q = 2 powe iteations left), and ight) shows the bound ɛ X ˆX F X ˆX Tace distance Fidelity X X F X X F X, X ) F X, X ) Iteati on Figue 4: The table left) shows eo metics fo the noisy ank-1 16-qubit ecovey. The figue ight) shows the convegence ate fo the same simulation. method aveages just 59.2 seconds. The ight subfigue shows that the low-memoy implementation which has memoy equiement On)) still has only On 2 ) time complexity pe iteation. Figue 3 tests Theoem 1 by plotting the value of ɛ = X X 2 F / X X 2 F 1 which is bounded by ɛ) fo matices X that ae geneated by the iteates of the algoithm. The algoithm is set fo = 1 so X is the sum of a ank 2 tem, which includes the Nesteov tem, and the full ank gadient), but the plots conside a ange of and a ange of ovesampling paametes ρ. The plots use q = 0, 1 top ow, left to ight) and q = 2 bottom ow, left) powe iteations. Because X has ank l = + ρ, it is possible fo ɛ < 0, as we obseve in the plots when is small and ρ is lage. Fo two powe iteations, the eo is excellent. In all cases, the obseved eo ɛ is much bette than the bound ɛ shown bottom ow, ight) fom Theoem 1, suggesting that it may be possible to have a moe efined analysis. Finally, to test scaling to vey lage data, we compute a 16 qubit state n = 65536), using a known quantum state as input, with ealistic quantum mechanical petubations global depolaizing noise of level γ = 0.01; see [FGLE12]) as well as AWGN to give a SNR of 30 db, and p = 5n = measuements. The fist iteation uses Lanczos and all subsequent iteations use RandomizedEIG using ρ = 5 and q = 3 powe iteations. On a cluste with 10 computes, the mean time pe iteation is 401 seconds. The table in Fig. 4 left) shows the eo metics of the ecoveed matix, and Fig. 4 ight) plots the convegence ate of the Fobenius-nom eo and tace distance. Figue 5 epots the median eo on 20 test poblems acoss a ange of p. Hee, X is only appoximately 8

9 Eo in Fobenius nom, median convex SVP, x0=0 splitting, x0=eiga T y)) AltMinSense, oiginal not symmetic) eo due to AWGN eo due to non low ank Numbe of measuements divided by n*) Figue 5: Accuacy compaison of seveal algoithms, as a function of numbe of samples p. Each point is the median of the esults of 20 simulations. low ank and y is contaminated with noise. We compae the convex appoach [FGLE12], the AltMinSense appoach [JNS12], and a standad splitting appoach. AltMinSense and the convex appoach have poo accuacy; the accuacy of AltMinSense can be impoved by incopoating symmety, but this changes the algoithm fundamentally and the theoetical guaantees ae lost. The splitting appoach, if initialized coectly, is accuate, but lacks guaantees. Futhemoe, it is slowe in pactice due to slowe convegence, though fo some simple poblems i.e., no convex constaints C) it is possible to acceleate using L-BFGS [Lau12]. 6 Conclusion Randomization is a poweful tool to acceleate and scale optimization algoithms, and it can be igoously included in algoithms that ae obust to small eos. In this pape, we leveage andomized appoximations to emove memoy bottlenecks by meging the two-key steps of most ecovey algoithms in affine ank minimization poblems: gadient calculation and low-ank pojection. Unfotunately, the cuent black-box appoximation guaantees, such as Theoem 1, ae too pessimistic to be diectly used in theoetical chaacteizations of ou appoach. Fo futue wok, motivated by the ovewhelming empiical evidence of the good pefomance of ou appoach, we plan to diectly analyze the impact of andomization in chaacteizing the algoithmic pefomance. Acknowledgment VC and AK s wok was suppoted in pat by the Euopean Commission unde Gant MIRG , ERC Futue Poof, SNF , and ARO MURI W911NF SRB is suppoted by the Fondation Sciences Mathématiques de Pais. The authos thank Alex Gittens fo his insightful comments and Yi-Kai Liu and Steve Flammia fo helpful discussions. A Poofs Poof of Theoem 2. Thee ae thee aspects to the poof. Even without appoximate SVD calculations, the poblem is non-convex, so we must leveage the R-RIP to pove that iteates convege. Mixed in with this calculation is the appoximate natue of ou ank l point X i+1, whee we will apply the bounds fom Theoem 1. Finally, we elate X i+1 to its ank vesion X i+1. An impotant definition fo ou subsequent developments is the following: 9

10 Definition 2 ɛ-appoximate low-ank pojection). Let X be an abitay matix. Fo any ɛ > 0, P ɛ,l X) povides a ank-l matix appoximation to X such that whee P X) agmin Y:ankY) X Y F. E P ɛ,l X) X 2 F 1 + ɛ) P X) X 2 F, 11) Let X i be the putative ank solution at the i-th iteation, X be the ank matix we ae looking fo and X i+1 be the ank l matix, obtained using appoximate SVD calculations. Define L := 21 + δ +l ) and M := 2 ). Then, we have: f X i+1 ) = fx i ) + fx i ), Xi+1 X i + A X i+1 X i ) 2 F fx i ) + fx i ), Xi+1 X i + L 2 X i+1 X i 2 F = fx i ) 1 2L fx i) 2 F + L 2 X i+1 X i 2 F L fx i), Xi+1 X i + 1 ) L 2 fx i) 2 F = fx i ) 1 2L fx i) 2 F + L 2 X i+1 X i 1 ) L fx i) 2 F. 12) By constuction X i+1 P ɛ,l Xi 1 L fx i) ) since the step-size is µ = 1/L), so, fo X i+1 P Xi 1 L fx i) ), E X i+1 X i 1 L fx i)) 2 F 1 + ɛ) X i+1 X i 1 L fx i)) 2 F 1 + ɛ) X X i 1 L fx i)) 2 F 13) by the definition of P ) since ankx ) = ). Combining 13) with 12), we obtain: Ef X i+1 ) fx i ) 1 2L fx i) 2 F + L ɛ) X X i + 1 L fx i) 2 F = fx i ) 1 1 2L fx i) 2 F ɛ) 2L fx i) 2 F + fx i ), X X i + L ) 2 X X i 2 F [ 1 + ɛ) fx i ) + fx i ), X X i + L ] 2 X X i 2 F + ɛ 2L fx i) 2 F 14) whee we use the fact that fx i ) 0 in the last inequality. Due to the esticted stong convexity of f that follows fom the esticted isomety popety, we have: fx ) fx i ) + fx i ), X X i + M 2 X X i 2 F fx ) M 2 X X i 2 F fx i ) + fx i ), X X i which, combined with 14), leads to: [ Ef X i+1 ) 1 + ɛ) Due to the R-RIP, fx ) + L M X X i 2 F 2 ] + ɛ 2L fx i) 2 F 15) X X i 2 F AX X i ) ) Now define a constant C and assume fx i ) = y AX i 2 2 > C 2 ε 2 2 if the assumption fails, it means X i is aleady close to X ). In paticula, in the noiseless case ε = 0, we may pick C abitaily lage and set all 1/C tems to zeo. AX X i ) 2 F = y AX i ) ε 2 2 = y AX i ) ε ε, y AX i ) fx i ) + ε ε 2 y AX i ) 2 fx i ) + ε C fx i) 17) 10

11 Substituting 17) and 16) into 15), expanding the values of L and M, and noting that fx ) = y AX ) 2 2 = ε 2 2, gives [ Ef X i+1 ) 1 + ɛ) ε δ +l + δ 2 fx i ) + ε )] C fx i) + ɛ 2L fx i) 2 F 18) [ δ+l + δ ɛ) ) fx i ) δ ) ] +l + δ 2 ε ɛ C 2L fx i) 2 F 19) We bound fx i ) using ou assumption on the magnitude of A : fx i ) 2 F = 4 A y AX i )) 2 F 4 A 2 y AX i ) 2 2 = 4 A 2 fx i ) 4 mn p fx i) 20) Fo quantum tomogaphy, we even have AA = mn p I, so the inequality holds with equality and m = n). Combining 19) with 20) and by the definition of L, we obtain: [ Ef X δ+l + δ 2 i+1 ) 1 + ɛ) ) fx i ) δ ) ] +l + δ 2 ε 2 ɛ 2 + mn C 1 + δ +l p fx i) 21) ɛ = mn 1 + δ +l p ɛ)δ +l + δ )) fx i ) ɛ) 1 + δ ) +l + δ 2 ε ) C }{{}}{{} θ τ Note that if an exact SVD computation is used, then not only is ɛ = 0 but also X i+1 is ank, so we ae done and can use θ = θ and τ = τ. To finish the poof, we now elate EfX i+1 ) to Ef X i+1 ). In the algoithm, X i+1 is the output of RandomizedSVD, and X i+1 is the intemediate value UΣV on line 10 of Algo. 1. Given X i+1 with ank X i+1 ) = l >, X i+1 is defined as the best ank- appoximation to X i+1. 4 Thus, the following inequality holds tue: X i+1 X F = X i+1 X i+1 + X i+1 X F X i+1 X i+1 F + X i+1 X F 2 X i+1 X F 23) since X i+1 X i+1 F X X i+1 F. In paticula, since the above is valid fo any value of the andom vaiable X i+1, E X i+1 X 2 F E 4 X i+1 X 2 F. This bound is pessimistic and in pactice the constant is close to 1 athe than 4. We will again assume that f X i+1 ), fx i+1 ) C 2 ε 2 2, and C > 2, since othewise the cuent point is a good-enough solution. We have: fx i+1 ) = y AX i+1 ) 2 2 = AX X i+1 ) + ε 2 2 = AX X i+1 ) ε AX X i+1 ), ε which, if 1 2/C 0, implies fx i+1 ) = AX X i+1 ) ε y AX i+1 ) ε, ε = AX X i+1 ) ε y AX i+1 ), ε + 2 ε, ε AX X i+1 ) ε y AX i+1 ) 2 ε 2 2 ε 2 2 AX X i+1 ) 2 2 ε C fx i+1) 1 1 2/C AX X i+1 ) /C ε ) 4 If we include a convex constaint C then instead of defining X i+1 = P X i+1 ) we have X i+1 = P C P X i+1 )). In this case, P C P X i+1 )) X F = P C P X i+1 ) X ) F P X i+1 ) X F. The fist equality follows fom X C and the second is tue since the pojection onto a non-empty closed convex set is non-expansive. Hence the esult in 23) still applies when we include the C constaints. 11

12 By the R-RIP assumption, we have: Using 23) and 25) in 24), we obtain: AX X i+1 ) δ 2 ) X X i+1 2 F. 25) fx i+1 ) 41 + δ 2) 1 2/C X i+1 X 2 1 F 1 2/C ε ) Using the R-RIP popety again, the following sequence of inequalities holds: X i+1 X 2 F A X i+1 X ) 2 F 1 δ +l 1 + 2/C f 1 δ X 1 i+1 ) + ε ) +l 1 δ +l whee the second inequality is obtained following same motions as 17). Combining 26)-27) with 22), we obtain: EfX i+1 ) 41 + δ 2) 1 2/C 1 + 2/C 41 + θ δ2 ) fx i ) + 1 δ +l 1 2/C 1 + 2/C τ δ ) 2) 1 δ +l 1 2/C 1 1 ε δ +l 1 2/C }{{}}{{} θ τ Now we simplify the esult to make it moe intepetable. Define ρ = l. Let c be the smallest intege such that l c 1) and fo simplicity, assume l = c 1)) so that δ +l = δ c and δ +l + δ 2 2δ c. By Theoem 1, ɛ ρ 1 = c 2) 1. Fo conceteness, take C 4 so that 1 + 2/C 3/2 and 1 2/C) 1 2. Then θ δ 2 ɛ mn 1 δ c 1 + δ c p ɛ) 3δ ) c 28) and τ δ δ 2 1 δ c 1 + ɛ) 1 δ c ɛ) 1 + δ ) 2 + δ c δ ) 2) 1 δ c 1 + 2δ ) ) c ) Refeences [BCKK13] S. Becke, V. Cevhe, C. Koch, and A. Kyillidis, Spase pojections onto the simplex, ICML, to appea, [BM03] S. Bue and R.D.C. Monteio, A nonlinea pogamming algoithm fo solving semidefinite pogams via low-ank factoization, Math. Pog. seies B) ), no. 2, [CR09] E.J. Candes and B. Recht, Exact matix completion via convex optimization, Found. Comput. Math ), [FGLE12] [GK09] [GLF + 10] [GM11] S.T. Flammia, D. Goss, Y.K. Liu, and J. Eiset, Quantum tomogaphy via compessed sensing: eo bounds, sample complexity, and efficient estimatos, New J. Phys ), no. 9, R. Gag and R. Khandeka, Gadient descent with spasification: an iteative algoithm fo spase ecovey with esticted isomety popety, ICML, ACM, D. Goss, Y.-K. Liu, S. T. Flammia, S. Becke, and J. Eiset, Quantum state tomogaphy via compessed sensing, Phys. Rev. Lett ), no. 15, D. Goldfab and S. Ma, Convegence of fixed-point continuation algoithms fo matix ank minimization, Foundations of Computational Mathematics ), no. 2,

13 [HMT11] [JNS12] [KC12] N. Halko, P. G. Matinsson, and J. A. Topp, Finding stuctue with andomness: Stochastic algoithms fo constucting appoximate matix decompositions, SIAM Rev ), no. 2, P. Jain, P. Netapalli, and S. Sanghavi, Low-ank matix completion using altenating minimization, ACM Symp. Theoy Comput., Anastasios Kyillidis and Volkan Cevhe, Matix ecipes fo had thesholding methods, axiv pepint axiv: ). [Lau12] S. Laue, A hybid algoithm fo convex semidefinite optimization, ICML, [Liu11] Y. K. Liu, Univesal low-ank matix ecovey fom Pauli measuements, NIPS, 2011, pp [LRS + 11] [MJD10] [NC10] [Nes83] [NT09] [RFP10] [RR13] [WYZ10] J. Lee, B. Recht, R. Salakhutdinov, N. Sebo, and J. A. Topp, Pactical lage-scale optimization fo max-nom egulaization, NIPS, R. Meka, P. Jain, and I. S. Dhillon, Guaanteed ank minimization via singula value pojection, NIPS, M.A. Nielsen and I.L. Chuang, Quantum computation and quantum infomation, Cambidge univesity pess, Y. Nesteov, A method fo unconstained convex minimization poblem with the ate of convegence O1/k 2 ), Doklady AN SSSR, tanslated as Soviet Math. Docl ), D. Needell and J. Topp, CoSaMP: Iteative signal ecovey fom incomplete and inaccuate samples, Appl. Comput. Hamon. Anal ), B. Recht, M. Fazel, and P.A. Pailo, Guaanteed minimum-ank solutions of linea matix equations via nuclea nom minimization, SIAM eview ), no. 3, B. Recht and C. Ré, Paallel stochastic gadient algoithms fo lage-scale matix completion, Math. Pog. Comput., to appea 2013). Z. Wen, W. Yin, and Y. Zhang, Solving a low-ank factoization model fo matix completion by a nonlinea successive ove-elaxation algoithm, Mathematical Pogamming Computation 2010),