A Unified Theorem on SDP Rank Reduction

Transcription

1 A Unifie heorem on SDP Ran Reuction Anthony Man Cho So, Yinyu Ye, Jiawei Zhang November 9, 006 Abstract We consier the problem of fining a low ran approximate solution to a system of linear equations in symmetric, positive semiefinite matrices. Specifically, let A,..., A m R n n symmetric, positive semiefinite matrices, an let b,..., b m 0. We show that if there exists a symmetric, positive semiefinite matrix X to the following system of equations: A i X = b i for i =,..., m then for any fixe =,..., Olog m, there exists an X 0 0 of ran at most such that: β b i A i X 0 α b i for i =,..., m where: log m α = + O, β = Ω m / 3 log m/ log log m Ω log m otherwise log m for = O log log m Moreover, such an X 0 can be foun in ranomize polynomial time. his complements a result of Barvino [] an provies a unifie treatment of an generalizes several results in the literature [3, 6, 7, 8]. Introuction In this note we consier the problem of fining a low ran approximate solution to a system of linear equations in symmetric, positive semiefinite ps matrices. Specifically, let A,..., A m R n n be symmetric ps matrices, an let b,..., b m 0. Consier the following system of linear equations: A i X = b i for i =,..., m; X 0, symmetric It is well nown [] see also [, 9] that if is feasible, then there exists a solution X 0 of ran no more than m. However, in many applications, such as graph realization [0] an Department of Computer Science, Stanfor University, Stanfor, CA E mail: manchoso@cs.stanfor.eu Department of Management Science an Engineering, Stanfor University, Stanfor, CA E mail: yinyu-ye@stanfor.eu IOMS OM, Stern School of Business, New Yor University, New Yor, NY 00. E mail: jzhang@stern.nyu.eu

2 imension reuction [7], it is esirable to have a low ran solution, say, a solution of ran at most, where is fixe. Of course, such a low ran solution may not exist, an even if it oes exist, one may not be able to fin it efficiently. hus, it is natural to as whether one can efficiently fin an X 0 0 of ran at most where is fixe such that X 0 satisfies approximately, i.e.: βm, n, b i A i X 0 αm, n, b i for i =,..., m for some functions α an β 0, ]. he quality of the approximation will be etermine by how close α an β are to. Our main result is the following: heorem Let A,..., A m R n n be symmetric ps matrices, an let b,..., b m 0. Suppose that there exists an X 0 such that A i X = b i for i =,,..., m. Let r = min{ m, n}. hen, for any, there exists an X 0 0 with ranx 0 such that: βm, n, b i A i X 0 αm, n, b i for i =,..., m where: an an fm = Remars: log4mr + αm, n, = log4mr + βm, n, = 5e 4e for log4mr otherwise m / for log m log logm log fm/ m 4 log4mr for log m log logm < 4 log4mr for > 4 log4mr 3 log m log logm. Moreover, such an X 0 can be foun in ranomize polynomial time. 3 4 a From the efinition of r, we see that the bouns above can be mae inepenent of n an the rans of A,..., A m. b Note that fm/ 3/ in the region > log m log logm. c If max i m rana i = O, then the lower boun can be sharpene to Ω m / for all {,..., 4 log4mr}; see the proof of Proposition. he constants can be improve if we only consier one sie inequalities. It turns out that heorem provies a unifie treatment of an generalizes several results in the literature:

3 a Metric Embeing Let l m be the space Rm equippe with the Eucliean norm, an let l be the space of infinite sequences x = x, x,... of real numbers such that x / j x j <. Given an n point set V = {v,..., v n } in l m, we woul lie to embe it into a low imensional Eucliean space as faithfully as possible. Specifically, we say that a map f : V l is an D embeing where D if there exists a number r > 0 such that for all u, v V, we have: r u v fu fv D r u v he goal is to fin an f such that D is as small as possible. It is nown [3, 7] that for any fixe, an O n / log n / embeing into l exists. We now show how to erive this result from heorem. Let e i be the i th stanar basis vector in l, an efine E ij = e i e j e i e j for i < j n. Let U be the m n matrix whose i th column is the vector v i, where i =,..., n. hen, it is clear that the matrix X = U U satisfies the following system of equations: E ij X = v i v j for i < j n Now, heorem implies that we can fin an X 0 0 of ran at most such that: Ω n 4/ log n v i v j E ij X 0 O v i v j for i < j n Upon taing the Cholesy factorization X 0 = U0 U 0, we recover a set of points u,..., u n l such that: Ω n / log n v i v j u i u j O v i v j for i < j n as esire. We shoul point out that by using ifferent techniques, Matouše [7] was able to show that in fact an Θn embeing into l exists for the cases where =,. We remar that if we o not restrict the imension of the range of f, then by the Johnson Linenstrauss lemma [3, 4], for any ɛ > 0, there exists an + ɛ embeing of V into l, where = Oɛ log n. In [, Chapter V, Proposition 6.], Barvino generalizes this result an shows that if the assumptions of heorem are satisfie, then for any ɛ 0, an 8ɛ log4m, there exists an X 0 0 of ran at most such that: ɛb i A i X 0 + ɛb i for i =,..., m hus, heorem complements Barvino s result an generalizes the corresponing results in the stuy of bi Lipschitz embeings into low imensional Eucliean space [3, 7]. b Quaratic Optimization with Homogeneous Quaratic Constraints Consier the following optimization problems: vmaxqp = maximize x Ax subject to x A i x i =,..., m 5 vminqp = minimize x Ax subject to x A i x i =,..., m 6 3

4 where A,..., A m are symmetric positive semiefinite matrices. Both of these problems arise from various applications see [6, 8] an are NP har. heir natural SDP relaxations are given by: vmaxsp = maximize A X subject to A i X i =,..., m X 0 vminsp = minimize A X subject to A i X i =,..., m X It is clear that if X = xx is a ran feasible solution to 7 resp. 8, then x is a feasible solution to 5 resp. 6. Now, let Xmaxsp be an optimal solution to 7. It has been shown in [8] that one can extract a ran matrix X 0 from Xmaxsp such that i X 0 is feasible to 7 an ii A X 0 Ω log m vmaxqp. We now erive a similar result using heorem. By efinition, the matrix Xmaxsp satisfies the following system: A X maxsp = v maxsp, A i X maxsp = b i for i =,..., m As we shall see from the proof of heorem, one can fin a ran matrix X 0 0 such that: E [ A X 0] = v maxsp, A i X 0 Olog m b i for i =,..., m It follows that the matrix X 0 = Ω log m Ω log m vmaxsp Ω log m vmaxqp. X 0 0 is feasible to 7, an that E [A X 0] = In a similar fashion, if Xminsp is an optimal solution to 8, then one can extract a ran matrix X 0 0 from X minsp such that X 0 = Om X 0 is feasible for 8 an E [A X 0 ] = Om vminqp, thus recovering a result of Luo et al. [6]. In [6] the authors also consier a complex version of 5 an 6, in which the matrices A an A i are complex Hermitian an the components of the ecision vector x can tae on complex values. hey show that if Xmaxsp resp. X minsp is an optimal solution to the corresponing SDP relaxation 7 resp. 8, then one can extract a complex ran solution that achieves Ω log m resp. Om times the optimum value. Our result shows that these bouns are also achievable for the real version of 7 an 8 if we allow the solution matrix to have ran at most. Proof of the Main Result We first mae some stanar preparatory moves see, e.g., [, 6, 8]. Let X 0 be a solution to the system. By a result of Barvino [] an Patai [9], we may assume that r 0 ranx < 4

5 m. Let X = UU for some U R n r 0, an set A i = U A i U R r 0 r 0, where i =,..., m. hen, we have A i 0, rana i min{rana i, r 0 }, an b i = A i X = U A i U I = A i I = ra i Moreover, if X 0 0 satisfies the inequalities: βm, n, b i A i X 0 αm, n, b i for i =,..., m then upon setting X 0 = UX 0 U 0, we see that ranx 0 ranx 0, an i.e. X 0 satisfies the inequalities in. establish the following: A i X 0 = U A i U X 0 = A i X 0 hus, in orer to establish heorem, it suffices to heorem Let A,..., A m R n n be symmetric ps matrices, where n < m. hen, for any, there exists an X 0 0 with ranx 0 such that: βm, n, ra i A i X 0 αm, n, ra i for i =,..., m where αm, n, an βm, n, are given by 3 an 4, respectively. he proof of heorem relies on the following estimates of a chi square ranom variable. Proposition Let ξ,..., ξ n be i.i.. stanar Gaussian ranom variables. Let α, an β 0, be constants, an set U n = n i= ξ i. Note that U n χ n. hen, the following hol: Pr U n α n [ ] α n [ n α exp = exp α + log α ] 9 Pr U n β n [ β exp β ] n [ n = exp β + log β ] 0 Proof o establish 9, we let t [0, / an compute: Pr U n α n = Pr { exp [ t U n α n ] } E [ exp [ t U n α n ]] by Marov s inequality = exp tα n E [ exp tξ ] n by inepenence = exp tα n t n/ Let f : [0, / R be given by ft = exp tα n t n/. hen, we have: f t = exp tα n α n t n/ + exp tα n n t n/+ an hence f is minimize at t = α /. Note that t 0, / whenever α,. hus, we conclue that: Pr U n α n [ ] α f t n = α exp 5

6 o establish 0, we procee in a similar fashion. For t 0, we have: Pr U n β n = Pr { exp [ t β n U n ] } E [ exp [ t β n U n ]] by Marov s inequality = exp tβ n E [ exp tξ ] n by inepenence = exp tβ n + t n/ Now, let f : [0, R be given by ft = exp tβ n + t n/. hen, we have: f t = exp tβ n β n + t n/ exp tβ n n + t n/+ an hence f is minimize at t = β /. Moreover, we have t > 0 whenever β <. It follows that: Pr U n β n [ ] β f t n = β exp as esire. In the sequel, let be a given integer. Consier the following ranomize proceure for generating an X 0 0 of ran at most : Algorithm Proceure GenSoln Input: An integer. Output: An ps matrix X 0 of ran at most. : generate i.i.. Gaussian ranom variables ξ j i with mean 0 an variance /, an efine ξ j = ξ j,..., ξj n, where i =,..., n; j =,..., : return X 0 = j= ξj ξ j We remar that the above proceure is ifferent from those in [6, 8]. Let X 0 0 be the output of GenSoln. he following propositions form the heart of our analysis. Proposition Let H R n n be a symmetric positive semiefinite matrix. Consier the spectral ecomposition H = r = λ v v, where r = ranh an λ λ λ r > 0. Set λ = λ /λ + + λ r. hen, for any β 0,, we have: [ ] [ ] Pr H X 0 βrh r exp β + log β r exp + log β On the other han, if β satisfies eβ log r /5, then can be sharpene to: Pr H X 0 βrh 5eβ 6

7 Proof We first establish. Let q = λ v. hen, we have H = r = q q. Observe that q ξj is a Gaussian ranom variable with mean 0 an variance σ l q, e l where el is the l th coorinate vector. Moreover, we have: σ = = q e l = rh an E q ξ j = σ = l j= Hence, we conclue that: Pr q ξ j [ ] βσ = Pr U β exp β + log β j= for =,..., r Now, observe that H X 0 = r = j= q ξ j. Hence, we conclue that: Pr H X 0 βrh Pr q ξ j [ ] βσ r exp β + log β = j= o establish, we procee as follows. Clearly, we have H X 0 = r = j= λ v ξ j. Now, observe that u = v ξj,j N 0, I r. Inee, v ξj is a Gaussian ranom variable, as it is the sum of Gaussian ranom variables. Moreover, we have: E [ v ξj] = 0 an E [ v ξ j v l ξj ] = v v l = {=l} It follows that H X 0 has the same istribution as r = j= λ ξ j, where ξ j are i.i.. Gaussian ranom variables with mean 0 an variance /. Now, we compute: Pr H X 0 βrh = Pr λ ξ j β Define: = j= = Pr = j= = λ ξ j β p r, λ, β Pr = j= hen, by Proposition, we have: p r, λ, β Pr λ r ξ j β = Pr = j= λ λ ξ j β = j= where λ / = λ λ ξ j β r eβ r λ r r λ r = r/ 7

8 On the other han, we have: p r, λ, β r Pr = j= r λ ξ j β Pr = j= λ λ r ξ j β λ r Now, observe that: whence: λ r r λ = = p r, λ, β p r, λ :r λ, r β λ r It then follows from an easy inuctive argument that: p r, λ, β { } eβ / min r λ 3 Let α = p r, λ, β /. Note that α 0,. By 3, we have λ α / eβ for =,..., r. Upon summing over an using the fact that r = λ =, we obtain: = α / eβ 4 If r =, then we have α eβ. Henceforth, we shall assume that r. Note that for any α 0,, the function t tα /t is ecreasing for all t, since we have: t tα /t = log α t t 3 α /t < 0 Hence, it follows that: = α / r α + tα /t t = log/α α + e t t 5 log/α t r where we use the change of variable z = t log/α in the last step. Using the expansion: e t t = t j t j! = t + t j j +! j 0 j 0 8

9 we compute: log/α log/α r e t t t = log r + j 0 = log r + j 0 t j+ j + j +! log/α log/α r log j+ /α j + j +! r j+ log r + j 0 log j+ /α j +! = log r + α log r + α 6 Upon combining 4, 5 an 6, we conclue that: eβ α + log r which, together with the assumption that eβ log r /5, implies that α 5eβ/. Proposition 3 Let H R n n be a symmetric positive semiefinite matrix. Consier the spectral ecomposition H = r = λ v v, where r = ranh an λ λ λ r > 0. hen, for any α >, we have: [ ] Pr H X 0 αrh r exp α + log α 7 Proof As before, let q = λ v. hen, using the arguments in the proof of Proposition, we conclue that: Pr q ξ j [ ] ασ = PrU α exp α + log α for =,..., r j= Now, observe that H X 0 = r = j= q ξ j. Hence, we have: [ ] Pr H X 0 αrh r exp α + log α as esire. Proof of heorem We first establish the upper boun. We write α = +α for some α > 0. Using the inequality log + x x x / + x 3 /3, which is vali for all x > 0, it is easy to show that: α for α α + log α = α + log + α 6 9 α for 0 < α < 6 9 8

10 log4mn Let =. If, then set α = ; otherwise, set α =. In the former case, we have α, an hence by Proposition 3 an the boun in 9, for each i =,..., m, we have: Pr A i X 0 αra i rana i exp α 4m where the last inequality follows from the fact that rana i n. In the latter case, we have α 0,, an a similar calculation shows that: Pr A i X 0 αra i rana i exp for each i =,..., m. Hence, we conclue that: α 4m Pr A i X 0 αm, n, ra i for all i =,..., m 4 = where αm, n, is given by 3. Next, we establish the lower boun. We consier the following cases: Case : log m log logm Let β = 5em / in Proposition. Since r < m, we have: eβ log r < log m 0m/ 0 < 5 by our choice of. It follows that of Proposition applies, an we conclue that: Pr A i X 0 βra i / m for i =,..., m ogether with 0, we have: Pr βra i A i X 0 αm, n, ra i for all i =,..., m 3 4 / > 0 for all. Case : log m log logm < 4 log4mn Suppose that = log m log logm 4e for some >. Let β = log m 3/ in Proposition. Upon noting that m 3/ = log 3/ m an using of Proposition, we have: Pr A i X 0 βra i rana i eβ ranai ogether with 0, we have: m 3/ < Pr βra i A i X 0 αm, n, ra i for all i =,..., m m > 0

11 for all. Case 3: > 4 log4mn We write β = β for some β 0,. Using the inequality log x x x /, which is vali for all x [0, ], we have: β + log β = β + log β β Let β = 4 log4mn /. By assumption, we have β 0,. By of Proposition, for each i =,..., m, we have: Pr A i X 0 βra i rana i exp β 4 4m It follows that: Pr βra i A i X 0 αm, n, ra i for all i =,..., m = his completes the proof of heorem. 3 A Refinement In this section we show how heorem can be refine using the following set of estimates for a chi square ranom variable: Fact Laurent, Massart [5] Let ξ,..., ξ n be i.i.. stanar Gaussian ranom variables. Let a,..., a n 0, an set: n a = max a i, a = a i i n Define V n = n i= a iξ i i=. hen, for any t > 0, we have: Pr V n a t + a t e t / Pr V n a t e t / Fact allows us to use the conition number of the given matrix H to compute the eviation probabilities in Propositions an 3. o carry out this program, let us first recall some notations. Let H be a symmetric positive semiefinite matrix. Define r = ranh, an let K = λ /λ r be the conition number of H, where λ λ λ r > 0 are the eigenvalues of H. Set λ = λ /λ + + λ r. We then have the following proposition: Proposition 4 he following inequalities hol: a r λ K r ; b λ r +K + KK r +K ;

12 c + r K λ λ ; λ K λ. Proof a he first inequality follows from the fact that r j= λ j =. o establish the secon inequality, suppose to the contrary that λ > K/r i. hen, we have λ r > /r, whence r j= λ j > r /r + K/r >, which is a contraiction. b Let λ r = x. hen, we have λ = Kx. o boun λ, we first observe that for x < u v < Kx an ɛ min{u x, Kx v} > 0, we have u ɛ + v + ɛ > u + v. his implies that the vector λ that maximizes λ satisfies r λ r +K λ r =, or equivalently, λ r = r +K. his in turn yiels: λ r r + K + K r + K = KK + r + K r + K as esire. c We have: λ λ = + λ j λ j= + r K as esire. We compute: λ λ = λ + j= λ j λ λ + r λ + λ K K r r + K K where we use the fact that λ = λ K/r in the secon inequality. Using Fact an Proposition 4, we obtain the following refinements to heorem : heorem Uner the setting of heorem an the aitional assumptions that min i r i = Ωlog m an K max i m K i = O r, the event: occurs with constant probability. {A i X 0 Θ ra i for i =,..., m}

13 Proof that: Using the fact that r λ = j= = an setting t = β, we conclue by λ Pr = j= λ ξ j β = Pr = j= = Pr = j= β exp 4 exp [ Ωβr i ] λ ξ j λ ξ j λ β λ t Hence, by taing β = Θ, we conclue that: Pr A i X 0 Θ ra i = O/m for i =,..., m which in turn implies the theorem. heorem 3 Uner the setting of heorem an the aitional assumptions that min i r i = Ωlog m an K max i m K i = O, the event: occurs with constant probability. {A i X 0 Θ ra i for i =,..., m} Proof Using the arguments in the proof of heorem, we see that: Pr H X 0 αrh = Pr λ ξ j α Let t = = j= It then follows from an the efinition of t in 3 that: λ + λ α λ λ 3 Pr H X 0 αrh exp t / 4 Note that: t = λ i λ i + r i K i + λ i λ i α α + K i by equation 3 by Proposition 4c, = Ω αr i 3

14 Hence, by taing α = Θ, we conclue that: Pr A i X 0 Θ ra i = O/m for i =,..., m which in turn implies the theorem. 4 Conclusion In this note we have consiere the problem of fining a low ran approximate solution to a system of linear equations in symmetric, positive semiefinite matrices. As we have emonstrate, our result provies a unifie treatment of an generalizes several results in the literature. A main ingreient in our analysis is a set of tail estimates of a chi square ranom variable. We believe that these estimates coul be of inepenent interest. As a further illustration of our techniques, suppose that we are given positive semiefinite matrices A of ran r, where =,..., K. Consier a napsac semiefinite matrix equality: K A X = b, X 0 for =,..., K = Our goal is to fin a ran one matrix X 0 0 for each X such that: β b K A X 0 α b = hen, our result implies that the istortion rates woul be on the orer of logk r as oppose to K r obtaine from the stanar analysis where the terms are treate as K r inepenent equalities. References [] Alexaner I. Barvino, Problems of Distance Geometry an Convex Properties of Quaratic Maps, Discrete an Computational Geometry 3:89 0, 995. [] Alexaner Barvino, A Course in Convexity, Grauate Stuies in Mathematics Volume 54, American Mathematical Society, 00. [3] Sanjoy Dasgupta, Anupam Gupta, An Elementary Proof of the Johnson Linenstrauss Lemma, echnical Report R 99 06, International Computer Science Institute, Bereley, CA, 999. [4] W. B. Johnson, J. Linenstrauss, Extensions of Lipschitz Mapping into Hilbert Space, Contemporary Mathematics 6:89 06, 984. [5] B. Laurent, P. Massart, Aaptive Estimation of a Quaratic Functional by Moel Selection, he Annals of Statistics 85:30 338,

15 [6] Zhi Quan Luo, Nicholas D. Siiropoulos, Paul seng, Shuzhong Zhang, Approximation Bouns for Quaratic Optimization with Homogeneous Quaratic Constraints, to appear in SIAM Journal on Optimization, 006. [7] Jiří Matouše, Bi Lipschitz Embeings into Low Dimensional Eucliean Spaces, Commentationes Mathematicae Universitatis Carolinae 33: , 990. [8] A. Nemirovsi, C. Roos,. erlay, On Maximization of Quaratic Form over Intersection of Ellipsois with Common Center, Mathematical Programming, Series A 86: , 999. [9] Gábor Patai, On the Ran of Extreme Matrices in Semiefinite Programs an the Multiplicity of Optimal Eigenvalues, Mathematics of Operations Research 3: , 998. [0] Anthony Man Cho So, Yinyu Ye, A Semiefinite Programming Approach to ensegrity heory an Realizability of Graphs, Proceeings of the 7th Annual ACM SIAM Symposium on Discrete Algorithms, pp ,