On the local stability of semidefinite relaxations

On the local stability of semidefinite relaxations Diego Cifuentes Department of Mathematics Massachusetts Institute of Technology Joint work with Sameer Agarwal (Google), Pablo Parrilo (MIT), Rekha Thomas (U. Washington). arxiv:1710.04287 Real Algebraic Geometry and Optimization - ICERM - 2018 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 1 / 21

Nearest point problems Given a variety X R n, and a point θ R n, x x θ 2 s.t. x X A variety is the zero set of some polynomials X := {x R n : f 1 (x) = = f m (x) = 0} Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 2 / 21

Nearest point problems Given a variety X R n, and a point θ R n, x x θ 2 s.t. x X A variety is the zero set of some polynomials X := {x R n : f 1 (x) = = f m (x) = 0} This problem is nonconvex, and computationally challenging. SDP relaxations have been successful in several applications. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 2 / 21

Nearest point problems Given a variety X R n, and a point θ R n, x x θ 2 s.t. x X A variety is the zero set of some polynomials X := {x R n : f 1 (x) = = f m (x) = 0} This problem is nonconvex, and computationally challenging. SDP relaxations have been successful in several applications. Goal Study the behavior of SDP relaxations in the low noise regime: when x is sufficiently close to X. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 2 / 21

Nearest point problems Many different applications Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 3 / 21

Nearest point to the twisted cubic x X x θ 2, where X := {(x 1, x 2, x 3 ) : x 2 = x 2 1, x 3 = x 1 x 2 } The twisted cubic X can be parametrized as t (t, t 2, t 3 ). Its Lagrangian dual is the following SDP: max γ,λ 1,λ 2 R γ, s.t. γ+ θ 2 θ 1 λ 1 θ 2 λ 2 θ 3 θ 1 1 2λ 1 λ 2 0 0. λ 1 θ 2 λ 2 1 0 λ 2 θ 3 0 0 1 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 4 / 21

Nearest point to the twisted cubic x X x θ 2, where X := {(x 1, x 2, x 3 ) : x 2 = x 2 1, x 3 = x 1 x 2 } 3 Nearest point to the twisted cubic 1.0 2 zero duality gap 0.8 3 1 0 1 3 = 3 1 0.6 0.4 duality gap 2 0.2 3 0.0 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 1 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 4 / 21

Nearest point problem to a quadratic variety Theorem If θ X is a regular point then there is zero-duality-gap for any θ R n that is sufficiently close to θ. Applications: Triangulation problem [Aholt-Agarwal-Thomas] Nearest (symmetric) rank one tensor Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 5 / 21

Parametrized QCQPs Consider a family of quadratically constrained programs (QCQPs): x R N g θ (x) h i θ(x) = 0 for i = 1,..., m (P θ ) where g θ, hθ i are quadratic, and the dependence on θ is continuous. The Lagrangian dual is an SDP. Goal: Given θ for which the SDP relaxation is tight, analyze the behavior as θ θ. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 6 / 21

Parametrized QCQPs Consider a family of quadratically constrained programs (QCQPs): x R N g θ (x) h i θ(x) = 0 for i = 1,..., m (P θ ) where g θ, hθ i are quadratic, and the dependence on θ is continuous. The Lagrangian dual is an SDP. Goal: Given θ for which the SDP relaxation is tight, analyze the behavior as θ θ. Example: For a nearest point problem g θ (x) := x θ 2, h i (x) independent of θ The problem is trivial for any θ X. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 6 / 21

SDP relaxation of a (homogeneous) QCQP Primal problem Dual problem x R N max λ R m x T G θ x x T H i θx = b i i = 1,..., m d(λ) := i λ ib i Q θ (λ) 0 where Q θ (λ) is the Hessian of the Lagrangian (P θ ) (D θ ) Q θ (λ) := G θ + i λ i H i θ S N. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 7 / 21

SDP relaxation of a (homogeneous) QCQP Primal problem Dual problem x R N max λ R m x T G θ x x T H i θx = b i i = 1,..., m d(λ) := i λ ib i Q θ (λ) 0 (P θ ) (D θ ) Problem statement Assume that val(p θ ) = val(d θ ), i.e., θ is a zero-duality-gap parameter. Find conditions under which val(p θ ) = val(d θ ) when θ is close to θ. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 7 / 21

Characterization of zero-duality-gap Given x θ primal feasible, its Lagrange multipliers are: Lemma λ Λ θ (x θ ) λ T h θ (x θ ) = g θ (x θ ) Q θ (λ)x θ = 0. Let x θ R N, λ R m. Then x θ is optimal to (P θ ) and λ is optimal to (D θ ) with val(p θ ) = val(d θ ) iff: 1 h θ (x θ ) = 0 (primal feasibility). 2 Q θ (λ) 0 (dual feasibility). 3 λ Λ θ (x θ ) (complementarity). Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 8 / 21

Characterization of zero-duality-gap Given x θ primal feasible, its Lagrange multipliers are: Lemma λ Λ θ (x θ ) λ T h θ (x θ ) = g θ (x θ ) Q θ (λ)x θ = 0. Let x θ R N, λ R m. Then x θ is optimal to (P θ ) and λ is optimal to (D θ ) with val(p θ ) = val(d θ ) iff: 1 h θ (x θ ) = 0 (primal feasibility). 2 Q θ (λ) 0 (dual feasibility). 3 λ Λ θ (x θ ) (complementarity). Proof. If Q θ (λ)x θ = 0 and h θ (x θ ) = 0, then 0 = x T θ Q θ (λ)x θ = x T θ G θ x θ + i λ i x T θ H i x θ = g θ (x θ ) d(λ). Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 8 / 21

Characterization of zero-duality-gap Lemma Let θ be a zero-duality-gap parameter with ( x, λ) primal/dual optimal. Assume that 1 Q θ( λ) has corank-one (strict-complementarity) 2 x θ feasible for (P θ ), λ θ Λ θ (x θ ) s.t. (x θ, λ θ ) θ θ ( x, λ). Then there is zero-duality-gap when θ is close to θ. Proof. Q θ (λ θ ) has a zero eigenvalue (Q θ (λ θ )x θ = 0). Q θ (λ θ ) Q θ( λ) (the dependence on θ is continuous). Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 9 / 21

Characterization of zero-duality-gap Lemma Let θ be a zero-duality-gap parameter with ( x, λ) primal/dual optimal. Assume that 1 Q θ( λ) has corank-one (strict-complementarity) 2 x θ feasible for (P θ ), λ θ Λ θ (x θ ) s.t. (x θ, λ θ ) θ θ ( x, λ). Then there is zero-duality-gap when θ is close to θ. Proof. Q θ (λ θ ) has a zero eigenvalue (Q θ (λ θ )x θ = 0). Q θ (λ θ ) Q θ( λ) (the dependence on θ is continuous). Q θ( λ) has N 1 positive eigenvalues. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 9 / 21

Characterization of zero-duality-gap Lemma Let θ be a zero-duality-gap parameter with ( x, λ) primal/dual optimal. Assume that 1 Q θ( λ) has corank-one (strict-complementarity) 2 x θ feasible for (P θ ), λ θ Λ θ (x θ ) s.t. (x θ, λ θ ) θ θ ( x, λ). Then there is zero-duality-gap when θ is close to θ. Proof. Q θ (λ θ ) has a zero eigenvalue (Q θ (λ θ )x θ = 0). Q θ (λ θ ) Q θ( λ) (the dependence on θ is continuous). Q θ( λ) has N 1 positive eigenvalues. Q θ (λ θ ) also has N 1 positive eigenvalues (continuity of eigenvalues). Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 9 / 21

Characterization of zero-duality-gap Lemma Let θ be a zero-duality-gap parameter with ( x, λ) primal/dual optimal. Assume that 1 Q θ( λ) has corank-one (strict-complementarity) 2 x θ feasible for (P θ ), λ θ Λ θ (x θ ) s.t. (x θ, λ θ ) θ θ ( x, λ). Then there is zero-duality-gap when θ is close to θ. Proof. Q θ (λ θ ) has a zero eigenvalue (Q θ (λ θ )x θ = 0). Q θ (λ θ ) Q θ( λ) (the dependence on θ is continuous). Q θ( λ) has N 1 positive eigenvalues. Q θ (λ θ ) also has N 1 positive eigenvalues (continuity of eigenvalues). Q θ (λ θ ) 0, so there is zero-duality-gap. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 9 / 21

Nearest point to a quadratic variety x X x θ 2, where X := {x R n : f 1 (x) = = f m (x) = 0} Theorem Let θ be a regular point of X, i.e. rank f ( θ) = codim X. Then there is zero-duality-gap for θ close to θ. Proof. Since θ X, then x = θ, and λ = 0. Need to find λ θ Λ θ (x θ ) s.t. λ θ θ θ 0. Regularity implies λ θ 2 σ( f ) x θ θ θ θ 0. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 10 / 21

Nearest point to a quadratic variety x X x θ 2, where X := {x R n : f 1 (x) = = f m (x) = 0} Theorem Let θ be a regular point of X, i.e. rank f ( θ) = codim X. Then there is zero-duality-gap for θ close to θ. Proof. Since θ X, then x = θ, and λ = 0. Need to find λ θ Λ θ (x θ ) s.t. λ θ θ θ 0. Regularity implies λ θ 2 σ( f ) x θ θ θ θ 0. Remark: The theorem generalizes to the case of strictly convex objective. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 10 / 21

Guaranteed region of zero-duality-gap x X x θ 2, where X := {x R 3 : x 2 = x 2 1, x 3 = x 1 x 2 } 6 4 zero duality gap 2 3 0 2 Y 4 6 1.5 1.0 0.5 0.0 0.5 1.0 1.5 1 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 11 / 21

Application: Triangulation [Aholt-Agarwal-Thomas] Problem Given noisy images û j R 2 of an unknown point, u U u j û j 2 where U is the multiview variety of the cameras. j Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 12 / 21

Application: Triangulation [Aholt-Agarwal-Thomas] Problem Given noisy images û j R 2 of an unknown point, u U u j û j 2 where U is the multiview variety of the cameras. j If either n = 2, or n 4 and the camera centers are not coplanar, then U is defined by the (quadratic) epipolar constraints. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 12 / 21

Application: Triangulation [Aholt-Agarwal-Thomas] Problem Given noisy images û j R 2 of an unknown point, u U u j û j 2 where U is the multiview variety of the cameras. j If either n = 2, or n 4 and the camera centers are not coplanar, then U is defined by the (quadratic) epipolar constraints. The regularity condition is easy to check. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 12 / 21

Application: Triangulation [Aholt-Agarwal-Thomas] Problem Given noisy images û j R 2 of an unknown point, u U u j û j 2 where U is the multiview variety of the cameras. j If either n = 2, or n 4 and the camera centers are not coplanar, then U is defined by the (quadratic) epipolar constraints. The regularity condition is easy to check. Under low noise the SDP relaxation is tight. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 12 / 21

Application: Rank one approximation Problem Given a tensor ˆx R n 1 n l, consider x X x ˆx 2 where X is the variety of rank one tensors (Segre). Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 13 / 21

Application: Rank one approximation Problem Given a tensor ˆx R n 1 n l, consider x X x ˆx 2 where X is the variety of rank one tensors (Segre). The Segre variety is defined by quadratics (2 2 ors). Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 13 / 21

Application: Rank one approximation Problem Given a tensor ˆx R n 1 n l, consider x X x ˆx 2 where X is the variety of rank one tensors (Segre). The Segre variety is defined by quadratics (2 2 ors). Thus, the SDP relaxation is tight under low noise. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 13 / 21

Application: Rotation synchronization Problem Given a graph G = (V, E) and matrices ˆR ij R d d for ij E, R 1,...,R n SO(d) R j ˆR ij R i 2 F ij E Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 14 / 21

Application: Rotation synchronization Problem Given a graph G = (V, E) and matrices ˆR ij R d d for ij E, R 1,...,R n SO(d) R j ˆR ij R i 2 F ij E The objective function is strictly convex. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 14 / 21

Application: Rotation synchronization Problem Given a graph G = (V, E) and matrices ˆR ij R d d for ij E, R 1,...,R n SO(d) R j ˆR ij R i 2 F ij E The objective function is strictly convex. Thus, the SDP relaxation is tight under low noise. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 14 / 21

Application: Rotation synchronization Problem Given a graph G = (V, E) and matrices ˆR ij R d d for ij E, R 1,...,R n SO(d) R j ˆR ij R i 2 F ij E The objective function is strictly convex. Thus, the SDP relaxation is tight under low noise. Similar tightness results have been shown [Fredriksson-Olsson], [Rosen-Carlone-Bandeira-Leonard], [Wang-Singer]. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 14 / 21

Application: Orthogonal Procrustes Problem Given matrices A R m 1 n, B R m 1 m 2, C R k m 2, X St(n,k) AXC B 2 F where St(n, k) is the Stiefel manifold. The objective function is strictly convex. Thus, the SDP relaxation is tight under low noise. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 15 / 21

Nearest point to non-quadratic varieties Any variety can be described by quadratics by using auxiliary variables. Example: Nearest point problem to the curve y 2 2 = y 3 1 can be written as y θ 2, s.t. y 2 = y 1 z, y 1 = z 2, y 2 z = y1 2. y R 2,z R The objective is not strict convex. 1.5 1.0 0.5 0.0 z 0.5 1.0 0.5 0.0 0.5 1.0 1.5 y 1 1 2.0 2 0 y 2 1.5 2 1 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 16 / 21

Stability of SDP relaxations of (arbitrary) QCQPs Consider a general family of QCQPs: x R N g θ (x) h i θ(x) = 0 for i = 1,..., m (P θ ) Let θ be a zero-duality-gap parameter: val(p θ ) = val(d θ ). There are bad cases, where the SDP relaxation is non-informative. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 17 / 21

Stability of SDP relaxations of (arbitrary) QCQPs Consider a general family of QCQPs: x R N g θ (x) h i θ(x) = 0 for i = 1,..., m (P θ ) Let θ be a zero-duality-gap parameter: val(p θ ) = val(d θ ). There are bad cases, where the SDP relaxation is non-informative. We introduce a Slater-type condition that guarantees zero-duality-gap nearby θ. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 17 / 21

Stability of SDP relaxations of QCQPs Let θ be a zero-duality-gap parameter with ( x, λ) primal/dual optimal. Assumption (restricted Slater) There is µ R m s.t. the quadratic function Ψ µ (x) := i µ ih ī (x) satisfies: θ Ψ µ ( x) = 0, and Ψ µ is strictly convex on ker Q θ( λ). Theorem Under the restricted Slater assumption and some regularity conditions, there is zero-duality-gap when θ is close to θ. Moreover, the SDP recovers the imizer. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 18 / 21

Stability of SDP relaxations of QCQPs Let θ be a zero-duality-gap parameter with ( x, λ) primal/dual optimal. Assumption (restricted Slater) There is µ R m s.t. the quadratic function Ψ µ (x) := i µ ih ī (x) satisfies: θ Ψ µ ( x) = 0, and Ψ µ is strictly convex on ker Q θ( λ). Theorem Under the restricted Slater assumption and some regularity conditions, there is zero-duality-gap when θ is close to θ. Moreover, the SDP recovers the imizer. Applications (ongoing): Higher levels of SOS/Lasserre hierarchy. For instance: system identification, noisy deconvolution, camera resectioning, homography estimation, approximate GCD. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 18 / 21

Using geometry to derive smaller SDP relaxations Primal problem x X x T Gx X = {x : x T H i x = b i for i = 1,..., m} Dual problem max λ R m,q S N i λ ib i Q = G + i λ ih i Q 0 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 19 / 21

Using geometry to derive smaller SDP relaxations Primal problem x X x T Gx X = {x : x T H i x = b i for i = 1,..., m} Dual problem Let ˆx 1,, ˆx S X max λ R m,q S N i λ ib i ˆx T j Qˆx j = ˆx j T Gˆx j + i λ i ˆx j T H i ˆx j for j = 1,..., S Q 0 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 19 / 21

Using geometry to derive smaller SDP relaxations Primal problem x X x T Gx X = {x : x T H i x = b i for i = 1,..., m} Dual problem Let ˆx 1,, ˆx S X max λ R m,q S N i λ ib i ˆx j T Qˆx j = ˆx j T Gˆx j + i λ ib i for j = 1,..., S Q 0 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 19 / 21

Using geometry to derive smaller SDP relaxations Primal problem x X x T Gx X = {x : x T H i x = b i for i = 1,..., m} Dual problem Let ˆx 1,, ˆx S X max γ R,Q S N γ ˆx j T Qˆx j = ˆx j T Gˆx j + γ for j = 1,..., S Q 0 SDP is smaller, e.g., the multipliers λ R m disappear. relaxation is stronger. Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 19 / 21

Example: Orthogonal Procrustes Problem Given matrices A R m 1 n, B R m 1 m 2, C R k m 2, X St(n,k) AXC B 2 F where St(n, k) is the Stiefel manifold. n r Equations SDP Gröbner Sampling SDP variables constraints time(s) basis (s) variables constraints time(s) 5 3 682 233 0.65 0.03 137 130 0.11 6 4 1970 576 1.18 9.94 326 315 0.14 7 5 4727 1207 3.56-667 651 0.24 8 6 9954 2255 13.88-1226 1204 0.45 9 7 19028 3873 42.14-2081 2052 1.10 10 8 33762 6238 124.43-3322 3285 2.48 Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 20 / 21

Summary We analyzed the local stability of SDP relaxations. Found sufficient conditions for zero-duality-gap nearby θ. Many applications (triangulation, rank one approximation, rotation synchronization, orthogonal Procrustes). If you want to know more: D. Cifuentes, S. Agarwal, P. Parrilo, R. Thomas, On the local stability of semidefinite relaxations, arxiv:1710.04287. D. Cifuentes, C. Harris, B. Sturmfels, The geometry of SDP-exactness in quadratic optimization, arxiv:1804.01796. D. Cifuentes, P. Parrilo, Sampling algebraic varieties for sum of squares programs, arxiv:1511.06751. Thanks for your attention! Cifuentes (MIT) Stability of semidefinite relaxations ICERM 18 21 / 21