Facial Reduction for Cone Optimization

Transcription

1 Facial Reduction for Cone Optimization Henry Wolkowicz Dept. Combinatorics and Optimization University of Waterloo Tuesday Apr. 21, 2015 (with: Drusvyatskiy, Krislock, (Cheung) Voronin)

2 Motivation: Loss of Slater CQ/Facial reduction optimization algorithms key is KKT system; (Slater s CQ/strict feasibility for convex conic optimization) Slater CQ holds generically Surprisingly, for many conic opt, SDP relaxations, arising from applications Slater s fails, e.g., SNL, POP, Molecular Conformation, QAP, GP, strengthened MC Slater fails = : -unbounded dual solutions; -theoretical and numerical difficulties (in particular for primal-dual interior-point methods). solutions? - theoretical facial reduction (Borwein, W. 81) - preprocess for regularized smaller problem (Cheung, Schurr, W. 11) - take advantage of degeneracy (e.g. recent: Krislock, W. 10; Cheung, Drusvyatskiy, Krislock, W. 14; Reid, Wang, W. Wu 15 )

3 3 Outline: Regularization/Facial Reduction 1 2 Abstract convex program LP case CP case Cone optimization/sdp case 3 Applications: SNL, Polyn Opt., QAP, GP, Molec. conformation...

4 4 Facial Reduction on (dual) LP, Ax = b, x 0 Theorem of alternative, ˆx s.t. Aˆx = b, ˆx > 0 iff A y 0, b y = 0, = y = 0 A full row rank Linear Programming Example, x R 5 min ( ) [ ] x s.t. x = ( ) ( ) 1, x 0 1 Sum the two constraints (use y T = (1 1) in (**)): 2x 1 + x 4 + x 5 = 0 = x 1 = x 4 = x 5 = 0 yields equivalent simplified problem: min 6x 2 x 3 s.t. x 2 + x 3 = 1, x 2, x 3 0

5 Facial Reduction on Primal, A T y c Linear Programming Example, y R 2 max ( 2 6 ) y 1 1 s.t y , ( active ) set {2, 3, 4} 3/2 is optimal, p 1/2 = 6 [ ] weighted last two rows sum to zero: set of implicit equalities: P e := {3, 4} Facial reduction to 1 dim. after substit. for y ( ) ( ) ( ) y1 1 1 = + t, max { 2 + 8t : 1 t }, t = 1 2. y 2 5

6 6 General Case? Abstract convex program Cone optimization/sdp case Can we do facial reduction in general? Is it efficient/worthwhile? applications?

7 Abstract convex program Cone optimization/sdp case Background/Abstract convex program (ACP) inf x f (x) s.t. g(x) K 0, x Ω where: f : R n R convex; g : R n R m is K -convex K R m closed convex cone; Ω R n convex set a K b b a K, a K b b a int K g(αx + (1 αy)) K αg(x) + (1 α)g(y), x, y R n, α [0, 1] Slater s CQ: ˆx Ω s.t. g(ˆx) int K (g(x) K 0) guarantees strong duality (near) loss of strict feasibility, nearness to infeasibility, correlates with number of iterations & loss of accuracy

8 Abstract convex program Cone optimization/sdp case Back to: Case of Linear Programming, LP Primal-Dual Pair: A onto, m n, P = {1,..., n} (LP-P) max s.t. b y A y c (LP-D) min c x s.t. Ax = b, x 0. Slater s CQ for (LP-P) / Theorem of alternative ŷ s.t. c A (( ŷ > 0, ) c A ŷ > 0, i P =: P lt) i iff Ad = 0, c d = 0, d 0 = d = 0 ( ) implicit equality constraints: i P e Find 0 d to ( ) with max number of non-zeros (exposes minimal face containing feasible slacks) di > 0 = (c A y) i = 0, y F y (i P e ) (where F y is primal feasible set) 8

9 Abstract convex program Cone optimization/sdp case Make implicit-equalities explicit/ Regularizes LP Facial Reduction: A y f c; minimal face f R n + (LP reg -P) max s.t. b y (A lt ) y c lt (LP reg -D) (A e ) y = c e min s.t. (c lt ) x lt + (c e ) x e [A ( lt A e] x lt ) x e = b x lt 0, x e free Mangasarian-Fromovitz CQ (MFCQ) holds (after deleting redundant equality constraints!) ( i P lt i P e ) ŷ : (A lt ) ŷ < c lt (A e ) ŷ = c e (A e ) is onto MFCQ holds iff dual optimal set is compact Numerical difficulties if MFCQ fails; in particular for interior point methods! Modelling issue? 9

10 10 Abstract convex program Cone optimization/sdp case Case of ordinary convex programming, CP (CP) sup y b y s.t. g(y) 0, where b R m ; g(y) = ( g i (y) ) R n, g i : R m R convex, i P Slater s CQ: ŷ s.t. g i (ŷ) < 0, i (implies MFCQ) Slater s CQ fails implies implicit equality constraints exist P e := {i P : g(y) 0 = g i (y) = 0} Let P lt := P\P e and g lt := (g i ) i P lt, g e := (g i ) i P e

11 Abstract convex program Cone optimization/sdp case Rewrite implicit equalities to equalities/ Regularize CP (CP) is equivalent to g(y) f 0, (CP reg ) where F e := {y : g e (y) = 0}. f is minimal face sup b y s.t. g lt (y) 0 y F e or (g e (y) = 0) Then F e = {y : g e (y) 0}, so is a convex set! Slater s CQ holds for (CP reg ) ŷ F e : g lt (ŷ) < 0 modelling issue again?

12 12 Faithfully convex case Abstract convex program Cone optimization/sdp case Faithfully convex function f (Rockafellar 70 ) f affine on a line segment only if affine on complete line containing the segment (e.g. analytic convex functions) F e = {y : g e (y) = 0} is an affine set Then: F e = {y : Vy = V ŷ} for some ŷ and full-row-rank matrix V. Then MFCQ holds for regularized (CP reg ) sup b y s.t. g lt (y) 0 Vy = V ŷ

13 13 Abstract convex program Cone optimization/sdp case Faces of Cones - Useful for Charact. of Opt. Face A convex cone F is a face of convex cone K, denoted F K, if x, y K and x + y F = x, y F Polar (Dual) Cone K := {φ : φ, k 0, k K } Conjugate Face If F K, the conjugate face of F is F c := F K K If x ri(f), then F c = {x} K.

14 14 Recall: (ACP) Abstract convex program Cone optimization/sdp case inf x f (x) s.t. g(x) K 0, x Ω polar cone: K = {φ : φ, y 0, y K }. K f := face(f) minimal face containing feasible set F. Lemma (Facial Reduction; find exposing vector φ) Suppose x is feasible. Then the LHS system { (Ω x) + } φ, g( x) φ K + implies K, φ, g( x) = 0 f φ K. Proof line 1 of system implies x global min for convex function φ, g( ) on Ω; i.e., 0 = φ, g( x) φ, g(x) 0, x F; implies g(f) φ K.

15 Abstract convex program Cone optimization/sdp case Semidefinite Programming, SDP, S n + K = S n + = K : nonpolyhedral, self-polar, facially exposed (SDP-P) (SDP-D) v P = sup y R m b y s.t. g(y) := A y c S n + 0 v D = inf x S n c, x s.t. Ax = b, x S n + 0 where: PSD cone S+ n S n symm. matrices c S n, b R m A : S n R m is an onto linear map, with adjoint A Ax = (trace A i x) = ( A i, x ) R m, A y = m i=1 A iy i S n 15 A i S n

16 16 Abstract convex program Cone optimization/sdp case Slater s CQ/Theorem of Alternative (Assume feasibility: ỹ s.t. c A ỹ 0.) ŷ s.t. s = c A ŷ 0 (Slater) iff Ad = 0, c, d = 0, d 0 = d = 0 ( )

17 17 Abstract convex program Cone optimization/sdp case Regularization Using Minimal Face Borwein-W. 81, f P = face F s P ; min. face of feasible slacks (SDP-P) is equivalent to the regularized (SDP reg -P) v RP := sup y { b, y : A y fp c} f p is miniminal face of primal feasible slacks {s 0 : s = c A y} f p S n + Lagrangian Dual DRP Satisfies Strong Duality: (SDP reg -D) v DRP := inf x { c, x : A x = b, x f P 0} = v P = v RP and v DRP is attained.

18 18 SDP Regularization process Abstract convex program Cone optimization/sdp case Alternative to Slater CQ Ad = 0, c, d = 0, 0 d S n + 0 ( ) Determine a proper face f p f = QS n +Q T S n + Let d solve ( ) with compact spectral decomosition d = Pd + P, d + 0, and [P Q] R n n orthogonal. Then c A y S n + 0 = c A y, d = 0 (implicit rank reduction, n < n) = F s P Sn + {d } = QS n + Q S n +

19 19 Regularizing SDP Abstract convex program Cone optimization/sdp case at most n 1 iterations to satisfy Slater s CQ. to check Theorem of Alternative Ad = 0, c, d = 0, 0 d S n + 0, ( ) use stable auxiliary problem (AP) min δ,d δ s.t. [ Ad c, d ] 2 δ, trace(d) = n, d 0. Both (AP) and its dual satisfy Slater s CQ.

20 20 Auxiliary Problem Abstract convex program Cone optimization/sdp case (AP) min δ,d δ s.t. [ Ad c, d ] 2 δ, trace(d) = n, d 0. Both (AP) and its dual satisfy Slater s CQ... but... Cheung-Schurr-W 11, a k = 1 step CQ Strict complementarity holds for (AP) iff k = 1 steps are needed to regularize (SDP-P). k = 1 always holds in LP case.

21 21 Conclusion Part I Abstract convex program Cone optimization/sdp case Minimal representations of the data regularize (P); use min. face f P (and/or implicit rank reduction) ideal goal: a backwards stable preprocessing algorithm to handle (feasible) conic problems for which Slater s CQ (almost) fails Puzzle? Minimal representations are needed for stability in cone optimization. But adding redundant constraints to quadratic models before lifting often strengthens SDP relaxation.

22 22 Part II: Applications of SDP where Slater s CQ fails Instances SDP relaxations of NP-hard comb. opt. Quadratic Assignment (Zhao-Karish-Rendl-W. 96 ) Graph partitioning (W.-Zhao 99 ) Strengthened Max-Cut (Anjos-W 02 ) Low rank problems Systems of polynomial equations (Reid-Wang-W.-Wu 15) Sensor network localization (SNL) problem (Krislock-W. 10, Krislock-Rendl-W. 10) Molecular conformation (Burkowski-Cheung-W. 11 ) general SDP relaxation of low-rank matrix completion problem

23 SNL ( K-W 10, D-K-V-W 14 ) Highly (implicit) degenerate/low-rank problem - high (implicit) degeneracy translates to low rank solutions - take advantage of degeneracy; fast, high accuracy solutions SNL - a Fundamental Problem of Distance Geometry; easy to describe - dates back to Grasssmann 1886 r : embedding dimension n ad hoc wireless sensors p 1,..., p n R r to locate in R r ; m of the sensors p n m+1,..., p n are anchors (positions known, using e.g. GPS) pairwise distances D ij = p i p j 2, ij E, are known within radio range R > 0 P = [ p 1... p n ] = [ X A ] R r n 23

24 24 Sensor Localization Problem/Partial EDM Sensors and Anchors 10 9 Initial position of points sensors anchors sens anch # sensors n = 300, # anchors m = 9, radio range R = 1.2

25 25 Underlying Graph Realization/Partial EDM NP-Hard Graph G = (V, E, ω) node set V = {1,..., n} edge set (i, j) E ; ω ij = p i p j 2 known approximately The anchors form a clique (complete subgraph) Realization of G in R r : a mapping of nodes v i p i R r with squared distances given by ω. Corresponding Partial Euclidean Distance Matrix, EDM { d 2 D ij = ij if (i, j) E 0 otherwise (unknown distance), d 2 ij = ω ij are known squared Euclidean distances between sensors p i, p j ; anchors correspond to a clique.

26 26 Connections to Semidefinite Programming (SDP) D = K (B) E n, B = K (D) S n S C (centered Be = 0) P = [ p 1 p 2... p n ] M r n ; B := PP S n + (Gram matrix of inner products); rank B = r; let D E n corresponding EDM ; e = ( ) (to D E n ) D = ( p i p j 2 n ( 2) i,j=1 ) n = pi T p i + pj T p j 2pi T p j i,j=1 = diag (B) e + e diag (B) 2B =: K (B) (from B S n + ).

27 27 Euclidean Distance Matrices; Semidefinite Matrices Moore-Penrose Generalized Inverse K B 0 = D = K(B) = diag (B) e + e diag (B) 2B E D E = B = K (D) = 1 2JoffDiag (D) J 0, Be = 0 Theorem (Schoenberg, 1935) A (hollow) matrix D (with diag (D) = 0, D S H ) is a Euclidean distance matrix if and only if B = K (D) 0. And!!!! embdim (D) = rank ( K (D) ), D E n (1)

28 28 Popular Techniques; SDP Relax.; Highly Degen. Nearest, Weighted, SDP Approx. (relax/discard rank B) min B 0 H (K (B) D) ; rank B = r; typical weights: H ij = 1/ D ij, if ij E, H ij = 0 otherwise. with rank constraint: a non-convex, NP-hard program SDP relaxation is convex, BUT: expensive/low accuracy/implicitly highly degenerate (cliques restrict ranks of feasible Bs) Instead: (Shall) Take Advantage of Degeneracy! clique α, α = k (corresp. D[α]) with embed. dim. = t r < k = rank K (D[α]) = t r = rank B[α] rank K (D[α]) + 1 = rank B = rank K (D) n (k t 1) = Slater s CQ (strict feasibility) fails

29 29 Basic Single Clique/Facial Reduction Matrix with Fixed Principal Submatrix For Y S n, α {1,..., n}: Y [α] denotes principal submatrix formed from rows & cols with indices α. D E k, α 1:n, α = k Define E n (α, D) := { D E n : D[α] = D }. (completions) Given D; find a corresponding B 0; find the corresponding face; find the corresponding subspace. if α = 1 : k; embedding dim embdim ( D) = t r [ ] D D =.

30 30 BASIC THEOREM for Single Clique/Facial Reduction Let: D := D[1:k] E k, k < n, embdim ( D) = t r be given; B := K ( D) = ŪBSŪ B, ŪB M k t, Ū B ŪB = I t, S S++ t be full rank orthogonal decomposition of Gram matrix; ] [ ] 1 U B := [ŪB k e M k (t+1) UB 0, U :=, and 0 I [ ] n k V M n k+t+1 be orthogonal. U e U e Then the minimal face: face K ( E n (1:k, D) ) ( ) = US+ n k+t+1 U S C = (UV )S+ n k+t (UV )

31 31 The minimal face face K ( E n (1:k, D) ) = ( ) US+ n k+t+1 U S C = (UV )S+ n k+t (UV ) Note that the minimal face is defined by the subspace L = R (UV ). We add 1 k e to represent N (K ); then we use V to eliminate e to recover a centered face.

32 Facial Reduction for Disjoint Cliques Corollary from Basic Theorem let α 1,..., α l 1:n pairwise disjoint sets, wlog: α i = (k i 1 + 1):k i, k 0 = 0, α := l i=1 α i = 1: α let Ū i R α i (t i +1) with full column rank satisfy e R (Ūi) and k i 1 t i +1 n k i k i 1 I 0 0 U i := α i 0 Ū i 0 R n (n α i +t i +1) n k i 0 0 I The minimal face is defined by L = R (U): t t l +1 n α α 1 Ū U := α l 0... Ū l 0 Rn (n α +t+1), n α I where t := l i=1 t i + l 1. And e R (U). 32

33 33 Sets for Intersecting Cliques/Faces α 1 := 1:( k 1 + k 2 ); α 2 := ( k 1 + 1):( k 1 + k 2 + k 3 ) α 1 α 2 k 1 k 2 k3

34 34 Two (Intersecting) Clique Reduction/Subsp. Repres. Let: Then α 1, α 2 1:n; k := α 1 α 2 for i = 1, 2: D i := D[α i ] E k i, embedding dimension t i ; B i := K ( D i ) = ŪiS i Ūi, Ūi M k i t i, Ū i Ū i = I ti, S i S t i ++ ; 1 U i := [Ūi e ] ki M k i (t i +1) ; and Ū M k (t+1) ([ ]) ([ ]) satisfies U1 R (Ū) = R 0 0 R I k1 0, with I k3 0 U Ū Ū = I t+1 2 ] [Ū U := 0 0 I n k M n (n k+t+1) and [ V be orthogonal. 2i=1 face K ( E n (α i, D i ) ) = ( US n k+t+1 + U ) S C = (UV )S n k+t + (UV ) ] U e U e M n k+t+1

35 35 Expense/Work of (Two) Clique/Facial Reductions Subspace Intersection for Two Intersecting Cliques/Faces Suppose: U 1 0 I 0 U 1 = U 1 0 and U 2 = 0 U 2 0 I 0 U 2 Then: U 1 U := U 1 or U := U 2 (U 2 ) U 1 U 1 (U 1 ) U 2 U 2 U 2 (Q 1 =: (U 1 ) U 2, Q 2 = (U (Efficiently) satisfies 2 ) U 1 orthogonal/rotation) R (U) = R (U 1 ) R (U 2 )

36 36 Two (Intersecting) Clique Explicit Delayed Completion Let: Hypotheses of intersecting Theorem (Thm 2) holds D i := D[α i ] E k i, for i = 1, 2, β α 1 α 2, γ := α 1 α 2 D := D[β] with embedding dimension r B := K ( D), k (t+1) Ū β := Ū(β, :), where Ū M satisfies intersection equation of Thm 2 [ V Ū e e ] M t+1 be orthogonal. Ū Z := (JŪβ V ) B((JŪβ V ) ). THEN t = r in Thm 2, and Z S r + is the unique solution of the equation (JŪβ V )Z (JŪβ V ) = B, and the exact completion is D[γ] = K ( PP ) where P := UV Z 1 2 R γ r

37 37 Completing SNL (Delayed use of Anchor Locations) Rotate to Align the Anchor Positions [ ] P1 Given P = R n r such that D = K (PP ) P 2 Solve the orthogonal Procrustes problem: min s.t. A P 2 Q Q Q = I P 2 A = UΣV SVD decomposition; set Q = UV ; (Golub/Van Loan 79, Algorithm ) Set X := P 1 Q

38 38 Summary: Facial Reduction for Cliques Using the basic theorem: each clique corresponds to a Gram matrix/corresponding subspace/corresponding face of SDP cone (implicit rank reduction) In the case where two cliques intersect, the union of the cliques correspond to the (efficiently computable) intersection of the corresponding faces/subspaces Finally, the positions are determined using a Procrustes problem

39 39 Results (from 2010) - Random Noisless Problems 2.16 GHz Intel Core 2 Duo, 2 GB of RAM Dimension r = 2 Square region: [0, 1] [0, 1] m = 9 anchors Using only Rigid Clique Union and Rigid Node Absorption Error measure: Root Mean Square Deviation RMSD = ( 1 n n i=1 p i p true i 2 ) 1/2

40 40 Results - Large n (SDP size O(n 2 )) n # of Sensors Located n # sensors \ R CPU Seconds # sensors \ R RMSD (over located sensors) n # sensors \ R e 16 5e 16 6e 16 3e e 16 4e 16 3e 16 3e e 16 5e 16 4e 16 4e 16

41 41 Results - N Huge SDPs Solved Large-Scale Problems # sensors # anchors radio range RMSD Time e 16 25s e 16 1m 23s e 16 3m 13s e 16 9m 8s Size of SDPs Solved: N = ( ) n (# vrbls) 2 E n (density of G ) = πr 2 ; M = E n ( E ) = πr 2 N (# constraints) Size of SDP Problems: M = [ 3, 078, , 315, , 709, , 969, 790 ] N = 10 9 [ ]

42 Noisy SNL Case 200 Sensors; [-0.5,0.5] box; noise 0.05; radio range 0.1 use sum of exposing vectors rather than intersection of faces obtained from cliques to do facial reduction use motivation: roundoff error cancels show video

43 43 Thanks for your attention! Facial Reduction for Cone Optimization Henry Wolkowicz Dept. Combinatorics and Optimization University of Waterloo Tuesday Apr. 21, 2015 (with: Drusvyatskiy, Krislock, (Cheung) Voronin)