Lecture 14: and Applications 11th March 2019
Outline LP SOCP SDP LP SOCP SDP 1 / 21
Conic LP SOCP SDP Primal Conic Program: min c T x s.t. Ax K b (CP) : b T y s.t. A T y = c (CD) y K 0 Theorem. (Strong Conic ) If (CP) is bounded below and strictly feasible, i.e., x 0, s.t. Ax 0 > K b, then (CD) is solvable and Opt(CD) = Opt(CP). 2 / 21
Example: LP LP SOCP SDP Primal LP: Dual LP: min c T x s.t. Ax b (LP-P) b T y s.t. A T y = c (LP-D) y 0 3 / 21
LP SOCP SDP Example: SOCP Primal SOCP: min x s.t. c T x Dual SOCP: A i x b i 2 d T i x e i, i = 1,..., m (SOCP P) λ R m u i R n i 1,i=1,...,m s.t. m bi T u i + e T λ i=1 m (A T i u i + d i λ i ) = c (SOCP D) i=1 u i 2 λ i, i = 1,..., m 4 / 21
Example: SDP LP SOCP SDP Primal SDP: Dual SDP: min x s.t. c T x n x i A i B 0 i=1 (SDP P) Y tr(by ) s.t. tr(a i Y ) = c i i = 1,..., n (SDP D) Y 0 5 / 21
LP SOCP SDP Example: Variant of SDP Primal SDP: Dual SDP: min Y B Y s.t. A i Y = c i i = 1,..., n (SDP P ) Y 0 x s.t. c T x n B x i A i 0 i=1 (SDP D ) 6 / 21
LP SOCP SDP Application 1: Use SDP duality to show that for any B S+: n { } x T Bx : x 2 = 1 = λ (B) x R n Reformulation with rank-1 constraint: : X tr(bx ) s.t. tr(x ) = 1 (P) X = xx T X tr(bx ) s.t. tr(x ) = 1 (SDP-r) X 0 Exact Recovery: Opt(SDP-r) = Opt(P) 7 / 21
Application 1: LP SOCP SDP : Dual to SDP relaxation: X tr(bx ) s.t. tr(x ) = 1 (SDP-r) X 0 λ (B) = min x λ s.t. λi B 0 (SDP-d) 8 / 21
LP SOCP SDP Application 2: Consider undirected weighted graph G = (V, E, W ). Here V = n,, W = {w ij } (i,j) E with w ij 0. (NP-Hard): 1 n n w ij (1 x i x j ) X 4 i=1 j=1 s.t. x i { 1, 1}, i = 1,..., n (MAXCUT) 9 / 21
LP SOCP SDP Application 2: Reformulation with Rank-1 Constraint: 1 n n w ij 1 x 4 4 tr(wx ) : i=1 j=1 s.t. X ii = 1, i = 1,..., n (MAXCUT ) x X = xx T 1 4 n i=1 j=1 n w ij 1 4 tr(wx ) s.t. X ii = 1, i = 1,..., n (SDP-r) X 0 GW Theorem. [Goemans & Williamson, 1995] MAXCUT Opt(SDP-r) 1.1383 MAXCUT 10 / 21
LP SOCP SDP Nesterov s π 2 Theorem Consider any QP with Q 0 in the form and its SDP relaxation x x T Qx s.t. x i { 1, 1}, i = 1,..., n (QP) X tr(qx ) s.t. X ii = 1, i = 1,..., n (SDP-r) X 0 Nesterov s π 2 Theorem. [Nesterov, 1998] If Q 0, Opt(QP) Opt(SDP-r) π 2 Opt(QP) 11 / 21
LP SOCP SDP Nesterov s π 2 Theorem Remark. MAXCUT is a special case: Q ij = w ij, i j, and Q ii = n j=1 w ij. Proof Sketch: Let ξ N (0, X ), where X is optimal to (SDP-r). Let ζ = sign(ξ), ζ { 1, 1} n. Opt(QP) E[ζ T Qζ] = tr ( Q 2 π arcsin(x ) ). Note asin(x ) X, where arcsin is inverse of sine. Hence, Opt(QP) 2 π Opt(SDP-r) Q. What about when Q is indefinite? (Nemirovski, Roos, Terlaky, 1998) Opt(SDP-r) O(1) ln(n)opt(qp) 12 / 21
LP SOCP SDP Application 3: Quadratic constrained quadratic programming: min x T Q 0 x + 2q T 0 x + c 0 s.t. x T i Q i x i + 2q T i x + c i 0, 1 i m (QCQP) Reformulation with rank-1 constraint: min x,x tr(a 0 X ) s.t. tr(a i X ) 0, 1 i m (QCQP ) [ xx X = T ] x x T 1 [ ] Qi q Here A i = i, i = 0, 1,..., m q T i c i 13 / 21
LP SOCP SDP Application 3: SDP relaxation: min X tr(a 0 X ) s.t. tr(a i X ) 0, 1 i m (SDP-r) X 0 Dual of SDP relaxation: X n+1,n+1 = 1 λ 0,t s.t. t A 0 + λ i A i [ ] 0 0 0 0 t (SDP-d) Remark. Opt(SDP-d) Opt(SDP-r) Opt(QCQP) 14 / 21
LP SOCP SDP S-Lemma S-Lemma. Suppose A, B S n and x T 0 Ax 0 > 0 for some x 0. Then x T Bx 0, x : x T Ax 0 holds true if and only if Remark. λ 0 : B λa. Note Farkas Lemma only applies to convex functions. Here the quadratic functions are not necessarily convex. Can not generalize to more than one constraint: x T Bx 0, x : x T A i x 0, i = 1, 2 λ 1 0, λ 2 0, B λ 1 A 1 + λ 2 A 2. 15 / 21
LP SOCP SDP Proof of S-Lemma First prove that x T Bx 0, x : x T Ax 0 implies that tr(bx ) 0, X 0 : tr(ax ) 0. (why?) Equivalently, Opt(P) = 0 min tr(bx ) s.t. tr(ax ) 0 (P) X 0 This is guaranteed if and only if the dual is feasible: λ,y 0 s.t. B = λa + Y (D) λ 0, Y 0 16 / 21
LP SOCP SDP Application 4: Consider the linear dynamical system: dx dt = Ax(t) + Bu(t), x(0) = x 0 y(t) = Cx(t) with the sector constraint (α < β): σ(y(t), u(t)) = 2(βy(t) u(t)) T (u(t) αy(t)) 0. The system is stable iff P S n such that the Lyapunov function V (t) = x(t) T Px(t) is non-increasing, i.e., dv (t) dt < 0, x(t), u(t) : σ(cx(t), u(t)) 0. 17 / 21
Application 4: LP SOCP SDP Note [ ] T [ dv (t) x(t) A = T ] [ ] P + PA PB x(t) dt u(t) B T P 0 u(t) [ ] T [ x(t) 2αβC σ(cx(t), u(t)) = T C (α + β)c T ] [ ] x(t) u(t) (α + β)c 2 u(t) The system is stable iff the LMI is feasible: [ A T P + PA 2αβC T C PB + (α + β)c T ] B T 0 P + (α + β)c 2 18 / 21
LP SOCP SDP Application 4: Now consider the linear dynamical system: dx dt = Ax(t) + Bu(t), x(0) = x 0 y(t) = Cx(t) with the unity-bounded constraint (α < β): u i (t) y i (t), i = 1, 2,..., p. The system is stable if the LMI is feasible: P S n, D = diag(λ 1,..., λ p ) such that [ A T P + PA + C T ] DC PB B T 0. P D 19 / 21
LP SOCP SDP More Machine Learning Low-rank matrix factorization; k-means for clustering; Graphical lasso for estimating covariance matrix; Robust optimization and chance constraint programs; Trust region methods; Polynomial optimization; Optimal control; Signal Processing MIMO detection in signal processing; Stochastic block models for community detection; 20 / 21
References LP SOCP SDP Ben-Tal & Nemirovski (2013), Chapters 3.1-3.6 21 / 21