Lecture: Introduction to LP, SDP and SOCP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html wenzw@pku.edu.cn Acknowledgement: this slides is based on Prof. Farid Alizadeh lecture notes
Linear Programming (LP) Primal Dual min c 1 x 1 +... + c n x n max b 1 y 1 +... + b m y m s.t. a 11 x 1 +... + a 1n x n = b 1... a m1 x 1 +... + a mn x n = b m x i 0 s.t. a 11 y 1 +... + a m1 y m c 1... a 1n y 1 +... + a mn y m c n 2/29
3/29 Linear Programming (LP) more succinctly Primal (P) Dual (D) min c x max b y s.t. Ax = b x 0 s.t. A y + s = c s 0
4/29 Weak duality Suppose x is feasible to (P) (y, s) is feasible to (D) Then 0 x s because x i s i 0 = x (c A y) = c x (Ax) y = c x b y = duality gap
5/29 Key Properties of LP Strong duality: If both Primal and Dual are feasible then at the optimum c x = b y x s = 0 complementary slackness: This implies x s = x 1 s 1 +... + x n s n = 0 and therefore x i s i = 0
6/29 complementarity Putting together primal feasibility, dual feasibility and complementarity together we get a square system of equations Ax = b A y + s = c x i s i = 0 for i = 1,..., n At least in principle this system determines the primal and dual optimal values
Algebraic characterization 7/29 We can define x s = (x 1 s 1,..., x n s n ) and L x : y (x 1 y 1,..., x n y n ) i.e. L x = Diag(x) We can write complementary slackness conditions as x s = L x s = L x L s 1 = 0 1, the vector of all ones, is the identity element: x 1 = x
Semidefinite Programming (SDP) 8/29 X Y means that the the symmetric matrix X Y is positive semidefinite X is positive semidefinite a Xa 0 for all vector a X = B B all eigenvalues of X is nonnegative
Semidefinite Programming (SDP) X, Y = ij X ijy ij = Tr(XY) Primal (P) min C 1, X 1 +... + C n, X n max Dual (D) b 1 y 1 +... + b m y m s.t. A 11, X 1 +... + A 1n, X n = b 1... A m1, X 1 +... + A mn, X n = b m X i 0 s.t. A 11 y 1 +... + A m1 y m + S 1 = c 1... A 1n y 1 +... + A mn y m + S n = c n S i 0 9/29
Simplified SDP 10/29 For simplicity we deal with single variable SDP: Primal (P) Dual (D) min C, X max s.t. A 1, X = b 1... A m, X = b m X 0 s.t. A single variable LP is trivial b y y i A i + S = C i S 0 But a single matrix SDP is as general as a multiple matrix
Weak duality in SDP 11/29 Just as in LP X, S = C, X b y Also if both X 0 and S 0 then because S 1/2 XS 1/2 0 X, S = Tr(XS 1/2 S 1/2 ) = Tr(S 1/2 XS 1/2 ) 0 Thus X, S = C, X b y 0
12/29 Complementarity Slackness Theorem X 0 and S 0 and X, S = 0 implies XS = 0 Proof: X, S = Tr(XS 1/2 S 1/2 ) = Tr(S 1/2 XS 1/2 ) Thus Tr(S 1/2 XS 1/2 ) = 0. Since S 1/2 XS 1/2 0, then S 1/2 XS 1/2 = 0 = S 1/2 X 1/2 X 1/2 S 1/2 = 0 X 1/2 S 1/2 = 0 = XS = 0
Algebraic properties of SDP 13/29 For reasons to become clear later it is better to write complementary slackness conditions as XS + SX 2 = 0 It can be shown that if X 0 and S 0, then XS = 0 iff XS + SX = 0
Algebraic properties of SDP 14/29 Definition: X S = XS+SX 2 The binary operation is commutative X S = S X is not associative: X (Y Z) (X Y) Z in general But X (X X) = (X X) X. Thus X p = X p is well defined In general X (X 2 Y) = X 2 (X Y) The identity matrix I is identity w.r.t Define the operator L X : Y X Y, thus X S = L X (S) = L X (L S (I))
Constraint Qualifications 15/29 Unlike LP we need some conditions for the optimal values of Primal and Dual SDP to coincide Here are two: If there is primal-feasible X 0 (i.e. X is positive definite) If there is dual-feasible S 0 When strong duality holds X, S = 0
KKT Condition 16/29 Thus just like LP The system of equations A i, X = b i, for i = 1,..., m y i A i + S = C i X S = 0 Gives us a square system
Second Order Cone Programming (SOCP) 17/29 For simplicity we deal with single variable SOCP: Primal (P) Dual (D) min c x max b y s.t. Ax = b x Q 0 s.t. A y + s = c s Q 0 the vectors x, s, c are indexed from zero If z = (z 0, z 1,..., z n ) and z = (z 1,..., z n ) z Q 0 z 0 z
Illustration of SOC 18/29 Q = {z z 0 z }
19/29 Weak Duality in SOCP The single block SOCP is not as trivial as LP but it still can be solved analytically weak duality: Again as in LP and SDP If x, s Q 0, then x s = c x b y = duality gap x s = x 0 s 0 + x s x s + x s since x, s Q 0 x s + x s Cauchy-Schwartz inequality 0
Complementary Slackness for SOCP 20/29 Given x Q 0, s Q 0 and x s = 0. Assume x 0 > 0 and s 0 > 0 We have ( ) ( ) x 2 0 s 2 0 n i=1 x 2 i n s 2 i x0 2 i=1 n i=1 ( ) x s = 0 x 0 s 0 = i s 2 i x2 0 s 2 0 x i s i 2x 2 0 = Adding (*), (**), (***), we get 0 ( ) 2 n i=1 x i + s ix 0 s 0 n i=1 2x i s i x 0 s 0 This implies x i s 0 + x 0 s i = 0, for i = 1,..., n
Illustration of SOC 21/29 When x Q 0, s Q 0 are orthogonal both must be on the boundary in such a way that their projection on the x 1,..., x n plane is collinear
Strong Duality 22/29 at the optimum c x = b y x s = 0 Like SDP constraint qualifications are required If there is primal-feasible x Q 0 If there is dual-feasible s Q 0
23/29 Complementary Slackness for SOCP Thus again we have a square system Ax = b, A y + s = c x s = 0 x 0 s i + s 0 x i = 0
Algebraic properties of SOCP 24/29 Let us define a binary operation for vectors x and s both indexed from zero x 0 s 0 x s x 1. s 1. = x 0 s 1 + s 0 x 1. x n s n x 0 s n + s 0 x n
Algebraic properties of SOCP 25/29 The binary operation is commutative x s = s x is not associative: x (y z) (x y) z in general But x (x x) = (x x) x. Thus x p = x p is well defined In general x (x 2 y) = x 2 (x y) The identity matrix I is identity w.r.t e = (1, 0,..., 0) is the identity: x e = x
Algebraic properties of SOCP 26/29 Define the operator L x : y x y ( x0 x L x = Arw(x) = ) x x 0 I x s = Arw(x)s = Arw(x)Arw(s)e
Summary 27/29 Properties LP SDP SOCP ( ) x binary operator x s = (x i s i ) X S = XS+SX 2 x s = s x 0 s + s 0 x identity 1 I e = (1, 0,..., 0) associative yes no no L X y Diag(x)y Y XY+YX 2 y Arw(x)y Primal feasibility Ax = b A i, X = b i Ax = b dual feasility A y + s = c i y ia i + S = C A y + s = c complementarity L x L s 1 = 0 L X (L S (I)) = 0 L x L s e = 0
28/29 Conic LP A set K R n is a proper cone if It is a cone: If x K = ax K for all α 0 It is convex: x, y K = αx + (1 α)y K for α [0, 1] It is pointed: K ( K) = {0} It is closed It has non-empty interior in R n dual cone: K = {x for all z K, x, z 0}
Conic LP 29/29 Conic-LP is defined as the following optimization problem: Primal (P) Dual (D) min c x max s.t. Ax = b x K s.t. For LP K is the nonnegative orthant b y A y + s = c s K For SDP K is the cone of positive semidefinite matrices For SOCP K is the circular or Lorentz cone In all three cases the cones are self-dual K = K