EE 227A: Conve Optimization and Applications March 6, 2012 Lecture 14: Optimality Conditions for Conic Problems Lecturer: Laurent El Ghaoui Reading assignment: 5.5 of BV. 14.1 Optimality for Conic Problems 14.1.1 Conic problem and its dual The conic optimization problem in standard equality form is: c T : A = b, K. where K is a proper cone, for eample a direct product of cones that are one of the three types: positive orthant, second-order cone, or semidefinite cone. Let K be the cone dual K, which we define as K := {λ : K, λ T 0}. (14.1) All cones we mentioned (positive orthant, second-order cone, or semidefinite cone), are selfdual, in the sense that K = K. The Lagrangian of the problem is given by L(, λ, y) = c T + y T (b A) λ T (14.2) The last term is added to take account of the constraint K. From the very definition of the dual cone: { 0 if K, ma λ K λt = + otherwise. Thus, we have p = min = min ma L(, λ, y) y,λ K ma y,λ K ct + y T (b A) λ T d := ma g(λ, y) y,λ K (14.3) where { y g(λ, y) = min c T + y T (b A) λ T = T b if c A T y λ = 0, otherwise 14-1
The dual for the problem is: Eliminating λ, we can simplify the dual as: d = ma y T b : c A T y λ = 0, λ K. d = ma y T b : c A T y K. 14.1.2 Conditions for strong duality We now summarize the results stated in past lectures. Strong duality hold when either: The primal is strictly feasible, i.e. : A = b, int(k). This also implies that the dual problem is attained. The dual is strictly feasible, i.e. y : c A T y int(k ). This also implies that the primal problem is attained. If both the primal and dual are strictly feasible then both are attained (and p = d ). 14.1.3 KKT conditions for conic problems Assume p = d and both the primal and dual are attained by some primal-dual triplet (, λ, y ). Then, p = c T = d = g(λ, y ) = min L(, λ, y ) L(, λ, y ) = c T λ T + y T (b A ) c T = p. (14.4) The last term in the fourth line is equal to zero which implies λ T = 0. Thus the KKT conditions are: K, A = b, λ K, λ T = 0, c A T y λ = 0, that is, L(, λ, y) = 0. Eliminating λ from the above allows us to get rid of the Lagrangian stationarity condition, and gives us the following theorem. 14-2
Theorem 1 (KKT conditions for conic problems). The conic problem admits the dual bound p d, where c T : A = b, K. d = ma y T b : c A T y K. If both problems are strictly feasible, then the duality gap is zero: p = d, and both values are attained. Then, a pair (, y) is primal-dual optimal if and only if the KKT conditions hold. 1. Primal feasibility: K, A = b, 2. Dual feasibility: c A T y K, 3. Complementary slackness: (c A T y) T = 0, 14.1.4 KKT conditions for SDP Consider the SDP p = min c T : F () := F 0 + with c R n, F i S n, i = 0,..., m. The Lagrangian is and the dual problem reads m i F i 0, i=1 L(, Y ) = c T Tr F ()Y, d = ma Y 0 min L(, Y ) = ma Y Tr F 0 Y : Tr F i Y = c i, i = 1,..., n, ; Y 0. The KKT conditions for optimality are as follows: 1. F () 0, 2. Y 0, Tr(F i Y ) = c i, i = 1,..., n, 3. Tr(F ()Y ) = 0. The last condition can be epressed as F ()Y = 0, according to the following result: Let F, Y S n. If F 0 and Y 0 then Tr(F Y ) = 0 is equivalent to F Y = 0. Proof: Let Y 1/2 be the square root of Y (the unique positive semi-definite solution to Z 2 = Y ). We have Tr F Y = Tr F = 0, where F := Y 1/2 F Y 1/2. Since F 0, we have F 0. The trace of F being zero then implies that F = 0. 14-3
Using the eigenvalue decomposition, we can reduce the problem to the case when Y is diagonal. Let us assume that ( ) ( ) Λ 0 F11 F Y =, F = 12 0 0 F12 T F 22 where Λ 0 is diagonal, and contains the eigenvalues of Y, and the matri F 11 is of the same size as Λ (which is equal to the rank of Y ). The condition F = Y 1/2 F Y 1/2 = 0 epresses as ( ) ( ) ( ) ( ) Λ 1/2 0 F11 F 0 = 12 Λ 1/2 0 Λ 0 0 F12 T = 1/2 F 11 Λ 1/2 0. F 22 0 0 0 0 Since Λ 0, we obtain F 11 = 0. But F 0 then implies F 12 = 0 (use Schur complements), thus ( ) ( ) 0 0 Λ 0 F Y = = 0, 0 F 22 0 0 as claimed. Thus the last KKT condition can be written as F ()Y = 0. Theorem 2 (KKT conditions for SDP). The SDP admits the dual bound p d, where c T : F () := F 0 + m i F i 0 i=1 d = ma Y Tr F 0 Y : Tr F i Y = c i, i = 1,..., n, Y 0. If both problems are strictly feasible, then the duality gap is zero: p = d, and both values are attained. Then, a pair (, Y ) is primal-dual optimal if and only if the KKT conditions hold. 1. Primal feasibility: F () 0, 2. Dual feasibility: Tr F i Y = c i, i = 1,..., n, Y 0, 3. Complementary slackness: F ()Y = 0, Recall LP duality for a problem of the from min {c T : A b} with dual variables y has the KKT conditions were i : y i (b A) i = 0. This can be compactly written as diag(y) diag(b A) = 0 which is similar to the KKT conditions for SDP (with Y = diag(y)). This should come as no surprise as SDP problems include LP problems as a special case. 14-4
14.1.5 Eamples Largest eigenvalue problem. matri A S n : Let us use the KKT conditions to prove that, for any given ma { T A : T = 1} = ma X 0, Tr X=1 Tr AX = λ ma(a), where λ ma denotes the largest singular value. Duality theory for SDP immediately tells us that the second equality holds. Indeed, the SDP p = ma Tr AX : X 0, Tr X = 1 (14.5) X admits the following dual (see lecture 11): p d := min t t : ti A. Using the eigenvalue decomposition of A, it is easy to show that d = λ ma (A). It remains to prove the first equality. We observe that ma { T A : T = 1} = ma X Tr AX : X 0, Tr X = 1, rank(x) = 1. (To see this, set X = T.) Thus we need to show that at optimum, the rank of the primal variable X in (14.5) is one. The pair of primal and dual problems are both strictly feasible, hence the KKT condition theorem applies, and both problems are attained by some primal-dual pair (X, t), which satisfies the KKT conditions. These are X 0, ti A, and (ti A)X = 0. These conditions prove that any non-zero column of X satisfies (ti A) = 0 (in other words, is an eigenvector associated with the largest eigenvalue). Let us normalize so that 2 = 1, so that Tr T = 1. We have (ti A) T = 0, which proves that the feasible primal variable X = T 0, Tr X = 1, is feasible and optimal for the primal problem (14.5). Since X has rank one, our first equality is proved. Eercises 1. XXX 14-5