1 Review of last lecture and introduction

Transcription

1 Semidefinite Programming Lecture 10 OR 637 Spring 2008 April 16, 2008 (Wednesday) Instructor: Michael Jeremy Todd Scribe: Yogeshwer (Yogi) Sharma 1 Review of last lecture and introduction Let us first of all recall semidefinite programming problems in standard form and their duals. The problem in standard form, labelled by (P), is Min C X subject to (P) AX = b X 0. The semidefinite problem in dual form is Max b y subject to (D) A y + S = C S 0. For a little review of the last lecture: we saw examples where strong duality can fail. Moreover, there are examples where one of (P) and (D) has a finite optimum value and the other is infeasible (we did not discuss such examples). But in this lecture we will look at the positive side of things. In particular, we will show that there is no duality gap (between the primal and dual for semidefinite programming). For this, we of course have to make some assumptions (given the examples from last time). We will assume existence of optimal solutions by assuming that (P) or (D) satisfies what will be called Slater conditions (see the hypotheses of Theorem 5). To introduce the Slater conditions, let us make some definitions. Definition 1. Let us define F(P ) and F (P ) as follows. F(P ) := { X SIR n n : AX = b, X 0, F (P ) := { X SIR n n : AX = b, X 0. Similarly, define the set of feasible solutions and the set of strictly feasible solutions to the dual program. F(D) := { (y, S) IR m SIR n n : A y + S = C, S 0, F (D) := { (y, S) IR m SIR n n : A y + S = C, S 0. At a very informal level, we want to show that if either of the two sets F (P ) or F (D) is nonempty, then there is no duality gap. (See Theorem 5 for a precise statement.) 1

2 2 Separating hyperplane theorems To prove the no-duality-gap theorem for semidefinite programming assuming Slater conditions, we will need to use a few separating hyperplane theorems which we now state. The first theorem states that if a point is outside of a closed convex set, then there is a hyperplane which strictly separates the two. Theorem 2. If C IR p is a closed, convex, and non-empty subset of IR p, and x C, then there exists a y IR p and η IR such that y x < η < inf { y x : x C. Proof. See OR 630 lecture notes for September 15, Available via cornell.edu/~miketodd/or630/lec07.pdf. The next theorem states that if there are two closed convex sets which don t extend infinitely in some common direction, then they can be strictly separated by a hyperplane. Theorem 3. If C 1 and C 2 are both closed, convex, and non-empty and have no recession direction in common, then their disjointness (C 1 C 2 = ) implies that there exists a y IR p, η IR such that sup { y x : x C 1 < η < inf { y x : x C 2. Proof Sketch. We only give the methodology, suppressing all the details. The idea is to reduce the theorem to Theorem 2. We consider the set C 2 C 1 = {x y : x C 2, y C 1, which may not necessarily be closed, but the absence of a recession direction ensures that it is closed and we get the result by applying Theorem 2. Remark 4. A few remarks are in order for the theorem above. First of all, the cone K of recession directions for C is the set of (nonzero) directions d IR p such that if x C, then x + λd C for all λ 0. Also note that the condition on recession directions above is a necessary one. For example, consider C 1 := {(x 1, x 2 ) IR 2 : x 1 x 2 1, x 1 > 0, C 2 := {(x 1, x 2 ) IR 2 : x 2 0. These two sets satisfy all the conditions of Theorem 3 except the last one: the direction d = (1, 0) is a recession direction for both the sets. In this case, there is no hyperplane strictly separating C 1 and C 2. 2

3 3 Strong duality for semidefinite programs We are now ready to state the strong duality theorem for semidefinite programs. Theorem 5. Suppose F(P ), and F (D). Then 1. The set of optimal solutions to (P) is non-empty, and compact, 2. There is no duality gap. Proof. We need to prove two statements. To prove the nonemptiness of the set of optimal solutions, we will use the Weierstrass theorem, which is stated below. Theorem 6 (Weierstrass). A continuous real-valued function defined on a nonempty compact set attains its minimum and maximum. Let us look at the first part now. Compactness of the set of optimal solutions (Item 1) Let us fix solutions ˆX F(P ) and (ŷ, Ŝ) F (D). The set of optimal solutions to (P) is the same as the set of optimal solutions to min{c X : AX = b, C X C ˆX, X 0. (1) But we note that for any feasible solution X to (1), C X C ˆX C X b ŷ C ˆX b ŷ Ŝ X Ŝ ˆX (by using the feasibility of the primal and dual) But the set {X 0 : Ŝ X Ŝ ˆX is compact by using Fact 13 from Lecture 4. The feasible region of the above problem (1) is the intersection of this compact set with some hyperplanes. Therefore, the feasible region of problem 1 is compact. Therefore, (P) attains its minimum (continuous function on a compact set, see Theorem 6), and let its optimal value is ζ. Then, the set of optimal solutions to (P ) is {X : AX = b, Ŝ X Ŝ ˆX, X 0, C X ζ which is compact, proving the item 1 of the theorem. 3

4 No duality gap (Item 2) To prove that there is no duality gap, we need to prove that a dual solution can come as close to ζ as we would like. That is, for every ɛ > 0, there exists a dual solution (ŷ ɛ, Ŝɛ) such that b ŷ ɛ ζ ɛ. To this end, we define F 1 = {X SIR n n : X 0, and F 2 = {X SIR n n : AX = b, C X ζ ɛ. Note that F 1 and F 2 are both closed and convex. (The first one is the semidefinite cone, and the second one is an intersection of hyperplanes and halfspaces.) Also, F 1 F 2 =, since any element which is in both of them is a feasible solution for the primal program, and we know that there cannot be a primal feasible solution with objective value less than ζ. Also note that F 1 and F 2 don t have any recession direction in common. We can see this by contradiction: let D be a common recession direction. Then, D must satisfy D 0, A D = 0, and C D 0. But such a D will prove that if ˆX is an optimal solution to (P) then ˆX + λd is also an optimal solution, for any λ > 0 proving that the set of optimal solutions to the primal program is unbounded, a contradiction. If F 2, then the sets F 1 and F 2 satisfy all the conditions of Theorem 3. Hence we have the existence of a symmetric matrix S SIR n n and σ IR such that S X 1 > σ > S X 2 for all X 1 F 1 and X 2 F 2. (2) If F 2 = on the other hand, we can simply choose S = I and σ = 1 giving us the above condition (2). We now make a few assertions about σ and S. σ < 0. This is because 0 F 1. S 0. Let us prove this by contradiction. If there is a z IR n with z Sz < 0, then S (λzz ) < σ for sufficiently large λ, violating the condition (2) for X = λzz F 1. Using these conditions, we have ( ) ( 1)AX = ( 1)b, C X ζ ɛ = S X σ < 0. (Note that we can infer S X < σ, but we will only need a weaker implication.) The above inequality statement implies that a linear inequality can be inferred from a set of equalities and inequalities. It means that the target linear inequality can be written as a linear combination of the original inequalities. Hence, there must exist y IR m, η IR such that m y i A i + ηc = S m y i b i + η(ζ ɛ) σ 4

5 η 0. We claim that η > 0. If not, then we have A y + S = 0 b y σ < 0 ( = b y > 0 ). This is not possible since otherwise (ŷ, Ŝ) + λ(y, S) would be a feasible solution to the dual for all λ > 0, with objective function value approaching + as λ, a contradiction, since the primal optimal value gives an upper bound on any feasible solution value of the dual (weak duality). Therefore, we have η > 0, and by scaling we can assume that η = 1. We have A y + S = C, b y > ζ ɛ S 0 (from the property of S above). This gives us the desired dual feasible solution that comes within ɛ of the primal optimum value. Remark 7. Let us make a quick remark about the proof above. In the proof, we used all along that Ŝ 0, and that was very important. Without this, we have seen that a duality gap may be present. 4 Conversion from primal to dual and back We will take a little digression and prove that the semidefinite program in standard (primal) form can be formulated as one in dual form and vice versa. 4.1 Digression: Independent A i s Let us first assume without (much) loss of generality that A 1, A 2,..., A m are linearly independent. If not, then consider the following cases. If A y = 0 = b y: In this case, let A 1, A 2,..., A k form a basis for the space spanned by all A i s. Eliminate the last m k constraints in (P) to keep only linearly independent constraints. Also, eliminate the last m k components of y in (D) and get an equivalent problem. If A y = 0 and b y > 0 for some y: In such case, (P) is infeasible and if (D) is feasible, it is unbounded. 5

6 Let us get back to the primal-dual conversion. Let A be the space spanned by A i s, i.e., A = {A y : y IR m SIR n n. Furthermore, let F 1, F 2,..., F l be a basis for A. Then we have the following two straightforward facts. X A X = z i F i, X A F i X = 0 i = 1, 2,..., l. Let us first look at how to convert a primal problem into a dual problem. Choose D SIR n n with AD = b and set g i = C F i for i = 1, 2,..., l. We then have (P) min{c X : AX = b, X 0 min {C X : A(X D) = 0, X 0 { min C X : X D = z i F i, X 0 min { C (D z i F i ) : { C D max g i z i : z i F i + X = D, X 0 z i F i + X = D, X 0, where the last equivalence follows from the definition of the g i s. We can do something similar for (D), resulting in the following. (D) max { b y : A y + S = C, S 0 max{(ad) y : A y + S = C, S 0 max{(a y) D : A y + S = C, S 0 max{(c S) D : A y + S = C, S 0 max{(c S) D : S C A, S 0 C D min{d S : F i (S C) = 0, for i = 1, 2,..., l, S 0 C D min{d S : F i S = g i for i = 1, 2,..., l, S 0, which is a problem in primal form. 6