Strong Duality Results September 22, 2008
Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation Issue Multiple Dual Choices Convex Optimization 1
Primal Problem General Convex Problem minimize f(x) subject to g j (x) 0, j = 1,..., m a T i x = b i, i = 1,..., r x X Assumption used throughout the lecture Assumption 1. The objective f and all g j are convex The set X R n is nonempty and convex The optimal value f is finite Convex Optimization 2
Slater Condition for Inequality Constrained Problem Lecture 8 Convex Primal Problem minimize f(x) subject to g j (x) 0, x X j = 1,..., m Slater Condition There exists a vector x X domf such that g j ( x) < 0 for all j = 1,..., m. Observation: The Slater condition is a condition on the constraint set only when domf = R n Theorem Let Assumption 1 and the Slater condition hold. Then: There is no duality gap, i.e., q = f The set of dual optimal solutions is nonempty and bounded Convex Optimization 3
Proof Consider the set V R m R given by V = {(u, w) g(x) u, f(x) w, x X} The Slater condition is illustrated in the figures below. The set V is convex by the convexity of f, g j s, and X. Convex Optimization 4
Proof continues The vector (0, f ) is not in the interior of the set V. Suppose it is, i.e., (0, f ) intv. Then, there exists an ɛ > 0 such that (0, f ɛ) V contradicting the optimality of f. Thus, either (0, f ) bdv or (0, f ) V. By the Supporting Hyperplane Theorem, there exists a hyperplane passing through (0, f ) and supporting the set V : there exists (µ, µ 0 ) R m R with (µ, µ 0 ) 0 such that µ T u + µ 0 w µ 0 f for all (u, w) V (1) This relation implies that µ 0 and µ 0 0. Suppose that µ 0 = 0. Then, µ 0 and relation (1) reduces to inf (u,v) V µt u = 0 On the other hand, by the definition of the set V, since µ 0 and µ 0, we have inf (u,v) V µt u = inf x X µt g(x) µ T g( x) < 0 - a contradiction Convex Optimization 5
Hence, µ 0 > 0 (the supporting hyperplane is nonvertical), and by dividing with µ 0 in Eq. (1), we obtain with µ 0. Therefore, inf (u,v) V { µt u + w} f { q( µ) = inf f(x) + µ T g(x) } f with µ 0 x X implying that q f. By the weak duality [q f ], it follows that q = f and µ is a dual optimal solution. Convex Optimization 6
We now show that the set of dual optimal solutions is bounded. For any dual optimal µ 0, we have { q = q( µ) = inf f(x) + µ T g(x) } x X f( x) + µ T g( x) Therefore, f( x) + max 1 j m {g j( x)} m j=1 min 1 j m { g j ( x)} m j=1 µ j f( x) q implying that µ j µ m j=1 µ j f( x) q min 1 j m { g j ( x)} Convex Optimization 7
Example l -Norm Minimization Equivalent to minimize Ax b The vector ( x, t ) given by minimize t subject to a T j x b j t 0, j = 1,..., m b j a T j x t 0, j = 1,..., m (x, t) R n R x = 0 and t = ɛ + max 1 j m b j for some ɛ > 0 satisfies the Slater condition We refer to a vector satisfying the Slater condition as a Slater vector Convex Optimization 8
Convex Objective Over Linear Constraints Def. A vector x 0 is in the relative interior of X when there exists a ball B r (x 0 ) = {x R n x x 0 r} such that B r (x 0 ) aff(x) X The relative interior of a convex set X is the set of points that are interior with respect to the affine hull of X. Primal Problem with Linear Inequality Constraints minimize f(x) subject to Ax b, x X Theorem Let Assumption 1 hold. Assume that there exists a vector x such that A x b and x relint(x) relint(domf) Then There is no duality gap (q = f ) and A dual optimal solution exists Convex Optimization 9
Implications When domf = R n, the relative interior condition has to do only with the constraint set. The relative interior condition is always satisfied (a) When domf = R n and X = R n : convex function over R n and linear constraints (b) LP s: X = R n and f(x) = c T x. In these cases, by the preceding theorem: Strong duality holds and optimal dual solution exists Convex Optimization 10
Examples Examples not satisfying the relative interior condition minimize e x 1 x 2 subject to x 1 0, x R 2 Here, X = R 2 and the feasible set C is C = {x x 1 = 0, x 2 R}. The domain of f is the set {x x 1 0, x 2 0}, thus, relint(x) relint(domf) = {x x 1 > 0, x 2 > 0}. However, none of the feasible vectors lies in relint(x domf). Hence, the condition fails. Furthermore, there is a gap - Homework. Consider the same constraint set with an objective f(x) = x 1. The condition is not satisfied. In this case, there is no gap, but a dual optimal solution does not exist. Convex Optimization 11
Slater for a General Convex Problem Lecture 8 minimize f(x) subject to g j (x) 0, j = 1,..., m Ax = b, Dx d x X Theorem Let f be finite, and assume that: There is a feasible vector satisfying the Slater condition, i.e., there is x R n such that g( x) 0, A x = b, D x d, x X domf There is a vector satisfying the linear constraints and belongs to the relative interior of X, i.e., there is x R n such that A x = b, D x d, x relint(x) relint(domf) Then: There is no duality gap, i.e., q = f The set of dual optimal solutions is nonempty Note: Conditions only on the constraint set Theorem says nothing about the existence of primal optimal solutions Convex Optimization 12
Quadratic Objective over Quadratic Constraint minimize x T Q 0 x + a T 0x + b 0 subject to x T Q j x + a T j x + b j 0, x R n j = 1,..., m Theorem Let each Q i be symmetric positive semidefinite matrix. Let f be finite. Then, there is no duality gap [q = f ] and the primal optimal set is nonempty. Note: The theorem says nothing about the existence of dual optimal solutions Opposed to the Slater and the relative interior condition, which say nothing about the existence of the primal optimal solutions Example minimize x subject to Ax = b x R n Convex Optimization 13
Representation Issue minimize x 2 subject to x x 1 x X, X = {(x 1, x 2 ) x 2 0} The relaxation of the inequality constraint results in a dual problem, for which the dual value q(µ) is for any µ 0 (verify yourself). Thus, q =, while f = 0. However, a closer look into the constraint set reveals C = {(x 1, x 2 ) x 1 0, x 2 = 0} Thus the problem is equivalent to minimize x 2 subject to x 1 0, x 2 = 0 x R 2 There is no gap for this problem!!! Why? The duality gap issue is closely related to the representation of the constraints [the model for the constraints] Convex Optimization 14
Multiple Choices for a Dual Problem minimize f(x) subject to g j (x) 0, j = 1,..., m Ax = b, Dx d x X There are multiple choices for the set of inequalities to be relaxed How to choose the right one? There is no general rule Keep box constraints, sign constraints if any Keep constraints for which the dual function can be easily evaluated Will for all of them no duality-gap relation hold? What do we know about duality gaps when multiple dual choices exist? Convex Optimization 15
Relax-All Rule minimize f(x) subject to g j (x) 0, j = 1,..., m x R n A linear equality is represented by two linear inequalities. Theorem Let f be finite. Consider a dual corresponding to the relaxation of all the constraints, and assume that there is no duality gap. Then, there is no duality gap when partially relaxing the constraints. Convex Optimization 16