Lecture 5. The Dual Cone and Dual Problem

IE 8534 1 Lecture 5. The Dual Cone and Dual Problem

IE 8534 2 For a convex cone K, its dual cone is defined as K = {y x, y 0, x K}. The inner-product can be replaced by x T y if the coordinates of the vectors are given. Clearly, K is always a closed set, regardless if K is closed or not.

IE 8534 3 In light of the dual cone, the problem of finding a lower bound on the optimal value for the standard conic optimization problem can be done as follows. Introduce the Lagrangian multiplier y for the equality constraint b Ax = 0, and s for the conic constraint x K. The objective becomes c T x + y T (b Ax) s T x = y T b + (c A T y s) T x. This will be a lower bound on the optimal value whenever c A T y s = 0 and s K. The quest to find the best lower bound leads to the dual problem for the standard conic optimization problem: maximize subject to b T y A T y + s = c s K. (1)

IE 8534 4 As an example, for the standard SDP problem (primal), minimize subject to C X A i X = b i, i = 1,..., m X 0, its dual problem is in the form of maximize subject to m i=1 b iy i m i=1 y ia i + S = C S 0.

IE 8534 5 The so-called weak duality relationship is immediate, by the construction of the dual problem. This is shown in the next theorem. Theorem 1 Let x be feasible for the primal problem, and (y, s) be feasible to its dual. Then, b T y c T x. Moreover, if b T y = c T x then x T s = 0, and they are both optimal to their own respective problems. If a pair of primal-dual feasible solutions x and (y, s) both exist and they satisfy x T s = 0 then we call them a pair of complementary optimal solutions.

IE 8534 6 A very useful theorem is the following. Theorem 2 Let x be feasible for the primal problem, and (y, s) be feasible to its dual, and x int K and s int K. Then, the sets of optimal solutions for both problems are nonempty and compact, and any pair of primal-dual optimal solutions will be complementary to each other.

IE 8534 7 Let us take some moments to study the structures of the dual cones. Let us start with the topological structures. If rel (K) is denoted as the relative interior of K, then (rel (K)) = K. Let the summation of two sets A and B be A + B = {z z = x + y with x A, y B}. It is clear that the sum of two convex sets remains convex. However, the sum of two closed convex sets may not be closed any more.

IE 8534 8 Example 1 The set R + 1 0 1 + SOC(3) is convex but not closed. For instance, the point (0, 1, 0) T is not in the set, but it is in the closure of the set, since for any ϵ > 0, we may write 1 ϵ 1 = 1 1 ϵ ϵ 0 + + ϵ 1, 0 1 where the second term is in SOC(3). 1 ϵ

IE 8534 9 However, if A and B are both polyhedra, then A + B remains a polyhedron, hence closed. It is interesting to note that the set-summation and the set-intersection operations are a duality pair: Proposition 1 Suppose that K 1 and K 2 are convex cones. It holds that (K 1 + K 2 ) = K 1 K 2. Proof. For y (K 1 + K 2 ), it follows that 0 (x 1 + x 2 ) T y for all x 1 K 1 and x 2 K 2. Since 0 K 2, it follows that 0 (x 1 ) T y for all x 1 K 1, thus y K 1, and similarly y K 2. Hence, y K 1 K 2. Now, take any y K 1 K 2, and any x 1 K 1 and x 2 K 2. We have (x 1 + x 2 ) T y = (x 1 ) T y + (x 2 ) T y 0, and so y (K 1 + K 2 ). This proves (K 1 + K 2 ) = K 1 K 2 as desired.

IE 8534 10 The following bipolar theorem can be considered as a cornerstone for convex analysis. Theorem 3 Suppose that K is a convex cone. Then (K ) = cl K. Proof. For any x K, by the definition of the dual cone, x T y 0 for all y K, and so K (K ). Therefore cl K (K ). The other containing relationship (K ) cl K can be shown by contradiction. Suppose such is not true then there exists z (K ) \ cl K. By the separation theorem for convex cones, there is vector c such that c T z < 0, and c T x 0 x cl K. The last relation suggests that c K, which contradicts with the first relation because, as z (K ), it should lead to c T z 0, but not the opposite. This contradiction completes the proof.

IE 8534 11 This proof suggests that the bipolar theorem for convex cone is equivalent to the separation theorem itself. Suppose that K is a closed convex cone, and v K. Then, the bipolar theorem asserts that v (K ), and this implies that there is y K such that v T y < 0, and x T y 0 for any x K because y K, hence a separation exists.

IE 8534 12 In case K is closed, the bipolar theorem is what in folklore understood as the dual of the dual is the primal itself again. Combining Theorem 3 and Proposition 1, it follows that cl (K 1 + K 2 ) = (K 1 K 2 ). (2) It is also useful to compute the effect of linear transformation on a convex cone in the context of dual operation. Let A be an invertible matrix. Then, we have (A T K) = A 1 K, (3) by observing that y (A T K) (A T x) T y 0 x K Ay K y A 1 K.

IE 8534 13 A convex cone is called solid if it is full dimensional. Mathematically, K R n is solid span K = R n, or K + ( K) = R n. A convex cone is called pointed if it does not contain any nontrivial subspace; i.e. K ( K) = {0}. The following result proves to be useful: Proposition 2 If K is solid then K is pointed, and if K is pointed then K is solid.

IE 8534 14 Proof. Suppose that K is solid, i.e. there is a ball B(x 0 ; δ) K with radius δ > 0 and centered at x 0, and at the same suppose that there is a line Rc K, then 0 s(x 0 + t) T c, t δ and s R, which is clearly impossible. This contradiction shows that if K is solid then K must be pointed. By the bipolar theorem, the second part follows by symmetry.

IE 8534 15 It is easy to compute the following dual cones (R n +) = R n +. SOC(n) = SOC(n). (S n n + ) = S n n +, and (H n n + ) = H n n +. The above standard cones are so regular that their corresponding dual cones coincide with the original cones themselves (the primal cones). In general, they are not the same. As an example, consider the following L p cone, where p 1, L p (n + 1) = t x t R, x Rn, t x p.

IE 8534 16 One can show that (L p (n + 1)) = L q (n + 1) where 1/p + 1/q = 1. Now, the self-duality of SOC(n + 1) is due to the fact that SOC(n + 1) = L 2 (n + 1). In convex analysis, the dual object for a convex function f(x) is known as the conjugate of f(x), defined as f (s) = sup{( s) T x f(x) x dom f}, where dom f stands for the domain of the function f. The conjugate function is also known as the Legendre-Fenchel transformation.

IE 8534 17 Examples of some popular conjugate functions are: 1. For f(x) = 1 2 xt Qx + b T x where Q 0, the conjugate function is f (s) = 1 2 (s + b)t Q 1 (s + b). 2. For f(x) = n i=1 ex i, the conjugate function is f (s) = n i=1 s i log( s i ) + n i=1 s i. 3. For f(x) = n i=1 x i log x i, the conjugate function is f (s) = n i=1 e1+s i. 4. For f(x) = c T x, the conjugate function is f (s) = 0 for s = c and f (s) = + when s c. 5. For f(x) = n i=1 log x i, the conjugate function is f (s) = n n i=1 log s i. 6. For f(x) = max 1 i n x i, the conjugate function is f (s) = 0 if n i=1 s i 1, and f (s) = + otherwise.

IE 8534 18 If f(x) is strictly convex and differentiable, then f (s) is also strictly convex and differentiable. The famous bi-conjugate theorem states that f = cl f. There is certainly a relationship between the two dual objects: the dual cone and the conjugate function. Let us now set out to find the exact relationship. First of all, suppose that f is a convex function, mapping from its domain in R n to R. Its epigraphy is a set in R n+1 defined as epi (f) := t t f(x), t R, x dom (f) x.

IE 8534 19 It is well known that f is a convex function if and only if epi (f) is a convex set. It turns out that H (epi (f )) is the dual of H (epi (f)), after taking the closure and reorienting itself. To be precise, we have: Theorem 4 Suppose that f is a convex function, and p K := cl q p > 0, q pf(x/p) 0. x Then, K = cl u v s v > 0, u vf (s/v) 0.

IE 8534 20 Let L 2 := 0, 1 1, 0 and L := L 2 I n = [L 2, 0 2,n ; 0 n,2, I n ]. The effect of applying L to x is to swap the position of x 1 and x 2. Equivalently, Theorem 4 can be stated as (H (epi (f))) = cl L (H (epi (f ))). (4)

IE 8534 21 In linear programming, it is known that whenever the primal and the dual problems are both feasible then complementary solutions must exist. This important property, however, is lost, for a general conic optimization model, as the following example shows. Consider minimize x 11 subject to x 12 + x 21 = 1 x 11, x 12 x 21, x 22 0. This problem is clearly feasible. However, there is no attainable optimal solution.

IE 8534 22 Its dual problem is maximize subject to y y 0, 1 1, 0 + s 11, s 12 s 21, s 22 = 1, 0 0, 0 s 11, s 12 s 21, s 22 0, or, more explicitly, maximize subject to y 1, y y, 0 0. The dual problem is feasible and has an optimal solution; however, a pair of primal-dual complementary optimal solutions fail to exist, although both primal and dual problems are feasible.

IE 8534 23 One may guess that the unfortunate situation was caused by the non-attainable optimal solution. Next example shows that even if both primal and dual problems are nicely feasible with attainable optimal solutions, there might still be a positive duality gap, meaning that the complementarity condition does not hold. minimize x 12 x 21 + 2x 33 subject to x 12 + x 21 + x 33 = 1 x 22 = 0 x 11, x 12, x 13 x 21, x 22, x 23 x 31, x 32, x 33 0.

IE 8534 24 The dual of this problem is maximize y 1 subject to 0, 1 y 1, 0 1 y 1, y 2, 0 0, 0, 2 y 1 0. The primal problem has an optimal solution, e.g. 1, 0, 0 X = 0, 0, 0, with the optimal value equal to 2. The dual 0, 0, 1 problem also is feasible, and has an optimal solution e.g. (y1, y2) = ( 1, 1), with optimal value equal to 1. Both problems are feasible and have attainable optimal solutions; however, their optimal solutions are not complementary to each other, and there is a positive duality gap: c T x b T y = (x ) T s > 0.

IE 8534 25 Key References: Yu. Nesterov and A. Nemirovsky, Interior Point Polynomial Methods in Convex Programming, Studies in Applied Mathematics 13, SIAM, 1994. J.F. Sturm, Theory and Algorithms for Semidefinite Programming. In High Performance Optimization, eds. H. Frenk, K. Roos, T. Terlaky, and S. Zhang, Kluwer Academic Publishers, 1999. S. Zhang, A New Self-Dual Embedding Method for Convex Programming, Journal of Global Optimization, 29, 479 496, 2004.