Lecture 5. The Dual Cone and Dual Problem

Transcription

1 IE Lecture 5. The Dual Cone and Dual Problem

2 IE For a convex cone K, its dual cone is defined as K = {y x, y 0, x K}. The inner-product can be replaced by x T y if the coordinates of the vectors are given. Clearly, K is always a closed set, regardless if K is closed or not.

3 IE In light of the dual cone, the problem of finding a lower bound on the optimal value for the standard conic optimization problem can be done as follows. Introduce the Lagrangian multiplier y for the equality constraint b Ax = 0, and s for the conic constraint x K. The objective becomes c T x + y T (b Ax) s T x = y T b + (c A T y s) T x. This will be a lower bound on the optimal value whenever c A T y s = 0 and s K. The quest to find the best lower bound leads to the dual problem for the standard conic optimization problem: maximize subject to b T y A T y + s = c s K. (1)

4 IE As an example, for the standard SDP problem (primal), minimize subject to C X A i X = b i, i = 1,..., m X 0, its dual problem is in the form of maximize subject to m i=1 b iy i m i=1 y ia i + S = C S 0.

5 IE The so-called weak duality relationship is immediate, by the construction of the dual problem. This is shown in the next theorem. Theorem 1 Let x be feasible for the primal problem, and (y, s) be feasible to its dual. Then, b T y c T x. Moreover, if b T y = c T x then x T s = 0, and they are both optimal to their own respective problems. If a pair of primal-dual feasible solutions x and (y, s) both exist and they satisfy x T s = 0 then we call them a pair of complementary optimal solutions.

6 IE A very useful theorem is the following. Theorem 2 Let x be feasible for the primal problem, and (y, s) be feasible to its dual, and x int K and s int K. Then, the sets of optimal solutions for both problems are nonempty and compact, and any pair of primal-dual optimal solutions will be complementary to each other.

7 IE Let us take some moments to study the structures of the dual cones. Let us start with the topological structures. If rel (K) is denoted as the relative interior of K, then (rel (K)) = K. Let the summation of two sets A and B be A + B = {z z = x + y with x A, y B}. It is clear that the sum of two convex sets remains convex. However, the sum of two closed convex sets may not be closed any more.

8 IE Example 1 The set R SOC(3) is convex but not closed. For instance, the point (0, 1, 0) T is not in the set, but it is in the closure of the set, since for any ϵ > 0, we may write 1 ϵ 1 = 1 1 ϵ ϵ ϵ 1, 0 1 where the second term is in SOC(3). 1 ϵ

9 IE However, if A and B are both polyhedra, then A + B remains a polyhedron, hence closed. It is interesting to note that the set-summation and the set-intersection operations are a duality pair: Proposition 1 Suppose that K 1 and K 2 are convex cones. It holds that (K 1 + K 2 ) = K 1 K 2. Proof. For y (K 1 + K 2 ), it follows that 0 (x 1 + x 2 ) T y for all x 1 K 1 and x 2 K 2. Since 0 K 2, it follows that 0 (x 1 ) T y for all x 1 K 1, thus y K 1, and similarly y K 2. Hence, y K 1 K 2. Now, take any y K 1 K 2, and any x 1 K 1 and x 2 K 2. We have (x 1 + x 2 ) T y = (x 1 ) T y + (x 2 ) T y 0, and so y (K 1 + K 2 ). This proves (K 1 + K 2 ) = K 1 K 2 as desired.

10 IE The following bipolar theorem can be considered as a cornerstone for convex analysis. Theorem 3 Suppose that K is a convex cone. Then (K ) = cl K. Proof. For any x K, by the definition of the dual cone, x T y 0 for all y K, and so K (K ). Therefore cl K (K ). The other containing relationship (K ) cl K can be shown by contradiction. Suppose such is not true then there exists z (K ) \ cl K. By the separation theorem for convex cones, there is vector c such that c T z < 0, and c T x 0 x cl K. The last relation suggests that c K, which contradicts with the first relation because, as z (K ), it should lead to c T z 0, but not the opposite. This contradiction completes the proof.

11 IE This proof suggests that the bipolar theorem for convex cone is equivalent to the separation theorem itself. Suppose that K is a closed convex cone, and v K. Then, the bipolar theorem asserts that v (K ), and this implies that there is y K such that v T y < 0, and x T y 0 for any x K because y K, hence a separation exists.

12 IE In case K is closed, the bipolar theorem is what in folklore understood as the dual of the dual is the primal itself again. Combining Theorem 3 and Proposition 1, it follows that cl (K 1 + K 2 ) = (K 1 K 2 ). (2) It is also useful to compute the effect of linear transformation on a convex cone in the context of dual operation. Let A be an invertible matrix. Then, we have (A T K) = A 1 K, (3) by observing that y (A T K) (A T x) T y 0 x K Ay K y A 1 K.

13 IE A convex cone is called solid if it is full dimensional. Mathematically, K R n is solid span K = R n, or K + ( K) = R n. A convex cone is called pointed if it does not contain any nontrivial subspace; i.e. K ( K) = {0}. The following result proves to be useful: Proposition 2 If K is solid then K is pointed, and if K is pointed then K is solid.

14 IE Proof. Suppose that K is solid, i.e. there is a ball B(x 0 ; δ) K with radius δ > 0 and centered at x 0, and at the same suppose that there is a line Rc K, then 0 s(x 0 + t) T c, t δ and s R, which is clearly impossible. This contradiction shows that if K is solid then K must be pointed. By the bipolar theorem, the second part follows by symmetry.

15 IE It is easy to compute the following dual cones (R n +) = R n +. SOC(n) = SOC(n). (S n n + ) = S n n +, and (H n n + ) = H n n +. The above standard cones are so regular that their corresponding dual cones coincide with the original cones themselves (the primal cones). In general, they are not the same. As an example, consider the following L p cone, where p 1, L p (n + 1) = t x t R, x Rn, t x p.

16 IE One can show that (L p (n + 1)) = L q (n + 1) where 1/p + 1/q = 1. Now, the self-duality of SOC(n + 1) is due to the fact that SOC(n + 1) = L 2 (n + 1). In convex analysis, the dual object for a convex function f(x) is known as the conjugate of f(x), defined as f (s) = sup{( s) T x f(x) x dom f}, where dom f stands for the domain of the function f. The conjugate function is also known as the Legendre-Fenchel transformation.

17 IE Examples of some popular conjugate functions are: 1. For f(x) = 1 2 xt Qx + b T x where Q 0, the conjugate function is f (s) = 1 2 (s + b)t Q 1 (s + b). 2. For f(x) = n i=1 ex i, the conjugate function is f (s) = n i=1 s i log( s i ) + n i=1 s i. 3. For f(x) = n i=1 x i log x i, the conjugate function is f (s) = n i=1 e1+s i. 4. For f(x) = c T x, the conjugate function is f (s) = 0 for s = c and f (s) = + when s c. 5. For f(x) = n i=1 log x i, the conjugate function is f (s) = n n i=1 log s i. 6. For f(x) = max 1 i n x i, the conjugate function is f (s) = 0 if n i=1 s i 1, and f (s) = + otherwise.

18 IE If f(x) is strictly convex and differentiable, then f (s) is also strictly convex and differentiable. The famous bi-conjugate theorem states that f = cl f. There is certainly a relationship between the two dual objects: the dual cone and the conjugate function. Let us now set out to find the exact relationship. First of all, suppose that f is a convex function, mapping from its domain in R n to R. Its epigraphy is a set in R n+1 defined as epi (f) := t t f(x), t R, x dom (f) x.

19 IE It is well known that f is a convex function if and only if epi (f) is a convex set. It turns out that H (epi (f )) is the dual of H (epi (f)), after taking the closure and reorienting itself. To be precise, we have: Theorem 4 Suppose that f is a convex function, and p K := cl q p > 0, q pf(x/p) 0. x Then, K = cl u v s v > 0, u vf (s/v) 0.

20 IE Let L 2 := 0, 1 1, 0 and L := L 2 I n = [L 2, 0 2,n ; 0 n,2, I n ]. The effect of applying L to x is to swap the position of x 1 and x 2. Equivalently, Theorem 4 can be stated as (H (epi (f))) = cl L (H (epi (f ))). (4)

21 IE In linear programming, it is known that whenever the primal and the dual problems are both feasible then complementary solutions must exist. This important property, however, is lost, for a general conic optimization model, as the following example shows. Consider minimize x 11 subject to x 12 + x 21 = 1 x 11, x 12 x 21, x This problem is clearly feasible. However, there is no attainable optimal solution.

22 IE Its dual problem is maximize subject to y y 0, 1 1, 0 + s 11, s 12 s 21, s 22 = 1, 0 0, 0 s 11, s 12 s 21, s 22 0, or, more explicitly, maximize subject to y 1, y y, 0 0. The dual problem is feasible and has an optimal solution; however, a pair of primal-dual complementary optimal solutions fail to exist, although both primal and dual problems are feasible.

23 IE One may guess that the unfortunate situation was caused by the non-attainable optimal solution. Next example shows that even if both primal and dual problems are nicely feasible with attainable optimal solutions, there might still be a positive duality gap, meaning that the complementarity condition does not hold. minimize x 12 x x 33 subject to x 12 + x 21 + x 33 = 1 x 22 = 0 x 11, x 12, x 13 x 21, x 22, x 23 x 31, x 32, x 33 0.

24 IE The dual of this problem is maximize y 1 subject to 0, 1 y 1, 0 1 y 1, y 2, 0 0, 0, 2 y 1 0. The primal problem has an optimal solution, e.g. 1, 0, 0 X = 0, 0, 0, with the optimal value equal to 2. The dual 0, 0, 1 problem also is feasible, and has an optimal solution e.g. (y1, y2) = ( 1, 1), with optimal value equal to 1. Both problems are feasible and have attainable optimal solutions; however, their optimal solutions are not complementary to each other, and there is a positive duality gap: c T x b T y = (x ) T s > 0.

25 IE Key References: Yu. Nesterov and A. Nemirovsky, Interior Point Polynomial Methods in Convex Programming, Studies in Applied Mathematics 13, SIAM, J.F. Sturm, Theory and Algorithms for Semidefinite Programming. In High Performance Optimization, eds. H. Frenk, K. Roos, T. Terlaky, and S. Zhang, Kluwer Academic Publishers, S. Zhang, A New Self-Dual Embedding Method for Convex Programming, Journal of Global Optimization, 29, , 2004.