March 18,1996 DUALITY AND SELF-DUALITY FOR CONIC CONVEX PROGRAMMING 1 Zhi-Quan Luo 2, Jos F. Sturm 3 and Shuzhong Zhang 3 ABSTRACT This paper consider

Transcription

1 March 8,99 DUALITY AND SELF-DUALITY FOR CONIC CONVEX PROGRAMMING Zhi-Quan Luo, Jos F. Sturm and Shuzhong Zhang ABSTRACT This paper considers the problem of minimizing a linear function over the intersection of an ane space with a closed convex cone. In the rst half of the paper, we give a detailed study of duality properties of this problem and present examples to illustrate these properties. In particular, we introduce the notions of weak/strong feasibility or infeasibility for a general primal-dual pair of conic convex programs, and then establish various relations between these notions and the duality properties of the problem. In the second half of the paper, we propose a self-dual embedding with the following properties: Any weakly centered sequence converging to a complementary pair either induces a sequence converging to a certicate of strong infeasibility, or induces a sequence of primaldual pairs for which the amount of constraint violation converges to zero, and the corresponding objective values are in the limit not worse than the optimal objective value(s). In case of strong duality, these objective values in fact converge to the optimal value of the original problem. When the problem is neither strongly infeasible nor endowed with a complementary pair, we completely specify the asymptotic behavior of an indicator in relation to the status of the original problem, namely whether the problem () is weakly infeasible, () is feasible but with a positive duality gap, () has no duality gap nor complementary solution pair. KEY WORDS. conic convex programming, semidenite programming, duality, self-duality, interior point method. The research of the rst author is supported by the Natural Sciences and Engineering Research Council of Canada, Grant No. OPG99, and was performed during a research leave to the Econometric Institute, Erasmus University Rotterdam. Room 5/CRL, Department of Electrical and Computer Engineering, McMaster University, Hamilton, Ontario, L8S L, Canada. Econometric Institute, Erasmus University Rotterdam, The Netherlands.

2 Introduction A conic convex program is an optimization problem for which the objective is linear and the constraint set is given by theintersection of an ane space with a closed convex cone. As such, it contains as special cases the linear programming problem, the quadratic programming problem, and most notably, the semidenite programming problem. Lately, the latter problem has been the focus of many studies due to its application to combinatorial optimizations and due to the applicability of interior point methods to solve this class of problems, see e.g. [8, 9,,,,, ]. A common assumption used in the interior point methods for conic convex programming is that a feasible interior starting point exists and is known. This requirement can be partially alleviated when an infeasible interior point method is used, as in Kojima-Shindoh-Hara [8] and Lin-Saigal [9], or when a self-dual embedding is used as in Potra-Sheng []. Nonetheless, these references only consider the complementarity version of the problem, which in general is not equivalent to the original conic convex optimization problem unless the Slater condition holds. This is unfortunate since in many applications the Slater condition may fail to hold, and in such circumstances it is usually not sucient to merely conclude whether or not a complementary solution exists. Instead, we must determine either that the problem is infeasible/unbounded, or else generate a sequence whose objective values approach the inmum of the original conic convex program. Clearly, this latter task requires a careful study of duality theory for conic convex programming in the absence of constraint qualications. There have been several studies of duality theory for conic convex programming, most of which assume a constraint qualication (in particular Slater's condition), see Dun [], Nesterov and Nemirovsky [] and Alizadeh []. Wolkowicz [] and Ramana et al. [] manage to dispense with any constraint qualication by working with a nonstandard primal-dual pair. It is currently not known though, whether the interior point methods can cope with this nonstandard duality. The diculty with the study of duality theory for conic convex programming lies in the nonlinearity of the conic constraint. Without any constraint qualication, the problem can exhibit many new duality features which are not present in linear programming. The same is true with the theory of self-duality. The latter theory forms the basis of a homogeneous self-dual model for conic convex programming. Such a model is particularly useful when interior point methods are used to solve the conic convex programming problem. The contributions of this paper are three fold. First, we give a detailed study of duality theory for conic convex programming in the absence of constraint qualications. Our study is based on generalizations of Farkas' lemma which provides characterizations for strong and weak infeasibility. Second, we present a self-dual embedding of the conic convex programming problem that is well suited for interior-point solution methods. In particular, we thoroughly analyze the limiting behavior of a weakly centered sequence for the self-dual embedding. Such a sequence is generated by most interior point methods. It is shown that a weakly centered sequence can provide informa-

3 tion regarding the status of the original conic convex programming problem, namely whether the problem has a complementary pair, or is strongly infeasible, or is weakly infeasible, or feasible but with a positive duality gap,or has no duality gap nor complementary solution pair. Third, we present many interesting examples from semidenite programming which reveal some delicate features of duality and self-duality theory for the general conic convex programming problem. This paper is organized as follows. In Section, we formally state the conic convex programming problem and the relevant terminology. Section is devoted to the study of duality relations for the standard conic primal-dual pair in the absence of constraint qualications. In Section we start with a generalization of the Goldman-Tucker type self-dual feasibility model. We then dene an extended self-dual programming problem, and analyze the limiting behavior of so-called weakly centered solution sequences for this extended self-dual programming problem. These results are illustrated by some examples from semidenite programming. Finally, we conclude the paper in Section 5. Notation. Given a set S, we let cl (S) int S and rel S denote the closure of S, the interior of S and the relative interior of S respectively. If S is a subset of < n and A is an m n matrix, then the image of S under the linear mapping A is denoted by AS, i.e. AS = fy < m j y = Ax for some x Sg: Terminologies and Preliminaries In this paper, we consider the general conic convex programming problem (P ) inf c T x s:t: x ; c A x K where c < n, A is a linear subspace of < n and K is a closed convex cone in < n. Let K denote the dual cone of K, i.e. K := fs < n j s T K< + g:

4 The cone ;K is also known as the polar cone of K [5]. Denote the orthogonal complement ofa (i.e. the dual cone of A) by A?. It is well known that (K ) = K and (A? )? = A, because A and K are closed [5]. A seemingly more general form of conic convex programming is (P ) inf f T x s:t: x ; g A x K where f g < n. However, if we denote the orthogonal projection matrices onto A and A? by P A and P A? respectively, it follows that (P ) is equivalent to (P) with Associated with (P) is a dual problem, c = P A f + P A?g: (D) inf c T s s:t: s ; c A? s K : We see that this dual is also a conic convex programming problem, and that dualizing the dual problem (D) yields the primal problem (P). As a side remark, we mention that the dual of (P )is (D ) inf g T s s:t: s ; f A? s K which is clearly equivalent to (D) for arbitrary f c + A? and g c + A. Denote the feasible set of (P) by F P := K\(c + A) and the feasible set of (D) by F D := K \ (c + A? ): Let p := inf c T F P d := inf c T F D denote the optimal solution values of (P) and (D) respectively. We denote the primal and dual optimal solution sets by F P := fx F P j c T x = p g F D := fs F D j c T s = d g:

5 Denition Problem (P) is said to be solvable if F P = : A special case of nonsolvability occurs when p = ;. In this case, wesay that (P) is unbounded. We say that (P) is feasible if the set of feasible solutions F P is nonempty. The interior solution sets for (P) and (D) are o F P = F P \ int K and respectively. The term strong feasibility points. o F D = F D \ int K denotes the case that (P) has so-called interior feasible Denition Problem (P) is said to be strongly feasible if o F P = : Denition Problem (P) is said to be weakly feasible if F P = but o F P = : Strong feasibility isalsoknown as Slater's constraint qualication. If x F P and s F D, then x T s = c T x + c T s ;kck : () This relation immediately implies the following weak duality result. Lemma (Weak duality) If F P = and F D = then p + d kck : We remark that for f c + A?, g c + A and f? g there holds x T s = f T x + g T s: () Denition The primal-dual pair (P), (D) is said to be strongly dual if p + d = kck : Denition 5 The primal-dual pair (P), (D) is said to be weakly dual if p + d > kck :

6 We introduce a set of -feasible solutions as follows: Similarly, we dene F P () :=fx KjkP A?(x ; c)k g : F D () :=fs K jkp A (s ; c)k g : Clearly, we have F P = F P () F P () and F D = F D () F D (), for all. Notice that F P () = for all > if and only if the distance between A + c and K is zero. Unlike the (nite) linear programming case, it is possible that F P () =, for all > but F P = fa + cg\k=. In this case, we say that (P) is weakly infeasible. We will provide some examples of weakly infeasible problems in Section. and Section.. Denition Problem (P) is said to be weakly infeasible if F P = but F P () = 8 >: Denition Problem (P) is said to be strongly infeasible if F P () = for some >: Clearly, (P) is not strongly infeasible is equivalent todist(a + c K) =. The notions of weak and strong infeasibility of dual problem (D) can be dened symmetrically. The following lemma characterizes strong infeasibility for (P). Lemma Problem (P) is not strongly infeasible, i.e. F P () = 8 > if and only if P A?c cl(p A?K): Proof: We have F P () = 8> if and only if there exists a sequence x () x () ::: in K such that which isequivalent to lim P A?(x (i) ; c) = i! P A?c cl (P A?K): In the remainder of this section, we collect some useful facts that will be used later in the development of duality and self-duality theory. Below is a well-known elementary property of a closed convex cone K. 5

7 Lemma int K = if and only if K is pointed, i.e. K\(;K) =fg. Moreover, if s int K and x Kthen (s ) T x = ) x =: () Lemma If K is a pointed closed convex cone with nonempty interior in the sense that K\(;K) =fg int K = then K\K is also a pointed closed convex cone with nonempty interior. Proof: That the cone K\K is pointed, closed and convex follows immediately from the fact that K and K are pointed closed convex cones. To show that int (K\K ) =, we will use a result of Rockafellar [5] (Corollary.. therein) which states that if K and K are nonempty convex cones, then K \K =(K + K ) : Applying this result with K = K and K = K,wehave K\K =(K + K ) : Using Lemma, it remains only to show thatk + K is pointed. Suppose it is not, then there exist f Kand g K such that ;(f + g) K+ K f + g = : Assume without loss of generality thatf =. The above relation implies that ;f = g ; (f + g) K + K and hence ;f ; f K for some f K: Since K is pointed, we have f = ;f so that (f + f ) T (;f ; f ) < f + f K ;f ; f K which contradicts the denition of a dual cone. Finally, we will need the following lemma to calculate the orthogonal complement of a linear subspace. Lemma 5 Suppose A is a linear subspace in < n. Let M < nn beaninvertible matrix. Then where M ;T := (M T ) ;. (MA)? = M ;T A?

8 Proof: Let y < n be any xedvector. We note the following relations: y (MA)? () y T (Mx)= 8x A () (M T y) T x = 8x A () M T y A? () y M ;T A? : The proof is complete. Relations of the Conic Primal-Dual Pair Semidenite and conic convex programming duality has been studied by Dun [], Wolkowicz [], Ramana et al. [], Nesterov and Nemirovsky [] and Alizadeh [], among others. A Slater condition or similar constraint qualication is usually assumed in order to arrive at duality results that show great similarity with the linear programming duality [,, ]. However, in [] it is shown that constraint qualications can be circumvented by requiring K to be a minimal cone for c + A, i.e. c + A\rel(K) = : Such a minimal cone formulation always exists, but it is often not possible to compute the minimal cone for (P) in practice. The approach of [] does not explicitly use minimal cones, but makes use of an extended Lagrange-Slater dual program. However, we are not aware of any solution method that uses the extended dual rather than the standard dual (D). Duality relations for the standard pair (P), (D) are studied in [, ]. In this section, we will derive the duality results by means of a generalized Farkas' lemma. The generalized Farkas' lemma will be of importance in itself when we analyze self-dual embeddings in Section.. Farkas-type lemmas The following lemma is a generalization of Farkas' lemma to conic convex programming. In previous generalizations of Farkas' lemma, it is usually assumed that P A?K is closed, see Hurwicz [], Ben-Israel [] and Alizadeh []. Alternatively, Wolkowicz [] extended Farkas' lemma under the assumption that K is a minimal cone for A?, i.e. A? \ relk =. The formulation below however, does not have any requirements on (P). This fact will become essential in the remainder of this paper.

9 Lemma (First Farkas' lemma) F P is not strongly infeasible in the sense that F P () = 8 > if and only if c T (A? \K ) < + : Proof: It is easy to see that a linearly transformed convex cone is also a convex cone. Therefore, we can apply the bi-polar theorem (see Rockafellar [5], Theorem.), which states that By Lemma and the relation () we have where cl (P A?K) =(P A?K) : () F P () = 8 > () P A?c cl (P A?K) () c T P A?(P A?K) < + (5) (P A?K) = f < n j T P A?K< + g Combining the above relation with (5) we obtain = f < n j P A? K g = (K \A? )+A: F P () = 8 > () c T (K \A? ) < + : We letd D be the set of dual improving directions, viz. D D := fs K \A? j c T s ;g: () We know from Lemma that (P) is strongly infeasible if and only if D D =. We dene an -relaxed set by D D () :=fs K jkp A sk c T s ;+g: () Next we proceed to derive acharacterization of primal infeasibility in terms of the set D D (). To thisend,we need to homogenize the region F P to form the cone (x x ) x F P x > : x 8

10 This cone is contained in the linear subspace B, " I c B := In particular, we have This implies that By Lemma 5, there holds x F P () (x x T x ) T B\(K < ++ ): F P = () B\(K< + )=(A\K) fg B? = " () () h h I ;c T i # (A<): (8) ; (B \(K< + )) = fg i ; (B \(K< + )) < + : (9) # (A? fg) = " # I A? : () ;c T By Lemma and the relation (9), we obtain the second version of Farkas' lemma which provides a characterization of the primal infeasibility F P =. Lemma (Second Farkas' lemma) We have F P = if and only if D D () = 8 >: Proof: By (9) we have F P = () h ; i (B \(K< + )) < + : In light of Lemma, this implies that F P = if and only if the distance from B? +( ;) T to (K < + ) is zero. Using the denition of dual cone, we can verify that (K < + ) = K < +. Thus, F P = is equivalent to ( s s! K < + P B s s! +!! ) = 8 >: Using the expression () for B?,we conclude that F P = is equivalent to ) kpa (s)k jc T s + s +j = 8 >: ( s s! K < + 9

11 By the denition () of D D () and a rescaling of variable s if necessary, we see that the above condition is further equivalent to This completes the proof. D D () = 8 >: We summarize the results of Lemma and Lemma in the following theorem. Theorem (Characterization of feasibility and weak/strong infeasibility) Let F P = K\(c + A) and let D D be givenby().then F P F P F P is strongly infeasible if and only if D D is feasible. is weakly infeasible if and only if D D is weakly infeasible. is feasible if and only if D is strongly infeasible. The proof of Theorem follows immediately from Lemma and Lemma. Theorem will be used to derive duality theorems in the next subsection.. Duality theorems It is well known that if (P) is a linear programming problem and p is nite, then strong duality holds, i.e. p + d ;kck =. Our objective is to generalize the strong duality result of linear programming to conic convex programming. Notice that, for general conic convex programming, it is possible that p is nite but (D) is weakly infeasible (see e.g. Example 8 in Section.). This means that we should allow an arbitrarily small constraint violation for the dual and dene d () :=infc T F D (): Before stating our strong duality result, we derive the following weak duality. Lemma 8 (Weak duality) If ; <p < then p + lim # d () ;kck : Proof: Let s() F D (), >, be a sequence with lim c T s() =limd (): # #

12 Dene u() :=P A (s() ; c): By denition of F D (), we have ku()k. By denition of p,weknow that for any >, there exists x( ) F P with c T x( ) <p + : () As x( ) Kand s() K,wehave x( ) T s() =c T s()+(x( ) ; c) T s(): Using the fact that (x( ) ; c) Aand s() c + u()+a?,we further obtain If we let #, we see that Combining this with (), the lemma follows. (c + u()) T x( ) c T (c + u()) ; c T s(): c T x( ) = lim # (c + u()) T x( ) lim(c T (c + u()) ; c T s()) # = kck ; lim # d (): Based on the extended Farkas' lemma and the above weak duality result, we obtain a strong duality theorem: Theorem There holds If p =, then either (D) is strongly infeasible and or (D) is not strongly infeasible and lim # d () = lim # d () =;: If ; <p < then p +lim # d () ;kck =: If p = ;, then(d) is infeasible and lim # d () =:

13 Proof: It is obvious that if (D) is strongly infeasible, then lim # d () =. Now consider the case that (D) is not strongly infeasible. We obtain from Lemma that c T (A\K) < + : () If p =, then (P) is infeasible. By the denition (8) of B, this is equivalent with (B \(K< + )) = (A\K) fg: () Combining () and (), it follows that h i c T (B \(K< + )) = c T (A\K) < + for any <. Applying Lemma to the above relation, and using (), we obtain that for any > and > there exists s F D () and s < + with Since s and <is arbitrary, wehave s ; + c T (s ; c) : d () =; for all >: This concludes the case p =. Now consider the case p = ;. Let x F P be an arbitrary vector, and let s() F D () bea sequence such that c T s() d ()+ for all >: By the denition of F D (), there exists some u() with ku()k such thats() ; c + u() A? and s() K. Since (x ; c) Aand x K,we obtain Letting # in the above relation yields x T s() =c T x + c T s() ; (x ; c) T u() ;kck : c T x + lim # d () ;kck : Since p = ;, wecanchoose x F P such that c T x!;. This implies lim # d () =. Now the infeasibility of (D) follows immediately. It only remains to consider the case ; <p <. Since in this case (P) is feasible, but not unbounded (p > ;), we know that there cannot exist a primal improving direction, i.e. c T (A\K) < + () and as p := inf c T F P wehave c T F P p + < + : (5)

14 Combining () and (5) yields c T x x p for all (x x ) K< + with (x ; x c) A. Stated dierently, wehave h c T ;p i (B\(K< + )) < + : Applying Lemma and (), we conclude that for any >and > there exists s F D () and s < + with s + p + c T (s ; c) : This shows that p + d () ;kck Combining this with Lemma 8 proves the theorem. for all >: For the case that p is nite, we see from the above theorem that p + d ;kck =ifd () is lower semicontinuous. The examples in Section. show that this is in general not the case. Our next goal is to derive asucient condition for d () to be lower semicontinuous. We need several lemmas. Lemma 9 Let s () int K and < + then is compact. K\fx < n j (s () ) T x g Proof: Suppose to the contrary that there exists a sequence x (i) with lim i! x (i) K\fx < n j (s () ) T x g i = ::: =. Let y be a cluster point of the bounded sequence ( ) x (i) x (i) i = ::: : By construction, kyk = and we have y Kdue to the closedness of K. Moreover, there holds (s () ) T y lim But this contradicts the fact that y K y =. i! x (i) =:

15 Corollary Let s () int K and < +. The following two statements are equivalent:. There exists x F P such that (s () ) T x.. For each > there exists x F P () such that (s () ) T x +. Proof: \ ) " is obvious. To prove the other direction, suppose that the second statement holds. Then there exists a sequence x (i) K, i = :::,such that and lim P A?(x (i) ; c) = () i! lim sup(s () ) T x (i) : () i! Let := sup i (s () ) T x (i). We know from Lemma 9 that the sequence x () x () ::: is contained in a compact set, viz. K\fx < n j (s () ) T x g: Hence, this sequence has a cluster point ink, say x K. From () it follows that x;c A, hence x F P : Finally, we know from () that (s () ) T x : o The above corollary essentially states that if F D = (the Slater condition), then any lower level set of the primal must be either nonempty or strongly infeasible. Corollary Let <and suppose o F D =. Then 9x F P such that c T x if and only if for any > 9x F P () such that c T x + : Proof: Let s () o F D be xed. Then the identity (),viz. (s () ) T x = c T x + c T s () ;kck 8x F P

16 implies the one-to-one correspondence between the level sets of c T x and (s () ) T x. Now we invoke Corollary. Hence, we can strengthen the duality result for the case in which the Slater condition holds. We thus arrive at the following well known strong duality theorem [,, ]. Theorem (Strong duality) Suppose that o F D =. Then either d = ; and F P = or p + d = kck and F P = : Proof: If d = ;, we obtain from Lemma that F P =. Now consider d > ;. We want to show that d () islower semicontinuous. Let s o F D and let s() F D () be a sequence with lim c T s() =limd (): # # Let u() :=P A (s() ; c), and notice that ku()k. Since s o F D, there exists some >such that We claim that s () := ky ; sk ) y K : ; (s() ; u()) + s F D 8 ( ): Indeed, since (s() ; u()) A? + c and s A? + c, it follows that s () A? + c. Furthermore, we can write s () = ; s()+ s ; ; u() : Notice that the vector s ; ; u() is within distance from s, thus it is in the cone K. Since s() K, it follows s () must also belong to K. This shows s () F D. Now wehave that d lim # c T s () = lim # c T s() =lim # d (): On the other hand, it is obvious that lim # d () d, and hence d () is lower semicontinuous. We thus obtain from Theorem that p + d ;kck =: 5

17 Finally, we have from Corollary that c T x = p for some x F P : We obtain from Theorem the well known fact that if F o P F o D =, then F P F D = and (x ) T s = c T (x + s ; c) = for all (x s ) F P FD: We say that in this case the conic convex programming problem has a complementary solution: Denition 8 A complementary solution is a pair (x s) F P F D such that x T s =: Obviously, any complementary solution is an optimal solution. The converse however, is in general not true unless o F P = or o F D =. This is because without the latter condition, there may exist a positive duality gap and as a result there cannot exist a complementary solution. Moreover, strong duality is necessary, but not sucient for the existence of a complementary solution. The duality relations are summarized in the following table. primal feasible primal infeasible strong weak weak strong dual strong A s.d., (P)+(D) solvable s.d., (P) solvable (D) unbounded (D) unbounded feasible weak B possible possible (D) unbounded dual weak C possible possible infeasible strong D possible The acronym `s.d.' in the above table stands for `strongly dual'. All entries in the table represent possible combinations of the status of the primal and dual problem. Only if we cannot conclude anything more, we explicitly mention that the entry represents a possible state. Due to the complete symmetry of the conic convex programming duality, the table is symmetric, so we only need to consider the upper-right block. The entries in the rst row of the table are denoted by `A', `A', `A' and `A', in the second row by `B', `B', and so on. The entries `A', `A', `A' and `A', are due to Theorem. Lemma implies entry `B'. The possibility of states `A' and `B' will be demonstrated in Section. by Examples and respectively. The entries `C' and `C' will be illustrated by Examples and respectively. Finally, the possibility of the entries in the table where weak infeasibility is not involved, is easily demonstrated by a -dimensional linear programming problem:

18 Example Let n =, c <, K = K = < + and A = f(x x ) j x =g A? = f(s s ) j s =g: We see that (P) is strongly feasible if c >, weakly feasible if c = and strongly infeasible if c <. Similarly, (D) is strongly feasible if c >, weakly feasible if c =and strongly infeasible if c <. Weak infeasibility does not exist in linear programming. However, weakly infeasible problems are easily constructed in semidenite programming, which is an important class of conic convex programming.. Semidenite programming Before we discuss some examples of semidenite programming problems, we have tointroduce some notation. For given positiveinteger n, we denote the space of n n symmetric matrices by S. Given X and Y in S, the standard inner-product is dened as X Y =trxy: The Frobenius norm of a matrix X Sis kxk F = p X X. If X Sis positive (semi-) denite, we writex (X ). Since the dimension of the space of n n symmetric matrices is n, n := n(n +) we can construct for S an orthonormal basis U of cardinality n, U = fu U ::: U n g: By orthonormality we mean that U i U j =fori = j and ku i k F =fori = ::: n. The conic convex problem (P) is a semidenite programming problem (SDP) if K = ( x < n j nx i= x i U i We letx =vec(x U) denote the coordinate vector of a matrix X S with respect to the basis U, i.e. x i = U i X for i = ::: n: It is easily veried that for X Y S we have X Y =trxy =vec(x U) T vec(y U): ) :

19 Suppose that A < mn is a matrix with kernel A, i.e. There holds We dene and A = fx < n j Ax =g: A? = fs < n j s = A T y for some y < m g K = K: A (i) := nx j= C := a ij U j nx i= We cannow rewrite the SDP (P) in matrix form as with dual c i U i for i = ::: m: (SDPP) inf C X s.t. A i (X ; C) = for i = ::: m X (SDPD) inf C S s.t. S = C + S : mx i= y i A (i) for some y < m If we allow multiple semideniteness constraints, say X () X (), we arrive at a seemingly more general form of semidenite programming. Yet, these problems can be cast in the form (SDPP), see e.g. Vandenberghe and Boyd [9]. This is true in particular if semideniteness constraints are combined with nonnegativity constraints. We will now give an example where the primal is weakly infeasible and the dual is strongly infeasible. Example Consider m = n =and " # C = ; The primal is weakly infeasible, p = inf and the dual is strongly infeasible, d =inf ( ( A () = x ; x X = s ; s S = " # A () = " " # ) = x " s s ; # ) = : # : 8

20 It is also possible that the primal and the dual are both weakly infeasible. Example Consider m = n =and C = 5 A () = 5 A () = 5 : The primal is weakly infeasible, p = inf 8 >< >: x +x X = x x x x 5 9 >= > = and the dual is also weakly infeasible, d = inf 8 >< >: s +s S = s s 9 >= 5 > = : In Section. we will encounter combinations of primal weak infeasibility with dual weak feasibility and dual strong feasibility. Self-Duality Almost all early interior point algorithms for linear programming assume strong feasibility of(p) and (D) and even the availability of an initial interior solution. A seminal approach for solving linear programs without any regularity assumptions is the self-dual embedding, developed by Ye-Todd- Mizuno []. In this approach, the primal and dual linear programs are combined in a single self-dual linear program, with an obvious initial feasible interior point, thus fullling a key requirement of the interior point methods. When applied to the self-dual formulation, the interior point methods generate a weakly centered sequence (to be dened later) that converges to a strictly complementary solution. Such a solution of the self-dual linear program can be used either to recover a solution of the original linear programming problem or to demonstrate its infeasibility. Another advantage of this self-dual formulation is that it only increases the problem dimension by. At the time of writing, the self-dual scheme was generalized to semidenite programming by Potra-Sheng []. However, this reference only solves the complementarity version of semidenite programming, which is not equivalent to semidenite optimization, except if (P) and (D) are assumed to be strongly feasible. Before we extend the Ye-Todd-Mizuno self-dual embedding to conic convex programming, we need to discuss a generalization of the Goldman-Tucker self-dual feasibility model. 9

21 . A self-dual feasibility model In the previous section, we have obtained a number of results for the primal problem (P). Due to the complete symmetry in duality, the results hold true for the dual (D) as well. In the sequel, we will consider the pair of problems (P) and (D) and we will refer to this primal-dual pair by the triplet (c A K). We say that a conic convex program (c A K) is infeasible if F P F D = i.e. if either the primal or the dual is infeasible. We let n D PD () := (x s) KK jkp A?x + P A sk c T (x + s) ;+ o and we dene D PD := D PD (): By Lemma, (c A K) is infeasible if and only if D PD () = 8 >: Furthermore, Lemma implies that (c A K) is strongly infeasible if and only if D PD =. We will now generalize the self-dual feasibility model of Goldman and Tucker [, 8] to conic convex programming. Denition 9 A conic convex program (c A K) is self-dual SD such that We let SD c = c SD A = A? SD K = K : A SD := I c I c ;c T ;c T ;kck if there exists a permutation matrix 5 (AA? <): Obviously, A SD is an (n + )-dimensional subspace of < n+ and by Lemma 5 its orthogonal complement is I c A? SD = I c ;c T ;c T ;kck 5 (A? A<): We introduce the closed convex cone K SD, K SD := KK < + < +

22 whose dual cone is K SD := K K< + < + : Now consider the following homogeneous feasibility model: or equivalently, Obviously, wehave if and only if (SD) min min Hence, (SD) is self-dual and Theorem There holds h h h x T s + x s = s:t: x ; x c A s ; x c A? c T x + c T s ; x kck + s = x K s K x s x T s T x s i T ASD \K SD : x T s T x s i T ASD \K SD s T x T s x i T A? SD \K SD : for h x T s T x s i T ASD \K SD : (8) (SD) has a solution with x > if and only if (c A K) has a complementary solution. (SD) has a solution with s > if and only if (c A K) is strongly infeasible. Proof: If x > then ( x x s x ) F P F D and we know from (8) that ( x x s x ) is a complementary solution. Conversely, if(x s ) is a complementary solution to (P), (D), then (x s ) A SD \K SD : If s >, then x =and so that ;s = c T x + c T s< s (x s) D PD :

23 Applying Lemma, it follows that (c A K) is strongly infeasible. Conversely, if(c A K) is strongly infeasible, then we know from Lemma that there exists (x s) D PD. For such a pair (x s) D PD we obviously have (x s ) A SD \K SD :. The extended self-dual program The self-dual feasibility model introduced in the previous subsection does not have a strongly feasible point, thus making it dicult to apply the interior point methods directly. The purpose of this subsection is to augment the aforementioned self-dual model by allowing a simple starting point for the interior point methods. Dispensing with any constraint qualication, we will discuss to what extent the original problem (c A K) can be solved, when the interior point method is applied to such an extended self-dual model. Throughout this subsection, we will make the following assumption on the cone K. Assumption The cone K is pointed and has a nonempty interior, i.e. K\(;K) =fg int K = : By Lemma, there exists some vector int (K\K ): Let us consider the following conic convex programming problem: (SD) min ( + kk )y s:t: x (x ; y )c + y + A (9) s (x ; y )c + y + A? () c T (x + s ; (x ; y )c ; y )+s = y () T (x + s)+x + s + y + z =(+y )( + kk ) () x K s K (x y s z ) < +: () We denote the vector of decision variables for (SD) by w, w := (x s x y s z )

24 and we letf SD denote the feasible set of (SD), with interior We see immediately that the vector and that (SD) has a trivial optimal solution o F SD := F SD \ int (KK < +): c SD := ( ) o F SD () w := ( +kk ): The optimal set of (SD) is Remark that if w F SD,then F SD := fw F SD j y =g: (x s x s ) A SD \K SD : This means that every optimal solution of (SD) induces an optimal solution of (SD). The reverse is also true if the solutions are normalized according to the constraint (). Self-dual programs that are similar to (SD) were proposed and studied for linear programming by Ye, Todd and Mizuno [] and Jansen, Roos and Terlaky []. Lemma The conic convex problem (SD) is self-dual. Dening the permutation SD by SD (x s x y s z )=(s x s z x y ) we have ( + kk )y = wt SD w = x T s + x s + y z for any w F SD. Proof: Let w F SD and introduce the variables x := x ; (x ; y )c ; y s := s ; (x ; y )c ; y : Then, x Aand s A?. From (), we have s = ;c T (x +s) ;kck x +(+kck )y (5) and from (), z =+kk +(;kk + T c)y ; ( + T c)x ; T (x +s) ; s :

25 Now substituting (5) for s yields z =+kk ;k ; ck y ; ( + T c ;kck )x ; ( ; c) T (x +s): () Now we dene r := ; c and let and A SD := We see from (9){() and (){() that I c r I c r ;c T ;c T ;kck +kck ;r T ;r T kck ; ( + T c) ;krk K SD := KK < + : 5 (AA? <<) F SD =(w + A SD ) \K SD =(c SD + A SD ) \K SD : By () and () we further have that (SD) is the conic convex problem (c SD A SD K SD ). Now dene the permutation SD by SD (x s x y s z )=(s x s z x y ): By Lemma 5, the orthogonal complement ofa SD is A? SD = I c r I c r ;c T ;c T ;kck +kck ;r T ;r T kck ; ( + T c) ;krk 5 (A? A<<) we have A? SD = SDA SD K SD = SDK SD c SD = SD c SD so that the program (c SD A SD K SD ) is self-dual. Finally, it follows from (w ; w )( SD w ; SD w ) = that We will now consider a sequence ( + kk )y = x T s + x s + y z : fw () w () :::gf SD with lim k! y(k) =: Applying Lemma 9 (or alternatively relation ()), it follows that this sequence is contained in a compact set and hence it has a limit. Moreover, any limit is an optimal solution to (SD) and hence to (SD).

26 Denition A sequence fw () w () :::gf SD is said to be weakly centered if there exists a scalar constant!> such that and lim k! y (k) =. x (k) s(k)!y (k) > for k = ::: () We remark that condition () is the minimal centrality condition of Todd and Ye []. This condition holds true for all path-following algorithms and for some potential reduction methods. In particular, Nesterov and Todd [] developed a framework of primal-dual interior point algorithms for self-scaled conic convex problems that generate a weakly centered sequence of iterates. Selfscaled conic convex programming is a rich subclass of conic convex programming and it includes linear programming and semidenite programming among others. It should be noted here that if (c A K) is a linear (semidenite, self-scaled) programming problem, then (SD) is also a linear (semidenite, self-scaled) programming problem. Following the argument of Guler and Ye [5], we can easily show that if (SD) has an optimal solution with the property x + s >, then any limit point of a weakly centered sequence for (SD) must have the same property. Theorem 5 Let w () w () ::: be aweakly centered sequence for(sd). Then lim inf k! x (k) > if and only if (c A K) has a complementary solution. lim inf k! s (k) > if and only if (c A K) is strongly infeasible. (P) is strongly infeasible if lim inf k! s (k) > and lim inf k! c T s (k) <. (D) is strongly infeasible if lim inf k! s (k) > and lim inf k! c T x (k) <. Moreover, the sequence w () w () ::: has a limit point. For any such limit point w () holds there If (c A K) has a complementary solution, then x () If (c A K) is strongly infeasible, then s () (x () s () ) F P F D: (x () s () ) D PD : 5

27 Proof: Following the argument ofguler and Ye [5] and using Lemma, we have for any w F SD, and for any k, = (w (k) ; w ) T SD (w (k) ; w ) = (w (k) ) T SD w (k) ; (w ) T SD w (k) +(w ) T SD w ( + kk )(y (k) + y ) ; (x s (k) + s x (k) ): Multiplying with x(k) +s(k) and using (), we obtain (y (k) +y ) ( + kk )(x (k) + s (k) ) ;!y(k) (x y (k) + y + s ): In particular, if we letw F SD,wehave y =andhence lim inf k! (x(k) + s (k) )! +kk (x + s ): This shows that if (SD) has a solution with x + s >, then any limit point w () of the sequence fw (k) j k = :::g satises x () + s () >. Using Theorem, we can then use such a limit point to decide whether (c A K) has a complementary solution and whether it is strongly infeasible. The remaining statements of the theorem are now easily veried using (9)-(). Let w () w () ::: be a weakly centered sequence for (SD). Theorem 5 characterizes the case that lim inf x (k) + s (k) > : k! The remainder of this section is dedicated to the remaining case where x (k) + s (k) =: lim k! Here, we need to distinguish three possibilities, viz. problem (c A K) is weakly infeasible, or feasible but there is no strong duality, or feasible and strong duality applies, but there exists no complementary solution pair.

28 We will develop an indicator on the basis of which we can often exclude one or two of the above possibilities. Before we introduce this indicator, we remark that since lim k! x (k) + s (k) =, there must exist k such that x (k) + s (k) <! 8k k : Consequently, there holds (cf. ()) x (k)!y(k) s (k) >y (k) > and s (k)!y(k) x (k) For k k, let us introduce the sequences ^w (k) := w (k) w (k) := x (k) ; y (k) and We see from (9)-() that Moreover, we have from (8) that >y (k) > for all k k : (8) s (k) ; y (k) w (k) ^ (k) := ^y (k) kk (k) := (x (k) +y (k) )( + kk + kck) : (^x (k) ^s (k) ) F P (^ (k) ) F D (^ (k) ) (9) (x (k) s (k) ) D PD ( (k) ): () lim k! ^y(k) = lim k! y(k) = lim k! ^(k) =: () The following lemma shows that ^w (k) provides information on the level sets of (P) and (D). Lemma Let fw (k) k = :::g be a weakly centered sequence with lim k! (x (k) + s (k) ) =. Then for any xed k k there holds Proof: inf(c +^y (k) )T F P c T (c +^y (k) ) ; ct^s (k) : Let x F P be an arbitrary primal feasible solution. As x Kand ^s (k) K,wehave x T^s (k) = c T^s (k) +(x ; c) T^s (k) so that using x c + A and ^s (k) c +^y (k) + A?, Hence, c T^s (k) +(x ; c) T (c +^y (k) ): (c +^y (k) )T x c T (c +^y (k) ) ; ct^s (k) :

29 The following theorem shows that a weakly centered sequence for (SD) provides useful information for the original conic convex problem (c A K) even if this latter problem is neither strongly infeasible nor endowed with a complementary solution pair. A crucial quantity here is the indicator, whichwe dene as follows: := lim sup ^s (k) : k! Theorem Let w () w () ::: be aweakly centered sequence with Then and Moreover, lim sup c T^x (k) + p k! lim k! (x(k) + s (k) )=: (^x (k) ^s (k) ) F P (^ (k) ) F D (^ (k) ) lim sup c T^s (k) + d () k! lim k! ^(k) =: () if (c A K) is feasible, then < () if (c A K) is feasible and strong duality applies, then if =, then (c A K) is weakly infeasible and lim ct^x (k) = p lim ct^s (k) = d = (5) k! k! (x (k) s (k) ) D PD ( (k) ) lim k! (k) =: () Proof: The relation () follows immediately from (9) and (). We will now show (). By the denition of p,weknow that for any > there exist x() F P such that Also, from () and Lemma we have Since > is arbitrary, this shows that c T x() p + : c T x() = lim (c +^y(k) k! x() lim sup(kck ; c T^s (k) ): k! p lim sup(kck ; c T^s (k) ): () k! 8

30 However, from () we have c T (^x (k) +^s (k) ; c)+^s (k) =(+ T c)^y (k) so that using (), p lim sup(kck ; c T^s (k) ) = lim sup(c T^x (k) +^s (k) ): k! k! By the primal-dual symmetry, d lim sup(kck ; c T^x (k) ) = lim sup(c T^s (k) +^s (k) ) k! k! which proves (). If strong duality holds, then() and the above relation imply lim inf ct^s (k) kck ; p = d lim sup(c T^s (k) +^s (k) ) k! k! which shows that lim k! c T^s (k) = d and = lim k! ^s (k) =. We thus obtain (5). It remains to show () and (). Suppose = lim k! ^s (k) =. Because we obtain for this case that x (k) = which together with () implies that x (k) s(k) (x (k) ; y (k) )(s(k) ; y (k) ) lim k! x(k) = lim k! ^s (k) lim k! (k) =: The relation () now follows from (). Applying Lemma we obtain (). ^s (k) = (8). Examples in semidenite programming Many recent papers discuss primal-dual interior point algorithms for semidenite programming (SDP), see [8, 9,,, ], among others. All these algorithms generate a sequence of weakly centered iterates, so that the results of Section. are applicable. We will illustrate the theory of weakly centered sequences for (SD) with some semidenite programming problems. We choose := vec(i U) where I is the n n identity matrix. In the following examples, we will use the matrix representation rather than the coordinate vectors the examples specify the matrices C A () A () ::: A (m) of the standard (SDPP) form. Moreover, the weakly centered sequence will be parameterized by a continuous parameter > such that lim # y () =: 9

31 We will only discuss those dicult cases where lim # (x ()+s ()) =. First, we consider a weakly infeasible problem. Example (Weakly infeasible) Let m = n =and " # " # C = A () = A () = The primal is weakly infeasible, p = inf ( x + x X = " " # ) = x # : and the dual is strongly feasible and unbounded, d =inf ( s + s S = " s s # ) = ;: We construct a weakly centered sequence for < = as follows: " # X() = I + S() = I + " ; # y () = x () = + s () = + z () =; ( + + ): We see that = lim! ^s () =, which indeed implies weak infeasibility, see Theorem. We have applied the predictor-corrector algorithm for semidenite programming with Nesterov- Todd directions [, ] to the self-dual embeddings of the SDP examples in this section. The plots below show the numerical results of this algorithm. The solid lines represent the primal values c T^x (k) and c T^x (k) +^s (k), whereas the dual values ct^s (k) and c T^s (k) +^s (k) are represented by dashed lines. Example.5 Example 5 5 objective values 5 5 objective values iterations iterations

32 The next example, which is from Vandenberghe and Boyd [9], gives a feasible problem, where strong duality fails to hold. Example 5 (Weakly dual) Let m = n =and C = 5 A () = so that the primal is solvable and weakly feasible, p =inf 8 >< >: x X = and the dual is solvable and weakly feasible, d = inf 8 >< >: s S = 5 A () = x ( ; x )= x x x s ; s s 9 >= 5 > = 9 >= 5 > = Remark that p + d ;kck F => so that strong duality fails. A weakly centered sequence for < = is given by X() = I + = 5 S() = I + ; y () = x () = + s () = + z () =; ( + ) so that =lim! ^s () =. 5 5 The third case is a problem where strong duality holds, but there exists no complementary solution pair (see Vandenberghe and Boyd [9]). Example (Strongly dual) Let m = n =and " # " C = A () = # A () = " # so that the primal is solvable and weakly feasible, p = inf ( x + x X = " x # ) =

33 and the dual is strongly feasible but not solvable, d = inf ( s + s S = " # ) s =: s Notice that p + d = kck F, but the dual has no optimal solution. A weakly centered sequence for < = is " # ; X() = I + S() = I + " # y () = x () = + s () = + z () =; ( + + ): Hence, = lim! s () =as we already knew beforehand from Theorem. 8 Example Example.5 objective values 5 objective values iterations iterations So far, we have seen a weakly infeasible problem with =, a feasible problem with only weak duality and ( ) and a strongly dual problem with =. The reader may wonder whether the value of the indicator completely characterizes the three cases that we consider. Unfortunately, this is not the case, as the next example shows. Example (Weakly infeasible) Consider " # C = The primal is weakly infeasible, p =inf ( A () = x X = " # A () = " " # ) = x # : and the dual is solvable and weakly feasible, d =inf ( s S = " s s # ) =:

34 We construct a weakly centered sequence for < = as " # X() = I + S() = I + " ; # y () = x () = + s () = + z () =; ( + ): We see that = lim! ^s () =. The reader may still wonder whether we can distinguish weak infeasibility from strong duality. It appears somewhat dicult indeed, to construct an example of an infeasible problem with =, but it does exist. Example 8 (Weakly infeasible) Let m = n =and C = 5 A () = 5 A () = 5 : The primal is solvable and weakly feasible, p =inf 8 >< >: x X = x x x ;x = x 9 >= 5 > = and the dual is weakly infeasible, d =inf 8 >< >: s S = s s s 9 >= 5 > = : We construct a weakly centered sequence for < = by X() = I + ; = ; = 5 S() = I + 5 y () = x () = + s () = + z () =; ( + + ) so that =lim! ^s () =.

35 Example 8 Example objective values objective values iterations iterations After Example 8 there is little hope that feasibility with lack of strong duality would imply >. Indeed, we can construct a feasible problem with only weak duality but =. Example 9 (Weakly dual) Let m = n =and C = 5 A() = A () = ; 5 A() = 5 : 5 It can be veried that p + d ;kck F = +; >, but one can construct a weakly centered sequence, similar to the one in Example 8, for which =lim! ^s () =. Our results on the indicator are summarized in the following table. ( ) weakly infeasible Example 8 Example Example weakly dual Example 9 Example 5 impossible strongly dual Example impossible impossible The possible combinations in the table are illustrated by Examples -9. The impossibility of the remaining combinations follows from Theorem.

36 5 Concluding Remarks Even for linear programming, solving a single self-dual embedding may not be enough to solve the original problem (P), see Ye, Todd and Mizuno []. In particular, if it is concluded that the primal-dual pair (c A K) is infeasible, it is in general not yet known whether (P) is infeasible or (D) is infeasible or both. However, we can determine whether or not (P) is feasible by solving a self-dual program for a feasibility formulation. To be more specic, let us dene a vector f D < n+ by! f D = ; and consider the following problem: o (P D ) min nfd Tx j x B\K n (D D ) min s (f D + B? ) \K o : We see that problem (P D ) is always feasible and we know from Lemma that (D D ) is strongly infeasible if and only if the original problem (P) is feasible. Suppose we embed the primal-dual pair (P D ), (D D ) in a self-dual problem (SD). Given a weakly centered sequence w (k) k = :::for this self-dual embedding, the feasibility andweak/strong infeasibility of (P) is completely characterized by the limiting behavior of x (k) and s (k), due to Theorem and Theorem 5. Moreover, we know from Theorem 5 that the strong infeasibility of(d D ) will be demonstrated by a solution Partitioning x = exists. x B\K fd T x <:!, we have = F P. Hence, we obtain a feasible solution to (P) if it References [] Alizadeh, F., \Interior point methods in semidenite programming with applications to combinatorial optimization," SIAM Journal on Optimization 5 (995) -5. [] Ben-Israel, A., \Linear equations and inequalities on nite dimensional, real or complex, vector spaces: a unied theory," Journal of Mathematical Analysis and Applications (99) [] Dun, R.J., \Innite programs," in: H.W. Kuhn and A.W. Tucker, eds. Linear Inequalities and Related Systems (Princeton University Press, Princeton, NJ, 95), 5-. [] Goldman, A.J. and Tucker, A.W., \Polyhedral convex cones," in: H.W. Kuhn and A.W. Tucker, eds. Linear Inequalities and Related Systems (Princeton University Press, Princeton, NJ, 95), 9-. 5

37 [5] Guler, O. and Ye, Y., \Convergence behavior of interior-point algorithms," Mathematical Programming (99) 5-8. [] Hurwicz, L., \Programming in linear spaces," in: K.J. Arrow, L. Hurwicz and H. Uzawa eds. Studies in Linear and Nonlinear Programming, II (Stanford University Press, Stanford, CA, 958), -. [] Jansen, B., Roos, C. and Terlaky, T., \The theory of linear programming: skew symmetric self-dual problems and the central path," Optimization 9 (5-), 99. [8] Kojima, M., Shindoh, S. and Hara, S., \Interior-point methods for the monotone linear complementarity problem in symmetric matrices," Research Reports on Information Sciences, B-8, Dept. of Information Sciences, Tokyo Institute of Technology, -- Oh-Okayama, Meguro-ku, Tokyo 5, Japan, 99. [9] Lin, C-J. and Saigal, R., \An infeasible predictor corrector method for semi-denite linear programming," Technical Report, Department of Industrial and Operations Engineering, The University of Michigan, Ann Arbor, MI, U.S.A., 995. [] Monteiro, R.D.C., \Primal-dual path following algorithms for semidenite programming," Technical Report, School of Industrial and Systems Engineering, Georgia Tech, Atlanta, Georgia, U.S.A., 995. [] Nesterov, Y. and Nemirovsky, A., \Interior point polynomial methods in convex programming," Studies in Applied Mathematics (SIAM, Philadelphia, PA, 99). [] Nesterov, Y. and Todd, M.J., \Primal-dual interior-point methods for self-scaled cones," Technical Report 5, School of Operations Research and Industrial Engineering, Cornell University, Ithaca, New York, 995. [] Potra, F.A., Sheng, R., \Homogeneous interior-point algorithms for semidenite programming," Technical Report, University of Iowa, Iowa, USA, 995. [] Ramana, M., Tuncel, L., and Wolkowicz, H., \Strong duality for semidenite programming," Technical Report CORR 85-, University of Waterloo, Waterloo, Canada, 995. [5] Rockafellar, R.T., Convex Analysis, Princeton University Press, Princeton, NJ, 9. [] Sturm, J.F. and Zhang, S., \Symmetric primal-dual path following algorithms for semidenite programming," Report 955/A, Econometric Institute, Erasmus University Rotterdam, Rotterdam, The Netherlands, 995. [] Todd, M.J. and Ye, Y., \Approximate Farkas lemmas and stopping rules for iterative infeasiblepoint algorithms for linear programming," Technical Report, School of Operations Research and Industrial Engineering, Cornell University, Ithaca, NY, USA, 99.