Faculty of Mathematical Sciences University of Twente University for Technical and Social Sciences P.O. Box 217 75 AE Enschede The Netherlands Phone +31-53-48934 Fax +31-53-4893114 Email memo@math.utwente.nl Memorandum No. 1493 An easy way to obtain strong duality results in linear, linear semidefinite and linear semi-infinite programming P. Pop and G.J. Still July 1999 ISSN 169-269
AN EASY WAY TO OBTAIN STRONG DUALITY RESULTS IN LINEAR, LINEAR SEMIDEFINITE AND LINEAR SEMI-INFINITE PROGRAMMING PETRICA POP AND GEORG STILL ABSTRACT In linear programming it is known that an appropriate nonhomogenious Farkas Lemma leads to a short proof of the strong duality results for a pair of primal and dual programs. By using a corresponding generalized Farkas lemma we give a similar proof of the strong duality results for semidefinite programs under constraint qualifications. The proof includes optimality conditions. The same approach leads to corresponding results for linear semi-infinite programs. For completeness, the proofs for linear programs and the proofs of all auxiliary lemmata for the semidefinite case are included. KEYWORDS linear programming, semidefinite programming, semi-infinite programming, duality MATHEMATICAL SUBJECT CLASSIFICATION 1991 9C5, 9C25, 9C34
1 1. STRONG DUALITY RESULTS IN LINEAR PROGRAMMING Consider the pair of primal and dual linear programs, P max c T x s.t. Ax b x R n D min y R bt y s.t. A T y = c ; y ; m where A is an.m n/-matrix.m n/ and c R n ; b R m.letv P denote the maximum value of the primal program P and v D the minimum value of the dual problem D. The feasible sets of P and D are abbreviated by F P and F D. Most commonly a homogeneous Farkas Lemma is used to prove optimality conditions for P and D. We will use the following non-homogeneous version to prove in one step strong duality and optimality conditions. Lemma 1. Let be given an.m n/-matrix B, an.k n/-matrix C and b R m ; c R k. Then precisely one of the following alternatives is valid. (a) There is a solution x R n of Bx b, Cx = c. (b) There exist vectors ¼ R m ;¼ ; ½ R k such that ( B ) ( T b ¼ + C T) ( T c ½ = 1). T This result is an easy corollary of a common version of Farkas Lemma (see Section 4 for a proof). We begin with the weak duality result. Lemma 2. (Weak Duality) Let be given x F P ; y F D. Then, (1) b T y c T x = y T.b Ax/ If in (1) we have b T y c T x =, thenx; y are solutions of P, D with v P = v D. Proof. For feasible x; y we find b T y c T x = b T y y T Ax = y T.b Ax/ or equivalently b T y c T x. The equal sign implies that y is minimal for D and x maximal for P with same value b T y = v D = c T x = v P. We now prove the strong duality result, the existence of solutions and optimality conditions. Theorem 1. (Strong Duality) The following holds. (a) Suppose F P. ThenF D = if and only if v P =. Suppose F D. ThenF P = if and only if v D =. (b) Suppose F P ; F D. Then P and D have solutions x and y satisfying c T x = b T y, i.e. v P = v D. Moreover, the following optimality conditions hold x F P solves P there exists y F D such that y T.b Ax/ = y F D solves D there exists x F P such that y T.b Ax/ = Proof. (a) We prove the first statement. Suppose F D =. Then there is no solution of A T y = c; y. By Lemma 1 there exist vectors ¼ ; ½such that ( ) ( ) ( ) A I c T ½ + ¼ = or A½ = ¼ and c T ½ = 1 1 Then, with x F P the vector x.t/ = x t½ is feasible for all t > with c T x.t/ = c T x + t for t. This implies v P =. On the other hand, if F D then
2 with a vector y F D by Lemma 2 it follows v P b T y. The proof of the other case is similar. (b) By Lemma 2 we have shown that x; y are solutions of P, D with b T y c T x = if we show that x; y satisfy the relations Ax b Iy (2) A T y = c c T x + b T y Suppose that this system does not have a solution x; y. ThenbyLemma1thereexist Þ and vectors ¼ x ;¼ y ;½such that AT b T ¼ x + I ¼ y + A c T ½ + c b Þ = 1 or (3) A T ¼ x = Þc; A. ½/ = Þb ¼ y ; b T ¼ x c T. ½/ = 1 We distinguish between two cases. Case Þ = Then in view of (3) with x F P ; y F D the vectors x.t/ = x t½; y.t/ = y + t¼ x are feasible with b T y.t/ c T x.t/ = b T y c T x t for t in contradiction to our assumption. Case Þ> Then by dividing relations (3) by Þ be obtain a solution of the system (2), a contradiction. This shows the first part of (b). The optimality conditions are obtained as follows. Suppose x is a solution of P. As shown, there exist a solution y of D with = b T y c T x = y T.b Ax/. On the other hand if for x F P the vector y F D satisfies y T.b Ax/ = thenbylemma2,x is a solution of P. The optimality conditions for y F D are obtained similarly.
3 2. STRONG DUALITY RESULTS IN SEMIDEFINITE PROGRAMMING In this section we will give a similar proof of the strong duality result and optimality conditions in semidefinite programming. Consider the pair of primal and dual linear semidefinite programs, P max x R c x s.t. A.x/ = B x i A i n D min B Y s.t. A i Y = c i ; i = 1;;n ; Y ; Y where B; A i are symmetric.m m/-matrices and c R n. We write Y for a positive semidefinite, and Y for a positive definite matrix Y. By B Y we denote the inner product B Y = ij b ijy ij (coinciding with the trace of BY). For convenience of notation we also have replaced c T x by c x. Let again v P ;v D be the maximum, minimum values of P, D, respectively and F P, F D the feasible sets. Points x F P ; Y F D are called strictly feasible if A.x/; Y are positive definite. We give a generalized nonhomogeneous Farkas Lemma (see Section 4 for a proof). For a given set S let cone.s/ denote the convex cone, lin.s/ the linear hull and clos.s/ the closure of S. Lemma 3. Let be given S ={.b k ;þ k / b k R q ; þ k R; k K}, K a possibly infinite set, and S 1 ={.c j ; j /; c j R q ; j R; j J}, J a finite set. Then precisely one of the following alternatives is valid with S = cone.s / + lin.s 1 /. (a) There is a solution ¾ of bk T¾ þ k; k K, c T j ¾ = j; j J. (b) We have ( 1) clos.s/. We need a result for semidefinite matrices. A proof is given in Section 4. Lemma 4. Let be given A; B. ThenA B and A B = if and only if A B =. If moreover A then A B = B =. The possibility to treat the semidefinite problem as a direct generalization of the linear case depends on the following observation. Let V m denote the compact set V m = {V = vv T v R m ; v = 1}. Then, in view of A vv T = v T Av it follows (4) A A V forall V V m We now proceed as in the case of linear programs. Lemma 5. (Weak Duality) Let be given x F P ; Y F D. Then, (5) B Y c x = Y A.x/ If in (1) we have B Y c x =, thenx; Y are solutions of P, D with v P = v D. Proof. For feasible x; Y we find B Y c x = B Y n x i A i Y = Y A.x/ or B Y c x. The equal sign implies that Y is minimal for D and x maximal for P with the same value B Y = v D = c x = v P. We give the prove of the strong duality results together with optimality conditions under usual constraint qualifications.
4 Theorem 2. (Strong Duality) The following holds. (a) Suppose P is strictly feasible. Then F D = if and only if v P =. Suppose D is strictly feasible. Then F P = if and only if v D =. (b) Suppose P and D are strictly feasible. Then, P and D have solutions x and Y satisfying c x = B Y. Moreover, the following optimality conditions hold x F P solves P there exists Y F D such that Y A.x/ = Y F D solves D there exists x F P such that Y A.x/ = Proof. In P we can assume that A i ; i = 1;;n are linearly independent. (a) Assuming F D, then with Y F D we obtain from Lemma 5, B Y v P,i.e. v P <. Suppose now that F D =, i.e. there is no solution Y of A i Y = c i ; i = 1;;n; Y V ; for all V V m By Lemma 3, ( 1) clos ( cone. V m ; // + lin {.A i ; c i /; i = 1;;n} ),i.e.there exist V ¹ k V m ;¼ ¹ k ; k K ¹, ½ ¹ i R such that k K ¹ ¼ ¹ k ( V ¹ k ) + ½ ¹ i ( ) ( Ai c i 1 Putting S ¹ = k K ¹ ¼ ¹ k V ¹ k and x¹ = ½ ¹ this is equivalent with ) for ¹ (6) x ¹ i A i + E ¹ = S ¹ ; c x ¹ = 1 + ž ¹ with ž¹ =.E ¹ ;ž ¹ / for ¹. In the expression.e ¹ ;ž ¹ / the element.e¹ ;ž ¹ / is to be seen as a vector in R m2 +1. With a strictly feasible x we have A.x/ and we can choose M > large enough such that Mž ¹ A.x/ E ¹ ; ¹ N. This implies Mž ¹ B N ( Mž ¹ x i + x ¹ ) i Ai ; c ( Mž ¹ x + x ¹) = 1 ž ¹ + Mž¹ c x Dividing by Mž ¹ and using ž ¹ we obtain B N ( xi + x¹ ) i Ai Mž ¹ ; c ( x + x¹ ) 1 Mž ¹ Mž ¹ 1 + c x M The other case can be proven similarly. (b) In view of Lemma 5 and using (4), to prove the first part of the statement, it is sufficient to show that there exist a solution x; Y of n x i A i V B V; V V m (7) Y V ; V V m Y A i = c i ; i = 1;;n n x ic i + B Y
Suppose that this system is not solvable. By Lemma 3 there exist Þ ¹, V l ¹ ; V k ¹ V m ;¼ ¹ k ;¼¹ l ; k K ¹ ; l L ¹, ½ ¹ i R such that for ¹ l L ¹ ¼ ¹ l A 1 V l ¹ A n V l ¹ B V l ¹ + ¼ ¹ k k K ¹ V ¹ k + ½ ¹ i A i c i + Þ¹ c 1 c n B Putting Y ¹ = l L ¹ ¼ ¹ l V l ¹, S¹ = k K ¹ ¼ ¹ k V k ¹, x¹ = ½ ¹ this is equivalent with A i Y ¹ Þ ¹ c i + ž ¹ i = ; i = 1;n; Þ ¹ B n (8) x¹ i A i + E ¹ = S ¹ ; B Y ¹ c x ¹ = 1 + ž ¹ ; where ž ¹ =.ž 1 ¹ ;;ž¹ n ; E¹ ;ž ¹ / for¹. Define the numbers ¹ = max{.y ¹ ; S ¹ / ; x ¹ ;Þ ¹ }. We distinguish between two cases. Case ¹ M; ¹ N Then, there exist convergent subsequences Y ¹ Y; S ¹ S; x ¹ x; Þ ¹ Þ and from (8) we find (9) A i Y = Þc i ; i = 1;;n; ÞB x i A i = S ; B Y c x = 1 We distinguish between two sub-cases. If Þ>then by dividing relations (9) by Þ be obtain a solution of the system (7), a contradiction. If Þ = then in view of (9) with x F P ; Y F D the vectors x.t/ = x + tx; Y.t/ = Y + ty are feasible with B Y.t/ c x.t/ = B Y c x t for t in contradiction to our assumption. Case ¹ ;¹ (for some subsequence) By dividing (8) by ¹ and taking converging subsequences we obtain with some Ŷ ; Ŝ ; ˆÞ ; ˆx; (1) A 1 Ŷ A n Ŷ Ŝ B Ŷ ˆx i A i c i +ˆÞ c 1 c n B = and max{.ŷ; Ŝ/ ; ˆx ; ˆÞ}=1. It now follows ˆÞ>. In fact for ˆÞ = by multiplying (1) with. x; Y; 1/, x; Y strict feasible we find using A i Y + c i = 1 5 (11) A.x/ Ŷ + Ŝ Y = with A.x/; Y In view of Lemma 4 it follows Ŷ = Ŝ = and by the linear independency of A i in (1) also ˆx =, a contradiction. The relation ˆÞ > implies that (8) is valid with Þ ¹ (some subsequence). Now we can choose Y ¹ ž such that with some M > (12) A i Y ¹ ž = ž ¹ i ; i = 1;;n and Y ¹ ž M ž ¹ for all ¹ N Thus, with strictly feasible x; Y there exists M > such that Y ž ¹ + Mž ¹ Y ; Mž ¹( ) (13) B x i A i E ¹ ; ¹ N
6 For Y ¹ + Y ¹ ž + Mž ¹ Y, x ¹ + Mž ¹ x we find using (8), (12), (13) and ž ¹ ;ž ¹ (14) A i.y ¹ + Y ¹ ž + Mž ¹ Y /.Þ ¹ + Mž ¹ /c i = ; i = 1;n;.Þ ¹ + Mž ¹ /B n ( x ¹ i + Mž ¹ x i ) Ai ; B.Y ¹ + Y ¹ ž + Mž ¹ Y / c.x ¹ + Mž ¹ x/ = 1 + ž ¹ + O.ž¹ / 1 2 for any fixed ¹ large enough. Since Þ ¹ we obtain Þ ¹ + Mž ¹ > forlarge¹. By dividing (14) by Þ ¹ + Mž ¹ > we have a solution of (7) in contradiction to our assumption. This shows the first part of (b). The optimality conditions are obtained as follows. Suppose x is a solution of P. As shown, there exist a solution Y of D with = B Y c x = Y A.x/. Lemma 4 implies Y A.x/ =. On the other hand if for x F P the vector Y F D satisfies Y A.x/ = and thus Y A.x/ = then, by Lemma 5, x is a solution of P. The optimality conditions for Y F D are obtained similarly. The proof of the semidefinite case is longer than the proof of the statements of Theorem 1 for linear programs. The reason is that the set S = cone.s 1 / + cone.s2 / + lin.s 1/ + cone {s } with S 1 ={.A 1 V; ; A n V; ; B V / V V m }, S 2 ={.; ; ; V; / V V m }, S 1 ={.; ; ; A i ; c i / i = 1;;n}, s =. c 1 ; ; c n ; B; / need not to be closed. This, although the strict feasibility assumptions in Theorem 2(b) imply that the set cone.s 1/ + cone.s2 / + lin.s 1/ is closed. Hence, in the proof of Theorem 2(b), the case ¹ cannot be excluded. This complication is not present in linear programming since cones generated by finitely many vectors are always closed. For further details on semidefinite programming, such as duality gaps, we refer to [3]. Commonly the duality results and optimality conditions for semidefinite problems are obtained by transforming the semidefinite programs into a more abstract coneconstrained form. Our approach avoids such a transformation by transforming the programs into a special case of a semi-infinite problem (see also Section 3). 3. STRONG DUALITY RESULTS IN SEMI-INFINITE PROGRAMMING In this section we briefly outline how the same approach can be applied to linear semiinfinite programs. A common linear semi-infinite problem is of the form, P max x R c x s.t. b.t/ x i a i.t/ ; for all t T ; n where c R n is a given vector and b.t/; a i.t/ C.T; R/, T a compact subset of a topological space. Again we have replaced c T x by c x. C.T; R/ denotes the space of real-valued functions f, continuous on T, with norm f = max{ f.t/ t T}. Note, that in view of (4) the semidefinite program in the previous section can be written as a semi-infinite program by defining b.t/ = t T Bt; a i.t/ = t T A i t; i = 1;;n; t T = {t R n t = 1} For f C.T; R/ we write f (f > ) if f.t/ (f.t/>) for all t T. The dual C.T; R/ of the space C.T; R/ is the space of all real-valued Borel measures y on T
(see [1]). We define f y = T f.t/dy.t/ ; f C.T; R/; y C.T; R/ The measure y is said to be non-negative (notation Y ) if f y forall f C.T; R/; f and positive (y > ) if f y > forall f C.T; R/; f ; f. The dual of P then reads D min b y s.t. a i y = c i ; i = 1;;n ; y y C.T;R/ As before let v P ;v D denote the values of P; D and F P ; F D the feasible sets. Elements x F P and y F D are said to be strictly feasible if a.x/ = b x i a i > and y > We introduce the set K + 1 ={f C.T; R/ f ; f 1}. With these settings we can proceed as in the semidefinite case. The full system for the solutions x of P, y of D corresponding to (7), for example, becomes in the semi-infinite case n x ia i.t/ b.t/; t T 7 q y ; q K + 1 a i y = c i ; i = 1;;n n x ic i + b y By considering some appropriate modifications in the proofs of Section 2 we can prove weak and strong duality results for semi-infinite programs along the same lines as in the semidefinite case. For shortness we only give the strong duality result. Theorem 3. (Strong Duality) The following holds. (a) Suppose P is strictly feasible. Then F D = if and only if v P =. Suppose D is strictly feasible. Then F P = if and only if v D =. (b) Suppose P and D are strictly feasible. Then, P and D have solutions x and y satisfying c x = b y. Moreover, the following optimality conditions hold x F P solves P there exists y F D such that a.x/ y y F D solves D there exists x F P such that a.x/ y = For further details on semi-infinite programming we refer to the paper [2]. 4. PROOFS OF THE AUXILIARY LEMMATA For completeness, in this section, the proofs of all auxiliary lemmata of Section 1 and Section 2 will be given. Proof of Lemma 1 We prove the statement with the help of the following common homogeneous version of Farkas Lemma Given a.m n/-matrix A and b R m, precisely one of the alternatives.a /,.b / is valid,.a / Ax ; b T x > is solvable.b / A T y = b; y is solvable
8 By introducing in the situation of Lemma 1 an auxiliary variable x n+1 the statement.a/ is equivalent with There exists a solution.x; x n+1 / of x n+1 > Bx x n+1 b Cx x n+1 c Cx+ x n+1 c This system.a / has the alternative.b / =.b/ There exist vectors ¼; ¼ + ;¼ such that with b =.; 1/ and ½ = ¼ + ¼ we have ( B T ) ( b ¼ + C T ) ( T c ½ = 1). T Proof of Lemma 4 A B = directly implies A B = tr.a B/ =. To prove the converse, consider the transformation of A; B to diagonal form, A = Þ i q i qi T ; B = þ j v j v T j ; where q i ;v j are the orthonormal eigenvectors and Þ i ;þ j the corresponding eigenvalues of A; B. Then with A B = tr.a B/ we find using Þ i þ j A B = Þ i þ j tr.q i qi T v j v T j / = Þ i þ j.v T j q i qi T v j / = Þ i þ j.qi T v j / 2 i; j=1 i; j=1 j=1 i; j=1 Moreover, A B = implies Þ i þ j.qi Tv j/ 2 = orþ i þ j.qi Tv j/ = foralli; j and then A B = Þ i þ j q i qi T v jv T j = Þ i þ j.qi T v j/ q i v T j = i; j=1 When A then in particular, the matrix A is regular and A B = implies B = A 1 =. Proof of Lemma 3 We prove the statement by using the following standard separation theorem Let S R q be a convex closed set and y R q. Then precisely one of the alternatives.a /,.b / holds,.a / There exist ¾ R q ;Þ Rsuch that ¾ T s Þ; s S; ¾ T y >Þ.b / y S It is easy to show that if.b/ is valid then.a/ cannot hold. Suppose now that.b/ is not true. By putting y =.; 1/, S = clos ( cone.s / + lin.s 1 / ) the condition.b / is not fulfilled. Thus by.a / there exist a vector.¾; ¾ q / R q ;Þ Rsuch that i; j=1 ¾ T b + ¾ q þ Þ for all.b;þ/ cone.s / (15) ¾ T c + ¾ q Þ for all.c;/ lin.s 1 / ¾ q > Þ With.c;/ lin.s 1 /;.b;þ/ cone.s / these relations also holds for ±t.c;/, t.b;þ/; t. This implies ¾ T c + ¾ q =, ¾ T b + ¾ q þ and we can choose Þ =. By dividing (15) by ¾ q we obtain with ¾ = ¾=¾ q the relation ¾ T b þ; ¾ T c = for all.b;þ/ cone.s /;.c;/ lin.s 1 /,i.e.(a). REFERENCES [1] Rudin W., Functional analysis, McGraw-Hill, (1973). [2] Hettich R., Kortanek K., Semi-infinite programming Theory, methods, and applications, SIAM Review, Vol.35, No.3, 38-429, (1993). [3] Vandenberghe L. and Boyd S., Semidefinite programming,, SIAM Review, 38, 49-95, (1996).