Chapter 6 The Karush-Kuhn-Tucker conditions 6.1 Introduction In this chapter we derive the first order necessary condition known as Karush-Kuhn-Tucker (KKT) conditions. To this aim we introduce the alternative theorems. 6.2 Alternative theorems and the Farkas Lemma We present in this section alternative theorems which allows to formulate the not existence of solution as the existence of an alternative system. Given two linear systems (I) e (II), an alternative theorem consists in proving that: System (I) has a solution if and only if system (II) does not have a solution. Among alternative theorems, one of the mst important that weare going to use to prove optimality conditions, is the Farkas Lemma. Theorem 6.3 (Farkas Lemma) Let B a p n matrix and g R n. System Bd 0 g T d < 0 (I) does not have a solution d R n if and only if system B T u = g u 0 (II) has a solution. u R p. 43
44 CHAPTER 6. THE KARUSH-KUHN-TUCKER CONDITIONS 6.4 The Karush-Kuhn-Tucker conditions Consider the problem min f (x) In section 5 we derive the first order optimality conditions (P POL) If x is a local minimizer of problem (P-POL) then there exists NO solution d R n of the system A I(x )d 0, f (x ) T d < 0. (6.1) where A I(x ) is the I(x ) n submatrix of A defined as A I(x ) = (a T i ) i I(x ) and I(x ) = {i : a T i x = b i }. Identifying the system (6.1) with system (I) of Farkas Lemma, we get There exists NO solution d R n to the linear system (6.1) if and only if there exists a solution of the system A T I(x ) u = f (x ), (6.2) u 0. where u R I(x ). Using this result we can state the optimality conditions for problem (P-POL) known as Karush-Kuhn-Tucker conditions (KKT). Theorem 6.5 (Karush-Kuhn-Tucker conditions) If x is a local minimizer of problem (P-POL). Then a multiplier λ R m exists that such that (i) f (x ) A T λ = 0, (iii) λ i (b i a T i x ) = 0 per i = 1,...,m. (iv) Ax b, Proof. Since x is a local minimizer, by definition it is feasible hence (iv) holds. Further there exists no solution to the system (6.1). Identifying the system (6.1) with system (I) of Farkas Lemma, we get that there exists NO solution che esiste to system(6.2). Let u i 0, i I(x ) a solution of the preceding system and define λ R m as: λ i = { u i peri I(x ) 0 peri / I(x ). (6.3)
6.4. THE KARUSH-KUHN-TUCKER CONDITIONS 45 By definition λ 0 and (ii) holds. Furthe (iii) derives immediately by the defintion (6.3) of λ. Indeed we have { λi (b i a T i x u i ) = (b i a T i x ) = 0 peri I(x ) 0(b i a T i x ) = 0 peri / I(x ). Finally,(6.2) can be written as f (x ) = and (i) holds. f (x ) = u i a i so that i I(x ) u i a i + 0 a i = A T λ i I(x ) i I(x ) The vector λ R m is called generalized Lagrange multiplier or KKT multiplier associated to constraints. Condition (iii) is called complementarity condition and states that at optimality the product λi (b i a i T x ) vanishes so that if the constraints is inactive the corresponding multiplier is zero. Since λ 0 and Ax b 0 condition (iii)can be equivalently written as λ T (b Ax ) = m i=1 λ i (b i a T i x ) = 0. We can express conditions above using the Lagrangian function. The Lagrangian function associated to problem (P-POL) is L(x,λ) = f (x) + λ T (b Ax) Condition (ii) states that in (x,λ ) the gradient x L(x,λ) = f (x) A T λ vanishes, so that the pair (x,λ ) ia a stationary point of the Lagrangian function. (Karush-Kuhn-Tucker conditions) Let x is a local minimizer of problem (P-POL). Then a multiplier λ R m exists that such that (i) x L(x,λ ) = 0 (stationary), (iii) λ T (b Ax ) = 0 (complementarity), (iv) Ax b, (feasibility).
46 CHAPTER 6. THE KARUSH-KUHN-TUCKER CONDITIONS Candidates to be minimizer of problem (P-POL) can be found by solving the KKT conditions. We consider the following definition. A point x is called KKT point or stationary point of problem (P-POL) if there exists λ R m such that. (i) A x b, (ii) L( x, λ) = 0, (iii) λ 0, (iv) λ T (b A x) = 0. under onvexity assumtpion on f we get Theorem 6.6 (Necessary and sufficient conditions for global optimzality) Let f be a continuously differentiable convex function in R n. A point x is a global minimizer of problem (P-POL) if and only if a multiplier λ R m exists that such that (i) Ax b, (ii) f (x ) A T λ = 0, (iv) λ T (b Ax ) = 0. If f is strictly convex and the conditions (i)-(iv) hold, then x is the unique global minimizer of f on S. We derive now the KKT conditions for a more general problem. min f (x) Dx = h where D is a p n matrix, A is a q n matrix, h R p, b R q. (P GEN) The Lagrangian function associated to (P-GEN) is with λ R q, µ R p. L(x,λ,µ) = f (x) + λ T (b Ax) + µ T (Dx h),
6.4. THE KARUSH-KUHN-TUCKER CONDITIONS 47 The KKT conditions for problem (P-GEN) are satisfied in (x,λ, µ ) R n R q R p when it holds: (i) Ax b, Dx = h (feasibility), (ii) x L(x,λ, µ ) = f (x ) A T λ + D T µ = 0 (stationarity), (iv) λ T (b Ax ) = 0 (complementarity). Theorem 6.7 (Karush-Kuhn-Tucker Conditions for (P-GEN)) Let x is a local minimizer of problem (P-GEN). Then multipliers λ R q, µ R p exist such that: (i) x L(x,λ, µ ) = 0, (iii) λ i (b i a T i x ) = 0 per i = 1,...,q. Consider the special case of Quadratic Programming (QP) problems min 1 2 xt Qx + c T x, (PQ) where Q is a n n symetric and positive semidefinite matrix, c R n. Theorem 6.8 (Optimality conditions for QP) Let Q be a n n symetric and positive semidefinite matrix. A point x is a global minimizer of problem (PQ) if and only if a multiplier λ R m exists that such that: (i) Ax b, (ii) Qx + c A T λ = 0, (iv) λ T (b Ax ) = 0. Further if Q is positive definite and 8i)-(iv) hold, then x is the unique global solution of problem (PQ).