LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality theory to I Characterize unbounded LPs I Resolution theorem and its converse: two ways to look at polyhedra IOE 610: LP II, Fall 2013 Duality Page 63 Duality theory for Linear Programming I Special case of Lagrangian Duality Theory for (general) optimization I Idea: I Find a relaxation of your optimization problem, to obtain a dual (i.e., optimistic) bound on its optimal bound I Try to find a tight dual bound through the above methodology IOE 610: LP II, Fall 2013 Duality Page 64
For linear programming (in standard form): (P) minimize s.t. c T x Ax = b x 0, where A 2< m n. Assume (P) has an optimal solution x?. I L(x, p) =c T x + p T (b Ax), where p 2< m I g(p) =min x 0 c T x + p T (b Ax) I Note: for any p, g(p) apple c T x? I Thus, to obtain a tight lower bound on c T x?,wemaywantto maximize p g(p), which is the dual problem I Exploring g(p) further, we conclude that the dual problem can be formulated as (D) maximize s.t. p T b p T A apple c T IOE 610: LP II, Fall 2013 Duality Page 65 For linear programming (in inequality form) (P) minimize c T x s.t. Ax b where A 2< m n. Assume (P) has an optimal solution x?. I L(x, p) =c T x + p T (b Ax), where p 2< m I g(p) =min x c T x + p T (b Ax) I For any p 0, g(p) apple c T x? I Thus, to obtain a tight lower bound on c T x?,wemaywantto maximize p 0 g(p), which is the dual problem I Exploring g(p) further, we conclude that the dual problem can be formulated as (D) maximize s.t. p T b p T A = c T p 0 IOE 610: LP II, Fall 2013 Duality Page 66
The dual problem for LP in general form I Introduce a variable p i in the dual problem for every non-sign constraint in the primal I We will have a constraint in the dual for every variable, and vice versa in the primal (P) minimize c T x (D) maximize p T b s.t. a T i x b i, i 2 M 1, s.t. p i 0, i 2 M 1 a T i x apple b i, i 2 M 2, p i apple 0, i 2 M 2 a T i x = b i, i 2 M 3, p i free, i 2 M 3 x j 0, j 2 N 1, p T A j apple c j, j 2 N 1 x j apple 0, j 2 N 2, p T A j c j, j 2 N 2 x j free, j 2 N 3, p T A j = c j, j 2 N 3 IOE 610: LP II, Fall 2013 Duality Page 67 The relations between primal and dual problems: PRIMAL minimize maximize DUAL b i 0 constraints apple b i apple 0 variables = b i free 0 apple c j variables apple 0 c j constraints free = c j Theorem 4.1 ( Dual of the dual is the primal ) If we transform the dual into an equivalent minimization problem and then form its dual, we obtain a problem equivalent to the original problem. IOE 610: LP II, Fall 2013 Duality Page 68
Properties of primal-dual pairs Theorem 4.3 (Weak duality) If x is a feasible solution to the primal problem and p is a feasible solution to the dual problem, then Corollary 4.1 p T b apple c T x. If the cost of the primal (dual) is 1 (+1), then the dual (primal) problem must be infeasible. Corollary 4.2 Let x and p be feasible solutions to the primal and the dual, respectively, and suppose that p T b = c T x.thenx and p are optimal solutions to the primal and the dual, respectively. IOE 610: LP II, Fall 2013 Duality Page 69 Strong duality and Complementary slackness Theorem 4.4 Strong duality Theorem 4.5 Complementary slackness We ll come back to it. IOE 610: LP II, Fall 2013 Duality Page 70
Duals of equivalent LPs are equivalent Atechnicalitybeforeweproceed: Theorem 4.2 Suppose that we have transformed a linear programming problem 1 into another linear programming problem 2, by a sequence of transformations of the following types: I Replace a free variable with the di erence of two nonnegative variables. I Replace an inequality constraint by an equality constraint involving a nonnegative slack variable. I If some row of the matrix A in a feasible standard form problem is a linear combination of the other rows, eliminate the corresponding equality constraint. Then the duals of 1 and 2 are equivalent, i.e., they are either both infeasible, or they have the same optimal cost. IOE 610: LP II, Fall 2013 Duality Page 71 Separating hyperplane theorem Theorem 4.11 (Separating hyperplane theorem) Let S be a nonempty closed convex subset of < n and let x? 62 S. Then 9c 2< n such that c T x? < c T x 8x 2 S. I Definition: AsetS < n is called closed if, for every sequence x 1, x 2,... of elements of S that converges to some x 2< n, x 2 S. I Intersection of closed sets is closed. I Every polyhedron is a closed set Theorem 4.10 (Weierstrass theorem) If f : < n!<is a continuous function, and if S < n is a nonempty, closed, and bounded set, then I There exists some x? 2 S such that f (x? ) apple f (x) 8x 2 S I There exists some y? 2 S such that f (y? ) f (x) 8x 2 S. Note: all three assumptions are important! IOE 610: LP II, Fall 2013 Separating hyperplanes and duality theory Page 72
A theorem of alternative Farkas lemma (also, Theorem 4.6) Let A 2< m n and let b 2< m. Then, exactly one of the following two alternatives holds: (a) 9x 0 such that Ax = b (b) 9p such that p T A 0 T and p T b < 0. Proof I It is easy to show that both alternatives cannot hold. I Suppose the first system does not have a solution. Let S = {Ax : x 0} = {y 2< m : 9x 0 s.t. y = Ax}. I S is nonempty, convex and closed (?), and b 62 S. I Thus, b and S can be separated by a hyperplane... IOE 610: LP II, Fall 2013 Separating hyperplanes and duality theory Page 73 Strong duality theorem Theorem 4.4 (Strong Duality Theorem) If a linear programming problem has an optimal solution, so does its dual, and the respective optimal costs are equal. Proof We will prove the theorem for the primal-dual pair of the form (P) minimize c T x (D) maximize b T p s.t. Ax b s.t. A T p = c, p 0 Proof outline: I Let x? be an optimal solution of (P), and let I be the set of indices of constraints active at x?. I There does not exist a vector d 2< n such that c T d < 0and a T i d 0 8i 2 I I By Farkas lemma, P i2i p ia i = c, p i solution. Let p i =0, i 62 I. I p is feasible for (D), and c T x? = b T p. 0, i 2 I has a IOE 610: LP II, Fall 2013 Separating hyperplanes and duality theory Page 74
Summary of relationship between (P) and (D) There are 4 possible outcomes for a primal-dual pair of LPs (P) and (D): I (P) and (D) are both infeasible I (P) is infeasible and (D) is unbounded I (P) is unbounded and (D) is infeasible I (P) and (D) both have optimal solutions of equal value IOE 610: LP II, Fall 2013 Separating hyperplanes and duality theory Page 75 Complementary slackness Theorem 4.5 (Complementary Slackness) Let x and p be feasible solutions to the primal and the dual problem, respectively. The vectors x and p are optimal solutions for the two respective problems i p i (a T i x b i )=0, 8i (c j p T A j )x j =0, 8j. IOE 610: LP II, Fall 2013 Separating hyperplanes and duality theory Page 76
Geometric interpretation of duality and complementary slackness (P) minimize c T x (D) maximize b T p s.t. a T P m i x b i, i =1,...,m s.t. i=1 a ip i = c p 0 Let x 2< n and let I = i : a T i x = b i I For x to be an optimal solution of (P), it is necessary and su cient that there exists p 2< m such that together they satisfy: (a) a T i x b i for all i (Primal feasiblity) (b) p i =0foralli 62 I (Complementary slackness) (c) P m i=1 a ip i = c (Dual feasiblity) (d) p 0 (Dual feasibility) I What if x is feasible, but not optimal? I What if x is an optimal BFS? A degenerate optimal BFS? IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 77 Strong Duality)Farkas lemma Farkas lemma (also, Theorem 4.6) Let A 2< m n and let b 2< m. Then, exactly one of the following two alternatives holds: (a) 9x 0 such that Ax = b (b) 9p such that A T p 0andb T p < 0 Proof Consider (P) minimize b T p (D) maximize 0 T x s.t. A T p 0 s.t. Ax = b x 0 I If (a) holds, (D) is feasible and has optimal value 0. By strong duality, (P) has optimal value 0, and (b) does not hold I If (a) does not hold, (D) is infeasible, and (P) is either infeasible or unbounded I (P) is feasible: take p = 0 I Hence, (P) is unbounded, i.e., (b) holds IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 78
Characterization of unbounded LPs I If we know an LP has a finite optimal value, search for an optimal solution can be restricted to a (finite set) of extreme points of the feasible region (If necessary, of an equivalent problem in standard form) I If an LP is unbounded, how can we find that out? what constitutes a proof of its unboundedness? I Can the search for a proof of unboundedness be restricted to some finite set? IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 79 Cones Definition 4.1 AsetC < n is a cone if x 2 C for all 0 and all x 2 C. Note: I 0 2 C if C 6= ;. I The only possible extreme point of a cone is 0 if it is the case, we say that the cone is pointed Theorem 4.12 (Special case of Thm. 2.6) Let C be a polyhedral cone defined by the constraints a T i x 0, i =1,...,m. Then the following are equivalent: (a) The 0 vector is an extreme point of C (b) C does not contain a line (c) There exist n vectors out of the family a 1,...,a m,whichare linearly independent. IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 80
Rays and recession cones I A recession cone of a set P 6= ; at point y 2 P is the set I If P is a polyhedron: {d : y + d 2 P 8 0} I P = {x : Ax b} 6= ;: the recession cone at y 2 P is: {d : A(y + d) b, 8 0} = {d : Ad 0} independent of the point y. I P = {x 0 : Ax = b}: the recession cone is {d 0 : Ad = 0}. Definition 4.2: Extreme rays (a) A nonzero element x of a polyhedral cone C < n is called an extreme ray if there are n 1 linearly independent constraints that are active at x. (b) An extreme ray of the recession cone associated with a nonempty polyhedron P is also called an extreme ray of P. IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 81 Observations about extreme rays I If x is an extreme ray, x with > 0 is also an extreme ray. We call two extreme rays equivalent if one is a positive multiple of the other. I If two extreme rays are equivalent, they correspond to the same n 1 active constraints. Any n 1 linearly independent constraints define a line and can lead to at most two non-equivalent extreme rays. I Since there is a finite number of ways we can choose n 1 active constraints, the number of extreme rays of a polyhedron is finite (we do not distinguish between equivalent ones). I A finite collection of extreme rays is a complete set of extreme rays is it contains exactly one representative from each equivalence class. IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 82
Characterization of unbounded LPs Theorem 4.13 Consider the problem of minimizing c T x over a pointed polyhedral cone C = {x : a T i x 0, i =1,...,m}. The optimal cost is equal to 1 i some extreme ray d of C satisfies c T d < 0. Theorem 4.14 Consider the problem of minimizing c T x s.t. Ax b, andassume that the feasible set has at least one extreme point. The optimal cost is equal to 1 if an only if some extreme ray d of the feasible set satisfies c T d < 0. IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 83 A solution approach for LPs (P) minimize c T x s.t. x 2 P 1. Check if P = ;. If so, (P) is infeasible; if not, continue. 2. For every extreme ray d of P, checkifc T d < 0. If so, (P) is unbounded; if not continue. 3. For every extreme point x if P, evaluatec T x. Pickanextreme point with the smallest value: it is an optimal solution of (P). IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 84
Representation of polyhedra Theorem 4.15 (Resolution theorem) Let P = {x : Ax b} be a nonempty polyhedron with at least one extreme point. Let x 1,...,x k be the extreme points, and let w 1,...,w r be a complete set of extreme rays of P. Let ( kx ) rx kx Q = ix i + j w j : i 0, j 0, i =1. Then Q = P. Corollary 4.4 i=1 j=1 i=1 Anonemptyboundedpolyhedronistheconvexhullofitsextremepoints. Corollary 4.5 If the cone C = {x : Ax 0} is pointed, then every element of C can be represented as a nonnegative linear combination of the extreme rays of C. IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 85 Key to proof of Resolution Theorem Showing Q P is easy. Suppose z 2 P, butz 62 Q. Consider and (LP z ) maximize 0 T + 0 T kx rx s.t. ix i + j w j = z i=1 kx i=1 i =1 j=1 0, 0 (LD z ) minimize z T p + q s.t. (x i ) T p + q 0 8i (w j ) T p 0 8j (LP z )isinfeasible,hence(ld z ) is unbounded. IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 86
Converse to the resolution theorem I AsetQ is finitely generated if it is specified in the form 8 9 < kx rx kx = Q = : ix i + j w j : i 0, j 0, i =1 ;, i=1 j=1 where x 1,...,x k and w 1,...,w r are some given elements of < n. I The resolution theorem claims that every polyhedron with at least one extreme point is finitely generated. Theorem 4.16 A finitely generated set is a polyhedron. In particular, a convex hull of finitely many vectors is a (bounded) polyhedron. i=1 IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 87 Impact of Resolution Theorem and it converse Every polyhedron with at least one extreme point can be given/described/represented two ways: I {x : Ax b} for some A 2< m n and b 2< m n I x = P k i=1 ix i + P r j=1 jw j : i 0, j 0, P k for some x 1,...,x k, w 1,...,w r 2< n. i=1 i =1 o IOE 610: LP II, Fall 2013 Interpretation and uses of duality theory Page 88