IE 5531: Engineering Optimization I Lecture 7: Duality and applications Prof. John Gunnar Carlsson September 29, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 1 / 30
Administrivia PS 2 posted this evening Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 2 / 30
Recap Any linear program minimize c T x s.t. Ax = b x 0 has a dual of the form maximize b T y s.t. A T y c If both problems are feasible, we have c T x b T y for all feasible x and y At optimality, we have c T x = b T y The dual of the dual is the primal Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 3 / 30
Duality relationships Primal minimize c T x maximize b T y Dual A A T b i 0 constraints b i 0 variables = b i free 0 c j variables 0 c j constraints free = c j Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 4 / 30
Today Weak & strong duality Duality examples Complementary slackness Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 5 / 30
Weak duality The weak duality theorem says that if x is a feasible point for the primal (minimization) problem and y is a feasible point for the dual (maximization) problem, then we have b T y c T x If optimal solutions exist for both problems, the strong duality theorem says that b T y = c T x Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 6 / 30
Proof of weak duality Let x and y be feasible primal-dual pairs for a standard-form LP, so that Ax = b; x 0; A T y c We nd that b T y = x T A T y c T x since x 0 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 7 / 30
Proof of strong duality Suppose that x is an optimal solution to a standard-form LP that we found via the simplex method, let B denote the basic indices at x, and let N denote the non-basic indices Let x B = A 1 B b and x N = 0 Recall the denition of the reduced cost vector r in the simplex method (the top row): r = c A ( ) T A 1 T B c B When the simplex method terminates, the reduced cost vector must be nonnegative, thus ) T c B 0 c A ( T A 1 B }{{} y Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 8 / 30
Proof of strong duality Dene y by setting y = ( ) A 1 T B c B as in the preceding; clearly we have and thus y is dual feasible Finally, note that b T y = b T ( A 1 B which completes the proof c A T y ) T c B = ( ) A 1 T b c B = c T B x = B ct x y must be optimal for the dual problem, by weak duality B Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 9 / 30
Unboundedness Note that if one of the problems is unbounded, then the other must be infeasible Suppose that c T x is unbounded below in a standard-form LP; weak duality says that for all x and y b T y c T x Suppose that y is feasible in the dual; we can choose x to make c T x arbitrarily small and thus b T y, a contradiction Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 10 / 30
Infeasibility On the other hand, it is not true that if one problem is infeasible, the other must be unbounded The following primal-dual pairs are both infeasible: with dual minimize x 1 + 2x 2 x 1 + x 2 = 1 2x 1 + 2x 2 = 3 maximize y 1 + 3y 2 y 1 + 2y 2 = 1 y 1 + 2y 2 = 2 s.t. s.t. Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 11 / 30
Feasibility and boundedness An obvious corollary: if the primal problem is infeasible and the dual problem has a feasible point, then it must be unbounded A less obvious corollary: unless both problems are infeasible, at least one of them must have an unbounded feasible set The following relationships between primal and dual LPs are possible: Primal Finite optimum Unbounded Infeasible Finite optimum Possible X X Dual Unbounded X X Possible Infeasible X Possible Possible Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 12 / 30
Duality example Consider a transportation problem: We have m warehouses, each containing an amount a i of a product We have n stores that need to receive an amount b j of that product It costs c ij to transport one unit of the product from warehouse i to store j How can we minimize our costs while satisfying all stores' demands? Assume that m i=1 a i = n j=1 b j (total supply equals total demand) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 13 / 30
Duality example The LP formulation is minimize m n c ij x ij s.t. i=1 j=1 m x ij = a i i j=1 n x ij = b j j i=1 x ij 0 i, j Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 14 / 30
Duality example The dual of the transportation problem is maximize m n a i u i + b j v j s.t. i=1 j=1 u i + v j c ij i, j Remember: ai is the amount of the product available at warehouse i bj is the amount of the product required at store j cij is the transportation cost from i to j Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 15 / 30
Duality example Since there is no sign constraint, we can set u i u i (for interpretation purposes): n m maximize b j v j a i u i s.t. j=1 i=1 v j u i c ij i, j Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 16 / 30
Duality example Since there is no sign constraint, we can set u i u i (for interpretation purposes): n m maximize b j v j a i u i s.t. j=1 i=1 v j u i c ij i, j Interpretation: think of the u i 's and v j 's as prices Specically, think of a new logistics company that wants to ship these products The company will buy the goods at each warehouse at a price u i and sell them back to the warehouses at a price v j In order to be competitive with the original company's transportation modes, the new company must have v j u i c ij i, j (otherwise the original transportation company won't use them) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 16 / 30
Maximum ow Recall from earlier the maximum ow problem: Given a directed, weighted graph G = (V, E, W ) and a pair of nodes s and t Think of the edge weights w ij as the capacity of that edge What's the largest amount of ow we can send from s to t, subject to the capacity constraints? Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 17 / 30
Maximum ow In lecture 1 we derived an LP formulation: let x ij denote the amount of ow across edge (i, j); the problem is maximize (s,i) E (i,j) E x si s.t. x ij w ij (i, j) E x ij = x ji i s, t (j,i) E x ij 0 (i, j) E Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 18 / 30
Maximum ow The dual of the maximum ow problem is minimize w ij z ij s.t. (i,j) E z ij = y i y j (i, j) E y s y t = 1 z ij 0 (i, j) E (see B&T 7.5 for derivation) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 19 / 30
Maximum ow The dual of the maximum ow problem is minimize w ij z ij s.t. (i,j) E (see B&T 7.5 for derivation) z ij = y i y j (i, j) E y s y t = 1 z ij 0 (i, j) E We're assigning values y i to the nodes, and incurring costs y i y j for all (i, j) E We can assume WLOG that y s = 1 and y t = 0 It is not hard to show that, in fact, y i {0, 1} for all i Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 19 / 30
Min cut The preceding problem is equivalent to nding a subset S of vertices, such that s S, that minimizes the cut length of S, i.e. i S:j / S w ij Thus, strong duality says that the maximum ow we can push through the network is equal to the smallest cut size of any subset S of vertices Corollary: if the value of a ow is equal to the value of some cut, then both are optimal Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 20 / 30
Max-ow min-cut theorem Input graph Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 21 / 30
Max-ow min-cut theorem Maximum ow Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 22 / 30
Max-ow min-cut theorem Minimum cut Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 23 / 30
Complementary slackness Consider a standard-form linear program and let A i denote column i of A We know that any constraint in the primal corresponds to a variable in the dual, and vice versa An important relationship between optimal primal and dual solutions is known as complementary slackness, or complementarity: Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 24 / 30
Complementary slackness Theorem Let x and y be feasible solutions to the primal and dual problem, respectively. The vectors x and y are optimal solutions for the two respective problems if and only if x i > 0 = A T i y = c i A T i y < c i = x i = 0 in other words, x i (A T i y c i ) = 0 i Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 25 / 30
Proof of complementary slackness Suppose that A T i x i > 0 = A T i y = c i y < c i = x i = 0 as in the theorem; then certainly ( A T y c) T x = = = 0 (y T A c T ) n i=1 x (y T A i c i ) x i Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 26 / 30
Proof of complementary slackness Since ( y T A c T ) x = 0, we know that 0 = (y T A c T ) = y T Ax c T x = y T b c T x c T x = b T y x which completes one direction of the proof by weak duality. Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 27 / 30
Proof of complementary slackness To prove the other direction, suppose that x and y are optimal, so that c T x = b T y (strong duality) We can write c T x b T y = 0 c T x (Ax) T y = 0 c T x y T Ax = 0 ( ) c T y T A x = 0 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 28 / 30
Proof of complementary slackness We can rewrite the equation ( ) c T y T A x = 0 as a summation: n i=1 Since x and y are feasible, we have so ( c i y T A i ) xi for all i (c i y T A i ) x i = 0 c i y T A i 0 i x i 0 i Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 29 / 30
Non-standard form LPs Complementarity holds for LPs that are not in standard form as well Let a T j denote the jth row of A and let A i denote the ith column of A A primal-dual pair x, y is optimal if and only if ) y j (a T j x b j = 0 j {1,..., m} ) (c i y T A i x i = 0 i {1,..., n} where A R m n In other words: take each variable, multiply it by its associated constraint; the product must be 0 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 30 / 30