1/39 The Strong Duality Theorem 1 Adrian Vetta 1 This presentation is based upon the book Linear Programming by Vasek Chvatal
2/39 Part I Weak Duality
3/39 Primal and Dual Recall we have a primal linear program max s.t. n c j x j n a ij x j b i i [m] = {1, 2,..., m} x j 0 j [n] = {1, 2,..., n} and its dual linear program min s.t. m b i y i m a ij y i c j j [n] y i 0 i [m]
4/39 Weak Duality Theorem The Weak Duality Theorem. For any primal feasible solution (x 1, x 2,..., x n ) and any dual feasible solution (y 1, y 2,..., y m ) we have n c j x j b i y i
5/39 Proof of Weak Duality Weak Duality Theorem. For any primal feasible solution (x 1, x 2,..., x n ) and any and any dual feasible solution (y 1, y 2,..., y m ) we have n c j x j m b i y i. Proof. n x j c j = n x j a ij y i y i n a ij x i y i b i
6/39 Part II Strong Duality
7/39 Strong Duality Theorem The Strong Duality Theorem. If the primal has an optimal solution (x 1, x 2,..., x n) then the dual has an optimal solution (y 1, y 2,..., y m) such that n c j xj = b i y i
8/39 An Example Consider the primal and its dual maximise 4x 1 + x 2 + 5x 3 + 3x 4 subject to x 1 x 2 x 3 + 3x 4 1 5x 1 + x 2 + 3x 3 + 8x 4 55 x 1 + 2x 2 + 3x 3 5x 4 3 x 1, x 2, x 3, x 4 0 minimise y 1 + 55y 2 + 3y 3 subject to y 1 + 5y 2 y 3 4 y 1 + y 2 + 2y 3 1 y 1 + 3y 2 + 3y 3 5 3y 1 + 8y 2 5y 3 3 y 1, y 2, y 3 0
9/39 Optimal Solutions maximise 4x 1 + x 2 + 5x 3 + 3x 4 subject to x 1 x 2 x 3 + 3x 4 1 5x 1 + x 2 + 3x 3 + 8x 4 55 x 1 + 2x 2 + 3x 3 5x 4 3 x 1, x 2, x 3, x 4 0 The primal has optimal solution (x 1, x 2, x 3, x 4 ) = (0, 14, 0, 5) with value 29. minimise y 1 + 55y 2 + 3y 3 subject to y 1 + 5y 2 y 3 4 y 1 + y 2 + 2y 3 1 y 1 + 3y 2 + 3y 3 5 3y 1 + 8y 2 5y 3 3 y 1, y 2, y 3 0 The dual has optimal solution (y 1, y 2, y 3 ) = (11, 0, 6) with value 29.
10/39 Solving the Primal Let s solve the primal by the simplex method. maximise 4x 1 + x 2 + 5x 3 + 3x 4 subject to x 1 x 2 x 3 + 3x 4 1 5x 1 + x 2 + 3x 3 + 8x 4 55 x 1 + 2x 2 + 3x 3 5x 4 3 x 1, x 2, x 3, x 4 0 Put into dictionary form. z = 4x 1 + x 2 + 5x 3 + 3x 4 x 5 = 1 x 1 + x 2 + x 3 3x 4 x 6 = 55 5x 1 x 2 3x 3 8x 4 x 7 = 3 + x 1 2x 2 3x 3 + 5x 4
11/39 Pivot 1 z = 4x 1 + x 2 + 5x 3 + 3x 4 x 5 = 1 x 1 + x 2 + x 3 3x 4 x 6 = 55 5x 1 x 2 3x 3 8x 4 x 7 = 3 + x 1 2x 2 3x 3 + 5x 4 Pivot (Bland s Rule) Add x 1 to basis. (Lowest index.) Remove x 5 from basis. (No choice.) z = 4(1 + x 2 + x 3 3x 4 x 5) + x 2 + 5x 3 + 3x 4 x 1 = 1 + x 2 + x 3 3x 4 x 5 x 6 = 55 5(1 + x 2 + x 3 3x 4 x 5) x 2 3x 3 8x 4 x 7 = 3 + (1 + x 2 + x 3 3x 4 x 5) 2x 2 3x 3 + 5x 4 z = 4 + 5x 2 + 9x 3 9x 4 4x 5 x 1 = 1 + x 2 + x 3 3x 4 x 5 x 6 = 50 6x 2 8x 3 + 7x 4 + 5x 5 x 7 = 4 x 2 2x 3 + 2x 4 x 5
2/39 Pivot 2 z = 4 + 5x 2 + 9x 3 9x 4 4x 5 x 1 = 1 + x 2 + x 3 3x 4 x 5 x 6 = 50 6x 2 8x 3 + 7x 4 + 5x 5 x 7 = 4 x 2 2x 3 + 2x 4 x 5 Pivot: Add x 2 to basis. (Lowest index.) Remove x 7 from basis. (No choice.) z = 4 + 5(4 2x 3 + 2x 4 x 5 x 7) + 9x 3 9x 4 4x 5 x 2 = 4 2x 3 + 2x 4 x 5 x 7 x 1 = 1 + (4 2x 3 + 2x 4 x 5 x 7) + x 3 3x 4 x 5 x 6 = 50 6(4 2x 3 + 2x 4 x 5 x 7) 8x 3 + 7x 4 + 5x 5 z = 24 x 3 + x 4 9x 5 5x 7 x 2 = 4 2x 3 + 2x 4 x 5 x 7 x 1 = 5 x 3 x 4 2x 5 x 7 x 6 = 26 + 4x 3 5x 4 + 11x 5 + 6x 7
3/39 Pivot 3 z = 24 x 3 + x 4 9x 5 5x 7 x 2 = 4 2x 3 + 2x 4 x 5 x 7 x 1 = 5 x 3 x 4 2x 5 x 7 x 6 = 26 + 4x 3 5x 4 + 11x 5 + 6x 7 Pivot: Add x 4 to basis. (No choice.) Remove x 1 from basis. (No choice.) z = 24 x 3 + (5 x 3 2x 5 x 7 x 1) 9x 5 5x 7 x 4 = 5 x 3 2x 5 x 7 x 1 x 2 = 4 2x 3 + 2(5 x 3 2x 5 x 7 x 1) x 5 x 7 x 6 = 26 + 4x 3 5(5 x 3 2x 5 x 7 x 1) + 11x 5 + 6x 7 z = 29 2x 3 11x 5 6x 7 x 1 x 4 = 5 x 3 2x 5 x 7 x 1 x 2 = 14 4x 3 5x 5 3x 7 2x 1 x 6 = 1 + 9x 3 + 21x 5 + 11x 7 + 5x 1
14/39 An Optimal Primal Solution z = 29 x 1 2x 3 11x 5 6x 7 x 2 = 14 2x 1 4x 3 5x 5 3x 7 x 4 = 5 x 1 x 3 2x 5 x 7 x 6 = 1 + 5x 1 + 9x 3 + 21x 5 + 11x 7 The optimal solution is (x 1, x 2, x 3, x 4 ) = (0, 14, 0, 5). The optimal value is z = 29. So the optimal primal solution is as claimed.
15/39 An Optimal Dual Solution z = 29 x 1 2x 3 11x 5 6x 7 x 2 = 14 2x 1 4x 3 5x 5 3x 7 x 4 = 5 x 1 x 3 2x 5 x 7 x 6 = 1 + 5x 1 + 9x 3 + 21x 5 + 11x 7 Recall that the optimal dual solution is (y 1, y 2, y 3 ) = (11, 0, 6). How does the simplex algorithm confirm this? Look at the top row: z = 29 x 1 0x 2 2x 3 0x 4 11x 5 0x 6 6x 7 So the dual variables (y 1, y 2, y 3 ) have values equal to the coefficients of (x 5, x 6, x 7 ) in z!
16/39 Example 2 Take the following primal and its dual max 5x 1 + 4x 2 + 3x 3 s.t. 2x 1 + 3x 2 + x 3 5 4x 1 + x 2 + 2x 3 11 3x 1 + 4x 2 + 2x 3 8 x 1, x 2, x 3 0 min 5y 1 + 11y 2 + 8y 3 s.t. 2y 1 + 4y 2 + 3y 3 5 3y 1 + y 2 + 4y 3 4 y 1 + 2y 2 + 2y 3 3 y 1, y 2, y 3 0 Recall the final dictionary was z = 13 3x 2 1x 4 1x 6 x 1 = 2 2x 2 2x 4 + x 6 x 3 = 1 + x 2 + 3x 4 2x 6 x 5 = 1 + 5x 2 + 2x 4 So the primal solution is (x 1, x 2, x 3 ) = (2, 0, 1) with value 13. The dual solution is (y 1, y 2, y 3 ) = (1, 0, 1) with value 13.
17/39 Example 3 Take the following primal and its dual max 5x 1 + 5x 2 + 3x 3 s.t x 1 + 3x 2 + x 3 3 x 1 + 3x 3 2 2x 1 x 2 + 2x 3 4 2x 1 + 3x 2 x 3 2 x 1, x 2, x 3 0 min 3y 1 + 2y 2 + 4y 3 + 2y 4 s.t. y 1 y 2 + 2y 3 + 2y 4 5 3y 1 y 3 + 3y 4 5 y 1 + 3y 2 + 2y 3 y 4 3 y 1, y 2, y 3, y 4 0 Recall the final dictionary was z = 10 1x 5 1x 6 2x 7 x 1 = 32 29 3 29 x 7 9 29 x 6 + 5 29 x 5 x 2 = 8 29 8 29 x 7 + 5 29 x 6 6 29 x 5 x 3 = 30 29 1 29 x 7 3 29 x 6 8 29 x 5 x 4 = 1 29 + 28 29 x 7 3 29 x 6 + 21 29 x 5 So the primal solution is (x 1, x 2, x 3 ) = ( 32 29, 8 29, 30 29 ) with value 10. The dual solution is (y 1, y 2, y 3, y 4 ) = (0, 1, 1, 2) with value 10.
18/39 Part III Proof of Strong Duality
19/39 Strong Duality Theorem The Strong Duality Theorem. If the primal has an optimal solution (x 1, x 2,..., x n) then the dual has an optimal solution (y 1, y 2,..., y m) such that n c j xj = b i y i
20/39 Intuition Look at the initial dictionary: And the final dictionary: z = 4x 1 + x 2 + 5x 3 + 3x 4 x 5 = 1 x 1 + x 2 + x 3 3x 4 x 6 = 55 5x 1 x 2 3x 3 8x 4 x 7 = 3 + x 1 2x 2 3x 3 + 5x 4 z = 29 x 1 2x 3 11x 5 6x 7 x 2 = 14 2x 1 4x 3 5x 5 3x 7 x 4 = 5 x 1 x 3 2x 5 x 7 x 6 = 1 + 5x 1 + 9x 3 + 21x 5 + 11x 7 The slack variable x 5 appears in one constraint in the initial dictionary. So the only way it can appear with a coefficient of 11 in the top row of the final dictionary is if we multiplied that constraint by 11. Similarly we must have multiplied the x 7 constraint by 6.
21/39 Intuition (cont.) Let s verify this. Add these constraints to the top row of the initial dictionary: z = 4x 1 + x 2 + 5x 3 + 3x 4 + 11 x 5 = 11 (1 x 1 + x 2 + x 3 3x 4 ) + 6 x 7 = 6 (3 + x 1 2x 2 3x 3 + 5x 4 ) z + 11x 5 + 6x 7 = 29 x 1 2x 3 Thus z = 29 x 1 2x 3 11x 5 6x 7
22/39 Intuition (cont.) maximise 4x 1 + x 2 + 5x 3 + 3x 4 subject to x 1 x 2 x 3 + 3x 4 1 5x 1 + x 2 + 3x 3 + 8x 4 55 x 1 + 2x 2 + 3x 3 5x 4 3 x 1, x 2, x 3, x 4 0 In terms of the original LP we have 11 (x 1 x 2 x 3 + 3x 4 ) 11 1 6 ( x 1 + 2x 2 + 3x 3 5x 4 ) 6 3 5x 1 + x 2 + 7x 3 + 3x 4 29 So 4x 1 + x 2 + 5x 3 + 3x 4 5x 1 + x 2 + 7x 3 + 3x 4 29
23/39 Intuition: Example 2 Look at the initial dictionary: And the final dictionary: z = 5x 1 + 4x 2 + 3x 3 x 4 = 5 2x 1 3x 2 x 3 x 5 = 11 4x 1 x 2 2x 3 x 6 = 8 3x 1 4x 2 2x 3 z = 13 3x 2 1x 4 1x 6 x 1 = 2 2x 2 2x 4 + x 6 x 3 = 1 + x 2 + 3x 4 2x 6 x 5 = 1 + 5x 2 + 2x 4 The slack variable x 4 appears in one constraint in the initial dictionary. So the only way it can appear with a coefficient of 1 in the top row of the final dictionary is if we multiplied that constraint by 1.
24/39 Intuition: Example 3 Look at the initial dictionary: And the final dictionary: z = 5x 1 + 5x 2 + 3x 3 x 4 = 3 x 1 3x 2 x 3 x 5 = 2 + x 1 3x 3 x 6 = 4 2x 1 + x 2 2x 3 x 7 = 2 2x 1 3x 2 + x 3 z = 10 1x 5 1x 6 2x 7 x 1 = 32 3 29 29 x7 9 29 x6 + 5 29 x5 x 2 = 8 29 8 29 x7 + 5 29 x6 6 29 x5 x 3 = 30 29 1 29 x7 3 29 x6 8 x 4 = 1 29 + 28 29 x7 3 29 29 x5 21 x6 + 29 x5 The slack variable x 7 appears in one constraint in the initial dictionary. So the only way it can appear with a coefficient of 2 in the top row of the final dictionary is if we multiplied that constraint by 2.
25/39 Proof Idea Does what we have seen always work? Run the simplex algorithm to find... The values of the decision variables (x 1, x 2,..., x n). The values of the slack variables (x n+1, x n+2,..., x n+m). The top row of the final dictionary n+m z = OPT γ k x k k=1 [We know γ k 0 for all k, and for variables in the basis γ k = 0.] Set the dual variables to be y i = γ n+i for all i 1, 2,..., m.
26/39 Proof of Strong Duality We only need to show two things: (1) That (y 1, y 2,..., y m) is feasible for the dual, i.e. a ij y i c j j [n] (2) It has value equal to the optimal primal value, i.e. n c j xj = b i y i
27/39 Proof of Strong Duality (cont.) Let z = OPT(primal) = So the final dictionary states that z = n e.g. z = 10 x 5 x 6 2x 7 n c j xj n+m c j xj γ k x k k=1 n+m = z γ k x k k=1 = OPT(primal) n+m k=1 γ k x k
8/39 Proof: Some Algebra nx n+m X c j x j = OPT γ k x k = OPT = OPT = OPT = OPT = OPT k=1 nx γ k x k k=1 n+m X k=n+1 γ k x k nx mx γ j x j γ n+i x n+i nx mx γ j x j yi x n+i nx mx γ j x j mx b i y i! + y i nx b i mx! nx a ij x j a ij y i γ j! x j
29/39 Proof: Dual Feasibility ( n c j x j = OPT b i y i ) + ( n m ) a ij yi γ j x j But this equality is true for all choices of {x 1, x 2,..., x n }. So it must be that c j = a ij y i But γ j 0, and so we have feasibility γ j (2) a ij y i c j
30/39 Proof: Dual Optimality ( n c j x j = OPT b i y i ) + ( n m ) a ij yi γ j x j But this equality is true for all choices of {x 1, x 2,..., x n }. So it must be that Thus we have strong duality OPT(primal) b i y i = 0 (1) b i y i = OPT(primal) = n c j xj
31/39 Part IV Complementary Slackness
2/39 Weak Duality Recall for any feasible solutions {x 1, x 2,..., x n } and {y 1, y 2,..., y m } n x j c j = n x j a ij y i y i n a ij x j y i b i
33/39 Strong Duality By strong duality, we must have equalities for the optimal solutions {x 1, x 2,..., x n} and {y 1, y 2,..., y m} n xj c j = = = n xj y i y i a ij y i n a ij xj b i
34/39 Some Observations P n x j c j = P n x j Pm a ijyi = P m y i Pn a ijxj P = m y b i So, by strong duality, we have We also have i x j c j = x j y i n a ij xi From these observations we obtain... = y i a ij y i b i
35/39 Complementary Slackness Complementary Slackness Theorem. A primal feasible solution (x 1, x 2,..., x n) and a dual feasible solution (y 1, y 2,..., y m) are both optimal if and only if and x j > 0 y i > 0 a ij y i = c j j [n] a ij xj = b i i [m]
36/39 Part V Consequences of Strong Duality
37/39 Running Times It is easy to check that the dual of the dual is the primal. So solving the dual will also give an optimal primal solution. Recall the # pivots is typically linear in the number of constraints. The # constraints in the dual is just the # variables, n, in the primal. Thus O(n) pivots are typically needed to solve the dual. So solving the dual can be far quicker than solving the primal!
38/39 Dual Variables We have seen many examples where the dual problem often has a nice combinatorial meaning. In economic problems often the primal variables correspond to allocations and the dual variables to prices.
39/39 Certificates The simplex algorithm gives us a certificate of optimality. Given a primal solution x the dual solution y proves x is optimal. [So always check that the x you found is optimal!] So a certificate let s us know when to stop searching. Certificates seem fundamental in assessing the hardness of problems...