Week 3 Linear programming duality This week we cover the fascinating topic of linear programming duality. We will learn that every minimization program has associated a maximization program that has the same value. This surprising connection will allow us to efficiently certify the optimality or the infeasibility of a given linear program. 3.1 Bounding the value of a linear program To develop some intuition, we will explore the idea of bounding the value of the following linear program: 2x 1 + 3x 2 6x 1 + 4x 2 10 x 1 + 5x 2 4 x 1, x 2 0 We can get a first easy upperbound by noting that for any feasible solution (x 1, x 2 ) we have 2x 1 + 3x 2 6x 1 + 4x 2 10, where the first inequality follows from the fact that x 1, x 2 0 and the second from the first constraint of the program. We can get another upperbound by noting that for any feasible solution (x 1, x 2 ) we have 2x 1 + 3x 2 2x 1 + 10x 2 = 2(x 1 + 5x 2 ) 2 4 = 8, where the second inequality follows from the second constraint of the program. We can get an even better upperbound if we combine the two constraints: 2x 1 + 3x 2 1 2 (6x 1 + 4x 2 ) + 1 5 (x 1 + 5x 2 ) 1 2 10 + 1 5 4 = 29 5. Taking a more systematic approach, we could ask to find multipliers y 1, y 2 0 for each constraint to obtain the best possible upperbound on the value of our program. Copyright 2015 Julián Mestre. This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International License. For the most recent version of these notes, please visit the class webpage.
week 3 linear programming duality discrete optimization 2 This leads to the following linear program 1 10y 1 + 4y 2 6y 1 + y 2 2 4y 1 + 5y 2 3 y 1, y 2 0 1 In order to have the inequality 2x 1 + 3x 2 y 1 (6x 1 + 4x 2 ) + y 2 (x 1 + 5x 2 ), we ask that 2 6y 1 + y 2, and 3 4y 1 + 5y 2, so that we get the upperbound The best possible upperbound is attained by y 1 = 26 7 and y 2 = 10 which yields a bound of 13 55. How good an upperbound is this? As 26, 10y 1 + 4y 2, which we should try to. good as it gets! Indeed, there is a solution to the original program with the same value 2. The auxiliary linear program is the dual 2 The solution x 1 = 17 13 and x 2 = 13 7 has value 55 13. Amazing! program of the original program, which is called primal. In the rest of this lecture, we will define formally how to derive the dual of a given program and to prove the remarkable property that the value of the primal and dual programs is the same, provided they are both feasible. 3.2 How to derive the dual program Let A R m n be a matrix with columns A 1,..., A n and rows a 1,..., a m. With each primal linear program we associate the dual program j c jx j a T i x b i i M 1 a T i x b i i M 2 a T i x = b i i M 3 x j 0 j N 1 x j 0 j N 2 x j free j N 3 i b iy i y T A j c j j N 1 y T A j c j j N 2 y T A j = c j j N 3 y i 0 i M 1 y i 0 i M 2 y i free i M 3 (P) (D) Of course, we must have and m = M 1 + M 2 + M 3 n = N 1 + N 2 + N 3. A variable x j or y i is free if it can take positive and negative values. We can define the dual of a maximizing program by reversing the direction of the transformation. In this way, taking the dual of the dual brings us back to the linear program we started from! 3.3 Weak duality The Weak Duality Theorem states that if (P) and (D) are feasible then the value of (P) is greater than or equal to the value of (D). Theorem 3.1. Let x and y be feasible solutions of (P) and (D) respectively. Then.
week 3 linear programming duality discrete optimization 3 Proof. One could say that the dual has been defined with the express purpose to make this proof go through: c j x j y T A j x j = y T Ax = y i a T i x y i b i. (3.1) j i i j The first inequality holds because, if j N 1 then x j 0 and c j y T A j, so c j x j y T A j x j ; if j N 2 then x j 0 and c j y T A j, so c j x j y T A j x j ; if j N 3 then x j free and y T A j = c j, so c j x j = y T A j x j. The second inequality can be justified in a similar fashion. Corollary 3.1. If (P) has unbounded objective then (D) is infeasible. Similarly, if (D) has unbounded objective then (P) is infeasible. 3.4 Strong duality The collorary follows from Theorem 3.1: Any feasible solution of (D) sets a lowerbound on the value of (P). Similarly, any solution of (P) sets an upperbound on the value of (D). The Strong Duality Theorem states that the value of the primal and the dual programs must be the same, provided the programs have bounded objective. Theorem 3.2. If a linear program is feasible and has bounded objective, so does its dual. Furthermore, the value of the two programs is the same. Proof. We will prove the theorem only for linear programs in standard form 3 : 3 In the Exercises we will show that it indeed holds for any linear program Ax = b The dual program is: y T A c T y free Suppose the primal program has bounded objective. Then if we run Simplex on the primal program, it will return an optimal basic feasible solution x. Let B be the basis associated with x. Recall that the termination criterion for Simplex was that reduced costs at B were non-negative; that is, c T c B T A 1 B A 0. Therefore, the solution y T = c T B A 1 B is dual feasible. The cost of this solution is = y T b = c B T A 1 B b = c B T x B =. From Theorem 3.1, it follows that y is an optimal dual solution. Furthermore, x and y have the same objective value.
week 3 linear programming duality discrete optimization 4 3.5 Complementary slackness Strong duality constrains the structure of the optimal solutions of primal and dual programs. We now show that a pair of optimal primal and dual solutions must obey the so-called complementary slackness property. Theorem 3.3. Let x and y be feasible solutions to (P) and (D) respectively. Then x and y are optimal if and only if i) x j = 0 or y T A j = c j for all j N 1 N 2 N 3, and ii) y i = 0 or a i T x = b i for all i M 1 M 2 M 3. Notice that this Theorem only provides interesting information for indices associated with inequality constraints (j N 1 N 2 and i M 1 M 2 ). For equality constraints (j N 3 and i M 3 ) the property holds trivially. Proof. Consider the proof of Theorem 3.1 applied to the optimal solution x and y. By Theorem 3.2 we know that if the solution are optimal then =, therefore, each of the inequalities in (3.1) must in fact, be met with equality. Conversely, if we have equality throughout the cost of the solutions must be the same. It is easy to check that the first inequality in (3.1) is met with equality if and only if i) holds. Similarly, the second inequality in (3.1) is met with equality if and only if ii) holds. 3.6 Certifying infeasibility If the program (P) and its dual (D) are feasible, Theorem 3.2 tells us that they must have the same value. Therefore, if one of them is feasible and it is has unbounded objective, then the other one must necessarily be infeasible. This connection between (P) and (D) can be exploited to design a simple certificate of infeasibility. Theorem 3.4 (Farkas lemma). Let A R m n and b R m. Then exactly one of the following alternatives holds, i) there exists x R n such that Ax = b and, or ii) there exists y R m such that A T y 0 and < 0. Proof. Let us first argue that both alternatives cannot hold simultaneously. Indeed, if Ax = b then y T Ax = y T b. Notice that and y T A 0, so y T A. On the other hand, we know that y T b < 0, os we get a contradiction. Now assume that there is no x R n such that and Ax = b. Then the following linear program is infeasible 0 x While the statement and the proof of Theorem 3.4 may seem abstract and unintuitive at first, there is a neat geometric interpretation of the Theorem statement. We can think of Ax as a positive linear combination of the columns of A. Therefore, the system is Ax = b, is infeasible if and only if b lies outside conical hull of the columns of A. Intuitively b lies outside this cone if and only if there is a hyperplane separating the two. The vector y is the normal direction of this hyperplane. {Ax : } A 1 A 2 A 3 Ax = b A 4 b y
week 3 linear programming duality discrete optimization 5 By Theorem 3.2, this means its dual must be either infeasible or have unbounded objective. The dual is the following program A T y 0 y free The program is feasible since y = 0 is a valid solution. Therefore, there must be a solution y such that A T y 0 and < 0. 3.7 Physical interpretation of duality The exposition in this section is not meant to be a rigorous mathematical argument, but rather a helpful image for building intuition. Feel free to skip this part if you find it confusing. Consider the linear program and its dual Ax b x free y T A = c T y 0 Let A have rows a T 1,..., at m. Picture a physical representation of the polyhedron P = {x : Ax b} where the hyperplanes a i x = b i are solid walls. Imagine placing a ball inside P and letting it fall under the influence of a gravitation force that pull the ball in the opposite direction of c. If the ball is small enough, after some time it will come to a rest near some corner x of P. The ball is stationary because the gravitational pull is perfectly cancelled by the forces exerted by the walls it rests on. In other words, there are coefficients y 1,..., y n 0 such that a 1 a 3 c y 3 a 3 y 1 a 1 c = i y i a i. Notice that y is a dual feasible solution. Furthermore, y i > 0 only for those hyperplanes that meet at x. Therefore, if a i x > b i then y i = 0. Therefore, In other words, for all i y i = 0 or a i x = b i. = i y i b i = i y i a i T x = y T Ax = c T x. This is precisely the complementary slackness condition ii) of Theorem 3.3 In other words, we get strong duality. Exercises 1. Each linear program can be classified into infeasible, feasible with unbounded objective, or feasible with bounded objective.
week 3 linear programming duality discrete optimization 6 Therefore, a pair of primal and dual programs can potentially be classified into nine possible types. Not all nine combination can be achieved. For each combination either argue why it cannot be achieved, or show example (a pair of primal and dual programs). Hint: it is possible to have an unfeasible-unfeasible primal-dual pair of programs. 2. Our proof of Theorem 3.2 only works for linear programs in standard form. Show that it also holds for programs of the form: Ax b First turn the above program into an equivalent 4 program in 4 We already did this in Week 1. standard form by introducing additional slack variables. Then show that the duals of the original program and the one in standard form are equivalent. 3. Our proof of Theorem 3.2 only works for linear programs in standard form. Show that it also holds for programs of the form: A T y c y 0 Notice that the proof of Theorem 3.2 does not work here because the program is not in standard form, but rather it is dual of a program in standard form. 4. Let A R m n and b R m. Prove that exactly one of the following alternatives holds, 5. 6. i) there exists x R n such that Ax b, or ii) there exists y R m such that A T y = 0 and y 0 and > 0. Prove that the system Ax = 0 and x > 0 is infeasible if and only if the system A T y 0 and A T y = 0 is feasible. Let a and a 1,..., a m be vectors in R n. Prove that the following statements are equivalent: (a) for all x R n such that we have a x max i a i x, or (b) there exists λ R m such that λ 0 and i λ i = 1 and a i λ ia i. Here x > 0 means every coordinate of x is strictly positive.
week 3 linear programming duality discrete optimization 7 Solutions of selected exercises 2. Our proof of Theorem 3.2 only works for linear programs in standard form. Show that it also holds for programs of the form: Ax b Solution. Call the above program (P). Its dual, which we will denote by (D), is A T y c y 0 We introduce an equivalent program (P ) in standard form by adding dummy variables that eat up the slack. Ax Iz = b z 0 We call the dual of this new program (D ): A T y c Iy 0 y free Notice that (D ) is in fact equivalent to (D), since Iy 0 is the same constraint as y 0. To finish the argument, we note that if (P) is feasible and has a bounded objective then (P ) is feasible and has bounded objective, because they are, in essence, the same program. Using the proof of Theorem 3.2 from the notes, we get that (D ) is feasible and has the same value of (P ). But since (D ) and (D) are equivalent, it follows that (D) is also feasible and has the same value as (P).