A Review of Linear Programming

A Review of Linear Programming Instructor: Farid Alizadeh IEOR 4600y Spring 2001 February 14, 2001 1 Overview In this note we review the basic properties of linear programming including the primal simplex method, duality theory, and dual simplex method. 2 Linear Programming And Convex Sets The Standard LP is defined as min s.t. c T x Ax = b x 0 Here c, x R n, A R m n m < n, b R m. We use standard form to ease discussion of the theory and algorithms pertaining to linear programming. There are other possible standard forms, for instance: max s.t. w T x Ax b Feasible set: {x : Ax = b, x 0} 1

2 LINEAR PROGRAMMING AND CONVEX SETS Review of Linear Programming Example 1 (Extreme points) One constraint x 1 + x 2 + x 3 = 1: Example 2 (Two equality constraints in R 3 ) Definition 2 A set C R n is convex if x C and y C, then every point z in the line segment from x to y also belongs to C. 2

2 LINEAR PROGRAMMING AND CONVEX SETS Review of Linear Programming The point z is in the line segment from x to y means z = α(y x)+x for some 0 α 1 z = αy + (1 α)x is called the convex combination of x,y. Definition 2 z = α 1 x 1 + α 2 x 2 + + α n x n where α 1,..., α n 0 α i = 1 called convex combination of x 1,..., x n. Definition 2 Convex hull of a finite set C R n, {x 1,..., x k } is conv{c} = {z R n : α 1,..., α k 0, αi = 1 such that z = α 1 x 1 + + α n x n }. Convex hull of an arbitrary set C is defined as conv{c} def = {α 1 x 1 + + α n 0x n : α i 0, α i = 1, x i C} Example 3 (convex hull) C 1 is not a convex set but the conv C is. it. Fact 2 In a general optimization problem, if the objective function is linear then min c T x s.t. x C has the same value as min c T x s.t. x conv C. In other words every optimization problem can be reduced to one which optimizes over a convex set. Unfortunately effectively determining Conv(C) is not that easy in general and may involve prohibitive computational costs. Now suppose that the set C over which we are optimizing is a a finite set of points with integer coordinates. That is suppose that our optimization problem is in fact an integer programming problem. Then, this fact says that we only need to worry about the convex hull of our integer valued points. But convex hull of a finite number of points is a polytope and optimization over polytopes is linear programming. Thus, 3

2 LINEAR PROGRAMMING AND CONVEX SETS Review of Linear Programming Corollary 2 Each integer program is equivalent to some linear programming problem. Again this fact should not give us much comfort from a computational point of view. The polytope obtained by taking the convex hull of our integer program may have a huge number of facets that is the resulting linear program may have a huge number of constraints (much larger than the original integer program.) Fact 2 If C 1 and C 2 are two convex sets then so is C 1 C 2. Definition 2 A point x C where C is a convex set is called extreme if x = 1x+0x 1 + x k is the only way that x can be written as convex combinations of other points in C. Example 4 (extreme points of two convex sets) Fact 2 Consider the optimization problem min c T x s.t. x C where C is convex. Then the optimum is attained at some extreme point of C. Now the feasible region of an LP F = {x : Ax = b, x 0} is a convex set, in fact a polyhedron. Thus a linear program has a finite number of extreme points. Thus to solve a linear programming problem it suffices to focus on the extreme points and try to find the one with the best objective value; that point will be the optimum over all points in the feasible set of the LP. Note 1 an optimal point does not necessarily have to be an extreme point, but among all these optimal points there will be at least one extreme point. 4

4 THE SIMPLEX ALGORITHM (IN MATRIX FORM) Review of Linear Programming 3 The point of the simplex method Since optimum occurs on an extreme point (which there are only a finite number of them in linear programming) we can sketch the following steps to find the optimum point: 1. First find some extreme point. 2. Check to see if it is optimal. 3. If not, move on to a neighboring extreme point which improves the objective function and repeat 1-3. We now have to give an algebraic characterization of these operations. What is an extreme point of a polyhedron in algebraic terms? What does it mean to move from point to a neighboring point? How do we decide that the current point doesn t have a better neighbor? Consider the feasible set of a standard LP: F = {x R n : Ax = b and x i 0} Fact 3 An extreme point of set F has at most m coordinates non-zero where m is the number of rows of matrix A. Equivalently at least n m coordinates are zero. Example 5 (Standard form LP) Consider a feasible set F = {x 1 + x 2 + x 3 = 3, and x i 0} Here m = 1, n = 3, and A = [1, 1, 1. As you can see from the picture in the previous page that the feasible set is a triangle and each of its extreme points have 2 nonzeros and one zero coordinate. Definition 3 A basic feasible solution of LP in standard form is a solution where n m coordinates are set equal to zero, and the remaining coordinates are non-negative. 4 The Simplex Algorithm (in Matrix Form) Consider the linear programming problem in the standard form: min s.t. c T x Ax = b x 0 (1) 5

4 THE SIMPLEX ALGORITHM (IN MATRIX FORM) Review of Linear Programming where A R m n, c, x R n, b R m and m < n. c T = (c 1,..., c n ) a 11... a 1n... } a m1... {{ a mn } A x 1. x n } {{ } x = b 1. b n } {{ } b (2) Remember that the optimum occurs at an extreme point of the feasible set and that at such a point at most m of the entries are nonzero. Thus in equation Ax = b, since A is m n, if we fix m n entries of x, then only m will be left unknown; but we have m equations and we can solve for the unknown entries. Thus at each extreme point of the the coordinates of x and the corresponding columns of A and coordinates of c are labeled as basic and nonbasic. Without loss of generality let us assume that the first m coordinates of x and c and the first m columns of A are the basic ones and the rest are nonbasic. Thus the matrix A is split into basic (B) and non-basic (N) submatrices. Similarly x = (x B ; x N ) and c = (c B ; c N ). Thus a basic feasible solution (BFS) is one where x = (x B ; x N = 0) The Simplex Algorithm states that one should find an extreme point and check if it is optimal. If it is not it should move on to one of its neighboring extreme points. A neighboring extreme point has the same basic/nonbasic structure except that one of basic and nonbasic coordinates are swapped. For example x 1 and x 2 are neighbors where represents any nonbasic coordinate. x 1 = (,, 0, 0, 0,, 0,, 0, 0, 0, 0) x 2 = (0,,, 0, 0,, 0,, 0, 0, 0, 0) We can then rewrite the equations as follows: Ax = b [B, N [ xb x N = b (3) Suppose that we know x N. Then the equation Bx B + Nx N = b can be solved for x B : x B = B 1 b B 1 Nx N The value of the objective function is z = c T x. But: Thus z = c T x = c B T x B + c N T x N = c B T B 1 (b Nx N ) + c N T x N z = c B T B 1 b + ( c N T c B T B 1 N ) x N. 6

4 THE SIMPLEX ALGORITHM (IN MATRIX FORM) Review of Linear Programming In particular, at a BFS: x N = 0 and z = c B T B 1 b How can we determine that x B is optimal? Since we are minimizing, then, if some entry, say j of r N = c N T c B T B 1 N is negative, by replacing x j = 0 with x j = α for some α > 0 we can improve the objective. In essence we are replacing x N which is zero with αe j where 0. e j = 1. 0 with 1 in the j th entry. (The vector r is the last row of the tableau when you solve LP by hand.) The larger α the larger is the decrease in the objective function. However, we may not be able to increase α indefinitely. While we are increasing the non basic variable x j we must maintain feasibility, that is Bx b + αne j = b or equivalently x B + αb 1 Ne j = B 1 b. Thus x B will change as α changes. Let us call the matrix T def = B 1 N, and its j th column t j. (T is what you get in the tableau when you are solving linear programs by hand.) x B + αt j = B 1 b Thus so far, we have determined that since r j < 0 we can increase the j th component of x N, that is this component is entering the basis. Now we must determine which component of x B has to leave (become zero). Since x B + αt j = B 1 b must always be true as we increase α, we must adjust components of x B. Let us look at it component by component: (x B ) i + αt ij = (B 1 b) i where T ij is the i, j entry of T (that is i th entry of t j.) Now we have two distinct cases: 1. if T ij 0 then by increasing α we must increase (x B ) i to keep the balance; therefore in this case there are no problems. 2. If T ij > 0 then by increasing α we must decrease (x B ) i to keep the balance. We must however be careful not to let (x B ) i to fall below zero and become infeasible. This means that α (x B ) i /T ij. Since α (x B ) i /T ij should hold for all i where T ij < 0 we can at most set α to α = (x B ) i min {i:t ij 0} T ij If this minimum is attained at index k then the k th entry of x B has to leave the basis. 7

4 THE SIMPLEX ALGORITHM (IN MATRIX FORM) Review of Linear Programming Example 6 (From basic indices to basic solution) Consider the following LP: min 2x 1 3x 2 2x 3 + x 4 x 1 + x 3 + 6x 4 = 2 x 2 + 2x 3 + x 4 = 3 x {1,2,3} 0 Let us assume that some how we have found that x T B = (x 1, x 2 ) Then [ [ [ [ 1 0 1 6 2 2 B =, N =, c 0 1 2 1 B =, c 3 N = 1, x B = B 1 b = [ 2 3, and x N = 0. Now r N = c N T c B T B 1 N = [ 2 8, We can organize this information in tableau as you have learned from your LP courses: x 1 x 2 x 3 x 4 B 1 b 1 0 1 6 2 0 1 2 1 3 0 0 2-8 5 Since r 4 < 0 we decide that by bringing x 4 to the basis we can improve the objective. To decide which basic variable should leave we check the largest value α can be, and which basic variable is responsible for it: α = min{2/6, 3/1} = 2 6 = 1 3. Thus x 1 has to leave the basis and x 4 enters leading to the new tableau: x 1 x 2 x 3 x 4 B 1 b -0.167 0 1.83 1 2.67 0.167 1 0.167 0 0.33 1.33 0 3.33 0 7.67 Can we improve the solution to the above example any further? The answer is no. To show this we need duality theory. 8

5 DUALITY Review of Linear Programming 5 Duality Consider the linear programming in standard form again: min s.t. c T x Ax = b x 0 (4) We will refer to it as the primal problem. Define the Dual of this LP as: max s.t. b T y A T y c (5) The constraint A T y c may be written as A T y + s = c where s 0 are some times called slack variables. The duality theory indicates that the values of the optimal solution in the first and second problems above are equal. Lemma 5 (Weak duality:) If x is feasible for the primal and (y, s) is feasible for the dual, then b T y c T x Proof: c T x b T y = c T x (Ax) T y = x T c x T A T y = x T (c A T y) = x T s 0 The last inequality holds because both x and s consist of nonnegative entries and thus their inner product has to be nonnegative. This simple lemma reveals a few key points. First for any primal-feasible x and dual feasible (y, s) the quantity x T s is called the duality gap.second, if for some primal-feasible x and dual-feasible (y, s) we have x T s = 0 then, necessarily, x is optimal for the primal and (y, s) is optimal for the dual. Some algorithms maintain feasible primal and dual solutions and strive to improve the solutions by reducing then duality gap. Now the question is that does every linear program has duality gap zero at the optimum? The answer is yes if both primal and dual are feasible. Theorem 5 (Strong Duality Theorem:) If there exists an optimal primal x and an optimal dual y, then b T y = c T x 9

5 DUALITY Review of Linear Programming 5.1 Table of Duality Proof: Suppose we apply the simplex method and after some iterations we come to the point where the last row of the tableau is all nonnegative that is c T N ct B B 1 N 0. We will prove that we are at the optimum. Recall that x = (x B, x N ) with x N = 0. Set y T = c T B B 1. claim: y is dual feasible solution and optimal. We first show that it is feasible that is c A T y 0 c A T y = [ cb c N [ B T N T y = [ cb B T y c N N T y = [ cb B T ( B T c B ) c N N T ( B T c B ) Thus y is dual feasible. To show that it is optimal we prove y T b = c T x: y T b = c B T B 1 b = c B T x B = c T x noting that x N = 0 the last equality is justified. [ 0 = r N 0 Note 2 In the simplex method, at each iteration we have access to the vector y T def = c T B B 1. This vector can be plugged into the dual, but it will not satisfy the constraints: In the proof we showed that c T y T A = (0, r T N ) which is not nonnegative in general. However at the optimum y will be dual feasible and the objective value is y T b = c T B B 1 b = z, the same value as the primal objective. Therefore y T = c T B B 1 is dual optimal solution. Theorem 5 (Complementary Slackness Theorem:) If x is the primal optimal solution and (y, s) is the dual optimal solution, then x i s i = 0 for i = 1... n. Proof: At the optimal solution: c T x = b T y c T x b T y = x T s = 0 Σx i s i = 0. But because x i 0 and s i 0 for all i i x is i = 0 implies x i s i = 0 for each and every i. Another way of expressing the complementary slackness theorem is the following: If the i th inequality in the dual is strict (a T i y < c i) then the corresponding x i = 0. If x i > 0 that is x i is basic then the corresponding inequality in the dual is satisfied with equality: a T i y = c i. 5.1 Table of Duality All linear programs have a corresponding dual and not just the ones in the standard form. In general every constraint on an LP corresponds to a variable on its dual and every variable corresponds to a constraint on the dual. In addition the right hand side vector becomes the objective vector in the dual and the objective vector goes to the right hand side. To better understand how the constraints in the primal effect the variables in the dual and how the variables in the primal effect the constraints in the dual. one can refer to the following table: 10

5 DUALITY Review of Linear Programming 5.2 Dual Simplex Method MIN MAX x i 0 C V O A x i 0 N R S x i unconstrained = T C x i 0 O V N x i 0 A S R T = x i unconstrained Example 7 (Arbitrary dual) The dual thus becomes: min 2x 1 3x 2 + x 3 x 4 s.t. x 1 + x 3 = 5 (y 1 ) x 1 2x 2 + x 3 4 (y 2 ) x 2 + x 3 x 4 6 (y 3 ) x 1 0, x 2 0 max 5y 1 + 4y 2 + 6y 3 s.t. y 1 + y 2 2 2y 2 + y 3 3 (x 1 ) y 1 + y 2 + y 3 = 1 (x 2 ) y 3 = 1 (x 3 ) y 2 0, y 3 0 5.2 Dual Simplex Method In the simplex method discussed in section 4 at each iteration we had an x which was basic and feasible (that is x 0) but not optimal (that is r N 0). The simplex method maintains feasibility while pushing toward optimality. Now let us view the simplex method from the point of view of the dual problem. Remember that each constraint in A T y c corresponds to a coordinate of x. At the optimum by complementary slackness theorem if x i > 0 then the i th constraint is satisfied with equality: a T i y = c i. Now at each iteration of simplex we have access to vector y T = c T B B 1. This vector in general is not feasible for the dual because, as we showed in the proof of the duality theorem 5 c T y T A = (c T B c T BB 1 B, c T N c T BB 1 N) = (0, r T N) 11

5 DUALITY Review of Linear Programming 5.2 Dual Simplex Method and since, if we are not at the optimum, then r N does not have all nonnegative components, A T y c is not satisfied. However, b T y = c T BB 1 b = c T bx B = z That is the objective function at this dual infeasible point is the same as the value of the objective function at the primal. Thus, from the point of view of the dual problem, the simplex method moves from one infeasible point y to another, at each iteration maintaining optimality condition (that is primal and dual values are equal) and attempting to get feasible (if r N 0 then y will be feasible.) This suggests an idea. Let us turn the tables in the primal problem. Suppose that we have a solution x, still basic (that is x = (x B, x N ) with x N = 0), but not necessarily feasible (that is x B 0.) Assume also that the optimality condition holds that is r T N = ct N ct B B 1 N 0. We are going to describe an algorithm that moves from one basic infeasible point to another neighboring one, at each iteration maintaining the optimality condition c T N yt N 0, and moving toward a point that is feasible; if we find such a point then it is primal-optimal. The strategy leads to the dual simplex method. In this method, we first choose the coordinate that has to leave the basis. This is easy enough: choose a coordinates i where (x B ) i < 0. Now we must decide which coordinate has to enter the basis. By removing i from the basis we are removing the i th row of the matrix B. Since y T = c T B B 1 this implies that y T new y T α(b 1 ) i. where in general A i. means the i th row of matrix A. However, we must maintain the optimality condition: c T N yt newn 0. That is: c T N ( y T α(b 1 ) i. ) N 0 c T N y T N + α(b 1 ) i. N 0 r T N + αt i. 0 Now let us look at the j th column of this inequality: r j + αt ij 0 If T ij 0 then this does not impose any condition on α: as α increases r j + αt ij remains positive. But if T ij < 0 we can increase α to at most r j /T ij, beyond that r j will become negative and optimality condition is violated. Therefore, α = min j { r j T ij : T ij < 0 The index j at which this minimum is achieved is the one that enters the basis. } 12

5 DUALITY Review of Linear Programming 5.2 Dual Simplex Method Example 8 (Dual simplex method) Consider the LP: min 3x 1 + 4x 2 + 5x 3 s.t. x 1 + 2x 2 + 3x 3 5 2x 1 + 2x 2 + x 3 6 x 1, x 2, x 3 0 which in standard form is min 3x 1 + 4x 2 + 5x 3 s.t. x 1 2x 2 3x 3 + x 4 = 5 2x 1 2x 2 x 3 + x 5 = 6 x 1, x 2, x 3, x 4, x 5 0 Now if we choose {4, 5} to be the indices of basic variables, then ( ) ( ) 1 0 1 2 3 B = B 1 =, N = 0 1 2 2 1 c T B = (0, 0), c T N = (3, 4, 5) x T B = ( 5, 4) y T = c T BB 1 = c T B = (0, 0) r T N = c T N y T N = c T N = (3, 4, 5) Clearly this solution is basic and infeasible, yet the optimality condition r N 0 holds. Notice that y is dual feasible, check it. This information can be organized in a tableau: x 1 x 2 x 3 x 4 x 5 B 1 b -1-2 -3 1 0-5 -2-2 -1 0 1-6 3 4 5 0 0 0 If we choose x 5 to leave the basis, then smallest r j / T ij is min{ 3/ 2, 4/ 2, 5/ 1} = 3/2 attained at index 1. Thus x 1 should enter the basis. Pivoting on the squared entry of the tableau we get: x 1 x 2 x 3 x 4 x 5 B 1 b 0-1 -5/2 1-1/2-2 1 1 1/2 0-1/2 3 0 1 7/2 0 3/2 9 Now we choose x 4 to leave the basis, and min{1/1, 7/2 the basis. Pivoting on the boxed element we get: 5/2, 3/2 x 1 x 2 x 3 x 4 x 5 B 1 b 0 1 1/2-1 -1/2 2 1 0-2 1-1 1 0 0 1 1 1 11 1/2 } = 1, thus we choose x 2 to enter Now we have both feasibility and optimality satisfied, and we have the solution. 13