IE 5531: Engineering Optimization I Lecture 5: The Simplex method, continued Prof. John Gunnar Carlsson September 22, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 1 / 32
Administrivia Lecture slides 4,5 posted http://www.tc.umn.edu/~jcarlsso/syllabus.html PS 1 posted this evening Xi Chen's oce hours: Tuesdays 10:00-12:00, ME 1124, Table B Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 2 / 32
Today: The Simplex Method, continued Comment on convexity Tableaus Gauss-Jordan pivoting Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 3 / 32
Absolute values The problem minimize n i=1 c i x i s.t. Ax b is easy only if c 0 The distinction is that n i=1 c i x i is a convex function in x Minimizing a convex function is easy, as is maximizing a concave function, but minimizing a concave function is generally dicult Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 4 / 32
Basic feasible solution Consider a polyhedron dened by {x : Ax = b, x 0} where A is an m n matrix and b R n The corner points are all characterized by taking m linearly independent columns, denoted by the indices B (the basic set), from A, solving A B x B = b and setting all other variables x N to 0 If all entries x B 0, then x is called a basic feasible solution (BFS), equivalently a vertex or corner point Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 5 / 32
Neighboring basic solutions Two basic solutions are neighboring or adjacent if they dier by exactly one basic (or nonbasic) variable A basic feasible solution is optimal if no better neighboring feasible solution exists Given a basic set B, the simplex method looks at all neighboring BFS's and nds a better one, if it exists Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 6 / 32
LP canonical form A standard-form LP is said to be in canonical form at a basic feasible solution if the objective coecients to all the basic variables are zero the constraint matrix for the basic variables form an identity matrix (with some permutation if necessary) If the LP is in canonical form, then it's easy to tell if the current BFS is optimal Thus, given a basic set B, we want to transform the LP into canonical form Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 7 / 32
An equivalent LP Given a basic set B, we nd that an equivalent problem to {minimize x : Ax = b, x 0} is minimize r T x s.t. Āx = b x 0 where r B = 0, r N = c N A T N ) (A 1 T c B B, Ā = A 1 A, b = A 1 B B b Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 8 / 32
Optimality test If r N 0 (equivalently r 0) at a BFS with basic variable set B, then the BFS is an optimal basic solution and A B is an optimal basis Note that the alternative linear program is in canonical form, so we can tell if we're optimal or not Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 9 / 32
Example Consider the example minimize x 1 +2x 2 +3x 3 x 4 s.t. x 1 +x 3 =1 x 2 +x 4 =1 x 1 +x 2 +x 5 =1.5 x 1, x 2, x 3, x 4, x 5 0 We set B = {1, 2, 3} so that x = (0.5, 1, 0.5, 0, 0), since 1 0 1 0 1 1 A B = 0 1 0 ; A 1 = 0 1 0 B 1 1 0 1 1 1 and therefore r N = c N A T N optimal ( ) (A 1 T 6 c B B = 3 ) ; this is not Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 10 / 32
Simplex tableau While performing the simplex algorithm, we maintain a simplex tableau that organizes the intermediate canonical form data: B r T c T B b basis indices Ā b Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 11 / 32
Simplex tableau example The problem minimize x 1 2x 2 s.t. x 1 +x 3 = 1 x 2 + x 4 = 1 x 1 + x 2 +x 5 = 1.5 x 1,x 2, x 3,x 4, x 5 0 has the following tableau for B = {3, 4, 5}: B 1 2 0 0 0 0 3 1 0 1 0 0 1 4 0 1 0 1 0 1 5 1 1 0 0 1 1.5 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 12 / 32
Finding a better neighbor point If one of the indices of r is negative, our basic set is not optimal We make an eort to nd a better neighboring basic solution (that diers by the current basic solution by exactly one basic variable), as long as the reduced cost coecient of the entering variable is negative Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 13 / 32
Changing basis With B = {3, 4, 5}, B 1 2 0 0 0 0 3 1 0 1 0 0 1 4 0 1 0 1 0 1 5 1 1 0 0 1 1.5 Try inserting variable x 1 into the basic set; the constraint says 1 0 1 0 0 0 x1 + 1 x2 + 0 x3 + 1 x4 + 0 x5 = }{{} 1 1 0 0 0 1 i.e. x 3 x 4 x 5 = 1 1 1.5 1 0 1 x1 1 1 1.5 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 14 / 32
Minimum ratio test The question is: how much can we increase x 1 while the current basic variable remain feasible (non-negative)? This is easy to gure out with the minimum ratio test (MRT): 1 Select the entering variable x e with reduced cost r e < 0 2 If Ā e 0 then the problem is unbounded 3 The MRT: What does θ represent? θ = min { bi Ā ie : Ā ie > 0 } Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 15 / 32
Minimum ratio test θ represents the largest amount that x e can be increased before one (or more) of the current basic variables x i becomes zero (and leaves the feasible set) Suppose that the minimum ratio is attained by one unique basic variable index o. Then x e is the entering basic variable and x o is the out-going basic variable: x o = b o ā oe θ = 0 x i = b i ā ie θ > 0 i 0 Thus the new basic set contains x e and drops x o Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 16 / 32
Tie breaking If the MRT does not give a single index, but instead a set of one or more, we choose one of these arbitrarily We say that the new basic feasible solution is degenerate because some of the basic variables x B just happen to be 0 We'll deal with this later; for now, we can just pretend that the degeneracies are actually ɛ > 0 and continue Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 17 / 32
The simplex algorithm Initialize the simplex algorithm with a feasible basic set B, so that x B 0. Let N be the remaining indices. Write the simplex tableau. 1 Test for termination. Find r e = min {r j } j N If r e 0, the solution is optimal. Otherwise, determine whether the column of Ā e contains a positive entry. If not, the objective function is unbounded below. Otherwise, let x e be the entering basic variable 2 Determine the outgoing variable. Use the MRT to determine the outgoing variable x o. 3 Update the basic set. Update B and A B and transform the problem to canonical form. Return to step 1. Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 18 / 32
Expanded simplex tableau B basis indices c ( ) A T A 1 T c B B c T b A 1 A B A 1 B b B A 1 B Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 19 / 32
Re-computing the tableau One arduous step in the preceding example is the re-computation of the tableau for every new basic set B We can bypass this step using Gauss-Jordan elimination, in much the same way as one solves a system of linear equations Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 20 / 32
Gauss-Jordan elimination Given a simplex tableau, an outgoing variable x o, and an entering variable x e, 1 Divide all the entries in the row corresponding to x o by element ā oe (the pivot element), so that ā oe 1. 2 For all i o, modify all other entries in the usual Gauss Jordan process: ā ij ā ij āoj ā oe ā ie 3 Modify the right-hand side and the objective function row in the same way. The above procedure allows us to nd the optimal basic set without computing A 1 at every step. B Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 21 / 32
Example Consider the problem (not in standard form) minimize 3x 1 + 4x 2 s.t. x 1 + x 2 4 2x 1 + x 2 5 x 1, x 2 0 We re-write this in standard form: minimize 3x 1 4x 2 s.t. x 1 +x 2 +x 3 =4 2x 1 +x 2 + x 4 =5 x 1, x 2, x 3, x 4 0 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 22 / 32
Example minimize 3x 1 4x 2 s.t. x 1 +x 2 +x 3 =4 2x 1 +x 2 + x 4 =5 x 1, x 2, x 3, x 4 0 The rst step is to choose a basic set B, and then build Ā = A 1 A B and so forth in the tableau. What's a good choice of B? Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 23 / 32
Example minimize 3x 1 4x 2 s.t. x 1 +x 2 +x 3 =4 2x 1 +x 2 + x 4 =5 x 1, x 2, x 3, x 4 0 The rst step is to choose a basic set B, and then build Ā = A 1 A B and so forth in the tableau. What's a good choice of B? If we set B = {3, 4}, then we nd that A B is just the identity matrix, so Ā = A! If we are given a problem of the form minimize x : Ax b, where b 0, this is always a good way to start Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 23 / 32
Tableau (I) B 3 4 0 0 0 (II) 3 1 1 1 0 4 (III) 4 2 1 0 1 5 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 24 / 32
Tableau (I) B 3 4 0 0 0 (II) 3 1 1 1 0 4 (III) 4 2 1 0 1 5 The canonical form shows us that this is not optimal. Pick x 1 as an entering basic variable Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 24 / 32
Tableau (I) B 3 4 0 0 0 (II) 3 1 1 1 0 4 (III) 4 2 1 0 1 5 The canonical form shows us that this is not optimal. Pick x 1 as an entering basic variable Using MRT, we see that 4/1 < 5/2, so x 4 will leave the basic set Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 24 / 32
Tableau (I) B 3 4 0 0 0 (II) 3 1 1 1 0 4 (III) 4 2 1 0 1 5 The canonical form shows us that this is not optimal. Pick x 1 as an entering basic variable Using MRT, we see that 4/1 < 5/2, so x 4 will leave the basic set Next, we do Gauss-Jordan elimination: 1 Divide (III) by 2 2 Add 3 (III) to (I) 3 Subtract (III) from (II) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 24 / 32
Tableau (I) B 0 5/2 0 3/2 15/2 (II) 3 0 1/2 1 1/2 3/2 (III) 1 1 1/2 0 1/2 5/2 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 25 / 32
Tableau (I) B 0 5/2 0 3/2 15/2 (II) 3 0 1/2 1 1/2 3/2 (III) 1 1 1/2 0 1/2 5/2 The canonical form shows us that this is not optimal. Pick x 2 as an entering basic variable Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 25 / 32
Tableau (I) B 0 5/2 0 3/2 15/2 (II) 3 0 1/2 1 1/2 3/2 (III) 1 1 1/2 0 1/2 5/2 The canonical form shows us that this is not optimal. Pick x 2 as an entering basic variable Using MRT, we see that 3/2 1/2 < 5/2 1/2, so x 3 will leave the basic set Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 25 / 32
Tableau (I) B 0 5/2 0 3/2 15/2 (II) 3 0 1/2 1 1/2 3/2 (III) 1 1 1/2 0 1/2 5/2 The canonical form shows us that this is not optimal. Pick x 2 as an entering basic variable Using MRT, we see that 3/2 1/2 < 5/2 1/2, so x 3 will leave the basic set Next, we do Gauss-Jordan elimination: 1 Multiply (II) by 2 2 Add (5/2) (III) to (I) 3 Subtract (1/2) (II) from (III) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 25 / 32
Tableau (I) B 0 0 5 1 15 (II) 2 0 1 2 1 3 (III) 1 1 0 1 1 1 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 26 / 32
Tableau (I) B 0 0 5 1 15 (II) 2 0 1 2 1 3 (III) 1 1 0 1 1 1 The canonical form shows us that this is not optimal. Pick x 4 as an entering basic variable Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 26 / 32
Tableau (I) B 0 0 5 1 15 (II) 2 0 1 2 1 3 (III) 1 1 0 1 1 1 The canonical form shows us that this is not optimal. Pick x 4 as an entering basic variable Using MRT, we see that x 1 will leave the basic set (since 3/ ( 1) < 0) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 26 / 32
Tableau (I) B 0 0 5 1 15 (II) 2 0 1 2 1 3 (III) 1 1 0 1 1 1 The canonical form shows us that this is not optimal. Pick x 4 as an entering basic variable Using MRT, we see that x 1 will leave the basic set (since 3/ ( 1) < 0) Next, we do Gauss-Jordan elimination: 1 Add (III) to (I) 2 Add (III) to (II) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 26 / 32
Tableau (I) B 1 0 4 0 16 (II) 2 1 1 1 0 4 (III) 4 1 0 1 1 1 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 27 / 32
Tableau (I) B 1 0 4 0 16 (II) 2 1 1 1 0 4 (III) 4 1 0 1 1 1 The canonical form shows us that this is optimal, so we're done The optimal basic set is B = {2, 4} and that x 2 = 4 and x 4 = 1 The objective function value is 16 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 27 / 32
Another example Consider the problem minimize 10x 1 12x 2 12x 3 s.t. x 1 + 2x 2 + 2x 3 20 2x 1 + x 2 + 2x 3 20 2x 1 + 2x 2 + x 3 20 x 1, x 2, x 3 0 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 28 / 32
Another example After introducing slack variables x 4, x 5, x 6 and setting B = {4, 5, 6}, the initial tableau is B 10 12 12 0 0 0 0 4 1 2 2 1 0 0 20 5 2 1 2 0 1 0 20 6 2 2 1 0 0 1 20 Pivot at (say) x 1, so x 5 leaves (x 6 does too, but don't worry about that) Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 29 / 32
Another example After exchanging B 0 7 2 0 5 0 100 4 0 3/2 1 1 1/2 0 10 1 1 1/2 1 0 1/2 0 10 6 0 1 1 0 1 1 0 Pivot at (say) x 3 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 30 / 32
Another example After exchanging Pivot at x 2, x 6 leaves B 0 4 0 2 4 0 120 3 0 3/2 1 1 1/2 0 10 1 1 1 0 1 1 0 0 6 0 5/2 0 1 3/2 1 10 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 31 / 32
Another example After exchanging and we're done B 0 0 0 18/5 8/5 8/5 136 3 0 0 1 2/5 2/5 3/5 4 1 1 0 0 3/5 2/5 2/5 4 2 0 1 0 2/5 3/5 2/5 4 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 22, 2010 32 / 32