The Simplex Algorithm: Technicalities 1

Transcription

1 1/45 The Simplex Algorithm: Technicalities 1 Adrian Vetta 1 This presentation is based upon the book Linear Programming by Vasek Chvatal

2 2/45 Two Issues Here we discuss two potential problems with the simplex method and how to avoid them. Termination. How can we ensure that the algorithm terminates? Initialialisation. How can we ensure that the algorithm starts?

3 3/45 Part I Termination: Cycling

4 4/45 Choosing Pivots Entering Variable. We may chose any non-basic variable with a positive coefficient in the top row of the dictionary to enter the basis. If there is more than one choice to enter, which do we pick? Leaving Variable. We may chose any basic variable whose non-negativity imposes the most stringent constraint on the entering variable to leave the basis. If there is more than one choice to leave, which do we pick?

5 5/45 Degeneracy Leaving Variable. We may chose any basic variable whose non-negativity imposes the most stringent constraint on the entering variable to leave the basis. If two basic variables x i, x j can be selected to leave the basis then their values are both forced to zero by the entering variable. Since one of them, say x i, must stay in the basis, we now have a basic variable whose value is zero. [Its constant term in the dictionary is 0.] Basic solutions with some basic variables of value zero are called degenerate. For a pivot at a degenerate solution, the objective value will not increase, nor will any of the variable values change.

6 6/45 Cycling So pivots at degenerate solutions do not lead to better solutions. Is it possible that the algorithm can get stuck (before it finds an optimal solution) in a sequence of degenerate solutions? YES. - If we are not careful in how we choose our pivots.

7 /45 An Example of Cycling Assume we use the following rules for selecting pivots: Entering Variable. Add the non-basic variable with the highest positive coefficient. Leaving Variable. In case of a tie, remove the basic variable with the smallest subscript. Apply these rules to the following example. maximise 10x 1 57x 2 9x 3 24x 4 1 subject to 2 x x x 3 + 9x x x x 3 + x 4 0 x 1 1 x 1, x 2, x 3, x 4 0

8 8/45 Dictionary Form maximise 10x 1 57x 2 9x 3 24x subject to x1 2 2 x2 5 x3 + 9x x1 3 2 x2 1 x3 + x4 0 2 x 1 1 x 1, x 2, x 3, x 4 0 Write in dictionary form. z = 10x 1 57x 2 9x 3 24x 4 x 5 = 1 2 x x x 3 9x 4 x 6 = 1 2 x x x 3 x 4 x 7 = 1 x 1

9 9/45 Pivot 1 z = 10x 1 57x 2 9x 3 24x 4 x 5 = 1 2 x x x 3 9x 4 x 6 = 1 2 x x x 3 x 4 x 7 = 1 x 1 Pivot: Add x 1 to basis. (We had no choice.) Remove x 5 from basis. (We had the choice of x 5 or x 6 ). z = 10(11x 2 + 5x 3 18x 4 2x 5) 57x 2 9x 3 24x 4 x 1 = 11x 2 + 5x 3 18x 4 2x 5 x 6 = 1 2 (11x2 + 5x3 18x4 2x5) x2 + 1 x3 2 x 7 = 1 (11x 2 + 5x 3 18x 4 2x 5) x4 z = 53x x 3 204x 3 20x 5 x 1 = 11x 2 + 5x 3 18x 4 2x 5 x 6 = 4x 2 2x 3 + 8x 4 + x 5 x 7 = 1 11x 2 5x x 4 + 2x 5

10 10/45 Pivot 2 z = 53x x 3 204x 3 20x 5 x 1 = 11x 2 + 5x 3 18x 4 2x 5 x 6 = 4x 2 2x 3 + 8x 4 + x 5 x 7 = 1 11x 2 5x x 4 + 2x 5 Pivot: Add x 2 to basis. (It has a higher coefficient than x 3.) Remove x 6 from basis. (We had no choice; x 1 is unconstrained by x 2 ). z = 53( 1 2 x3 + 2x x5 1 x6) + 41x3 204x3 20x5 4 1 x 2 = 2 x3 + 2x x5 1 4 x6 x 1 = 11( 1 2 x3 + 2x x5 1 x6) + 5x3 18x4 2x5 4 x 7 = 1 11( 1 2 x3 + 2x x5 1 x6) 5x3 + 18x4 + 2x5 4 z = 29 2 x 3 98x x x 6 x 2 = 1 2 x 3 + 2x x x 6 x 1 = 1 2 x 3 + 4x x x 6 x 7 = x 3 4x x x 6

11 1/45 Pivot z = x3 98x4 2 4 x x6 x 2 = 1 2 x3 + 2x x5 1 4 x6 x 1 = 1 2 x3 + 4x x x6 x 7 = x3 4x4 3 4 x x6 Pivot: Add x 3 to basis. (No choice.) Remove x 1 from basis. (We had the choice of choice of x 1 or x 2.) 29 z = 2 (8x x x6 2x1) 98x x5 4 4 x6 x 3 = 8x x5 11 x6 2x1 2 x 2 = 1 2 (8x x x6 2x1) + 2x x5 1 4 x6 x 7 = (8x x x6 2x1) 4x x x6 z = 18x x 5 93x 6 29x 1 x 3 = 8x x x 6 2x 1 x 2 = 2x x x 6 + x 1 x 7 = 1 x 1

12 12/45 Pivot 4 z = 18x x 5 93x 6 29x 1 x 3 = 8x x5 11 x6 2x1 2 x 2 = 2x x5 + 5 x6 + x1 2 x 7 = 1 x 1 Pivot: Add x 4 to basis. (It has a higher coefficient than x 5.) Remove x 2 from basis. (No choice as x 3 unconstrained by x 4.) z = 18( 1 4 x x x1 1 x2) + 15x5 93x6 29x1 2 1 x 4 = 4 x x x1 1 2 x2 x 3 = 8( 1 4 x x x1 1 x2) x5 11 x6 2x1 2 x 7 = 1 x 1 21 z = 2 x x 6 20x 1 9x 2 x 4 = 1 4 x x x x 2 x 3 = 1 2 x x 6 + 2x 1 4x 2 x 7 = 1 x 1

13 13/45 Pivot z = x 4 = 1 4 x x x1 1 2 x2 141 x5 x6 20x1 9x2 2 x 3 = 1 2 x x 7 = 1 x 1 x6 + 2x1 4x2 Pivot: Add x 5 to basis. (No choice.) Remove x 3 from basis. (We had the choice of x 3 or x 4.) z = (9x6 + 4x1 8x2 2x3) x6 20x1 9x2 2 2 x 5 = 9x 6 + 4x 1 8x 2 2x 3 x 4 = 1 4 (9x6 + 4x1 8x2 2x3) x x1 1 2 x2 x 7 = 1 x 1 z = 24x x 1 93x 2 21x 3 x 5 = 9x 6 + 4x 1 8x 2 2x 3 x 4 = x x x 2 + x 7 = 1 x x 3

14 14/45 Pivot 6 z = 24x x 1 93x 2 21x 3 x 5 = 9x 6 + 4x 1 8x 2 2x 3 x 4 = x x x2 + x 7 = 1 x x3 Pivot: Add x 6 to basis. (It has a higher coefficient than x 1.) Remove x 4 from basis. (No choice as x 5 unconstrained by x 6.) z = 24( 1 2 x x2 + 1 x3 x4) + 22x1 93x2 21x3 2 1 x 6 = 2 x x2 + 1 x3 x4 2 x 5 = 9( 1 2 x x2 + 1 x3 x4) + 4x1 8x2 2x3 2 x 7 = 1 x 1 z = 10x 1 57x 2 9x 3 24x 4 x 6 = 1 2 x x x 3 x 4 x 5 = 1 2 x x x 3 9x 4 x 7 = 1 x 1

15 15/45 A Cycle is Obtained z = 10x 1 57x 2 9x 3 24x 4 x 6 = 1 2 x x2 + 1 x3 x4 2 x 5 = 1 2 x x2 + 5 x3 9x4 2 x 7 = 1 x 1 But this is exactly the same as the original dictionary! z = 10x 1 57x 2 9x 3 24x 4 x 5 = 1 2 x x x 3 9x 4 x 6 = 1 2 x x x 3 x 4 x 7 = 1 x 1 We are stuck in a cycle (with top row coefficients not all negative). So the simplex algorithm did not terminate!

16 16/45 Part II Bland s Rule

17 17/45 Bland s Rule Can we stop the algorithm from cycling? YES. - If we are more careful in how we choose the entering and leaving variables. Bland s Rule. Given a choice, always choose the variable with the smallest subscript (for both entering and leaving variables).

18 18/45 Bland s Rule: Pivot 1 Let s see how this works with the previous example. z = 10x 1 57x 2 9x 3 24x 4 x 5 = 1 2 x x x 3 9x 4 x 6 = 1 2 x x x 3 x 4 x 7 = 1 x 1 Pivot: Add x 1 to basis. (We had no choice.) Remove x 5 from basis. (It has a lower subscript than x 6.) z = 53x x 3 204x 3 20x 5 x 1 = 11x 2 + 5x 3 18x 4 2x 5 x 6 = 4x 2 2x 3 + 8x 4 + x 5 x 7 = 1 11x 2 5x x 4 + 2x 5

19 19/45 Bland s Rule: Pivot 2 z = 53x x 3 204x 3 20x 5 x 1 = 11x 2 + 5x 3 18x 4 2x 5 x 6 = 4x 2 2x 3 + 8x 4 + x 5 x 7 = 1 11x 2 5x x 4 + 2x 5 Pivot: Add x 2 to basis. (It has a lower subscript than x 3.) Remove x 6 from basis. (We had no choice; x 1 is unconstrained by x 2.) z = 29 2 x 3 98x x x 6 x 2 = 1 2 x 3 + 2x x x 6 x 1 = 1 2 x 3 + 4x x x 6 x 7 = x 3 4x x x 6

20 0/45 Bland s Rule: Pivot z = x3 98x4 2 4 x x6 x 2 = 1 2 x3 + 2x x5 1 4 x6 x 1 = 1 2 x3 + 2x x x6 x 7 = x3 4x4 3 4 x x6 Pivot: Add x 3 to basis. (No choice.) Remove x 1 from basis. (It has a lower subscript x 2.) z = 18x x 5 93x 6 29x 1 x 3 = 8x x x 6 2x 1 x 2 = 2x x x 6 + x 1 x 7 = 1 x 1

21 21/45 Bland s Rule: Pivot 4 z = 18x x 5 93x 6 29x 1 x 3 = 8x x x6 9x1 2 x 2 = 2x x5 + 5 x6 + x1 2 x 7 = 1 x 1 Pivot: Add x 4 to basis. (It has a lower subscript than x 5.) Remove x 2 from basis. (No choice as x 3 unconstrained by x 4.) 21 z = 2 x x 6 20x 1 9x 2 x 4 = 1 4 x x x x 2 x 3 = 1 2 x x 6 + 2x 1 4x 2 x 7 = 1 x 1

22 22/45 Bland s Rule: Pivot z = x 4 = 1 4 x x x1 1 2 x2 141 x5 x6 20x1 9x2 2 x 3 = 1 2 x x 7 = 1 x 1 x6 + 2x1 4x2 Pivot: Add x 5 to basis. (No choice.) Remove x 3 from basis. (It has a lower subscript than x 4.) z = 24x x 1 93x 2 21x 3 x 5 = 9x 6 + 4x 1 8x 2 2x 3 x 4 = x x x 2 + x 7 = 1 x x 3

23 23/45 Bland s Rule: Pivot 6 z = 24x x 1 93x 2 21x 3 x 5 = 9x 6 + 4x 1 8x 2 2x 3 x 4 = x x x2 + x 7 = 1 x x3 So far all our pivots has been the same as before, but now... Pivot: Add x 1 to basis. (It has a lower subscript than x 6.) Remove x 4 from basis. (No choice as x 5 unconstrained by x 6.) z = 24x ( 2x 6 + 3x 2 + x 3 2x 4) 93x 2 21x 3 x 1 = 2x 6 + 3x 2 + x 3 2x 4 x 5 = 9x 6 + 4( 2x 6 + 3x 2 + x 3 2x 4) 8x 2 2x 3 x 7 = 1 ( 2x 6 + 3x 2 + x 3 2x 4) z = 20x 6 27x 2 + x 3 44x 4 x 1 = 2x 6 + 3x 2 + x 3 2x 4 x 5 = x 6 + 4x 2 + 2x 3 8x 4 x 7 = 1 + 2x 6 3x 2 x 3 2x 4

24 24/45 Bland s Rule: Pivot 7 z = 20x 6 27x 2 + x 3 44x 4 x 1 = 2x 6 + 3x 2 + x 3 2x 4 x 5 = x 6 + 4x 2 + 2x 3 8x 4 x 7 = 1 + 2x 6 3x 2 x 3 2x 4 Pivot: Add x 3 to basis. (No choice.) Remove x 7 from basis. (No choice as x 1, x 5 unconstrained by x 3.) z = 20x 6 27x 2 + (1 + 2x 6 3x 2 2x 4 x 7) 44x 4 x 3 = 1 + 2x 6 3x 2 2x 4 x 7 x 1 = 2x 6 + 3x 2 + (1 + 2x 6 3x 2 2x 4 x 7) 2x 4 x 5 = x 6 + 4x 2 + 2(1 + 2x 6 3x 2 2x 4 x 7) 8x 4 z = 1 18x 6 30x 2 46x 4 x 7 x 3 = 1 + 2x 6 3x 2 2x 4 x 7 x 1 = 1 4x 4 x 7 x 5 = 2 + 3x 6 2x 2 12x 4 2x 7

25 25/45 Bland s Rule: Breaking the Cycle z = 1 18x 6 30x 2 46x 4 x 7 x 3 = 1 + 2x 6 3x 2 2x 4 x 7 x 1 = 1 4x 4 x 7 x 5 = 2 + 3x 6 2x 2 12x 4 2x 7 All the coefficients in the top row are now negative. Thus, we break out of the degeneracy and obtain an optimal solution (x 1, x 2, x 3, x 4 ) = (1, 0, 1, 0) and z = 1.

26 26/45 Remarks Bland s Rule always works: use it and you obtain an optimal solution. There are other methods, e.g. Perturbation Methods, that also work. For proofs and discussions on these issues see Chvatal s book.

27 27/45 Part III The Starting Point?

28 28/45 Is the All-Zero Solution Feasible? Suppose we want to solve the linear program maximise subject to n j=1 c jx j n j=1 a ijx j b j i [m] x j 0 j [n] To run the simplex algorithm we need to start with a feasible solution. So far we have used the all-zero solution. But if some b j are negative then the all-zero solution is not feasible! E.g. maximise x 1 x 2 + x 3 subject to 2x 1 x 2 + 2x 3 4 2x 1 3x 2 + x 3 5 x 1 + x 2 2x 3 1 x 1, x 2, x 3 0

29 29/45 An Auxiliary Problem maximise subject to n j=1 c jx j n j=1 a ijx j b j i [m] x j 0 j [n] But our LP has a feasible solution if and only if the following auxiliary linear program has an optimal solution of value zero. minimise x 0 subject to n j=1 a ijx j x 0 b j i [m] x j 0 j {0, 1,..., n} So can we use an optimal solution to this auxiliary problem to find a feasible solution (and starting dictionary) for our LP? YES.

30 30/45 Finding a Starting Feasible Solution Let s see how this works on the problem maximise z = x 1 x 2 + x 3 subject to 2x 1 x 2 + 2x 3 4 2x 1 3x 2 + x 3 5 x 1 + x 2 2x 3 1 x 1, x 2, x 3 0 The auxiliary problem is then maximise w = x 0 subject to 2x 1 x 2 + 2x 3 x 0 4 2x 1 3x 2 + x 3 x 0 5 x 1 + x 2 2x 3 x 0 1 x 0, x 1, x 2, x 3 0

31 31/45 Solving the Auxiliary Problem: a Dictionary maximise w = x 0 subject to 2x 1 x 2 + 2x 3 x 0 4 2x 1 3x 2 + x 3 x 0 5 x 1 + x 2 2x 3 x 0 1 x 0, x 1, x 2, x 3 0 Let s apply the simplex method to this auxiliary problem. First put it in dictionary form. w = x 0 x 4 = 4 2x 1 + x 2 2x 3 + x 0 x 5 = 5 2x 1 + 3x 2 x 3 + x 0 x 6 = 1 + x 1 x 2 + 2x 3 + x 0

32 32/45 Solving the Auxiliary Problem: an Infeasible Solution w = x 0 x 4 = 4 2x 1 + x 2 2x 3 + x 0 x 5 = 5 2x 1 + 3x 2 x 3 + x 0 x 6 = 1 + x 1 x 2 + 2x 3 + x 0 The only term, i.e. x 0, in the top row has a negative coefficient. But this solution is not feasible for the auxiliary problem. To see this observe that x 5, x 6 < 0. But if we increase x 0 these variables will become non-negative (at the expense of decreasing the objective value). Let s make this pivot and see what happens...

33 33/45 A Strange Pivot: Pivot 0 w = x 0 x 4 = 4 2x 1 + x 2 2x 3 + x 0 x 5 = 5 2x 1 + 3x 2 x 3 + x 0 x 6 = 1 + x 1 x 2 + 2x 3 + x 0 Pivot: Add x 0 to basis. Remove x 5 from basis. (As x 5 was the most negative basic variable.) w = (5 + 2x 1 3x 2 + x 3 + x 5) x 0 = 5 + 2x 1 3x 2 + x 3 + x 5 x 4 = 4 2x 1 + x 2 2x 3 + (5 + 2x 1 3x 2 + x 3 + x 5) x 6 = 1 + x 1 x 2 + 2x 3 + (5 + 2x 1 3x 2 + x 3 + x 5) w = 5 2x 1 + 3x 2 x 3 x 5 x 0 = 5 + 2x 1 3x 2 + x 3 + x 5 x 4 = 9 2x 2 x 3 + x 5 x 6 = 4 + 3x 1 4x 2 + 3x 3 + x 5

34 34/45 Solving the Auxiliary Problem: Pivot 1 w = 5 2x 1 + 3x 2 x 3 x 5 x 0 = 5 + 2x 1 3x 2 + x 3 + x 5 x 4 = 9 2x 2 x 3 + x 5 x 6 = 4 + 3x 1 4x 2 + 3x 3 + x 5 Hooray: this is feasible for the auxiliary problem! Pivot: Add x 2 to basis. Remove x 6 from basis. w = 5 2x 1 + 3( x x x5 1 x6) x3 x5 4 x 2 = x x x5 1 4 x6 x 0 = 5 + 2x 1 3( x x x5 1 x6) + x3 + x5 4 x 4 = 9 2( x x x5 1 x6) x3 + x5 4 w = x x x x 6 x 2 = x x x x 6 x 0 = x x x x 6 x 4 = x x x x 6

35 35/45 Solving the Auxiliary Problem: Pivot 2 w = x x3 1 4 x5 3 4 x6 x 2 = x x x5 1 4 x6 x 0 = x1 5 4 x x x6 x 4 = x1 5 2 x x x6 Pivot: Add x 3 to basis. [Note: here it is quicker not to use Bland s rule.] Remove x 0 from basis. w = x1 + 5 ( x1 1 5 x x6 4 x0) x5 3 4 x6 x 2 = x1 + 3 ( x1 1 5 x x6 4 x0) x5 1 x6 x 3 = x1 1 5 x x6 4 5 x0 x 4 = x1 5 ( x1 1 5 x x6 4 x0) x x6 w = x 0 x 3 = x x x x 0 x 2 = x x x x 0 x 4 = 3 x 1 x 6 + 2x 0

36 36/45 The Auxiliary Problem: an Optimal Solution w = x 0 x 3 = x1 1 5 x x6 4 5 x0 x 2 = x x x6 3 5 x0 x 4 = 3 x 1 x 6 + 2x 0 All coefficients in the top row are now negative. So we have an optimal solution for the auxiliary problem. (x 0, x 1, x 2, x 3, x 4, x 5, x 6 ) = (0, 0, 11 5, 8 5, 3, 0, 0). The objective value is w = x 0 = 0. So this corresponds to a feasible solution to the original problem.

37 37/45 A Feasible Solution to the Original Problem maximise x 1 x 2 + x 3 subject to 2x 1 x 2 + 2x 3 4 2x 1 3x 2 + x 3 5 x 1 + x 2 2x 3 1 x 1, x 2, x 3 0 So we have an optimal solution for the auxiliary problem. (x 0, x 1, x 2, x 3, x 4, x 5, x 6 ) = (0, 0, 11 5, 8 5, 3, 0, 0). This gives the solution (x 1, x 2, x 3 ) = (0, 11 5, 8 5 ) to the original problem. This is feasible. [Check it!]

38 38/45 A Feasible Dictionary for the Original Problem maximise x 1 x 2 + x 3 subject to 2x 1 x 2 + 2x 3 4 2x 1 3x 2 + x 3 5 x 1 + x 2 2x 3 1 x 1, x 2, x 3 0 So we have a feasible solution for the original LP. How do we find a feasible starting dictionary? Consider the final dictionary for the auxiliary LP. w = x 0 x 2 = x x x x 0 x 3 = x x x x 0 x 4 = 3 x 1 x 6 + 2x 0 Remove the column for x 0 and add back the original objective...

39 39/45 A Feasible Dictionary for the Original Problem (cont.) maximise x 1 x 2 + x 3 subject to 2x 1 x 2 + 2x 3 4 2x 1 3x 2 + x 3 5 x 1 + x 2 2x 3 1 x 1, x 2, x 3 0 Remove the column for x 0 and add back the original objective. z = x 1 x 2 + x 3 x 2 = x x x6 x 3 = x1 1 5 x x6 x 4 = 3 x 1 x 6 z = x 1 ( x x5 + 1 x6) + ( x1 1 5 x5 + 3 x6) 5 x 2 = x x x6 x 3 = x1 1 5 x x6 x 4 = 3 x 1 x 6 This gives a feasible dictionary for the original LP. z = x1 3 5 x x6 x 2 = x x x6 8 x 3 = x1 1 5 x x6 x 4 = 3 x 1 x 6

40 40/45 The Two Phase Simplex Method So we use a two phase approach to solve the linear program. First we solve the auxiliary problem: Case I. If the optimal value is zero we have a feasible solution for the original. We then run the simplex algorithm for the second time. Case II. If the optimal value is less than zero the original problem is infeasible.

41 41/45 Part IV Efficiency

42 42/45 Termination With Bland s Rule the algorithm will terminate. To see this, observe that there are ( ) m+n m bases. The dictionary corresponding to each basis can appear only once. So there are at most ( ) m+n m iterations. But ( ) m+n m >> 2 m+n. Does the algorithm always make an exponential # pivots?

43 43/45 Typical Running Times The simplex method typically makes O(m) pivots, where m is the number of constraints. Each pivot takes time O(mn) with the dictionary method. [But there are actually faster ways to implement pivots.] So typically the algorithm is fast. On the other hand...

44 44/45 Exponential Running Times...all known pivot rules have pathological examples on which they make an exponential number of pivots. So the simple algorithm is a fast practical algorithm but has exponential worst case running time. In some sense the simplex algorithm is the exception that proves the rule: Polynomial algorithms good; exponential algorithms bad. In fact, recent theoretical results show the simplex algorithm isn t really an exception to this rule after all. [But that research is beyond the scope of this course...]

45 45/45 Polynomial Time Algorithms There are polynomial time algorithms for solving linear programs. The ellipsoid method and the interior point method are polytime algorithms. Conjecture. The simplex algorithm is polynomial time if, given a choice, it chooses the pivot variables at random.