Lecture 4: Algebra, Geometry, and Complexity of the Simplex Method. Reading: Sections 2.6.4, 3.5,

Transcription

1 Lecture 4: Algebra, Geometry, and Complexity of the Simplex Method Reading: Sections 2.6.4, 3.5,

2 Summary of the Phase I/Phase II Simplex Method We write a typical simplex tableau as z x 1 x 2... x n rhs 0 ā 11 ā ā 1n b1 0 ā 21 ā ā 2n b ā m1 ā m2... ā mn bm 1 ā m+1,1 ā m+1,2... ā m+1,n bm+1 where ā m+1,j = c j, b m+1 = z 0 if this is a max problem, and ā m+1,j = c j, b m+1 = z 0 if it is a min problem. Generic Simplex Pivot for Row r: P1: Choose as entering variable any x j with ā rj < 0 P2: Choose as leaving variable any x Bi with i < r, ā ij > 0 and bi /ā ij = min{ b k /ā kj ā kj > 0, k = 1,..., r 1} P3: Perform a pivot on entry ā ij. 2

3 Steps of the Phase I/Phase II Method Initialize: Perform a standard Gauss-Jordan reduction of the original equality system. If a redundant row is found delete it, and if an inconsistent row is found STOP, the LP is infeasible. Phase I: For row r = 1,..., m while b r < 0 do the following: 1. Attempt a generic simplex pivot. 2. If Step P1 fails, then STOP, the LP is infeasible 3. If Step P2 fails pivot on element ā rj. Go to the next row. 4. If neither P1 or P2 fails, perform the pivot and repeat. When rows 1,..., m have been successfully processed, the LP is feasible. Phase II: For r = m + 1 do the following: 1. Attempt a generic simplex pivot. 2. If Step P1 fails, then STOP, the current solution is optimal. 3. If Step P2 fails STOP, the LP is unbounded. 4. If neither P1 or P2 fails, perform the pivot and repeat. 3

4 Matrix form of a basic tableau Consider a general equality form LP tableau z x 1 x 2... x n rhs 0. a 11. a a 1n. b 1. 0 a m1 a m2... a mn b m 1 c 1 c 2... c n 0 or z x rhs 0 A b 1 c 0 We wish to give a matrix description of an arbitrary basic tableau associated with basic variables x B. To do this, we first partition of the tableau into basic and nonbasic columns: z x B x N rhs 0 B N b 1 c B c N 0 Then the associated basic tableau will look like this: basis z x B1... x Bm x N1... x Nn m rhs x B ā 1,N1... ā 1,Nn m b x Bm ā m,n1... ā m,nn m bm z c N1... c Nn m z 0 or basis z x B x N rhs x B 0 I N b z 1 0 c N z 0 We wish to describe N, b, c N, and z 0 in terms of B, N, b, c B, and c N. 4

5 We have already described the N matrix and b vector, using the equation or Ax = Bx B + Nx N = b B 1 Ax = x B + B 1 Nx N = B 1 b so that N = B 1 N and b = B 1 b. How about c N and z 0? Writing z = c B x B + c N x N and substituting x B = B 1 b B 1 Nx N, we get z = c B (B 1 b B 1 Nx N ) + c N x N = c B B 1 b + (c N c B B 1 N)x N or extending to all variables, z = c B B 1 b + (c c B B 1 A)x so that the complete tableau is or simply basis z x B x N rhs x B 0 I B 1 N B 1 b z 1 0 c N + c B B 1 N c B B 1 b basis z x rhs x B 0 B 1 A B 1 b z 1 c + c B B 1 A c B B 1 b 5

6 We define an important set of values ȳ 1,..., ȳ m for this tableau, called the shadow prices, by setting ȳ = c B B 1 Using this, and writing B 1 = S = tableau can be written s s 1m..... s mm s m1, the basis z x B x N rhs x B 0 I SN Sb z 1 0 c N + ȳn ȳb or basis z x rhs x B 0 SA Sb z 1 c + ȳa ȳb More on the role of shadow prices later. 6

7 Formulae for a Basic Tableau We can now give the formulae for the current tableau in terms of the original tableau and the S and ȳ values: (1) ā ij = s i1 a 1j + s i2 a 2j s im a mj i = 1,..., m j = 1,..., n (2) bi = s i1 b 1 + s i2 b s im b m i = 1,..., m (3) c j = c j ȳ 1 a 1j ȳ 2 a 2j... ȳ m a mj j = 1,..., n (4) z 0 = ȳ 1 b 1 + ȳ 2 b ȳ m b m Example: Consider the following LP: max cx Ax = b x 0 Where A, b, and c are given below: x 1 x 2 x 3 x 4 x 5 x 6 x 7 b A c Suppose we want to reconstruct the tableau with starting basis x B = (x 4, x 6, x 1 ). 7

8 We have B = and c B = (3, 4, 2) and so S = B 1 = 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 ȳ = c B S = ( 3/2, 5/2, 9/2). The final tableau has Ā = SA = 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/ = b = Sb = 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/ =

9 c = c ȳa = (2, 3, 4, 3, 1, 4, 6) ( 3/2, 5/2, 9/2) = (0, 6, 2, 0, 12, 0, 15) z 0 = ( 3/2, 5/2, 9/2) So the current tableau is = 8 basis z x 1 x 2 x 3 x 4 x 5 x 6 x 7 rhs x x x z We apply the formulae to a few entries: ā 23 = s 21 a 13 + s 22 a 23 + s 23 a 33 = (1/2)(3) + (1/2)( 1) + ( 1/2)(0) = 1 b3 = s 31 b 1 + s 32 b 2 + s 33 b 3 = ( 1/2)(3) + (1/2)(4) + (1/2)(5) = 3 c 5 = c 5 ȳ 1 a 15 ȳ 2 a 25 ȳ 3 a 35 = 1 ( 3/2)( 1) ( 5/2)( 1) (9/2)(2) = 12 z 0 = ȳ 1 b 1 + ȳ 2 b 2 + ȳ 3 b 3 = ( 3/2)(3) + ( 5/2)(4) + (9/2)(5) = 8 9

10 Matrix Form for Canonical Max Tableau The tableaus associated with a canonical max problem has some interesting special features. Consider the canonical max LP max z = c 1 x c n x n a 11 x a 1n x n b 1. a m1 x a mn x n b m x x n 0 This has starting tableau z x 1 x 2... x n s 1... s m rhs 0 a 11 a a 1n b 1. 0 a m1 a m2... a mn b m 1 c 1 c 2... c n or z x s rhs 0 A I b 1 c 0 0 where s = (s 1,..., s m ) is the vector of slack variables for the inequalities. 10

11 A general simplex tableau will look like this: z x 1 x 2... x n s 1... s m rhs 0. ā 11. ā ā 1n. ā 1,n ā 1,n+m.... b1. 0 ā m1 ā m2... ā mn ā m,n+1... ā m,n+m bm 1 c 1 c 2... c n c n+1... c n+m z 0 and if we apply the matrix formula to the extended system with  = (A I) and ĉ = (c, 0), we get tableau basis z (x, s) rhs x B 0 S(A I) Sb z 1 [(c, 0) ȳ(a I)] ȳb = basis z x s rhs x B 0 SA S Sb z 1 c + ȳa ȳ ȳb so that S is precisely the matrix under the slack variables, and ȳ is precisely the set of objective row values under the slack variables. Note: If the objective were a min (constraints still ) the tableau will look like basis z x s rhs x B 0 SA S Sb z 1 c ȳa ȳ ȳb and so the values under the slack variables will be the negatives of the shadow prices. 11

12 Example Woody s 3-variable problem has starting tableau basis z x 1 x 2 x 3 x 4 x 5 x 6 rhs x x x z The optimal tableau is basis z x 1 x 2 x 3 x 4 x 5 x 6 rhs x / x /4 0 2/3 2 x / z / so that the shadow prices are (5/2, 0, 5) which incidentally are also the negative of the reduced costs on the slack variables. The reduced costs on the x-variables are ( c 1, c 2, c 3 ) = (35, 60, 75) (5/2, 0, 5) = (0, 0, 10) which appears negated in the optimal tableau. 12

13 Change Vectors and Ascent Directions Consider an LP in equality form: max z = cx (P ) Ax = b x 0 and suppose ˆx and ˆx are two feasible (not necessarily basic) solutions for (P ) for which ˆx has a larger objective function than ˆx. We will concentrate on the change vector v such that ˆx = ˆx + v. Two key properties of the change vector v are: (a) Av = 0 (b) cv > 0 Proof: (a) Aˆx = b and Aˆx = b Aˆv = Aˆx Aˆx = b b = 0 (b) cˆx > cˆx cv = cˆx cˆx > 0 13

14 Pivot Directions as Change Vectors The simplex method involves moving from some BFS ˆx by choosing an entering variable x j in the associated tableau, and then increasing x j to some value 0, leaving all other nonbasic variables at 0. This resulted in new point x defined by ˆx j =, ˆx B i = b i ā ij, i = 1,..., m. ˆx k = 0, otherwise. Another way of looking at this is to say that x = ˆx + ˆv, where ˆx is the current BFS, and ˆv has components Then clearly ˆv j = 1, ˆv Bi = ā ij, i = 1,..., m. ˆv k = 0, otherwise. Aˆv = Bˆv B + N ˆv N = BĀ j + A j 1 and = B(B 1 A j ) + A j = A j + A j = 0 cˆv = c Bˆv B + c N ˆv N = c B Ā j + c j = c B (B 1 A j ) + c j = c j and thus so long as c j > 0 this satisfies the properties of change vectors given above. We call ˆv a basic change vector associated with basis B and nonbasic index j. 14

15 Basic Geometry of the Phase II Simplex Method Consider a canonical max LP, written in the following inequality form: (P I ) max z = c 1 x c n x n (slack) x 1 0 x 1... x n 0 x n a 11 x a 1n x n b 1. x n+1 a m1 x a mn x n b m x n+m and its standard equality form equivalent: max z = c 1 x c n x n a 11 x a 1n x n + x n+1 = b 1 (P E ) a m1 x a mn x n + x n+m = b m x x n 0 x n x n+m 0 Now associate each of the n + m variables in P E to the appropriate one of the n + m constraints in P I. 15

16 Relationships Between Algebra and Geometry P E basic feasible solution: A solution obtained by setting n nonbasic variables to 0 and solving for the remaining m basic variables. adjacent basic feasible solutions: A pair of BFSs whose bases differ by exactly one element. The set x, 0 defines those solutions obtained by pivoting from one BFS to an adjacent one in the direction of the basic change vector. optimal BFS: A BFS whose associated tableau has all reduced costs ative unbounded tableau: A tableau having a column whose reduced cost is negative and whose associated basic change vector has all ative components. P I corner-point solution: A solution obtained by setting n constraints to equality and finding the corresponding solution. edge: Set of feasible points satisfying exactly n 1 constraints at equality. optimal corner point: corner point whose associated level set contains the entire feasible region on the side. extreme half-line (extreme ray): an edge whose objective-function- increasing direction never leaves the feasible region 16

17 Phase II for Woody s Problem max z = 35x x 2 8x x x x 1 + 6x 2 48 x 1 0 x 2 0 basis z x 1 x 2 x 3 x 4 x 5 rhs x x x z x / x / x / z x /15 8/3 8 x / x /15 1/3 8 z /3 35/3 520 x / x /4 0 2/3 2 x / z /

18 Simplex Path for Woody s Problem tables tables chairs tables chairs tables chairs chairs

19 Phase II for 3-Variable Woody s max z = 35x x x 3 8x x x x x x 1 + 6x 2 + 9x 3 48 The Simplex Tableaus: x 1 0 x 2 0 x 3 0 basis z x 1 x 2 x 3 x 4 x 5 x 6 rhs x x x z x / x / / x / / z / / x /5 8/3 16 x / / x / /20 1/3 7 z / /2 35/3 470 x /3 1 4/15 8/3 8 x /3 0 1/ x /3 0 2/15 1/3 8 z /3 0 2/3 35/3 520 x / x /4 0 2/3 2 x / z /

20 Simplex Path for Woody s 3D Problem 4 desks tables chairs

21 The tableaus: An Unbounded LP max z = 35x x 2 8x x x x x 1 6x 2 48 x 1 0 x 2 0 basis z x 1 x 2 x 3 x 4 x 5 rhs x x x z x / x 2 0 4/ / x / z x /8 1/ x /6 1/ x /8 1/ z /8 15/ The change vector for the final tableau is ˆv = ( 1 10, 1 15, 0, 1, 1 10 ) 0 with cˆv = 15 8 > 0. 21

22 The Picture with the unbounded arrow in the direction ( 1 10, 1 15 ). 22

23 Geometry of the Phase I Simplex Method The Phase I Simplex Method simply tries to satisfy the set of inequalities one inequality at a time, by moving in the direction of the feasible half of each inequality, while continuing to stay on the feasible side of all of the previous inequalities. The stopping rules can be interpreted: br becomes ative: You have passed across the inequality into its feasible side. Step P1 fails: You have reached a point as close to the inequality as you can get, and therefore can never pass into the feasible region. Step P2 fails: The pivot on ā rj jumps you right onto the equality, so that it is immediately satisfied. Example: Consider the following system: 4x x x x 1 + 6x 2 48 x 1 0 x x C3 C1 C x

24 The Phase I Tableaus basis z x 1 x 2 x 3 x 4 x 5 rhs x (P2 fails) x x basis z x 1 x 2 x 3 x 4 x 5 rhs x / x (pivot, go to x / next row) basis z x 1 x 2 x 3 x 4 x 5 rhs x 2 0 1/3 1 1/ /3 x / x / (pivot) basis z x 1 x 2 x 3 x 4 x 5 rhs x / x /4 1/ (feasible) x /4 1/

25 The Figures 16 x2 16 x C3 8 C3 6 4 C1 C2 6 4 C1 C x1 0 x x2 16 x C3 8 C3 6 4 C1 C2 6 4 C1 C x1 0 x

26 On the other hand, if we add a fourth constraint to the above system: 4x x x x 1 + 6x x 1 + 8x 2 24 x 1 0 x x C C4 C3 x1 C

27 We take the same first three pivots basis z x1 x2 s1 s2 s3 s4 rhs x / x /4 1/ s /4 1/ s /4 1/ but now one more pivot gives basis z x1 x2 s1 s2 s3 s4 rhs x /4 0 1/3 0 7 x / s / s /2 0 1/ for which P1 fails, and the problem is declared infeasible. 16 x2 16 x C2 4 C C4 C3 x1 C1 0 2 C4 C3 x1 C The Farkas Lemma solution associated with this turns out to be ŷ = (1/2, 0, 1/3, 1) This has an interesting interpretation when applied to the original inequality system. Multiplying the first constraint by 1/2 and the third constraint by 1/3, and then adding the two ( ) constraints, we get implied surrogate constraint 3x 1 + 8x 2 62 which immediately shows the fourth constraint can never be satisfied. More on surrogate constraints later. 27

28 Multiple Optimal Solutions Suppose in Woody s problem we changed the cost of the second variable from 60 to 70. max z = 35x x 2 8x x x x 1 + 6x 2 48 x 1 0, x 2 0, 7 6 x2 5 4 C2 3 2 C3 1 0 C1 x

29 If we perform the Phase II simplex method on this system, after the same first two pivots as for Woody s original problem we arrive at tableau basis z x 1 x 2 x 3 x 4 x 5 rhs x /15 8/3 8 x / x /15 1/3 8 z /3 560 This is an optimal tableau, with optimal solution x 1 = 8, x 2 = 4, x 3 = 8, x 4 = 0, x 5 = 0, and objective value z = 560, but now we have a nonbasic variable, x 4, whose reduced cost is 0. What this means is that if we bring x 4 into the basis by pivoting on ā 14, we arrive at tableau basis z x 1 x 2 x 3 x 4 x 5 rhs x / x /4 0 2/3 2 x / z /3 560 This is also an optimal tableau, with different solution x 1 = 12, x 2 = 2, x 3 = 0, x 4 = 30, x 5 = 0, which is also optimal. Fact: If the optimal tableau has a nonbasic column with reduced cost 0, then a pivot on that column will result in an alternate optimal solution. 29

30 Tie Breaking and Degeneracy in the Simplex Method Usually tie-breaking can be done arbitrarily in the simplex method. This can lead to odd situations. For example, suppose we consider Woody s problem, with the pine resource changed to 112: max z = 35x x 2 8x x x x 1 + 6x 2 48 x 1 0, x 2 0, 8 6 x2 4 C2 2 0 C1 x1 C

31 After the first pivot we obtain the following tableau: basis z x 1 x 2 x 3 x 4 x 5 rhs x / x / x / z and when we do the ratio test on the x 1 column we get two candidates for blocking variable, x 3 and x 5. Suppose we arbitrarily choose x 3 to enter the basis. The next tableau is: basis z x 1 x 2 x 3 x 4 x 5 rhs x /15 8/3 0 x / x /15 1/3 8 z /3 35/3 520 with associated solution x 1 = 8, x 2 = 4, x 3 = 0, x 4 = 0, x 5 = 0, and objective function value z = 520. This is called a degenerate tableau, since a basic variable (x 3 ) also has value 0. A pivot is indicated in the x 4 column, but when we perform the ratio test, we find that = 0. What this means is that x 3 is the blocking variable, and a pivot on entry ā 41 give the tableau basis z x 1 x 2 x 3 x 4 x 5 rhs x / x /4 0 2/3 4 x / z / which has the same solution and objective value, but can now be verified to be optimal. What has occurred is a degenerate pivot. 31

32 Finiteness of the Simplex Method Theorem 4.1 If the simplex method performs no degenerate pivots, then it will always terminate in a finite number of steps. Proof: If a degenerate pivot does not occur, then the blocking ratio will always be positive, and so the succeeding solution x will have objective value z = z 0 + c j > z 0. Therefore the sequence of BFSs will have decreasing objective values and so we can never repeat a BFS. Now BFSs are determined by their basis set, and since there are only a finite (at most ( ) n m ) such sets, then a stopping tableau must be found in a finite number of pivots. 32

33 Unfortunately, this is not enough to guarantee the finiteness of the simplex method in the presence of degenerate pivots. In fact, the simplex method is not guaranteed to terminate under fairly reasonable tie-breaking rules. Consider the following degenerate tableau: basis z x 1 x 2 x 3 x 4 x 5 x 6 x 7 rhs x x /2 11/2 5/2 9 0 x /2 3/2 1/2 1 0 z If we solve this using the standard simplex method entering/leaving rules (most negative reduced cost/minimum ration), breaking ties by choosing the leftmost or topmost among the candidate pivot columns and rows, we get the following sequence of bases: [x 1 x 2 x 3 ], [x 1 x 4 x 3 ], [x 1 x 4 x 5 ], [x 1 x 6 x 5 ], [x 1 x 6 x 7 ], [x 1 x 2 x 7 ], [x 1 x 2 x 3 ],... The simplex method, using these rules, will never terminate. Such a sequence of bases is called a cycle. Corollary: If no cycle occurs in the simplex method, then the simplex method will terminate in a finite number of steps. 33

34 Bland s Rule Cycles are extremely rare (if not altogether nonexistent) in real-world problems. Nevertheless, they pose theoretical problems for any results that depend upon the simplex method for a proof. Luckily, there is a simple pivot rule called Bland s Rule that avoids cycles: Bland s Rule: Use the following rules to determine the entering and leaving variables for a simplex pivot: Among all of the variables eligible to enter the basis choose the one with smallest index. Among all of the variables eligible to leave the basis, again choose the one with the smallest index. In the cycling example given above, for the basis [x 1 x 6 x 5 ] instead of choosing the variable x 7 with the most negative c j value, Bland s Rule would choose the variable x 2 with the smallest index among those with c j < 0. This gives an optimal tableau. 34

35 Theorem 4.2 If Bland s Rule is used to choose the entering and leaving variables in the simplex method, then the simplex method will never cycle. Proof: Suppose we obtain cycle B 0, B 1,..., B k = B 0 of basis sets using Bland s Rule. Define a active variable to be one which enters the basis during a pivot in this cycle (and hence will necessarily leave the basis during another pivot in the cycle). Note that all active variables will have value 0 throughout the cycle, whether they are basic or nonbasic. Let t be the largest index of a active variable. We concentrate on two specific tableaus in the cycle: tableau T, with associated basis set B, where x t enters the basis, and tableau ˆT, with associated basis set ˆB, where x t leaves the basis. In tableau ˆT, suppose t = ˆB r, and let x s be the entering variable, that is, the pivot in tableau ˆT occurs on entry (r, s) There are two key points to notice about these tableaus in relation to Bland s Rule: Tableau T : Since x t enters the basis, then c t > 0, and c j 0 for every other active variable x j. Tableau ˆT : Since x s enters the basis, then ĉ s > 0, and since x t = x ˆBr leaves the basis, then â rs > 0, and â is 0 for every other active variable x j = x ˆBi. 35

36 Now consider the basic change vector associated with the pivot performed with respect to tableau ˆT : ˆv s = 1 ˆv ˆBi = â is i = 1,..., m ˆv j = 0 j ˆN \ {s} We know from the basic properties of change vectors that Aˆv = 0 and cˆv = ĉ s > 0. We compute the value of cˆv two ways. First note that cˆv = (c ỹa)ˆv = cˆv ỹ(aˆv) = cˆv ỹ 0 = cˆv = ĉ s > 0 where ỹ = c B B 1. If we write out the vector product, however, we get cˆv = j ˆB Ñ {s} ( x j active) c jˆv j every other index has either c j = 0 or ˆv j = 0 = c t â rs c j â is + c s 1 < 0 j= ˆB i ˆB Ñ\{t} ( c t > 0, â rs > 0) ( c j 0, â is 0) ( c s 0) a contradiction. It follows that no such cycle can exist. 36

37 Conclusions Theorem 4.3 The Phase I/Phase II Simplex Method, using Bland s anticycling rule, will always terminate in a finite number of steps. Corollary: Any LP is either infeasible, unbounded, or has an optimal solution. Corollary: For any equality form LP (P ): If (P ) has a feasible solution, then (P ) has a basic feasible solution. If (P ) has an optimal solution, then (P ) has a basic feasible optimal solution. 37

38 Computational Complexity of the Simplex Method How many simplex pivots are required to guarantee that a solution will always be reached? There are ( ) n m possible bases in an m n equality LP system. This number can grow as fast as 2 m. Again, there are laboratory examples of classes of LPs that exhibit exponential growth in the number of Phase II pivots, using almost any local pivot rules. In practice, the number of pivots to optimality seems to grow proportional to (linear in) m. Goal: To construct a polynomial-time algorithm for solving LPs. The interior point method is one such algorithm. (More on this method later in the course.) Big Open Question: Is there an implementation of the simplex method (effective choice of pivots) that will make it a polynomial-time algorithm? 38