Lecture 6 Simplex method for linear programming Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn No.1 Science Building, 1575
Outline Examples and standard form Fundamental theorem Simplex algorithm
Example: Transportation problem Schematics of transportation problem a 1 C ij b 1 a 2 b 2 a m Origin b n 1 b n Destination
Example: Transportation problem Formulation: m n min s = i=1 j=1 c ijx ij subject to the constraint m x ij b j, j = 1,..., n i=1 n x ij a i, i = 1,..., m j=1 x ij 0, i = 1,..., m; j = 1,..., n. where a i is the supply of the i-th origin, b j is the demand of the j-th destinations, x ij is the amount of the shipment from source i to destination j and c ij is the unit transportation cost from i to j. Optimization problem (Simplex method)
Linear programming Definition: If the minimized (or maximized) function and the constraints are all in linear form a 1x 1 + a 2x 2 + + a nx n + b. This type of optimization is called linear programming.
General form of constraints of linear programming The minimized function will always be min w = c T x (or max) x where c, x R n. There are 3 kinds of constraints in general: Type I: type constraint a i1 x 1 + a i2 x 2 + + a in x n b i Type II: = type constraint a j1 x 1 + a j2 x 2 + + a jn x n = b j Type III: type constraint a k1 x 1 + a k2 x 2 + + a kn x n b k
Examples: general form Example 1: (type III) min w = 100x 1 + 300x 2 + 400x 3 + 75x 4 s.t. x 1 + 5x 2 + 10x 3 + 0.5x 4 10000 x i 0, i = 1, 2, 3, 4 Example 2: (type I) max w = 7x + 12y s.t. 9x + 4y 360 4x + 5y 200 3x + 10y 300 x 0, y 0 Example 3: (type II) min w = x 1 + 3x 2 + 4x 3 s.t. x 1 + 2x 2 + x 3 = 5 2x 1 + 3x 2 + x 3 = 6 x 2 0, x 3 0
Standard form of constraints Standard form a 11x 1 + a 12x 2 + + a 1nx n = b 1. a m1x 1 + a m2x 2 + + a mnx n = b m x i 0, i = 1,..., n where b i 0 (i = 1,..., m). In matrix form Constraints min w = c T x (or max) x s.t. Ax = b, x 0 where A R m n, x R n, b R m, rank(a) = m n (This is not essential.) and b i 0 (i = 1,..., m).
Example 1: standard form Example 1: (type III) min w = 100x 1 + 300x 2 + 400x 3 + 75x 4 s.t. x 1 + 5x 2 + 10x 3 + 0.5x 4 10000 x i 0, i = 1, 2, 3, 4 Introduce surplus variable ( ) x 5 0, then the constraint becomes the standard form x 1 + 5x 2 + 10x 3 + 0.5x 4 x 5 = 10000 x i 0, i = 1, 2, 3, 4, 5
Example 2: standard form Example 2: max w = 7x + 12y s.t. 9x + 4y 360 4x + 5y 200 3x + 10y 300 x 0, y 0 Introduce slack variable ( ) x 1, x 2, x 3 0 and let x 4 = x, x 5 = y, then the constraint becomes the standard form max w = 7x 4 + 12x 5 x 1 +9x 4 + 4x 5 = 360 x 2 +4x 4 + 5x 5 = 200 x 3 +3x 4 + 10x 5 = 300 x i 0, i = 1, 2,..., 5
Example 3: standard form Example 3: min w = x 1 + 3x 2 + 4x 3 s.t. x 1 + 2x 2 + x 3 = 5 2x 1 + 3x 2 + x 3 = 6 x 2 0, x 3 0 Deal with the free variable x 1: Solving x 1 from one equation and substitute it into others. x 1 = 5 2x 2 x 3 then min w = 5 + x 2 + 3x 3 s.t. x 2 + x 3 = 4 x 2 0, x 3 0
Remark If some of b i < 0 in the primitive form, we can time 1 to both sides at first and introduce the slack and surplus variables again.
Outline Examples and standard form Fundamental theorem Simplex algorithm
Definitions For the standard form, n is called dimension, m is called order, variables x satisfying constraints s.t. Ax = b, x 0 are called feasible solution. Suppose rank(a) = m, and the first m columns of A are linearly independent, i.e. B = (a 1, a 2,..., a m) is nonsingular, where a i = (a 1i, a 2i,, a mi). Then call B a basis. The linear system Bx B = b has unique solution x B = B 1 b. Define x = (x B, 0), then x satisfies Ax = b. x is called a basic solution (the other x i are 0) with respect to B.
Definitions If there is 0 among x B, it is called a degenerate basic solution. If a basic solution is also a feasible solution, it is called a basic feasible solution. x i corresponding to column indices in B are called basic variable. The others are called non-basic variables. The number of the basic feasible solutions is less than C m n = n! m!(n m)!
Example Linear programming max w = 10x 1 + 11x 2 3x 1 + 4x 2 + x 3 = 9 5x 1 + 2x 2 + x 4 = 8 x 1 2x 2 + x 5 = 1 x i 0, i = 1, 2, 3, 4, 5 Choose B = (a 3, a 4, a 5) = I 3 3, then B is a basis, x = (0, 0, 9, 8, 1) is a non-degenerate basic solution. It satisfies the constraint, thus is a basic feasible solution. x 3, x 4, x 5 are basic variables.
Example Linear programming max w = 10x 1 + 11x 2 3x 1 + 4x 2 17 2x 1 + 5x 2 16 x i 0, i = 1, 2 The set of all the feasible solutions are called feasible region. 00 11 0000 1111 00000 11111 000000 111111 feasible region 00000000 11111111 000000000 111111111 0000000000 1111111111 0000000000 1111111111 00000000000 11111111111 000000000000 111111111111 000000000000 111111111111 0000000000000 1111111111111 00000000000000 11111111111111 00000000000000 11111111111111 000000000000000 111111111111111 000000000000000 111111111111111 0000000000000000 1111111111111111 00000000000000000 11111111111111111 00000000000000000 11111111111111111 000000000000000000 111111111111111111 0000000000000000000 1111111111111111111 0000000000000000000 1111111111111111111 This feasible region is a colorred convex polyhedron ( ) spanned by points x 1 = (0, 0), x 2 = (0, 16 5 ), x3 = (3, 2) and x4 = ( 17 3, 0).
Definitions A convex set S means for any x 1, x 2 S and λ [0, 1], then x = λx 1 + (1 λ)x 2 S. A non-convex set is shown below. Graphically, convex means any line segment x 1x 2 belongs to S if x 1, x 2 S. The vertices x 1 = (0, 0), x 2 = (0, 16 ),... are called extreme points 5 because there is no y 1, y 2 S, y 1 y 2 and 0 < λ < 1, such that x i = λy 1 + (1 λ)y 2.
Fundamental theorem Theorem (Fundamental theorem) Optimizing a linear objective function w = c T x is achieved at the extreme points in the feasible region colorblue if the feasible solution set is not empty and the optimum is finite. Z =C*X max 0000000000000000000000000000000000 1111111111111111111111111111111111 00000000000000000000000000000000000 11111111111111111111111111111111111 Z 000000000000000000000000000000000000 111111111111111111111111111111111111 min =C*X 000000000000000000000000000000000000 111111111111111111111111111111111111 000000000000000000000000000000000000 111111111111111111111111111111111111 0000000000000000000000000000000000000 1111111111111111111111111111111111111 00000000000000000000000000000000000000 11111111111111111111111111111111111111 00000000000000000000000000000000000000 11111111111111111111111111111111111111 000000000000000000000000000000000000000 111111111111111111111111111111111111111 000000000000000000000000000000000000000 111111111111111111111111111111111111111 000000000000000000000000000000000000000 111111111111111111111111111111111111111 0000000000000000000000000000000000000 1111111111111111111111111111111111111 0000000000000000000000000000000000 1111111111111111111111111111111111 00000000000000000000000000000000 11111111111111111111111111111111 000000000000000000000000000 111111111111111111111111111 000000000000000000000 111111111111111111111 00000000000000 11111111111111 00000000 11111111 01
Some basic theorems There are three cases for the feasible solutions of the standard form Empty set; Unbounded set; 0000 1111 0000000 1111111 00000000000 11111111111 00000000000000 11111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 000000000000000000 111111111111111111 000000000000000000 111111111111111111 000000000000000 111111111111111 000000000000 111111111111 0000000 1111111 00 11 unbounded convex Bounded convex polyhedron. A point in the feasible solution set is a extreme point if and only if it is a basic feasible solution.
Outline Examples and standard form Fundamental theorem Simplex algorithm
Simplex method Simplex method is first proposed by G.B. Dantzig in 1947. Simply searching for all of the basic solution is not applicable because the whole number is Cn m. Basic idea of simplex: Give a rule to transfer from one extreme point to another such that the objective function is decreased. This rule must be easily implemented.
Canonical form First suppose the standard form is Ax = b, x 0 One canonical form is to transfer a coefficient submatrix into I m with Gaussian elimination. For example x = (x 1, x 2, x 3) and ( ) ( ) 1 1 1 5 0 1 1 1 (A, b) = B = 1 2 0 4 1 2 0 4 then it is a canonical form for x 1 and x 3. One extreme point is x = (4, 0, 1), x 1 and x 3 are basic variables.
Transfer Now suppose A is in canonical form as the last example, then we transfer from one basic solution to another. Choose a 2 to enter the basis and a 1 leave the basis. ( ) ( ) 0 1 1 1 0 1 1 1 A = 1 2 0 4 0.5 1 0 2 ( ) 0.5 0 1 3 0.5 1 0 2 The canonical form for x 2 and x 3. The basic solution is x = (0, 2, 3). It is also a extreme point.
Transfer The transferred basic solution may be not feasible in general. 1. How to make the transferred basic solution feasible? 2. How to make the objective function decreasing after transfer?
How to make the transferred basic solution feasible? Assumption: All of the basic feasible solutions are non-degenerate. i.e. if x = (x 1, x 2,..., x m, 0,..., 0) is a basic feasible solution, then x i > 0. Suppose the basis is {a 1, a 2,..., a m} initially, and select a k (k > m) enter the basis. Suppose m a k = y ik a i i=1 then for any ɛ > 0 m ɛa k = ɛy ik a i i=1 x is a basic feasible solution m x ia i = b i=1
How to make the transferred basic solution feasible? We have m (x i ɛy ik )a i + ɛa k = b i=1 Because x i > 0, if ɛ > 0 is small enough, x = (x 1 ɛy 1k, x 2 ɛy 2k,..., x m ɛy mk, 0,..., 0, ɛ, 0,..., 0) is a feasible solution. To make it a basic solution we choose ɛ = min 1 i m { xi y ik yik > 0 } = xr y rk then x is a basic feasible solution, and let a r leave the basis. If y ik 0 for i = 1, 2,..., m, then for any ɛ > 0, x is feasible, thus the feasible region is unbounded in this case.
How to make the transferred basic solution feasible? Suppose n = 6 and the constraints 1 0 0 2 4 6 4 (A, b) = 0 1 0 1 2 3 3 0 0 1 1 2 1 1 One basis a 1, a 2, a 3, the basic feasible solution x = (4, 3, 1, 0, 0, 0) We want to choose a 4 enter the basis. The problem is to choose a i leave the basis. Compute x i y ik (k = 4) so let a 1 enter into basis. i 1 2 3 x i y ik 2 3 The new basic solution for x 2, x 3, x 4 is x = (0, 1, 3, 2, 0, 0).
How to make the objective function decreasing after transfer? The aim is to choose k such that the objective function decreasing after a k enter the basis. Suppose the canonical form is n x i + y ijx j = y i0, i = 1, 2,..., m j=m+1 where y i0 > 0. The basic feasible solution x = (y 10, y 20,..., y m0, 0,..., 0) the value of objective function m z 0 = c T Bx B = c k y k0 k=1
How to make the objective function decreased after transfer? For any feasible solution x = (x 1,..., x m, x m+1,..., x n), we have z = = m n n c k (y k0 y kj x j) + c jx j k=1 m c k y k0 + k=1 = z 0 + j=m+1 j=m+1 n j=m+1 c jx j j=m+1 n n m (c j c k y kj )x j k=1 n = z 0 + (c j z j)x j j=m+1 j=m+1 k=1 m ( c k y kj )x j where z j = c T By j = m k=1 c ky kj. If there exists j (m + 1 j n) such that r j = c j z j < 0, then when x j change from 0 to positive, the objective function will be decreased.
Simplex strategy Optimality criterion: If r j 0 for all j, then it is a optimal feasible solution. Unbounded criterion: If for some k (r k < 0), we have y jk 0 (j = 1, 2,, m), then min z =. Otherwise: We can choose the vector a k (r k < 0) to enter the basis and the vector a j ( y j0 y y jk = min i0 i y ik, y ik > 0) leave the basis.
Example Example min z = (3x 1 + x 2 + 3x 3) 2 1 1 x 1 2 1 2 3 x 2 5, x 0 2 2 1 6 Step 1: change into standard form x 3 2 1 1 1 0 0 1 2 3 0 1 0 2 2 1 0 0 1 min z = (3x 1 + x 2 + 3x 3) x 1 x 2 x 3 x 4 x 5 x 6 2 = 5, xi 0, i = 1, 2,..., 6 6
Example Step 2: Choose x 4, x 5, x 6 as basic variables, and compute the test number r 1 = c 1 z 1 = 3, r 2 = c 2 z 2 = 1, r 3 = c 3 z 3 = 3. set up simplex tableau Basis a 1 a 2 a 3 a 4 a 5 a 6 b a 4 2 1 1 1 0 0 2 a 5 1 2 3 0 1 0 5 a 6 2 2 1 0 0 1 6 r j 3 1 3 0 0 0 z 0 = 0
Example Step 3: Choose vector to enter the basis. Because r j < 0, j = 1, 2, 3, any one among a 1, a 2, a 3 could enter the basis. We choose a 2 (in general, a 1 or a 3 will be chosen because 3 is smaller). Step 4: Choose vector to leave the basis. Compute y i0 y ik, y ik > 0, k = 2, i = 1, 2, 3, we have Thus a 4 leave the basis. y 10 y 12 = 2, y 20 y 22 = 2.5, y 30 y 32 = 3 Step 5: Perform Gaussian elimination to obtain a new canonical form for basis a 2, a 5, a 6 and set up simplex tableau. Basis a 1 a 2 a 3 a 4 a 5 a 6 b a 2 2 1 1 1 0 0 2 a 5 3 0 1 2 1 0 1 a 6 2 0 1 2 0 1 2 r j 1 0 2 1 0 0 z 0 = 2
Example Step 6: Choose vector to enter the basis. Because r j < 0, j = 1, 3, any one among a 1, a 3 could enter the basis. We choose a 3. Step 7: Choose vector to leave the basis. Compute y i0 y ik, y ik > 0, k = 3, i = 1, 2, 3, we have (y i3 > 0, i = 1, 2) Thus a 5 leave the basis. y 10 y 13 = 2, y 20 y 23 = 1 Step 8: Perform Gaussian elimination to obtain a new canonical form for basis a 2, a 3, a 6 and set up simplex tableau. Basis a 1 a 2 a 3 a 4 a 5 a 6 b a 2 5 1 0 3 1 0 1 a 3 3 0 1 2 1 0 1 a 6 5 0 0 3 2 1 4 r j 7 0 0 3 2 0 z 0 = 4
Example Step 9: Choose vector to enter the basis. Because r j < 0, j = 1, 4, any one among a 1, a 4 could enter the basis. We choose a 1. Step 10: Choose vector to leave the basis. Compute y i0 y ik, y ik > 0, k = 1, i = 1, 2, 3, we have (y i1 > 0, i = 1) Thus a 2 leave the basis. y 10 y 11 = 1 5 Step 11: Perform Gaussian elimination to obtain a new canonical form for basis a 1, a 3, a 6 and set up simplex tableau. Basis a 1 a 2 a 3 a 4 a 5 a 6 b a 1 1 1 0 5 3 5 1 5 0 1 5 8 5 3 a 3 0 1 1 2 0 5 5 5 a 6 0 1 0 1 0 1 4 r j 0 7 5 0 6 5 3 0 z 5 0 = 27 5
Example Step 9: Choose vector to enter the basis. Because r j > 0, j = 1, 3, 6, so we obtain the optimal solution z = 27, and the corresponding extreme 5 point is x = ( 1 5, 0, 8, 0, 0, 4) 5
Initial basic feasible solution two step method An auxiliary problem (y R m ) m min z = i=1 y i Ax + y = b x 0, y 0 The initial basic feasible solution is trivial (x, y) = (0, b) Theorem: If the optimal feasible solution of the auxiliary problem is (x, 0), then x is a basic feasible solution of the primitive problem; if the optimal feasible solution of the auxiliary problem is (x, y ), y 0, then there is no feasible solution for the primitive problem.
What about the degenerate basic feasible solution? In general, the strategy of leaving and entering basis is chosen as 1. If more than one index j such that r j < 0, let 2. If choose a k to enter the basis; r k = min{r j r j < 0} { yi0 } min yik > 0 = yr 10 = = yr t0 y ik y r1 k y rt k and r 1 < < r t, then choose a r1 to leave the basis. For degenerate case, cycling will appear for this strategy!
What about the degenerate basic feasible solution? Bland s method: Change the strategy of leaving and entering basis into 1. If more than one index j such that r j < 0, let 2. If choose a k to enter the basis; k = min{j r j < 0} { yi0 } min yik > 0 = yr 10 = = yr t0 y ik y r1 k y rt k and r 1 < < r t, then choose a r1 to leave the basis (the same as before). Bland s method could eliminate the cycling, but it needs more computational effort.
Comment on simplex method In 1972, V. Klee and G. Minty constructed a linear programming problem which need O(2 n ) simplex steps! This shows simplex method is not a polynomial method. The first polynomial-time LP algorithm was devised by L. Khachian (USSR) in 1979. His ellipsoid method is O(n 6 ). Though his method is faster than simplex method theoretically, real implementations show counter results. In 1984, N. Karmarkar announced a polynomial-time LP method which is O(n 3.5 ). This begins the interior point revolution. Interior point method was faster than simplex for some very large problems, the reverse is true for some problems, and the two approaches are more or less comparable on others.
References 1. 2004 2. G. Hadley, Linear programming, Addison-Wesley Publishing Company, 1963. 3. Y. Ye, Interior point algorithms, theory and analysis, John Wiley & Sons, New York, 1997.