MATH 4211/6211 Optimization Linear Programming Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0
The standard form of a Linear Program (LP) minimize subject to c T x Ax = b x 0 where x R n is the unknown variable; A R m n is given, where m < n and rank(a) = m; b R m is given. Xiaojing Ye, Math & Stat, Georgia State University 1
Other forms of LP can be converted to the standard form. For example, suppose an LP is given as maximize subject to c T x Ax b x 0 Then we can rewrite it as an equivalent standard form minimize subject to ( c) T x Ax + y = b x, y 0 where [x; y] R n+m is the variable to solve for now, and y is called the slack variable. If [x ; y ] T is a solution of the new problem, then x is a solution of the original problem. Xiaojing Ye, Math & Stat, Georgia State University 2
Example. Convert the LP below into the standard form maximize x 2 x 1 subject to 3x 1 = x 2 5 x 2 2 x 1 0 x 1 := x 1, then x 1 0 x 1 0 x 2 = u v where u, v 0 x 2 2 2 u v 2 Introduce slack variables x 3, x 4 0 such that u v + x 3 = 2 and u v x 4 = 2 Xiaojing Ye, Math & Stat, Georgia State University 3
Example (cont). Now we obtain an equivalent problem: minimize x 1 u + v subject to 3x 1 + u v = 5 u v + x 3 = 2 u v x 4 = 2 x 1, u, v, x 3, x 4 0 Note that there are 5 variables now. Xiaojing Ye, Math & Stat, Georgia State University 4
Example. Suppose a factory wants to manufacture 4 products with the cost and budget (availability), as well as their profits, given in the table below. The goal is to maximize total profit. P1 P2 P3 P4 availability man weeks 1 2 1 2 20 kg of material A 6 5 3 2 100 boxes of material B 3 4 9 12 75 profit 6 4 7 5 Solution. Let x i be the quantity to manufacture product i, then the LP is maximize 6x 1 + 4x 2 + 7x 3 + 5x 4 subject to x 1 + 2x 2 + x 3 + 2x 4 20 6x 1 + 5x 2 + 3x 3 + 2x 4 100 3x 1 + 4x 2 + 9x 3 + 12x 3 75 x 1, x 2, x 3, x 4 0 Exercise: convert this into the standard form of LP. Xiaojing Ye, Math & Stat, Georgia State University 5
Geometric interpretation of the feasible set Ω = {x R n : Ax b, x 0}: Let a T i be the ith row of A for i = 1,..., m. Then a T i x b i is a half space. Hence Ax b is the intersection of m half spaces, which is a polyhedra. So is Ω. The objective c T x is a plane with slope defined on Ω, and the optimal point x is the point in Ω that has the minimum value c T x. Xiaojing Ye, Math & Stat, Georgia State University 6
Basic solutions of linear system Ax = b. Suppose A = [B D] where B R m m has rank(b) = m. Denote x B = B 1 b R m. Then x = is called a basic solution of Ax = b. [ ] xb 0 R n Xiaojing Ye, Math & Stat, Georgia State University 7
Definition. We introduce the following terms: x B R m : basic variables If x B has zero component(s), then x is called a degenerate basic solution B: basic columns of A. Note that in practice there are up to ( ) n m ways to choose the basic columns. x is called a feasible point if x Ω := {x R n : Ax = b, x 0}. x Ω is called an optimal (feasible) solution if c T x c T x for all x Ω. Xiaojing Ye, Math & Stat, Georgia State University 8
Properties of basic solutions Theorem (Fundamental Theorem of LP). For an LP with nonempty feasible set Ω, the following statements hold: (a) A basic feasible point exists. (b) If LP has solution, then there exists an optimal basic feasible solution. Xiaojing Ye, Math & Stat, Georgia State University 9
Proof. Part (a). Let x Ω (since Ω ), i.e., Ax = b and x 0. WLOG, assume x 1,..., x p > 0 and x p+1,..., x n = 0. Then where A = [a 1,..., a p,..., a n ]. Ax = x 1 a 1 + + x p a p = b If a 1,..., a p are linearly independent, then p m and x is a basic feasible point. If a 1,..., a p are linearly dependent, then y 1,..., y p R not all zero such that Ay = y 1 a 1 + + y p a p = 0 where y = [y 1,..., y p, 0,..., 0] T R n (WLOG we assume at least one y i > 0 otherwise take y = y), then we know A(x εy) = b for all ε. Xiaojing Ye, Math & Stat, Georgia State University 10
Proof (cont). Choose ε = min{x i /y i : y i > 0}, then x εy Ω and has only p 1 nonzero components. Repeat this until we find the basic columns of A and its corresponding basic feasible point. Xiaojing Ye, Math & Stat, Georgia State University 11
Proof (cont). Part (b). Let x Ω be optimal, and again WLOG assume x 1,..., x p > 0. If a 1,..., a p are linearly independent, then p m and x is a basic feasible solution. If a 1,..., a p are linearly dependent, then with the same argument as in (a), y 1,..., y p R not all zero such that Ay = y 1 a 1 + + y p a p = 0 If c T y 0 (say > 0), then by choosing ε min{ x i /y i : y i 0}, we have x εy Ω and c T (x εy) = c T x εc T y < c T x, which contradicts to the optimality of x. Hence c T y = 0. So we can choose ε in the same way as in (a) to get x εy which has p 1 nonzeros. Repeat to obtain a basic feasible solution. Xiaojing Ye, Math & Stat, Georgia State University 12
Theorem. Consider an LP with nonempty Ω, then x is an extreme point of Ω iff x is a basic feasible point. In theory, we only need to examine the basic feasible points of Ω and find the optimal basic feasible solution of the LP. However, there could be a large amount of extreme points to examine. Xiaojing Ye, Math & Stat, Georgia State University 13
Now we study the Simplex Method designed for LP. Recall basic row operations to linear system Ax = b: interchange two rows multiply one row by a real nonzero scalar Adding a scalar multiple of one row to another row Each of these operations corresponds to an invertible matrix E R m premultiplying to A. Xiaojing Ye, Math & Stat, Georgia State University 14
Also recall that, to solve a linear system Ax = b, we first form the augmented matrix [A, b], then apply a series of row operations, say E 1,..., E t, to the matrix to obtain E t E 1 [A, b] = [I, D, B 1 b] where D is such that E t... E 1 A = [I, D] and B = (E t E 1 ) 1 R m m is invertible. Then a particular solution of Ax = b is x = [B 1 b; 0] R n. In addition, note that [ Dx D ; x D ] R n is a solution of [I, D]x = 0 for any x D R n m. Therefore, any solution of Ax = b can be written for some x D R n m as [ B x = 1 ] [ ] b DxD + 0 x D Xiaojing Ye, Math & Stat, Georgia State University 15
Suppose we have applied basic row operations and converted Ax = b to [I, Y ]x = y 0 (called the canonical form) where Y = y 1 m+1 y 1 n..... y m m+1 y m n R m (n m) y 0 = y 10. y m0 R m Note that the column y q = [y 1q ;... ; y mq ] R m gives the coefficients to represent a q using a 1,..., a m, where a i is the ith column of A: a q = y 1q a 1 + + y mq a m = [a 1,..., a m ]y q Now we are using a 1,..., a m as basic columns, and x = [B 1 b; 0] is the corresponding basic solution to Ax = b. If B 1 b 0, then x is a basic feasible point of the LP. Xiaojing Ye, Math & Stat, Georgia State University 16
Now we want to move to another basic feasible point. To this end, we need to exchange one of a 1,..., a m with another a q (q > m) to form the new basic columns, find the corresponding basic solution x of Ax = b, making sure that x 0. Exchanging basic columns p and q where p m < q is simple: just apply basic row operations [I, Y ] so that the qth column becomes e p. Here e p R n is the vector with 1 as the pth component and 0 elsewhere. However, which a q we should choose to enter the basic columns, and which a p to leave? Xiaojing Ye, Math & Stat, Georgia State University 17
Since x = [y 10 ;... ; y m0 ; 0;... ; 0] R n + to Ax = b, we know is a basic solution corresponding y 10 a 1 + + y m0 a m = b Suppose we decide to let a q enter the basic columns, then due to a q = y 1q a 1 + + y mq a m = [a 1,..., a m ]y q we know for any ε 0 there is (y 10 εy 1q )a 1 + + (y m0 εy mq )a m + εa q = b Xiaojing Ye, Math & Stat, Georgia State University 18
Note that y q have positive and nonpositive components (if all are nonpositive then the problem is unbounded), then for ε > 0 gradually increases from 0, one of the coefficients (say p) first becomes 0. We will choose a p to leave the basic columns. More precisely, we choose ε = min i {y i0 /y iq : y iq > 0}, and obtain the basic columns a 1,..., â p,..., a m, a q Xiaojing Ye, Math & Stat, Georgia State University 19
Now we consider which a q to enter the basic column. Ideally, we should choose a q such that c T x decreases the most. Recall that the current point is x = [y 0 ; 0] R n. If we choose a q to enter, then the objective function becomes c T [ y0 εy q 0 where z i = c T [1:m] y i for i = 0, 1,..., m. ] + εe q = z 0 + ε[c q (c 1 y 1q + + c m y mq )] = z 0 + (c q z q )ε Xiaojing Ye, Math & Stat, Georgia State University 20
If c q z q < 0 for some q, then choosing a q to enter can further reduce the objective function since ε > 0. If there are more than one such q, then choose the one with smallest index or the one with smallest c q z q. If c q z q 0 for all q, then the optimal basic point is reached. These answered which a q to enter, and when the simplex method should stop. Xiaojing Ye, Math & Stat, Georgia State University 21
To simplify the process, we consider the matrix form of the simplex method. Given the standard form of an LP minimize subject to c T x Ax = b x 0 we form the tableau of the problem as the matrix [ ] [ ] A b B D b c T = 0 c T B ct D 0 R (m+1) (n+1) Note that the tableau contains all information of the LP. Xiaojing Ye, Math & Stat, Georgia State University 22
Premultiplying the matrix [ B 1 0 0 T 1 [ B 1 ] 0 0 T 1 ] [ B D b c T B ct D 0 Further premultiplying the matrix [ ] [ I 0 I B 1 D B 1 b c T B 1 c T B c T D 0 to the tableau, we obtain ] = [ I 0 c T B 1 ] = [ I B 1 D B 1 b c T B c T D 0 ] yields [ I B 1 D B 1 ] b 0 T c T D ct B B 1 D c T B B 1 b ] Xiaojing Ye, Math & Stat, Georgia State University 23
It is easy to check that [B 1 b; 0] R n is the basic solution c T D ct B B 1 D R n m contains the reduced cost coefficients, i.e., z q c q for q = m + 1,..., n c T B B 1 b R is the objective function Xiaojing Ye, Math & Stat, Georgia State University 24
According to the discussion above, the simplex method executes the following actions in order: find the index, say q, of the most negative component among c T D ct B B 1 D R n m ; find p = arg min i {y i0 /y iq : y iq > 0}; pivot the tableau about the (p, q) entry by basic row operations so that the column becomes 0 except the (p, q) entry which is 1. [1 : m, n + 1] gives the current basic solution (basic feasible point), the nonzeros in [m + 1, 1 : n] are the reduced cost coefficients, and the (m + 1, n + 1) entry is the negative of objective function. Xiaojing Ye, Math & Stat, Georgia State University 25
Example. Consider the following linear programming problem: maximize 7x 1 + 6x 2 subject to 2x 1 + x 2 3 Solve this LP using the simplex method. x 1 + 4x 2 4 x 1, x 2 0 Solution. We first convert this LP to the standard form: minimize 7x 1 6x 2 subject to 2x 1 + x 2 + x 3 = 3 x 1 + 4x 2 + x 4 = 4 x 1, x 2, x 3, x 4 0 Xiaojing Ye, Math & Stat, Georgia State University 26
Solution (cont). The tableau of this LP is 2 1 1 0 3 1 4 0 1 4 7 6 0 0 0 where the upper left 2 4 submatrix corresponds to A = [a 1,..., a 4 ], c = [ 7; 6, 0; 0], and b = [3; 4]. Note that A is already in the canonical form: a 3, a 4 are the basic columns, and x = [0; 0; 3; 4] is the basic solution of Ax = b. Since 7 is the most negative term, we will let a 1 enter the basic column. Comparing 3/2 and 4/1, we see the former is smaller and hence decide to let a 3 leave the basic column. Xiaojing Ye, Math & Stat, Georgia State University 27
Now pivoting about the (1, 1) entry, we obtain 1 2 1 2 0 3 2 1 0 7 2 2 1 1 5 2 0 5 2 7 2 0 21 2 Now we see 5/2 is the most negative term, so we let a 2 enter the basic columns. Comparing 3/2 5/2 = 3 and 1/2 7/2 = 5/7, we let a 4 leave the basic column. Then pivoting about the (2, 2) entry, we obtain 4 7 1 7 1 0 7 0 1 1 2 5 7 7 7 0 0 22 5 86 7 7 7 There are no negative entries in the last row, so we stop. The optimal solution is x = [ 8 7, 5 7, 0, 0] and optimal value is 86 7 of the LP in its standard form. These correspond to the optimal solution x 1 = 8/7, x 2 = 5/7, and optimal value 86/7 of the original problem. 8 Xiaojing Ye, Math & Stat, Georgia State University 28
Starting the Simplex Method The simplex method discussed above is referred to the Phase II which requires an initial point to be a basic feasible point. Now we consider the Phase I which finds one of such basic feasible points. To this end, we introduce artificial variables y R m and consider the following artificial problem: minimize subject to 1 T y Ax + Ey = b x 0, y 0 where 1 = [1,..., 1] T R m, E is diagonal with E ii = sign(b i ). Xiaojing Ye, Math & Stat, Georgia State University 29
The artificial problem minimize subject to 1 T y Ax + Ey = b x 0, y 0 has an obvious basic feasible point [x; y] = [0;... ; 0; b 1 ;... ; b m ] with which we can directly apply the Phase II simplex method. Xiaojing Ye, Math & Stat, Georgia State University 30
Moreover, the artificial problem in Phase I has a solution iff the original problem has nonempty feasible set Ω. In this case, the artificial problem above have an optimal objective value 0 and optimal solution [x ; y ], such that y = 0 and x is a basic feasible point of the original LP problem: If Ω is nonempty, then there exists x Ω, i.e., Ax = b and x 0. So [x; 0] solves the artificial problem in Phase I. If the artificial problem has a solution such that 0 T y = 0, then y = 0, and x satisfies Ax = b and x 0, which means x Ω. Xiaojing Ye, Math & Stat, Georgia State University 31