Linear Programming. Jie Wang. University of Massachusetts Lowell Department of Computer Science. J. Wang (UMass Lowell) Linear Programming 1 / 47

Transcription

1 Linear Programming Jie Wang University of Massachusetts Lowell Department of Computer Science J. Wang (UMass Lowell) Linear Programming 1 / 47

2 Linear function: f (x 1, x 2,..., x n ) = n a i x i, i=1 where a 1, a 2,..., a n are real numbers and x 1, x 2,..., x n are valuables. Linear equality: f (x 1, x 2,..., x n ) = b. Linear inequalities: f (x 1, x 2,..., x n ) > b. f (x 1, x 2,..., x n ) b. f (x 1, x 2,..., x n ) < b. f (x 1, x 2,..., x n ) b. Linear inequalities and linear equalities are often referred to as linear constraints. LP problems are to maximize (or minimize) a linear objective function with linear constraints. J. Wang (UMass Lowell) Linear Programming 2 / 47

3 LP Standard Form Standard form: Maximization + linear inequalities. maximize x 1 + x 2 subject to 4x 1 x 2 8 2x 1 + x x 1 2x 2 2 x 1 0 x 2 0. A minimization problem can be converted to an equivalent maximization problem, and vice versa. For example, maximizing f (x 1, x 2,..., x n ) under a set of constraints is equivalent to minimizing f (x 1, x 2,..., x n ) with the same set of constraints. J. Wang (UMass Lowell) Linear Programming 3 / 47

4 LP Slack Form Maximization + linear equalities with slack variables. maximize x 1 + x 2 subject to x 3 = 8 4x 1 + x 2 x 4 = 10 2x 1 x 2 x 5 = 2 5x 1 + 2x 2 x i 0 i = 1, 2, 3, 4, 5. Variables on the left-hand side of the equalities are basic variables and on the right-hand side nonbasic variables. Initially, basic variables are slack variables, and nonbasic variables are the original variables. These can be changed during the simplex procedure. The slack form turns an LP problem in a lower dimension with a set of inequality constraints to an equivalent LP problem in a higher dimension with a set of equality constraints. Equalities are much easier to handle than inequalities. J. Wang (UMass Lowell) Linear Programming 4 / 47

5 LP Applications An LP formulation for single-pair shortest path of (s, t). maximize d t subject to d v d u + w(u, v) for each edge u v E, d s = 0. Note: it s maximization, not minimization. An LP formulation for maximum flow. maximize v V f sv v V f vs subject to f uv c(u, v) for each u, v V v V f uv = v V f uv for each u V {s, t} f uv 0 for each u, v V. J. Wang (UMass Lowell) Linear Programming 5 / 47

6 Minimum-Cost Flow Each edge u v in a flow network has a capacity c(u, v) and a cost a(u, v). Want to send d units of flow from s to t while minimizing the total cost. minimize u v E a(u, v)f uv subject to for each u, v V : f uv c(u, v) for each u V {s, t}: v V f uv = v V f uv v V f sv v V f vs = d for each u, v V : f uv 0. J. Wang (UMass Lowell) Linear Programming 6 / 47

7 Multicommodity Flow Given a digraph G = (V, E) with k different commodities K 1, K 2,..., K k, where Each edge u v E has a capacity c(u, v) 0. G has no anti-parallel edges. K i = (s i, t i, d i ), representing the source, sink, and demand of commodity i. Let f i denote the flow for commodity i, where f iuv is the flow of commodity i from u to v. Let f uv = k i=1 f iuv denote the aggregate flow. Want to determine if such a flow exists. That is, there is no objective function to optimize. J. Wang (UMass Lowell) Linear Programming 7 / 47

8 Multicommodity Flow Continued optimize 0 subject to k i=1 f iuv c(u, v) for each u, v V v V f iuv v V f iuv = 0 for each u V {s i, t i } v V f i,s i,v v V f i,v,s i = d i f iuv 0 for each u, v V and i = 1, 2,..., k. J. Wang (UMass Lowell) Linear Programming 8 / 47

9 Feasible Solutions and Feasible Regions Feasible solutions are values of the variables that satisfy the constraints. Feasible region (a.k.a. simplex) is the set of feasible solutions, which is a convex polytope. A convex polytope is a geometric shape in d-dimensional space without any dents. J. Wang (UMass Lowell) Linear Programming 9 / 47

10 LP Algorithms The optimal solution occurs at the boundary of the simplex. Could be at exactly one vertex. In this case there is only one optimal solution. Could be at a line segment. In this case there are multiple optimal solutions. The simplex algorithm, while having exponential runtime in the worst case, is very efficient in practice. Other algorithms include (1) the ellipsoid method, the first-known polynomial-time algorithm; and (2) the Interior-Point method, which walks through the interior of the simplex instead of the vertices. J. Wang (UMass Lowell) Linear Programming 10 / 47

11 The Simplex Algorithm The simplex algorithm finds a basic solution from the slack form iteratively as follows: 1 Set each nonbasic variable to zero. 2 Compute the values of the basic variables from the equality constraints. If a nonbasic variable in a constraint causes a basic variable in the constraint to become 0, then the constraint is tight. 3 The goal of an iteration is to reformulate the linear program so that the basic solution (when the nonbasic variables are 0) has a greater objective value. This is done by (performing a pivot): Select a nonbasic variable x e (a.k.a. the entering variable); maximize it such that all constraints are satisfied. The variable x e becomes basic and another variable x l (a.k.a. the leaving variable) becomes nonbasic. 4 The simplex algorithm terminates when all of the coefficients appearing in the objective function become negative. J. Wang (UMass Lowell) Linear Programming 11 / 47

12 An Example Slack form: maximize 3x 1 + x 2 + 2x 3 subject to x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 x 1, x 2, x 3 0. maximize 3x 1 + x 2 + 2x 3 subject to x 4 = 30 x 1 x 2 3x 3 x 5 = 24 2x 1 2x 2 5x 3 x 6 = 36 4x 1 x 2 2x 3 x 1, x 2, x 3, x 4, x 5, x 6 0. J. Wang (UMass Lowell) Linear Programming 12 / 47

13 Simplex Algorithm Example Continued Set x 1 = 0, x 2 = 0, x 3 = 0 and get x 4 = 30, x 5 = 24, and x 6 = 36. Select entering variable x 1 and leaving variable x 6. Solve the corresponding equation for x 1 to get x 1 = 9 x 2 4 2x 3 4 x 6 4. Maximize it to get x 1 = 9. Substitute the right-hand side of x 1 into each remaining equation: z = 27 + x x 3 2 3x 6 4 x 1 = 9 x 2 4 x 3 4 x 6 4 x 4 = 21 3x 2 4 5x x 6 4 x 5 = 6 3x 2 2 4x 3 + x 6 2. J. Wang (UMass Lowell) Linear Programming 13 / 47

14 Simplex Algorithm Example Continued Set x 2, x 3, and x 6 to 0 to get z = 27, x 1 = 9, x 4 = 21, and x 5 = 6. Select entering variable x 3 and leaving variable x 5. Note that x 6 cannot be chosen for it will decrease the value of z. Solve for x 3 to get x 3 = 3 2 3x x 6 8 x 5 4. The maximum value for x 3 is 3 2. Substitute x 3 s left-hand side into the rest of the equations to obtain z = x 2 16 x x 6 16 x 1 = 33 4 x x 5 8 5x 6 16 x 3 = 3 2 3x x 6 8 x 5 4 x 4 = x x 5 8 x 6 16 J. Wang (UMass Lowell) Linear Programming 14 / 47

15 Simplex Algorithm Example Continued Set x 2, x 5, and x 6 to 0 to get z = 111 4, x 1 = 33 4, x 3 = 3 2, and x 4 = Select entering variable x 2 ; maximize it to get x 2 = 4. The leaving variable is x 3 : x 2 = 4 8x 3 3 2x x 6 3. Substitute x 2 s left-hand side into the rest of the equations to obtain: z = 28 x 3 6 x 5 6 2x 6 3 x 1 = 8 + x x 5 6 x 6 3 x 2 = 4 8x 3 3 2x x 6 3 x 4 = 18 x x 5 2 J. Wang (UMass Lowell) Linear Programming 15 / 47

16 Simplex Algorithm Example Continued Now all the coefficients are negative, meaning that there is no more room for improvement. The algorithm terminates with z = 28 by setting x 3, x 5, x 6 to 0. The basic solution is x 1 = 8, x 4 = 18, x 2 = 4, x 5 = 0, x 3 = 0, x 6 = 0. Note: When ties are present, choose the variable with the smallest index (Bland s Rule). J. Wang (UMass Lowell) Linear Programming 16 / 47

17 Pivoting Pivoting takes as input a slack form. Let N denote the set of indices of nonbasic variables and B the set of indices of basic variable in the slack form. Let the slack form be given by (N, B, A, b, c, v), where v is the objective value. Let l denote the index of the leaving variable x l and e the entering variable x e. Pivoting returns ( ˆN, ˆB, Â, ˆb, ĉ, ˆv) for the new slack form. J. Wang (UMass Lowell) Linear Programming 17 / 47

18 Pivoting Continued J. Wang (UMass Lowell) Linear Programming 18 / 47

19 How Will Variables Change in Pivoting? Lemma 29.1 In a call to Pivot(N, B, A, b, c, v, l, e) with a le 0, let x denote the basic solution after this call. Then 1 x j = 0 for each j ˆN. 2 x e = b l /a le. 3 x i = b i a ie ˆb e for each i ˆB {e}. Proof. The first statement is true because the basic solutions always sets nonbasic variables to 0. For these variables we have x i = ˆb i for i ˆB, since x i = ˆb i j ˆN âijx j. Since e ˆB, line 3 of Pivot gives For i ˆB {e}, line 9 gives This completes the proof. x e = ˆb e = b l /a le. x i = ˆb i = b i a ie ˆbe. J. Wang (UMass Lowell) Linear Programming 19 / 47

20 Simplex Code J. Wang (UMass Lowell) Linear Programming 20 / 47

21 Degeneracy Each iteration of the simplex algorithm either increases the objective value or leaves the objective value unchanged. The latter phenomenon is degeneracy. Example of degeneracy: z = x 1 + x 2 + x 3 x 4 = 8 x 1 x 2 x 5 = x 2 x 3. Suppose that x 1 is chosen to be the entering variable and x 4 the leaving variable. After pivoting: z = 8 + x 3 x 4 x 1 = 8 x 2 x 4 x 5 = x 2 x 3. J. Wang (UMass Lowell) Linear Programming 21 / 47

22 Degeneracy Continued Now x 3 is the only option for being the entering variable and x 5 leaving. After pivoting: z = 8 + x 2 x 4 x 5 x 1 = 8 x 2 x 4 x 3 = x 2 x 5. The objective value of 8 has not changed, and degeneracy occurs. Fortunately, x 2 can now be chosen to be entering and x 1 leaving. After pivoting: z = 16 x 1 2x 4 x 5 x 2 = 8 x 1 x 4 x 3 = x 2 x 5. J. Wang (UMass Lowell) Linear Programming 22 / 47

23 Cycling Degeneracy may lead to cycling: The slack forms at two different iterations are identical. If we instead choose x 2 entering and x 3 leaving, then after pivoting: This is a cycling. z = 8 + x 3 x 4 x 1 = 8 x 2 x 4 x 2 = x 3 + x 5. J. Wang (UMass Lowell) Linear Programming 23 / 47

24 Simplex Runtime Cycling is the only reason that the simplex algorithm might not terminate. By choosing the variable with the smallest index, the simplex algorithm always terminate. The simplex algorithm terminates in at most ( ) n+m m iterations. Proof. This is because B = m and there are n + m variables, implying that this are at most ( ) n+m m ways to choose B. In practice, the simplex algorithm runs extremely fast. J. Wang (UMass Lowell) Linear Programming 24 / 47

25 Duality The dual of a maximization LP problem (a.k.a. primal) is a minimization LP, and vice versa. Primal: maximize subject to n j=1 c jx j n j=1 a ijx j b i for i = 1, 2,..., m x j 0 for j = 1, 2,..., n. Dual: maximize subject to m i=1 b iy i m i=1 a ijy i c i for j = 1, 2,..., n y i 0 for i = 1, 2,..., m. J. Wang (UMass Lowell) Linear Programming 25 / 47

26 Primal-Dual Example Primal: maximize 3x 1 + x 2 + 2x 3 subject to x 1 + x 2 + 3x x 1 + 2x 2 + 5x x 1 + x 2 + 2x 3 36 x 1, x 2, x 3 0. Dual: minimize 30y y y 3 subject to y 1 + 2y 2 + 4y 3 3 y 1 + 2y 2 + y 3 1 3y 1 + 5y 2 + 2y 3 2 y 1, y 2, y 3 0. J. Wang (UMass Lowell) Linear Programming 26 / 47

27 Weak LP Lemma Lemma 29.8 Let x be any feasible solution to the primal LP and y be any feasible solution to the dual LP. Then n m c j x j b i y j. j=1 i=1 Proof. We have ( n n m ) c j x j a ij y i x j (Definition of c j from dual.) j=1 j=1 = i=1 i=1 m n a ij x j y i j=1 m b i y i. i=1 (Definition of b i from primal.) J. Wang (UMass Lowell) Linear Programming 27 / 47

28 Equality of Primal and Dual Corollary Let x be a feasible solution to a primal linear program (A, b, c), and let y be a feasible solution to the corresponding dual linear program. If n m c j x j = b i y i, j=1 then x and y are optimal solutions to the primal LP and dual LP, respectively. i=1 Proof. It follows from the Weak LP Lemma. J. Wang (UMass Lowell) Linear Programming 28 / 47

29 LP Duality Theorem Suppose that the simplex algorithm returns values x = (x 1, x 2,..., x n ) for the primal LP (A, b, c). Let N and B denotes the nonbasic and basic variables for the final slack form, let c denote the coefficients in the final slack form, and let y = (y 1, y 2,..., y m ) be defined as { c n+i if (n + i) N, y i = 0 otherwise. Then x is an optimal solution to the primal LP, y is an optimal solution to the dual LP, and n m c j x j = b i y i. j=1 i=1 J. Wang (UMass Lowell) Linear Programming 29 / 47

30 Proof It suffices to show that n c j x j = j=1 m b i y i. i=1 Suppose the simplex algorithm terminates with a final slack form and the objective function z = v + j N c j x j, where c j 0 for all j N. For the basic variables in the final slack form we can set their coefficients to 0. Namely, let c j = 0 for all j B. J. Wang (UMass Lowell) Linear Programming 30 / 47

31 Proof Continued We have z = v + j N c j x j = v + j N c j x j + j B c j x j (since c j = 0 for j B) = v + n+m j=1 c j x j (since N B = {1,..., n + m}). Since for j N we have x j = 0, we have z = v. Since each step in the simplex algorithm is an elementary operation, we know that the original objective function on x must be the same to that of the final slack form. Namely, n j=1 n+m c j x j = v + c j x j j=1 = v + j N = v. c j x j + j B c j x j J. Wang (UMass Lowell) Linear Programming 31 / 47

32 Proof Continued We now show that y is a feasible solution to the dual LP and m n b i y i = c j x j. n j=1 i=1 n+m c j x j = v + c j x j = v + = v + = v + j=1 n c j x j + j=1 n c j x j + j=1 n c j x j + j=1 n+m j=n+1 m i=1 j=1 c j x j c n+ix n+i m ( y i )x n+i (by definition of y i ) i=1 J. Wang (UMass Lowell) Linear Programming 32 / 47

33 Proof Continued = v + = v + = v + = ( v n c j x j + j=1 n c j x j j=1 n c j x j j=1 m ( ( y i ) b i i=1 m b i y i + i=1 m b i y i + i=1 m ) b i y i + i=1 n j=1 m n ) a ij x j j=1 i=1 j=1 n j=1 i=1 ( c j + n (a ij x j )y i m (a ij y i )x j m a ij y i )x j i=1 (by slack form) J. Wang (UMass Lowell) Linear Programming 33 / 47

34 Proof Continued This implies that m v b i y i = 0, c j + i=1 m a ij y i = c j for j = 1, 2,..., n. i=1 Thus, m i=1 b iy i = v = z. Now we show that y is feasible. Note that c j 0, we have m c j = c j + a ij y i This completes the proof. i=1 m a ij y i. i=1 J. Wang (UMass Lowell) Linear Programming 34 / 47

35 Revisit: LP Formulation for Maximum Flow We have seen an LP formulation for maximum flow, where the objective function (s,u) f su has a number of variables f su for (s, u) E. In the formulation, variable f uv is the value of the flow from u to v for (u, v) E. Recall that the flow network does not have an edge from t to s (the residual network may change this, but we don t deal with residual networks in the LP formulation). To make an LP dual simpler, we reformulate LP for maximum flow with one variable in the objective function. To do so, we create an edge (t, s) to E with capacity c(t, s) = +. J. Wang (UMass Lowell) Linear Programming 35 / 47

36 LP Formulation for Maximum Flow The following LP models the maximum flow problem (Note: we use indexing i and j instead of u and v to place ourselves in our comfort zone): Maximize f ts (only one variable) Subject to (i, j) E {(t, s)} : f ij c(i, j) (capacity constraint) i V : (i,j) E f ij (j,i) E f ji = 0 (flow conservation) (i, j) E : f ij 0 Note that there is no need to include constraint f ts c(t, s). J. Wang (UMass Lowell) Linear Programming 36 / 47

37 Formulation of the Dual Each primal constraint corresponds to a dual variable. Let y ij be the variable for constraint f ij c(i, j), for (i, j) E {(t, s)}. Let y i be the variable for constraint f ij f ji = 0 for i V. (i,j) E (j,i) E For the objective function we look at the constraints in the primal with nonzero right-hand side and identify dual variables for these constraints. The following is the objective of the dual: Minimize c(i, j)y ij. (i,j) E {(t,s)} J. Wang (UMass Lowell) Linear Programming 37 / 47

38 Formulation of the Dual Continued Each primal variable x must correspond to a dual constraint. To form a dual constraint, follow the steps below: 1 Identify all the primal constraints containing x. 2 For each primal constraint containing x, let a be the coefficient of x and y be the dual variable for this constraint. 3 Sum up all ay s to become the left-hand side of the dual constraint for x. 4 The value of the right-hand side of the dual constraint is determined by the coefficient of x. J. Wang (UMass Lowell) Linear Programming 38 / 47

39 Formulation of the Dual Continued For the primal variable f ij with i t and j s, we observe that (1) the coefficient in the primal objective is 0; and (2) f ij appears only in the following three primal constraints: j:(i,j) E i:(j,i) E f ij f ji j:(i,j) E i:(i,j) E f ij c(i, j) (y ij ) f ji = 0 (y i ) f ij = 0 (y j ) Thus, for the primal variable f ij with i t and j s, we have the following dual constraint: y ij + y i y j 0. The sign is used because the variable appears in one constraint with the sign and the rest with the = sign. J. Wang (UMass Lowell) Linear Programming 39 / 47

40 Formulation of the Dual Continued For the primal variable f ts, we note that (1) its coefficient in the primal objective is 1 and (2) it appears in the following two constraints: j:(t,j) E j:(s,j) E f tj f sj j:(j,t) E j:(j,s) E f jt = 0 (y t ) f js = 0. (y s ) Thus, the dual constraint w.r.t. variable f ts is y t y s = 1. The = sign is used because the primal constraints involving variable f ts are equations. J. Wang (UMass Lowell) Linear Programming 40 / 47

41 The Dual and the Minimum Cut Finally, the following is the dual for the maximum-flow LP: Minimize (i,j) E {(t,s)} c(i, j)y ij Subject to (i, j) E {(t, s)} : y ij + y i y j 0 y t y s = 1 y ij, y i 0 We know that the maximum flow is equal to the summation of the capacity of minimum cut, and any feasible solution to the primal is bounded above by a feasible solution to the dual. Thus, the optimal solution to the dual is the minimum cut. J. Wang (UMass Lowell) Linear Programming 41 / 47

42 Exercise As an exercise let s consider the previous dual as primal and obtain its dual. Let x ij be the dual variable for the following primal constraint: y ij + y i y j 0 for (i, j) E {(t, s)} Let x ts denote the dual variable for the following primal constraint: y t y s = 1. To form the dual objective, we note that only primal constraint y t y s = 1 has nonzero right-hand side. Thus, the objective for the dual is Maximize x ts. J. Wang (UMass Lowell) Linear Programming 42 / 47

43 Exercise Continued To form the dual constraint, we note that each primal varialbe y ij with (i, j) (t, s) appears in the following constraints: y ij + y i y j 0 and it has coeficient c(i, j) in the primal objective function. Thus, for this variable we have a dual constraint x ij c(i, j) for (i, j) (t, s). Each primal variable y i has coeffieint 0 in the primal objective function. For i {s, t}, y i appears in the following constraints: y ij + y i y j 0 y ji + y j y i 0 for (i, j) E {(t, s)} for (j, i) E {(t, s)} J. Wang (UMass Lowell) Linear Programming 43 / 47

44 Exercise Continued Thus, the dual constraint for the primal variable y i with i {s, t} is x ij x ji 0. j:(i,j) E {(t,s)} j:(j,i) E {(t,s)} The primal variable y s appears in the following constraints: y sj + y s y j 0 for (s, j) E y ts + y t y s 0. Thus, the dual constraint for y s is x sj x ts 0. j:(s,j) E Likewise, the dual constraint for y t is x ts x jt 0. j:(j,t) E J. Wang (UMass Lowell) Linear Programming 44 / 47

45 Exercise Continued View x ij as flow from node i to node j. Because for each i V, the total flow entering node i is at least the total flow going out of i and there is an edge from t to s, all the inequaliies in the dual constrains must be equalities, namely, for each i V, x ij x ji = 0. j:(i,j) E Finally, the dual problem is Maximize j:(j,i) E x ts Subject to (i, j) E {(t, s)} : x ij c(i, j) i V : j:(i,j) E x ij j:(j,i) E x ji = 0 (i, j) E : x ij 0. This is exactly the same as the maximum flow LP formulation. J. Wang (UMass Lowell) Linear Programming 45 / 47

46 A Sample Multiple-Choice Quesion Unlike problems in quizes, the final exam consists of only multiple-choice (including true/false) quesions. Example: 1. Which is the correct dual for the following primal LP: (a) Minimize 3x 1 + 8x 2 + 4x 3 Subject to x 1 + 5x 2 3x 4 7 3x x 3 + 5x 4 1 x 1, x 2, x 3, x 4 0. Maximize 7y 1 + y 2 Subject to y 1 3y 2 3 5y y 2 4 3y 1 + 5y 2 0 y 1, y 2, y 3, y 4 0. J. Wang (UMass Lowell) Linear Programming 46 / 47

47 Sample Multiple-Choince Quesions Continued (b) (c) Maximize 7y 1 + y 2 Subject to y 1 3y 2 3 3y 1 + 5y 2 0 y 1, y 2, y 3, y 4 0. (d) None of the above. Maximize 3y 1 + 8y 2 4y 3 Subject to y 1 3y 2 3 5y y 2 4 3y 1 + 5y 2 0 y 1, y 2, y 3, y 4 0. J. Wang (UMass Lowell) Linear Programming 47 / 47