INTEGER PROGRAMMING Integer Programming In many problems the decision variables must have integer values. Example:assign people, machines, and vehicles to activities in integer quantities. If this is the only deviation from linear programming, it is called an integer programming (IP) problem. If only somevariables are required to integer, the model is called a mixed integer programming (MIP) San Francisco Police Dep. problem is an IP problem. Wyndor Glass Co. problem could be an IP problem; how? 103 1
Integer Programming In integer programming, divisibility assumption must be dropped. Another area of application is in problem involving yes-or-no decisions, which have binary variables. These IP problems are called binary integer programming (BIP) problems. A very small example of a typical BIP problem is given in the following. 104 Prototype example California Manufacturing Company is considering expansion building a factory in Los Angeles, San Francisco or in both cities. One new warehouse can also be considered in a city where a new factory is being built. Maximum $10 million to invest. Objective: find feasible combination of alternatives that maximizes the total net present value. Decision number Yes-or or-no question Decision variable Net present value Capital required 1 Build factory in Los Angeles? x 1 $9 million $6 million 2 Build factory in San Francisco? x 2 $5 million $3 million 3 Build warehouse in Los Angeles? x 3 $6 million $5 million 4 Build warehouse in San Francisco? x 4 $4 million $2 million 105 2
BIP model All decisions variables have the binary form: x j 1 if decision j is yes, = j = 1, 2,3,4 0 if decision j is no, Z = total net present value of these decisions. Maximize Z = 9x 1 + 5x 2 + 6x 3 + 4x 4. Constraints: 6x 1 + 3x 2 + 5x 3 + 2x 4 10 x 3 + x 4 1 x 3 x 1 and x 4 x 2 x j is binary, for j = 1,2,3,4. Mutually exclusive alternatives Contingent decisions 106 BIP models Groups of yes-or-no decision often constitute groups of mutually exclusive alternatives: only onedecision in the group can be yes. Occasionally, decisions of the yes-or-no type are contingent decisions: decision that depend upon previous ones. Software options for solving BIP, IP or MIP models: Excel LINGO/LINDO MPL/CPLEX 107 3
BIP applications Investment analysis, such as the California Man. Co. Site selection, of factories, warehouses, energy power sources, etc. Designing a production and distribution network, or more generally the entire global supply-chain. Dispatching shipments, scheduling routes, vehicles and time period for departure and arrivals. Airline applications, as e.g. fleet assignment and crew scheduling. Scheduling interrelated activities, asset divestures, etc. 108 Formulation examples Example 1: making choices when decision variables are continuous. R&D Division of Good ProductsCo. has developed three possible new products. Requirement 1:from the three, at mosttwo can be chosen to be produced. Each product can be produced in either of two plants. However, management has imposed a restriction: Requirement 2:just one of the two plants can be chosen as the producer of the new products. 109 4
Example 1 Objectives:choose the products, the plant and the production rates of the chosen products to maximize total profit. Production time used for each unit produced Production time available per Product 1 Product 2 Product 3 week Plant 1 3 hours 4 hours 2 hours 30 hours Plant 2 4 hours 6 hours 2 hours 40 hours Unit profit 5 7 3 (10 3 euros) Sale potential 7 5 9 (units per week) 110 Formulation of the problem Similar to a standard product mix problem, such as the Wyndor Glass Co. It is such a problem, if we drop the two restrictions andby requiring that a product uses production hours in both plants. Let x 1, x 2, x 3 be the production rates of the respective products: Maximize Z = 5x + 7x + 3x subject to 1 2 3 3x + 4x + 2x 30 1 2 3 4x + 6x + 2x 40 1 2 3 x 7 x 1 2 5 x 9, x, x, x 0 3 1 2 3 111 5
Formulation of the problem For the real problem, Requirement 1add the constraint: Number of strictly positive variables (x 1, x 2, x 3 ) must be 2 This must be converted to an IP problem. It needs the introduction of auxiliary binary variables. Requirement 2 requires replacing the first two constraints to: Either 3x + 4x + 2x 30 1 2 3 or 4x + 6x + 2x 40 1 2 3 must hold. This again requires an auxiliary binary variable. 112 Auxiliary binary variables For Requirement 1, three auxiliary binary variables (y 1, y 2, y 3 )are introduced: y j 1 if x j > 0 can hold (can produce product j) = 0 if x j = 0 must hold (cannot produce product j) This is introduced in the model with the help of an extremely large positive number M, adding the constraints: x x x 1 1 2 2 3 3 1 + 2 + 3 2 j My My My y y y y is binary, for j = 1,2,3. 113 6
Auxiliary binary variables For Requirement 2, another auxiliary binary variable y 4 is introduced: y 4 1 if 4x1 + 6x2 + 2x3 40 must hold (choose Plant 1) = 0 if 3x1 + 4x2 + 2x3 30 must hold (choose Plant 2) This add the constraints: 3x + 4x + 2x 30 + My 4 1 2 3 4 4x + 6x + 2x 40 + M (1 y ) y 1 2 3 4 is binary 114 Complete model (MIP) Maximize Z = 5x + 7x + 3x 1 2 3 subject to x 7 1 2 3 1 1 2 2 3 3 1 2 3 1 2 3 4 1 2 3 4 5 0 y + y + y 2 3x + 4x + 2x My 30 4x + 6x + 2x + My 40 + M and x 0, for i = 1, 2,3 y i j x x x is binary, for j = 1, 2,3, 4 x x 9 My 0 My My 0 115 7
Solution MIP problem with 3 continuous and 4 binary variables. Optimal solution: y 1 = 1, y 2 = 0, y 3 = 1, y 4 = 1, x 1 = 5.5, x 2 = 0, x 3 = 9. That is, choose products 1 and 3 to produce, choose Plant 2 for production, choose production rates of 5.5 units per week for product 1 and 9 units per week for product 2. Resulting profit is 54500 per week. 116 Example 2 See Hillier s book (page 496). Profit from additional sales violates proportionality: 117 8
Example: Southwestern Airways Southwestern Airways needs to assign three crews to cover all the upcoming flights. Tableshows the flights in the first column. Other 12 columns show the 12 feasible sequences of flights for a crew. Numbers in each column indicate the order of the flights. Exactly three sequences must be chosen (one for each crew). More than one crew can be assigned to a flight, but it must be paid as if it was working. Last column show the cost of assigning a crew to a sequence of flights. 118 Data for Southwestern Airways Feasible sequence of flights Flight 1 2 3 4 5 6 7 8 9 10 11 12 1. San Francisco to Los Angeles 1 1 1 1 2. San Francisco to Denver 1 1 1 1 3. San Francisco to Seattle 1 1 1 1 4. Los Angeles to Chicago 2 2 3 2 3 5. Los Angeles to San Francisco 2 3 5 5 6. Chicago to Denver 3 3 4 7. Chicago to Seattle 3 3 3 3 4 8. Denver to San Francisco 2 4 4 5 9. Denver to Chicago 2 2 2 10. Seattle to San Francisco 2 4 4 5 11. Seattle to Los Angeles 2 2 4 4 2 Cost (1000 ) 2 3 4 6 7 5 7 8 9 9 8 9 119 9
Formulation of the problem Objective: minimize the total cost for the three crews. 12 feasible sequence of flights: 12 yes-or-no decisions: Should sequence jbe assigned to a crew? The 12 binary variables to represent the decisions are: x j 1 if sequence j is assigned to a crew = 0 otherwise 120 Formulation of the problem Minimize Z = 2x + 3x + 4x + 6x + 7x + 5x + 7x + 8x + 9x + 9x10 + 8x11 + 9x12 subject to x1 + x4 + x7 + x10 1 (SF to LA) x2 + x5 + x8 + x11 1 x3 + x6 + x9 + x12 1 x4 + x7 + x9 + x10 + x12 1 x1 + x6 + x10 + x11 1 x4 + x5 + x9 1 x7 + x8 + x10 + x11 + x12 1 x2 + x4 + x5 + x9 1 x5 + x8 + x11 1 x3 + x7 + x8 + x12 1 x + x + x + x + x 1 6 9 10 11 12 12 1 2 3 4 5 6 7 8 9 x j = 3 (assign three crews) j= 1 and x j is binary, for j = 1, 2,,12 121 10
Solution One optimal solution is: x 3 = 1 (assign sequence 3 to a crew) x 4 = 1 (assign sequence 4 to a crew) x 11 = 1 (assign sequence 11 to a crew) And all other x j = 0. Total cost is 18000. Another optimal solution is: x 1 = x 5 = x 12 = 1. 122 Discussion This example belongs to a class called set covering problems, with a number of potential activities(e.g. flight sequences) and characteristics(e.g. flights). S i is the set of all activities that possess characteristic i. A constraint is included for each characteristic i: j S i In set partitioning problems the constraint is j S j x x j j 1 = 1 123 11
Solving IP problems Integer problems are easy to solve? Difference to LP is that IP have far fewer solutions. IP problems have a finite number of feasible solutions. However: Finite numbers can be astronomically large! With n variables a BIP problem has 2 n solutions, having exponential growth. LP assuresthat a CFP solution can be optimal, guaranteeing the remarkable efficiency of the simplex method. LP problems are much easier to solve than IP problems. 124 Solving IP problems Consequently, most IP algorithms incorporate the simplex method. This is called the LP relaxation. Sometimes, the solution of the LP problem isthe solution of the IP problem, such as: Minimum cost flow problem, including transportation problem, assignment problem, shortest-path problem and maximum flow problem. Special structures (see examples 2 and 3): mutually exclusive alternatives, contingent decisions or setcovering constraints can also simplify the problem. 125 12
Solving IP problems Primary determinants of computational complexity: 1. number of integer variables, 2. these variables are binary or general integer variables, 3. any special structure in the problem. This is in contrast to LP, where number of constraints is much more important than the number of variables. As IP problems are much more difficult than LP, we could apply LP and roundthe obtained solution... Yes?? 126 Example 1 Maximize Z = x 2 subject to x1 + x2 0.5 x1 + x2 3.5, and x, x 0, integers. 1 2 127 13
Example 2 Maximize Z = x + 5x 1 2 1 2 subject to x1 + 10x2 20 x1 2 and x, x 0, integers. 128 Solving IP problems Thus, a better approach to deal with IP problems that are too large to be solved exactly are heuristic algorithms. Heuristics and metaheuristicsare extremely efficient for very large problems, but do not guaranteeto find an optimal solution. These algorithms will be discussed further later. Most popular traditional method for solving IP problems is the branch-and-bound technique. 129 14
Branch-and-bound applied to BIP Pure IP problems can consider some type of enumeration procedure. This should be done in a clever way such that only a tiny fraction of the feasible solutions is examined. Branch-and-boundwith a divide to conquertechnique can be used. dividing (branching) the problem into smaller and smaller subproblems until it can be conquered conquering (fathoming) by boundinghow good the best solution can be. If no optimal solution in subset: discard it. 130 Example: California Manuf, Co. Recall prototype example: Maximize Z = 9x 1 + 5x 2 + 6x 3 + 4x 4 subject to (1) 6x 1 + 3x 2 + 5x 3 + 2x 4 10 (2) x 3 + x 4 1 (3) x 1 + x 3 0 (4) x 2 + x 4 0 and (5) x j is binary, for j = 1, 2, 3, 4. 131 15
Branching Most straightforward way to divide the problem: fix the value of a variable at one set. e.g. x 1 = 0 for one subset and x 1 = 1 for another subset. Subproblem 1(fix x 1 = 0): Maximize Z = 5x 2 + 6x 3 + 4x 4 subject to (1) 3x 2 + 5x 3 + 2x 4 10 (2) x 3 + x 4 1 (3) x 3 0 (4) x 2 + x 4 0 (5) x j is binary, for j = 2, 3, 4. 132 Branching Subproblem 2(fix x 1 = 1): Maximize Z = 9 + 5x 2 + 6x 3 + 4x 4 subject to (1) 3x 2 + 5x 3 + 2x 4 4 (2) x 3 + x 4 1 (3) x 3 1 (4) x 2 + x 4 0 (5) x j is binary, for j = 2, 3, 4. 133 16
Branching Dividing (branching) into suproblems creates a tree with branches(arcs) for the Allnode. This is the solution treeor enumeration tree. Branching variable is the one used for branching. The branching continues or not after evaluating the subproblem. Other IP problems usually creates as many branches as needed. 134 Bounding A boundis needed for the best feasible solution of the subproblem. Standard way is to perform a relaxationof the problem, e.g. by deletingone set of constraints that makes the problem difficult. Most common is to require integer variables, so LP relaxation is the most widely used. 135 17
Bounding in example Example:for the whole problem, (5) is replaced by x j 1 and x j 0for j=1, 2, 3, 4. Using simplex: (x 1, x 2, x 3, x 4 ) = (5/6, 1, 0, 1), with Z = 16.5 Thus, Z 16.5 for all feasible solutions for BIP problem. Can be rounded to Z 16 (why?) LP relaxation for subproblem 1: (x 1, x 2, x 3, x 4 ) = (0, 1, 0, 1), with Z = 9 LP relaxation for subproblem 2: (x 1, x 2, x 3, x 4 ) = (1, 4/5, 0, 4/5), with Z = 16.5 136 Fathoming A subproblem can be conquered (fathomed, i.e. search tree is pruned) in three ways: 1. When the optimal solution for the LP relaxation of a subproblem is integer, it must be optimal. Example: for x 1 =0, (x 1, x 2, x 3, x 4 ) = (0, 1, 0, 1), is integer. It must be stored as first incumbent(best feasible solution found so far) for the whole problem, along with value of Z: Z * = value of Z for first incumbent In the example Z * = 9. Subproblem 1 is solved, so it is fathomed(dismissed). 137 18
Fathoming 2. As Z * = 9, we should not consider subproblems with bound 9. Thus, a problem is fathomed when bound Z * In Subproblem 2 that does not occur, the bound of 16is larger than 9. However, it can occur for descendants. As new incumbents with larger values of Z * are found, it becomes easier to fathom in this way. 3. If the simplex method finds that a subproblem s LP relaxation has no feasible solution, the subproblem has no feasible solution and can dismissed. 138 Summary of fathoming tests A subproblem is fathomed (dismissed) if Test 1:Its bound Z * or Test 2: Its LP relaxation has no feasible solutions or Test 3:Optimal solution for its LP relaxation is integer. If better, this solution becomes new incumbent, and Test 1 is reapplied for all unfathomed subproblems. 139 19
Fathoming in example Result of applying the three tests is in figure below. Subproblem 1is fathomed by test 3. 140 BIP branch-and-bound algorithm Initialization:Set Z * =. Apply bounding, fathoming and optimization steps described below to the whole problem. If not fathomed, perform iteration. Steps for each iteration: 1. Branching:Among the remaining subproblems, select the one created most recently. Branch from this node by fixing the next variable as either 0 or 1. 2. Bounding:For each new subproblem, obtain its bound by applying its LP relaxation. Round down Z for resulting optimal solution. 141 20
BIP branch-and-bound algorithm 3. Fathoming:For each new subproblem, apply the three fathoming tests, and discard subproblems that are fathomed by the tests. Optimality test:stop when there are no remaining subproblems. The current incumbent is optimal. Otherwise, perform another iteration. 142 Completing example Iteration 2.Remaining subproblems are for x 1 = 1. Subproblem 3(fix x 1 = 1, x 2 = 0): Maximize Z = 9 + 6x 3 + 4x 4 subject to (1) 5x 3 + 2x 4 4 (2) x 3 + x 4 1 (3) x 3 1 (4) x 4 0 (5) x j is binary, for j = 3, 4. 143 21
Example Subproblem 4(fix x 1 = 1, x 2 = 1): Maximize Z = 14 + 6x 3 + 4x 4 subject to (1) 5x 3 + 2x 4 1 (2) x 3 + x 4 1 (3) x 3 1 (4) x 4 1 (5) x j is binary, for j = 3, 4. 144 Example LP relaxation is obtained by replacing (5) by 0 x j 1 j = 3, 4.Optimal solutions are: LP relaxation for Subproblem 3: (x 1, x 2, x 3, x 4 ) = (1, 1, 0.8, 0), with Z = 13.8 LP relaxation for Subproblem 4: (x 1, x 2, x 3, x 4 ) = (1, 1, 0, 0.5), with Z = 16 Resulting bounds: Bound for subproblem 3: Z 13 Bound for subproblem 4: Z 16 145 22
Example All three fathoming tests fail, so both are unfathomed. 146 Iteration 3 Subproblem 4 has the larger bound, so next branching is done from (x 1, x 2 ) = (1, 1). Subproblem 5(fix x 1 = 1, x 2 = 1, x 3 = 0): Maximize Z = 14 + 4x 4 subject to (1) 5x 3 + 2x 4 1 (2), (4) x 4 1 (5) x 4 is binary 147 23
Iteration 3 (cont.) Subproblem 6(fix x 1 = 1, x 2 = 1, x 3 = 1): Maximize Z = 20 + 4x 4 subject to (1) 2x 4 4 (2) x 4 0 (4) x 4 1 (5) x 4 is binary LP relaxation: replace (5)by 0 x 4 1.Optimal solutions are: LP relaxation for subproblem 5: (x 1, x 2, x 3, x 4 ) = (1, 1, 0, 0.5), Z = 16 LP relaxation for subproblem 6: No feasible solutions. Bound for subproblem 5: Z 16 148 Iteration 3 (concl.) Subproblem 6 is fathomed by test 2, but not Subproblem 5. 149 24
Iteration 4 Node created most recently is selected for branching: x 4 = 0: (x 1, x 2, x 3, x 4 ) = (1,1,0,0) is feasible, with Z = 14 x 4 = 1: (x 1, x 2, x 3, x 4 ) = (1,1,0,1) is infeasible. First solution passes test 3 and second passes test 2 for fathoming. First solution is better than incumbent, so it becomes new incumbent, with Z * = 14 Reapplying fathoming test 1 to remaining branch to remaining Subproblem 3, it is fathomed: Bound = 13 Z * = 14. 150 Solution tree after Iteration 4 151 25
Other options in B&B Branchingcan be done e.g. from the best boundrather than from the most recently created subproblem. Boundingis done by solving a relaxation. Another possible one is e.g. the Lagrangian relaxation. Fathoming criteria can be generally stated as: Criterion 1:feasible solutions of subproblem must have Z Z * Criterion 2: the subproblem has no feasible solutions, or Criterion 3:an optimal solution of subproblem has been found. Branch-and-bound can find a nearly optimal solution. 152 Branch-and-bound for MIP General form of the problem: Maximize Z = j= 1 j= 1 subject to a x b, for i = 1,2,, m, and n n j j ij j i x 0, for j = 1,2,, n j x is integer, for j = 1,2,, I; I n. j c x 153 26
Branch-and-bound for MIP Similar to BIP algorithm. Solving LP relaxationsare the basis for bounding and fathoming. Four changes are needed: 1. Choice of branching variable. Only integer variables that have a noninteger valuein the optimal solution for the LP relaxation can be chosen. 154 Branch-and-bound for MIP 2. As integer variables can have a large number of possible values, create just two new subproblems: x j* : noninteger value of optimal solution for LP relaxation. [x j* ] = greatest integer x j*. Range of variables for two new subproblems: x j* [x j* ] and x j* [x j* ] + 1. Each inequality becomes an additional constraint. Example: x j* = 3.5, then: x j* 3 and x j* 4. When changes 1.and 2.are combined, a recurring branching variable can occur, see figure. 155 27
Recurring branching variable 156 Branch-and-bound for MIP Changes needed: 3. Bounding step: value of Zwas rounded downin BIP algorithm. Now some variables are not integerrestricted so bound is value of Z without rounding. 4. Fathoming test 3: optimal solution for the subproblem s LP relaxation must only be integerfor integer-restricted variables. 157 28
MIP branch-and-bound algorithm Initialization:Set Z * =. Apply bounding, fathoming and optimization steps described below to the whole problem. If not fathomed, perform iteration. Steps for each iteration: 1. Branching: Among the remaining subproblems, select the one created most recently. From integer variables that have a noninteger value in the optimal solution for the LP relaxation chose the first one. Let x j be this variable and x j* its value. Branch from this creating two subproblems by adding the respective constraints: x j* [x j* ] and x j* [x j* ] + 1. 158 MIP branch-and-bound algorithm 2. Bounding:For each new subproblem, obtain its boundby applying its LP relaxation. Use Z without rounding for resulting optimal solution. 3. Fathoming:For each new subproblem, apply the three fathoming tests, and discard subproblems that are fathomed by the tests. Test 1: Its bound Z *, where Z * is value of Zfor current incumbent. Test 2: Its LP relaxation has no feasible solutions. 159 29
MIP branch-and-bound algorithm 3. Fathoming (cont.): Test 3: Optimal solution for its LP relaxation is integerfor integer-restrictedvariables. (if this solution is better it becomes new incumbent, and test 1 is reapplied for all unfathomed subproblems). Optimality test:stop when there are no remaining subproblems. The current incumbent is optimal. Otherwise, perform another iteration. See MIP example in PL#7 and in page 518 of Hillier s book. 160 Branch-and-cut approach to BIP Branch-and-bound was develop and refined in the 1960s and early 1970s. Can solve problems up to 100 variables. Branch-and-cut approach was introduced in the mid 1980s, which can solve problems with thousands variables. Only solve large problems if they are sparse (less than 5 or even 1% of nonzerovalues in the functional constraints). Uses a combination of: automatic problem processing, generation of cutting planes and B&B techniques. 161 30
Automatic problem processing for BIP Computer inspection of IP formulation to spot reformulationsthat make the problem quicker to solve: Fixing variables:identify variables that can be fixed at 0 or 1, because other value cannot be optimal. Eliminating redundant constraints: identify and eliminate constraints that are automatically satisfied by solutions that satisfy all other constraints. Tightening constraints:tighten constraints in a way that reduces feasible region of LP relaxation. 162 Tightening constraints LP relaxation including feasible region. LP relaxation after tightening constraint. 163 31
Generating cutting planes for BIP Cutting plane(or cut) is a new functional constraint that reduced feasible region for LP relaxation. Procedure for generating cutting planes: 1. Consider functional constraint in form. 2. Find a group of Nvariables such that a) Constraint is violated if every variable in group = 1and all other variables = 0. b) It is satisfied if value of anyvariables changes from 1 to 0. 3. Resulting cutting plane: sum of variables in group N 1. 164 Constraint Programming Combination of artificial intelligence with computer programming languages in the mid-1980s. Flexibility in stating (nonlinear) constraints: 1. Mathematical constraints, e.g., x + y < z. 2. Disjunctive constraints, e.g., times of certain tasks cannot overlap. 3. Relational constraints, e.g., at least three tasks should be assigned to a certain machine. 165 32
Stating constraints 4. Explicit constraints, e.g., xand yhave same domain {1,2,3,4,5}, but (x, y) must be (1, 1), (2, 3)or (4, 5). 5. Unary constraints, e.g. zis integer between 5 and 10. 6. Logical constraints, e.g., if x = 5, y {6, 7, 8}. Standard logical functions such as IF, AND, OR, NOT can be used. Constraint programming applies domain reduction and constraint propagation. The process creates a tree search. 166 Example Consider: x 1 {1,2}, x 2 {1,2}, x 3 {1,2,3}, x 4 {1,2,3,4,5} Constraints: 1. All variables must have different values; 2. x 1 +x 3 = 4. Apply domain reduction and constraint propagation to obtain feasible solutions: x 1 {1}, x 2 {2}, x 3 {3}, x 4 {4,5}. 167 33
Constraint Programming Steps in Constraint Programming: 1. Formulate a compact model of the problem by using a variety of constraint types (most not of IP type). 2. Efficiently find feasible solutions that satisfy all these constraints. 3. Search among feasible solutions for an optimal one. Strength of constraint programming is in first two steps, whereas the main strength of IP is in step 3. Current research: integrate CP and IP! 168 34