Interior-Point versus Simplex methods for Integer Programming Branch-and-Bound Samir Elhedhli elhedhli@uwaterloo.ca Department of Management Sciences, University of Waterloo, Canada Page of 4 McMaster University, March 22, 24
. Motivation Interior-point methods are quite useful for solving largescale LPs, but they are not widely employed in branchand-bound algorithms because it is cumbersome to reoptimize an LP with an interior-point code after rows or columns have been added. E.L. Johnson, G. Nemhauser and M.W.P. Savelsbergh, Progress in linear programmingbased algorithms for integer programming: An exposition, Informs JOC,2:, 2. the simplex method has a significant advantage over interiorpoint methods within a branch and bound algorithm. In the branch-and-bound algorithm we can warm start the solution of each new subproblem with the optimal solution from the parent subproblem. Using the simplex method, a new optimal solution can easily be found after a few simplex iterations. With interior-point methods, warm starts are not as effective. J. E. Mitchell and B. Borchers Using an interior point method in a branch-and-bound Page 2 of 4 algorithm for integer programming, 99, revised 992, Rensselaer Polytechnic Institute, Troy, N.Y.
2. Branch-and-bound: LP bound Simplex Optimal Primal Optima Dual Basic NB branching constraints (Primal) Page 3 of 4 First Primal First Dual Basic N B Dual simplex
Replacing the simplex by an IPM is not going to work. (you may sacrifice warm starting). crossover is a prtial solution (Cplex) How should we apply IPMs? Save an approximate solution before termination (Gondzio et al. 998. Math. Prog.) Cutting plane/column generation context (Elhedhli and Goffin. 23. Math. Prog.) Page 4 of 4
3. A cutting plane context: Lagrangean Relaxation [MIP ] : min c T x s.t. Ax b (complicating) λ Bx d (easy) x x j integer for j J Lagrangean subproblem: [MIP λ ] : b T λ + min (c Ax) T λ s.t. Bx d x x j integer for j J Dual Lagrangean problem The dual master problem { max b T λ + v(ip λ ) } λ Page 5 of 4 max b T λ + λ s.t. (c Ax i ) T λ λ i I
max b T λ + λ s.t. λ + (Ax i ) T λ c T x i i I Where x i are the integer feasible points to the bounded set: {x : Bx d, x j integer for j J, x } The Dantzig-Wolfe master problem min s.t. i I (c T x i )α i α i (Ax i ) b i I α i = i I i I α i, i I. Page 6 of 4
4. Facts: The LR (DW) bound is at least as good as the LP bound. The subproblems are usually decomposable: solved much faster & allow the addition of multiple cuts (columns). Columns can be used as a basis for generating good feasible solutions (better incumbents). Exponential number of cuts (columns): need a cutting plane (column generation) approach May suffer from tailing effects. selected with care. Cuts (columns) should be Page 7 of 4
5. Using the LR bound within branch and bound: branchand-price Columns satisfying left branching Optimal Bound : zl Analytic center : a a a ( x, y, s ) Columns satisfying right branching Page 8 of 4 zl is valid Analytic center : a a ( y, s ) is dual feasible Dual IPM
6. Issues Careful branching: you may make the subproblerms more difficult to solve (considerably distort their structure) Early termination of the column gneeration/cutting plane scheme Warm starting Primal and dual heuristics Page 9 of 4
7. Example: The (binary) cutting stock problem Given: Rolls of length V and demand d l for items of lengths D l l =,..., L Find: the minimum number of rolls necessary to meet the demand D Page of 4 V D l If d l is either zero or one, we get the bin packing problem
8. MIP model for the bin packing problem { zk = when bin k is used Binary variables : y kl = when item l is assigned to bin k [BP P ] min s.t. K z k k= K y kl = l =,.., L Single sourcing constraints k= L D l y kl V z k k =,..., K Capacity constraints z k {, } k =,..., K; y kl {, } k =,..., K; l =,..., L; Page of 4 Question: Which decomposition to use? Answer: Depends on the quality of the bound.
9. The LP bound: LP The dual: [LP ] min s.t. K z k k= Dual variable K y kl = l ν l k= L D l y kl V z k k µ k z k k ρ k y kl k, l max s.t. L ν l K ρ k k= ν l D l µ k k, l V µ k ρ k = k µ k, ρ k k Solution: ρ k = ; ν l = D l V ; µ k = V ; L LP bound: D l V. Page 2 of 4
. Lagrangean relaxation of capacity constraints: LR2 min s.t. K z k k= K y kl = k= l L D l y kl V z k k :µ k z k {, } k y kl {, } k l min s.t. K k= L µ k D l y kl + K ( µ k V )z k K y kl = k= k= z k {, } k y kl {, } k l Optimal: µ k = V K L (Geoffrion): LR bound: V D ly kl = L k= V D l l K y kl = L k= D l V = LP Page 3 of 4
. Lagrangean relaxation of the single sourcing constraints: LR min s.t. K z k k= K y kl = l :λ l k= L D l y kl V z k k z k {, } k y kl {, } k l = The dual Lagrangean problem is : max λ Master problem: max Kθ + L s.t. λ l [KP λ ] min y,z ( λ l )y l + z l s.t. D l y l V z l y l, z =, l K identical - knapsacks λ l + Kv(KP λ ) L ( λ l )yl h + z h θ h =,..., H H is the index set of the integer solution to the sets l Page 4 of 4
{ (y l, z) : L } D l y l V z; y l, z =, ; l =,.., L Dual: the Dantzig-Wolfe master problem Bound: min s.t. H h= ( z h ) α h H α h = K h= H yl hα h = l =,.., L h= α h h =,..., H. v(lp ) = v(lr 2 ) v(lr ) Page 5 of 4
2. Lagrangean decomposition: LD min z k k s.t. x kl = k l Two sets of subproblems [SP k y ] min s.t. L x kl = y kl k l :π kl D l y kl V k k l y kl, z k =, k, l π kl y kl + z k L D l y kl V z k y kl, z k =, l The dual Lagrangean problem: K max v(spy k ) + λ k= \qquad L v(spx) l [SP l x] min s.t. K k= π kl x kl K x kl = k= x kl =, k Page 6 of 4
The full master problem max K k= s.t. θ k + l θ k + L θ l (π kl )y h kl zh k h =,..., H k y. θ l k π kl x h kl h =,..., Hl x. where H k y and H l x are as defined previously. Easy structure of [SPx] l = ( K L ) max π kl ykl h + zk h + π = max π k= K k= min h H k y min h H k y ( L ) π kl ykl h + zk h + Which yields a smaller master problem max K k= s.t. θ k + l θ k + L θ l L L ( ) min π kl x h h Hx l kl ( ) min π kl k=,..,k (π kl )y h kl zh k h =,..., H k y ; k =,..., K Page 7 of 4 θ l π kl k =,.., K; l =,.., L
3. Lagrangean decomposition: LD2 Instead of copying variable y, let us copy D l y kl, leading to [BP P ] min z k k s.t. y kl = k l x kl = D l y kl k; l :ω kl x kl V z k k l y kl, z k =, k; l x kl D l k; l Two sets of subproblems Page 8 of 4 [SPy k ] min s.t. K D l y kl k= y kl = k y kl =, l \qquad [SP l x] min s.t. L ω kl x kl + z k x kl V z k l z k =, k x kl D l k
The dual Lagrangean problem is max λ K v(spy k ) + k= The full master problem max K k= s.t. θ k + l θ l k θ k + L Ranking of the bounds θ l L v(spx) l (ω kl )y h kl zh k h =,..., H k y. ω kl x h kl h =,..., Hl x. v(lr2) = v(lp ) v(ld2) v(lr) = V (LD) Page 9 of 4
4. Using v(lr) in an interior-point branch-and-price framework Master problem max Kθ + L s.t. λ l L ( λ l )yl h + z h θ h =,..., H H { is the index set of the integer points: } L (y l, z) : D l y l V z; y l, z =, ; l =,.., L Dual: the Dantzig-Wolfe master problem min s.t. H h= ( z h ) α h H α h K h= H yl hα h = l =,.., L h= α h h =,..., H. Page 2 of 4
As H α h h= = K is redundant, ( we assume that we have a sufficient number of available bins), We get min s.t. H h= ( z h ) α h H yl hα h = l =,.., L h= α h h =,..., H. The Gilmore-Gomory formulation for the cutting stock problem Page 2 of 4
5. Cutting plane methods Relaxed master problem: max Kθ + L s.t. λ l L ( λ l )yl h + z h θ h H H Solution: Cutting Plane Method Get a query point λ from the relaxed master problem Check for optimality at the full master problem. If not optimal, add a new set of constraints Solve the K subproblems [SP λ ] m Get lower bound (z l ) : λ l + Kv(RMP ) Form a new relaxed master problem Page 22 of 4
6. ACCPM The choice of the query point gives the different variants of the cutting plane algorithms. Kelley s cutting plane method (Kelly, 96): the query point is the optimal solution of [RMP ]. Analytic Centre Cutting Plane Method (ACCPM): (Goffin, Haurie & Vial, 992) the query point is the analytic centre of the localization set: (θ, λ) : Kθ + m λ l z l ; F D (z l ) = θ + m yl hλ l z h ; h H, Page 23 of 4
7. ACCPM Kelley s (usually Simplex methods) ACCPM (interior point methods) Cut Localization Set ac New ac Lower bound New LB Page 24 of 4 Extreme point
8. Computing the Analytic Centre The analytic centre is the point that maximizes the distance from the boundaries of the localization set F q D (z l). is the unique point maximizing The weighted dual potential. The weighted primal potential The weighted primal dual potential satisfies the first order conditions Page 25 of 4 S x = Ne ν () Ã x =, x > (2) Ã T y + s = c, s > (3)
9. Waram starting Computing the analytic centre after adding cuts Primal IPM. Primal-dual IPM. Computing the analytic cemnte after branching Dual IPM. Page 26 of 4
2. The IP-B&P algorithm Columns satisfying left branching Optimal Bound : zl Analytic center : a a a ( x, y, s ) Columns satisfying right branching Page 27 of 4 zl is valid Analytic center : a a ( y, s ) is dual feasible Dual IPM
Initilaization: Intial upper bound: UB =. Initial set of nodes to explore: S = {}. Initial lower bound for node : LB =. Initial matrix for node : A is empty. Iteration: While there are nodes to explore (S is not empty) End while. Pick a node, say node n 2. Get a lower bound LB n from the full master problem. Use the matrix A n as a starting matrix. Use the lower bound LBn as an intial lower bound Get a lower bound LB n by applying a cutting plane/column generation method (ACCPM, Kelley s method or Bundle methods ). 3. If possible, generate a feasible solution. This gives an upper bound UB n 4. Update upper bound: UB = min (UB n, UB) 5. If LB n UB 6. Else Fathom node n: S = S\{n}. Branch: Create two new nodes n and n 2 : S = S {n, n 2 } 7. Save warm starting information: Use the branching rule to split the columns of A n into two matrices A n and A n 2. Use them as initial matrices for the child nodes n and n 2. The lower bound LB n is valid for the child nodes. Use it to initialize the lower bounds at child nodes n and n 2. LBn = LB n and LBn 2 = LB n. Page 28 of 4 Figure : The main steps of the branch-and-price algorithm
2. Branching rule Pick two items l and l 2 that are once put in the same bin k and once put in diffeent bins (search for an other bin k 2 that has one but not the other.) This corresponds to identifying the pattern x k l = x k l 2 = x k2 l = ; x k2 l 2 = Branching constraints x kl = x kl2 x kl + x kl2 Page 29 of 4
Page 3 of 4 l h
22. Generating feasible solutions Dual heuristics: Rounding α kh Rounding the original variables y kl = h y h kl α kh min δ s.t. n k= h H k Primal heuristics ( m ) c kl ykl h δ kh δ kh = h H k n ykl h δ kh = k= h H k δ kh Use the set of generated columns. k =,..., n l =,..., m h H k, k =,..., n Page 3 of 4
23. Numerical Testing Matlab 6. Cplex 7.5 Problems Triplets Random On a Sun Ultra-/44 workstation. Page 32 of 4
24. ACCPM: x 5 The progress of the bounds Bound 2.... upper bound lower bound best lower bound 2 3 4 5 6 7 8 9 Newton iters to compute the Analytic centre Newton iters 5 2 3 4 5 6 7 8 9 Newton iters to find a primal feasible point Newton iters 5 Page 33 of 4 2 3 4 5 6 7 8 9 Phase I Phase II Figure 2: The progress of the lower and upper bounds in ACCPM
25. Comparsion between AC-BP, K-BP and Cplex-MIP 7.5 LP Lag. AC- BP Cplex-MIP 7.5 K-BP bound bound Nodes CPU b Nodes Gap CPU b Nodes CPU b BinG 77 92.86 4.3 77. 4.5 BinG 2 83 95.83 9.6 5.8 9.5 BinG 3 93.3 96.9 6.7 82 7.4%() a 6 6.4 BinG 4 96.84 8.3 55.29 2.76 BinG 5 93.7 98.5 24.54 5327 3.84%() 6 23.38 BinG 6 89.2 8.69 223.93 26 2.6 BinG 7 87.74 2.4 46 2.47 25 3.62 BinG 8 97.7 99.7 3 2.79 457 2.43%() 6 37.5 BinG 9 9 99.5 38 3.2 5494.92%() 6. 3 7.86 BinG 92.98 4 5.7 27 22.95 4 8.9 BinG 2 93.87 48 7.8 458 3.93 52 23.42 BinG 4 9.9 99.35 6 7.78 66 9.86%(7) 6 62 3.4 BinG 5 93.42 99.38 67 2.32 9 7.89%(6) 6 66 39.67 BinG 7 96.55 99.89 69 28.75 527 6 7 5.3 BinG 2 97.43 93 57.69 45 6 7.8 Min 77 92.86 4.3 4.5 Max 97.43 93 57.69 7 7.8 Average 9.66 98.65 37.7 9.75 32.27 6.68 (.) a : Difference between lower and upper bound. (.) b : All CPU s in minutes. : Failed to find a feasible solution within the allowed time. # : Did not get past node. (.) d : found lower bound as a percentage of best lower bound. : Gap =. Page 34 of 4
26. Comparsion between AC-BP, K-BP and Cplex-MIP 7.5 LP Lag. AC- BP Cplex-MIP 7.5 K-BP bound bound Nodes CPU b Nodes Gap CPU b Nodes CPU b BinT 2.2..6 BinT 2.7 266.6.3 BinT 3.2 8686 %() a > 2.89 BinT 5.34 285 5.88%() > 2 3.58 BinT 6.94 399 5%() > 2 2.4 BinT 72 3.76 328 4.6%() > 2 34.2 BinT 8 6.4 276 3.7%() > 2 37.68 BinT 9 2.9 2439 3.33%() > 2 56.54 BinT 2 5.69 9865 2.94%() > 2 78.62 BinT 4 22.6 33 2.7%() > 2 #(99.72%) d > 2 BinT 2 2 34.3 798 2.5%() > 2 #(99.9%) > 2 BinT 4 36 55.96 533 2.2%() > 2 #(99.5%) > 2 BinT 5 34 88.8 56 > 2 #(99.52%) > 2 BinT 8 7 8.92 5594 > 2 #(82.75%) > 2 Min.2 c 2.2 c.6 c Max 36 5.69 c 276 c 78.62 c Average 9.29 4.6 c 499.7 4.23 c 25.92 c (.) a : Difference between lower and upper bound. (.) b : All CPU s in minutes. (.) c : Taken over the first 9 instances (those solved successfully by AC-BP and K-BP). : Failed to find a feasible solution within the allowed time. # : Did not get past node. (.) d : found lower bound as a percentage of best lower bound. : Gap =. Page 35 of 4
27. Comparsion between AC-BP and K-BP: Warm starting Problem A-BP K-BP SP SPrest CPU CPUrest SP SPrest CPU CPUrest BinG 6 4. 2 3.3. BinG 2 36 2.62.3 33.88.. BinG 3 5 2.93.7 55 2.4.23. BinG 4 69 2.7.2. 76 2.58.43. BinG 5 86 2.57.7. 4 2.59.84. BinG 6 94 2.59.24. 2 3.44.8.2 BinG 7 3 2.58.34.2 58 2.79 2.37.2 BinG 8 56 2.93.9.3 228.64 7.4.3 BinG 9 62 2.4.6.2 2 3.57 5.6.5 BinG 88 2.38.59.3 28.82 4.97.4 BinG 2 27 2 2.77.3 37 2.45 4.93.7 BinG 4 24 2.68 4.64.3 326 3.98 7.2. BinG 5 252 2.67 5.73. 37 3.5 22.3.2 BinG 7 339 2.8.8.8 442 2.9 28.46.9 BinG 2 425 2 24.. 564 7. Min 6 2. 2.64.3. Max 425 4 24..3 564 3.98 7..2 Average 6.6 2.6 3.52.4 23.67 2.67 7.52.4 # : Did not get past node. a : SPrest as a percent of SP b : CPUrest as a percent of CPU : No branching was done. Page 36 of 4
28. Comparsion between AC-BP and K-BP: Warm starting Problem A-BP K-BP SP SPrest CPU CPUrest SP SPrest CPU CPUrest BinT 2 22.2 24.6 BinT 2 48.7 62.3 BinT 3 74.2 96.87 BinT 5 42.3 67 3.53 BinT 6 78.88 299 2.24 BinT 72 2 3.66 336 33.95 BinT 8 247 6.26 355 37.38 BinT 9 268.92 49 56.4 BinT 2 3 5.47 47 78.7 BinT 35 2 7.56.8 52 2 # BinT 2 382 2 26.85.5 537 2 # BinT 4 456 2 4.89.7 554 2 # BinT 5 495 2 67.32.3 564 2 # BinT 8 64 2 95.84.68 539 2 # Min 22 2.2.5 24.6 Max 64 2 95.84.68 564 2 Average 244.5 2 4.88.2 338.8 59.4 # : Did not get past node. a : SPrest as a percent of SP b : CPUrest as a percent of CPU : No branching was done. Page 37 of 4
29. IP B&P vs Classical B&P: Nodes solved and CPU 35 CPU (mins) 3 25 2 5 Classical B&B (Kelley) IP B&B (ACCPM) 5 2 3 4 5 Problem number Number of nodes 2 8 6 4 2 Classical B&B (Kelley) IP B&B (ACCPM) Page 38 of 4 2 3 4 5 Problem number
3. IP B&P vs Classical B&P: Subproblems called. Subporoblems called at node 6 5 4 3 2 IP B&P (ACCPM) Classical B&P (Kelley) 2 3 4 5 Problem number Subporoblems called at rest of nodes 8 6 4 2 IP B&P (ACCPM) Classical B&P (Kelley) 2 3 4 5 Problem number Page 39 of 4
3. Conclusion Simpelx: LP bounding scheme IPMs: Lagrangean bounding scheme: Dantzig-Wolfe formulation Decomp. approaches Branch & Price ACCPM IP B&P IP B&P Branch & Bound IP B&B Page 4 of 4 Interior point methods
Warm starting can be done efficciently with IPMs. IPM-based Branch-and-Bound is promising to solve large MIPs Still to do: An efficient implementation (framework??) Branching Dual heuristics Preprocessing, valid cuts, etc Testing... Page 4 of 4