Mixed Integer Nonlinear Programming (MINLP)

Transcription

1 Tutorial: Mixed Integer Nonlinear Programming () Sven Leyffer MCS Division Argonne National Lab Jeff Linderoth ISE Department Lehigh University informs Annual Meeting San Francisco May 15, 2005

2 New Math MI NLP +

3 Tutorial Overview 1. Introduction, Applications, and Formulations 2. Classical Solution Methods 3. Modern Developments in 4. Implementation and Software

4 Motivation Examples Tricks Part I Introduction, Applications, and Formulations

5 Motivation Examples Tricks What How Why? The Problem of the Day Mixed Integer Nonlinear Program () minimize f(x, y) x,y subject to c(x, y) 0 x X, y Y integer f, c smooth (convex) functions X, Y polyhedral sets, e.g. Y = {y [0, 1] p Ay b} y Y integer hard problem f, c not convex very hard problem

6 Motivation Examples Tricks What How Why? Why the N? An anecdote: July, A young and frightened George Dantzig, presents his newfangled linear programming to a meeting of the Econometric Society of Wisconsin, attended by distinguished scientists like Hotelling, Koopmans, and Von Neumann. Following the lecture, Hotelling a pronounced to the audience: man But we all know the world is nonlinear! a in Dantzig s words a huge whale of a The world is indeed nonlinear Physical Processes and Properties Equilibrium Enthalpy Abstract Measures Economies of Scale Covariance Utility of decisions

7 Motivation Examples Tricks What How Why? Why the MI? We can use 0-1 (binary) variables for a variety of purposes Modeling yes/no decisions Enforcing disjunctions Enforcing logical conditions Modeling fixed costs Modeling piecewise linear functions If the variable is associated with a physical entity that is indivisible, then it must be integer Number of aircraft carriers to to produce. Gomory s Initial Motivation

8 Motivation Examples Tricks What How Why? A Popular Method Dantzig s Two-Phase Method for Adapted by Leyffer and Linderoth 1. Convince the user that he or she does not wish to solve a mixed integer nonlinear programming problem at all! 2. Otherwise, solve the continuous relaxation (N LP ) and round off the minimizer to the nearest integer. For 0 1 problems, or those in which the y is small, the continuous approximation to the discrete decision is not accurate enough for practical purposes. Conclusion: methods must be studied!

9 Motivation Examples Tricks What How Why? Example: Core Reload Operation (Quist, A.J., 2000) max. reactor efficiency after reload subject to diffusion PDE & safety diffusion PDE nonlinear equation integer & nonlinear model avoid reactor becoming overheated

10 Motivation Examples Tricks What How Why? Example: Core Reload Operation (Quist, A.J., 2000) look for cycles for moving bundles: e.g i.e. bundle moved from 4 to 6... model with binary x ilm {0, 1} x ilm = 1 node i has bundle l of cycle m

11 Motivation Examples Tricks What How Why? AMPL Model of Core Reload Operation Exactly one bundle per node: L M x ilm = 1 l=1 m=1 i I AMPL model: var x {I,L,M} binary ; Bundle {i in I}: sum{l in L, m in M} x[i,l,m] = 1 ; Multiple Choice: One of the most common uses of IP Full AMPL model c-reload.mod at leyffer/mac/

12 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Gas Transmission Problem (De Wolf and Smeers, 2000) Belgium has no gas! All natural gas is imported from Norway, Holland, or Algeria. Supply gas to all demand points in a network in a minimum cost fashion. Gas is pumped through the network with a series of compressors There are constraints on the pressure of the gas within the pipe

13 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Pressure Loss is Nonlinear Assume horizontal pipes and steady state flows Pressure loss p across a pipe is related to the flow rate f as p 2 in p 2 out = 1 Ψ sign(f)f 2 Ψ: Friction Factor

14 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Gas Transmission: Problem Input Network (N, A). A = A p A a A a : active arcs have compressor. Flow rate can increase on arc A p : passive arcs simply conserve flow rate N s N: set of supply nodes c i, i N s : Purchase cost of gas s i, s i : Lower and upper bounds on gas supply at node i p i, p i : Lower and upper bounds on gas pressure at node i s i, i N: supply at node i. s i > 0 gas added to the network at node i s i < 0 gas removed from the network at node i to meet demand f ij, (i, j) A: flow along arc (i, j) f(i, j) > 0 gas flows i j f(i, j) < 0 gas flows j i

15 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Gas Transmission Model subject to min j (i,j) A j N s c j s j f ij = s i i N sign(f ij )fij 2 Ψ ij (p 2 i p 2 j) = 0 (i, j) A p sign(f ij )fij 2 Ψ ij (p 2 i p 2 j) 0 (i, j) A a s i [s i, s i ] i N p i [p i, p i ] i N f ij 0 (i, j) A a

16 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Your First Modeling Trick Don t include nonlinearities or nonconvexities unless necessary! Replace p 2 i ρ i sign(f ij )f 2 ij Ψ ij (ρ i ρ j ) = 0 (i, j) A p f 2 ij Ψ ij (ρ i ρ j ) 0 (i, j) A a ρ i [ p i, p i ] i N This trick only works because 1. p 2 i terms appear only in the bound constraints 2. Also f ij 0 (i, j) A a This model is nonconvex: sign(f ij )fij 2 is a nonconvex function Some solvers do not like sign

17 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Dealing with sign( ): The NLP Way Use auxiliary binary variables to indicate direction of flow Let f ij F (i, j) A p z ij = { 1 fij 0 f ij F (1 z ij ) 0 f ij 0 f ij F z ij Note that Write constraint as sign(f ij ) = 2z ij 1 (2z ij 1)f 2 ij Ψ ij (ρ i ρ j ) = 0.

18 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Special Ordered Sets Sven thinks this NLP trick is pretty cool It is not how it is done in De Wolf and Smeers (2000). Heuristic for finding a good starting solution, then a local optimization approach based on a piecewise-linear simplex method Another (similar) approach involves approximating the nonlinear function by piecewise linear segments, but searching for the globally optimal solution: Special Ordered Sets of Type 2 If the multidimensional nonlinearity cannot be removed, resort to Special Ordered Sets of Type 3

19 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Portfolio Management N: Universe of asset to purchase x i : Amount of asset i to hold B: Budget { } min x R N + u(x) i N x i = B Markowitz: u(x) def = α T x + λx T Qx α: Expected returns Q: Variance-covariance matrix of expected returns λ: Risk aversion parameter

20 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing More Realistic Models b R N of benchmark holdings Benchmark Tracking: u(x) def = (x b) T Q(x b) Constraint on E[Return]: α T x r Limit Names: i N : x i > 0 K Use binary indicator variables to model the implication x i > 0 y i = 1 Implication modeled with variable upper bounds: x i By i i N i N y i K

21 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Even More Models Min Holdings: (x i = 0) (x i m) Model implication: x i > 0 x i m x i > 0 y i = 1 x i m x i By i, x i my i i N Round Lots: x i {kl i, k = 1, 2,...} x i z i L i = 0, z i Z + i N Vector h of initial holdings Transactions: t i = x i h i Turnover: i N t i Transaction Costs: i N c it i in objective Market Impact: i N γ it 2 i in objective

22 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Multiproduct Batch Plants (Kocis and Grossmann, 1988) M: Batch Processing Stages N: Different Products H: Horizon Time Q i : Required quantity of product i t ij : Processing time product i stage j S ij : Size Factor product i stage j B i : Batch size of product i N V j : Stage j size: V j S ij B i i, j N j : Number of machines at stage j C i : Longest stage time for product i: C i t ij /N j i, j

23 Motivation Examples Tricks Gas Transmission Portfolio Management Batch Processing Multiproduct Batch Plants s.t. min j M α j N j V β j j V j S ij B i 0 i N, j M C i N j t ij i N, j M Q i C i H B i i N Bound Constraints on V j, C i, B i, N j N j Z j M

24 Motivation Examples Tricks Variable Transformation Modeling Trick #2 Horizon Time and Objective Function Nonconvex. :-( Sometimes variable transformations work! v j = ln(v j ), n j = ln(n j ), b i = ln(b i ), c i = ln C i min α j e N j+β j V j j M s.t. v j ln(s ij )b i 0 i N, j M c i + n j ln(τ ij ) i N, j M Q i e C i B i H i N (Transformed) Bound Constraints on V j, C i, B i

25 Motivation Examples Tricks Variable Transformation How to Handle the Integrality? But what to do about the integrality? 1 N j N j j M, N j Z j M n j {0, ln(2), ln(3),......} { 1 nj takes value ln(k) Y kj = 0 Otherwise n j K ln(k)y kj = 0 j M k=1 K Y kj = 1 j M k=1 This model is available at leyffer/macminlp/problems/batch.mod

26 Motivation Examples Tricks Variable Transformation A Small Smattering of Other Applications Chemical Engineering Applications: process synthesis (Kocis and Grossmann, 1988) batch plant design (Grossmann and Sargent, 1979) cyclic scheduling (Jain, V. and Grossmann, I.E., 1998) design of distillation columns (Viswanathan and Grossmann, 1993) pump configuration optimization (Westerlund, T., Pettersson, F. and Grossmann, I.E., 1994) Forestry/Paper production (Westerlund, T., Isaksson, J. and Harjunkoski, I., 1995) trimloss minimization (Harjunkoski, I., Westerlund, T., Pörn, R. and Skrifvars, H., 1998) Topology Optimization (Sigmund, 2001)

27 Branch-and-Bound Outer Approximation Hybrid Methods Part II Classical Solution Methods

28 Branch-and-Bound Outer Approximation Hybrid Methods Classical Solution Methods for 1. Classical Branch-and-Bound 2. Outer Approximation & Benders Decomposition 3. Hybrid Methods LP/NLP Based Branch-and-Bound Integrating SQP with Branch-and-Bound

29 Branch-and-Bound Outer Approximation Hybrid Methods Definition Branch-and-Bound Solve relaxed NLP (0 y 1 continuous relaxation)... solution value provides lower bound Branch on y i non-integral Solve NLPs & branch until 1. Node infeasible Node integer feasible... get upper bound (U) 3. Lower bound U... y = 1 i y = 0 i infeasible integer feasible etc. etc. dominated by upper bound Search until no unexplored nodes on tree

30 Branch-and-Bound Outer Approximation Hybrid Methods Definition Variable Selection for Branch-and-Bound Assume y i {0, 1} for simplicity... (ˆx, ŷ) fractional solution to parent node; ˆf = f(ˆx, ŷ) 1. maximal fractional branching: choose ŷ i closest to 1 2 f +/ i max {min(1 ŷ i, ŷ i )} i 2. strong branching: (approx) solve all NLP children: minimize f(x, y) x,y subject to c(x, y) 0 x X, y Y, y i = 1/0 branching variable y i that changes objective the most: { max min(f + i i, fi )}

31 Branch-and-Bound Outer Approximation Hybrid Methods Definition Node Selection for Branch-and-Bound Which node n on tree T should be solved next? 1. depth-first search: select deepest node in tree minimizes number of NLP nodes stored exploit warm-starts (MILP/MIQP only) 2. best estimate: choose node with best expected integer soln min n T f p(n) + min { e + i (1 y i), e i y } i i:y i fractional where f p(n) = value of parent node, e +/ i = pseudo-costs summing pseudo-cost estimates for all integers in subtree

32 Branch-and-Bound Outer Approximation Hybrid Methods Definition Benders Decomposition Outer Approximation (Duran and Grossmann, 1986) Motivation: avoid solving huge number of NLPs Exploit MILP/NLP solvers: decompose integer/nonlinear part Key idea: reformulate as MILP (implicit) Solve alternating sequence of MILP & NLP NLP subproblem y j fixed: minimize f(x, y j ) x NLP(y j ) subject to c(x, y j ) 0 x X MILP NLP(y j ) Main Assumption: f, c are convex

33 Branch-and-Bound Outer Approximation Hybrid Methods Definition Benders Decomposition Outer Approximation (Duran and Grossmann, 1986) η let (x j, y j ) solve NLP(y j ) linearize f, c about (x j, y j ) =: z j new objective variable η f(x, y) (P ) MILP (M) f(x) (M) minimize z=(x,y),η η subject to η f j + fj T (z z j) y j Y 0 c j + c T j (z z j) y j Y x X, y Y integer SNAG: need all y j Y linearizations

34 Branch-and-Bound Outer Approximation Hybrid Methods Definition Benders Decomposition Outer Approximation (Duran and Grossmann, 1986) (M k ): lower bound (underestimate convex f, c) NLP(y j ): upper bound U (fixed y j ) NLP( y ) subproblem MILP master program NLP gives linearization MILP finds new y MILP infeasible? No Yes STOP stop, if lower bound upper bound

35 Branch-and-Bound Outer Approximation Hybrid Methods Definition Benders Decomposition Outer Approximation & Benders Decomposition Take OA cuts for z j := (x j, y j )... wlog X = R n η f j + f T j (z z j ) & 0 c j + c T j (z z j ) sum with (1, λ j )... λ j multipliers of NLP(y j ) η f j + λ T j c j + ( f j + c j λ j ) T (z z j ) KKT conditions of NLP(y j ) x f j + x c j λ j = 0... eliminate x components from valid inequality in y η f j + ( y f j + y c j λ j ) T (y y j ) NB: µ j = y f j + y c j λ j multiplier of y = y j in NLP(y j ) References: (Geoffrion, 1972)

36 Branch-and-Bound Outer Approximation Hybrid Methods LP/NLP Based Branch-and-Bound Integrating SQP and Branch-and-Bound LP/NLP Based Branch-and-Bound AIM: avoid re-solving MILP master (M) Consider MILP branch-and-bound interrupt MILP, when y j found solve NLP(y j ) get x j linearize f, c about (x j, y j ) add linearization to tree continue MILP tree-search... until lower bound upper bound

37 Branch-and-Bound Outer Approximation Hybrid Methods LP/NLP Based Branch-and-Bound Integrating SQP and Branch-and-Bound LP/NLP Based Branch-and-Bound need access to MILP solver... call back exploit good MILP (branch-cut-price) solver (Akrotirianakis et al., 2001) use Gomory cuts in tree-search preliminary results: order of magnitude faster than OA same number of NLPs, but only one MILP similar ideas for Benders & Extended Cutting Plane methods recent implementation by CMU/IBM group References: (Quesada and Grossmann, 1992)

38 Branch-and-Bound Outer Approximation Hybrid Methods LP/NLP Based Branch-and-Bound Integrating SQP and Branch-and-Bound Integrating SQP & Branch-and-Bound AIM: Avoid solving NLP node to convergence. Sequential Quadratic Programming (SQP) solve sequence (QP k ) at every node minimize f k + fk T d + 1 d 2 dt H k d (QP k ) subject to c k + c T k d 0 x k + d x X y k + d y Ŷ. Early branching: After QP step choose non-integral yi k+1, branch & continue SQP References: (Borchers and Mitchell, 1994; Leyffer, 2001)

39 Branch-and-Bound Outer Approximation Hybrid Methods LP/NLP Based Branch-and-Bound Integrating SQP and Branch-and-Bound Integrating SQP & Branch-and-Bound SNAG: (QP k ) not lower bound no fathoming from upper bound minimize f k + fk T d + 1 d 2 dt H k d subject to c k + c T k d 0 x k + d x X y k + d y Ŷ. Remedy: Exploit OA underestimating property (Leyffer, 2001): add objective cut f k + f T k d U ɛ to (QP k) fathom node, if (QP k ) inconsistent NB: (QP k ) inconsistent and trust-region active do not fathom

40 Branch-and-Bound Outer Approximation Hybrid Methods LP/NLP Based Branch-and-Bound Integrating SQP and Branch-and-Bound Comparison of Classical Techniques Summary of numerical experience 1. Quadratic OA master: usually fewer iteration MIQP harder to solve 2. NLP branch-and-bound faster than OA... depends on MIP solver 3. LP/NLP-based-BB order of magnitude faster than OA... also faster than B&B 4. Integrated SQP-B&B up to 3 faster than B&B number of QPs per node 5. ECP works well, if function/gradient evals expensive

41 Formulations Inequalities Dealing with Nonconvexity Part III Modern Developments in

42 Formulations Inequalities Dealing with Nonconvexity Importance of Relaxations Aggregation Modern Methods for 1. Formulations Relaxations Good formulations: big M s and disaggregation 2. Cutting Planes Cuts from relaxations and special structures Cuts from integrality 3. Handling Nonconvexity Envelopes Methods

43 Formulations Inequalities Dealing with Nonconvexity Importance of Relaxations Aggregation Relaxations z(s) def = min x S f(x) z(t ) def = min x T f(x) S Independent of f, S, T : z(t ) z(s) If x T = arg min x T f(x) And x T S, then x T = arg min x S f(x) T

44 Formulations Inequalities Dealing with Nonconvexity Importance of Relaxations Aggregation UFL: Uncapacitated Facility Location Facilities: J Customers: I min j J f j x j + i I f ij y ij j J y ij = 1 i I j J y ij I x j j J (1) i I OR y ij x j i I, j J (2) Which formulation is to be preferred? I = J = 40. Costs random. Formulation 1. 53,121 seconds, optimal solution. Formulation 2. 2 seconds, optimal solution.

45 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Valid Inequalities Sometimes we can get a better formulation by dynamically improving it. An inequality π T x π 0 is a valid inequality for S if π T x π 0 x S Alternatively: max x S {π T x} π 0 Thm: (Hahn-Banach). Let S R n be a closed, convex set, and let ˆx S. Then there exists π R n such that π T ˆx > max x S {πt x} π T x = π 0 S ˆx

46 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Nonlinear Branch-and-Cut Consider minimize x,y Note the Linear objective This is WLOG: f T x x + f T y y subject to c(x, y) 0 y {0, 1} p, 0 x U min f(x, y) min η s.t. η f(x, y)

47 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities It s Actually Important! We want to approximate the convex hull of integer solutions, but without a linear objective function, the solution to the relaxation might occur in the interior. No Separating Hyperplane! :-( min(y 1 1/2) 2 + (y 2 1/2) 2 y 2 s.t. y 1 {0, 1}, y 2 {0, 1} (ŷ 1, ŷ 2 ) η (y 1 1/2) 2 + (y 2 1/2) 2 η y 1

48 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Valid Inequalities From Relaxations Idea: Inequalities valid for a relaxation are valid for original Generating valid inequalities for a relaxation is often easier. ˆx π T x = π0 S T Separation Problem over T: Given ˆx, T find (π, π 0 ) such that π T ˆx > π 0, π T x π 0 x T

49 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Simple Relaxations Idea: Consider one row relaxations If P = {x {0, 1} n Ax b}, then for any row i, P i = {x {0, 1} n a T i x b i} is a relaxation of P. If the intersection of the relaxations is a good approximation to the true problem, then the inequalities will be quite useful. Crowder et al. (1983) is the seminal paper that shows this to be true for IP. : Single (linear) row relaxations are also valid same inequalities can also be used

50 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Knapsack Covers K = {x {0, 1} n a T x b} A set C N is a cover if j C a j > b A cover C is a minimal cover if C \ j is not a cover j C If C N is a cover, then the cover inequality x j C 1 j C is a valid inequality for S Sometimes (minimal) cover inequalities are facets of conv(k)

51 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Other Substructures Single node flow: (Padberg et al., 1985) S = x R N +, y {0, 1} N x j b, x j u j y j j N j N Knapsack with single continuous variable: (Marchand and Wolsey, 1999) S = x R +, y {0, 1} N a j y j b + x j N Set Packing: (Borndörfer and Weismantel, 2000) { } S = y {0, 1} N Ay e A {0, 1} M N, e = (1, 1,..., 1) T

52 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities The Chvátal-Gomory Procedure A general procedure for generating valid inequalities for integer programs Let the columns of A R m n be denoted by {a 1, a 2,... a n } S = {y Z n + Ay b}. 1. Choose nonnegative multipliers u R m + 2. u T Ay u T b is a valid inequality ( j N ut a j y j u T b). 3. j N ut a j y j u T b (Since y 0). 4. j N ut a j y j u T b is valid for S since u T a j y j is an integer Simply Amazing: This simple procedure suffices to generate every valid inequality for an integer program

53 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Extension to (Çezik and Iyengar, 2005) This simple idea also extends to mixed 0-1 conic programming f T z minimize z def =(x,y) subject to Az K b y {0, 1} p, 0 x U K: Homogeneous, self-dual, proper, convex cone x K y (x y) K

54 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Gomory On Cones (Çezik and Iyengar, 2005) LP: K l = R n + SOCP: K q = {(x 0, x) x 0 x } SDP: K s = {x = vec(x) X = X T, X p.s.d} Dual Cone: K def = {u u T z 0 z K} Extension is clear from the following equivalence: Az K b u T Az u T b u K 0 Many classes of nonlinear inequalities can be represented as Ax Kq b or Ax Ks b

55 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Using Gomory Cuts in (Akrotirianakis et al., 2001) LP/NLP Based Branch-and-Bound solves MILP instances: minimize η z def =(x,y),η subject to η f j + fj T (z z j) y j Y k 0 c j + c T j (z z j) y j Y k x X, y Y integer Create Gomory mixed integer cuts from η f j + fj T (z z j ) 0 c j + c T j (z z j ) Akrotirianakis et al. (2001) shows modest improvements Research Question: Other cut classes? Research Question: Exploit outer approximation property?

56 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Disjunctive Cuts for (Stubbs and Mehrotra, 1999) Extension of Disjunctive Cuts for MILP: (Balas, 1979; Balas et al., 1993) Continuous relaxation (z def = (x, y)) x C def = {z c(z) 0, 0 y 1, 0 x U} C def = conv({x C y {0, 1} p }) C 0/1 j def = {z C y j = 0/1} let M j (C) def = z = λ 0 u 0 + λ 1 u 1 λ 0 + λ 1 = 1, λ 0, λ 1 0 u 0 C 0 j, u 1 C 1 j P j (C) := projection of M j (C) onto z P j (C) = conv (C y j {0, 1}) and P 1...p (C) = C convex hull y

57 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Disjunctive Cuts: Example { minimize x (x 1/2) 2 + (y 3/4) 2 1, 2 x 2, y {0, 1} } x,y x Given ẑ with ŷ j {0, 1} find separating hyperplane C 0 j C 1 j { minimize z subject to z ẑ z P j (C) y ẑ = (ˆx, ŷ)

58 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Disjunctive Cuts Example z def = arg min z ẑ z s.t. λ 0 u 0 + λ 1 u 1 = z ( λ) 0 + λ 1 = 1( u ( ) ( u λ 0, λ 1 0 ) ) ẑ = (ˆx, ŷ) NONCONVEX

59 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities What to do? (Stubbs and Mehrotra, 1999) Look at the perspective of c(z) Think of z = µz P(c( z), µ) = µc( z/µ) Perspective gives a convex reformulation of M j (C): M j ( C), where µc i (z/µ) 0 C := (z, µ) 0 µ 1 0 x µu, 0 y µ c(0/0) = 0 convex representation

60 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Disjunctive Cuts Example C = x x y µ µ [ (x/µ 1/2) 2 + (y/µ 3/4) 2 1 ] 0 2µ x 2µ 0 y µ 0 µ 1 x µ C 0 j C 0 j C 1 j µ C 0 j C 1 j y yµ C 1 j x y

61 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Example, cont. C 0 j = {(z, µ) y j = 0} C 1 j = {(z, µ) y j = µ} Take v 0 µ 0 u 0 v 1 µ 1 u 1 min z ẑ s.t. v 0 + v 1 = z µ 0 + µ 1 = 1 (v 0, µ 0 ) C j 0 (v 1, µ 1 ) C j 1 µ 0, µ 1 0 Solution to example: ( ) ( x y = separating hyperplane: ψ T (z ẑ), where ψ z ẑ )

62 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Example, Cont x y = z ψ = ( 2x y 0.75 ) 0.198x y ẑ = (ˆx, ŷ)

63 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Nonlinear Branch-and-Cut (Stubbs and Mehrotra, 1999) Can do this at all nodes of the branch-and-bound tree Generalize disjunctive approach from MILP solve one convex NLP per cut Generalizes Sherali and Adams (1990) and Lovász and Schrijver (1991) tighten cuts by adding semi-definite constraint Stubbs and Mehrohtra (2002) also show how to generate convex quadratic inequalities, but computational results are not that promising

64 Formulations Inequalities Dealing with Nonconvexity Preliminaries MILP Inequalities Applied to Disjunctive Inequalities Generalized Disjunctive Programming (Raman and Grossmann, 1994; Lee and Grossmann, 2000) Consider disjunctive NLP minimize fi + f(x) x,y Y i subject to c i (x) 0 Y i B i x = 0 i I f i = γ i f i = 0 0 x U, Ω(Y ) = true, Y {true, false} p convex hull representation... x = v i1 + v i0, λ i1 + λ i0 = 1 λ i1 c i (v i1 /λ i1 ) 0, B i v i0 = 0 0 v ij λ ij U, 0 λ ij 1, f i = λ i1 γ i

65 Formulations Inequalities Dealing with Nonconvexity Difficulties Envelopes Bilinear Terms Dealing with Nonconvexities Functional nonconvexity causes serious problems. Branch and bound must have true lower bound (global solution) Underestimate nonconvex functions. Solve relaxation. Provides lower bound. If relaxation is not exact, then branch

66 Formulations Inequalities Dealing with Nonconvexity Difficulties Envelopes Bilinear Terms Dealing with Nonconvex Constraints If nonconvexity in constraints, may need to overestimate and underestimate the function to get a convex region

67 Formulations Inequalities Dealing with Nonconvexity Difficulties Envelopes Bilinear Terms Envelopes f : Ω R Convex Envelope (vex Ω (f)): Pointwise supremum of convex underestimators of f over Ω. Concave Envelope (cav Ω (f)): Pointwise infimum of concave overestimators of f over Ω.

68 Formulations Inequalities Dealing with Nonconvexity Difficulties Envelopes Bilinear Terms Branch-and-Bound Global Optimization Methods Under/Overestimate simple parts of (Factorable) Functions individually Bilinear Terms Trilinear Terms Fractional Terms Univariate convex/concave terms General nonconvex functions f(x) can be underestimated over a region [l, u] overpowering the function with a quadratic function that is 0 on the region of interest L(x) = f(x) + n α i (l i x i )(u i x i ) i=1 Refs: (McCormick, 1976; Adjiman et al., 1998; Tawarmalani and Sahinidis, 2002)

69 Formulations Inequalities Dealing with Nonconvexity Difficulties Envelopes Bilinear Terms Bilinear Terms The convex and concave envelopes of the bilinear function xy over a rectangular region R def = {(x, y) R 2 l x x u x, l y y u y } are given by the expressions vexxy R (x, y) = max{l y x + l x y l x l y, u y x + u x y u x u y } cavxy R (x, y) = min{u y x + l x y l x u y, l y x + u x y u x l y }

70 Formulations Inequalities Dealing with Nonconvexity Difficulties Envelopes Bilinear Terms Worth 1000 Words?

71 Formulations Inequalities Dealing with Nonconvexity Difficulties Envelopes Bilinear Terms Summary : Good relaxations are important Relaxations can be improved Statically: Better formulation/preprocessing Dynamically: Cutting planes Nonconvex : Methods exist, again based on relaxations Tight relaxations is an active area of research Lots of empirical questions remain

72 Special Ordered Sets Implementation & Software Issues Part IV Implementation and Software

73 Special Ordered Sets Implementation & Software Issues Implementation and Software for 1. Special Ordered Sets 2. Implementation & Software Issues

74 Special Ordered Sets Implementation & Software Issues Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 Special Ordered Sets of Type 1 SOS1: λ i = 1 & at most one λ i is nonzero Example 1: d {d 1,..., d p } discrete diameters d = λ i d i and {λ 1,..., λ p } is SOS1 d = λ i d i and λ i = 1 and λ i {0, 1}... d is convex combination with coefficients λ i Example 2: nonlinear function c(y) of single integer y = iλ i and c = c(i)λ i and {λ 1,..., λ p } is SOS1 References: (Beale, 1979; Nemhauser, G.L. and Wolsey, L.A., 1988; Williams, 1993)...

75 Special Ordered Sets Implementation & Software Issues Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 Special Ordered Sets of Type 1 SOS1: λ i = 1 & at most one λ i is nonzero Branching on SOS1 1. reference row a 1 <... < a p e.g. diameters 2. fractionality: a := a i λ i 3. find t : a t < a a t+1 4. branch: {λ t+1,..., λ p } = 0 or {λ 1,..., λ t } = 0 a < a t a > a t+1

76 Special Ordered Sets Implementation & Software Issues Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 Special Ordered Sets of Type 2 SOS2: λ i = 1 & at most two adjacent λ i nonzero Example: Approximation of nonlinear function z = z(x) z(x) breakpoints x 1 <... < x p function values z i = z(x i ) piece-wise linear x = λ i x i x z = λ i z i {λ 1,..., λ p } is SOS2... convex combination of two breakpoints...

77 Special Ordered Sets Implementation & Software Issues Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 Special Ordered Sets of Type 2 SOS2: λ i = 1 & at most two adjacent λ i nonzero Branching on SOS2 1. reference row a 1 <... < a p e.g. a i = x i 2. fractionality: a := a i λ i 3. find t : a t < a a t+1 4. branch: {λ t+1,..., λ p } = 0 or {λ 1,..., λ t 1 } x < a t x > a t

78 Special Ordered Sets Implementation & Software Issues Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 Special Ordered Sets of Type 3 Example: Approximation of 2D function u = g(v, w) Triangularization of [v L, v U ] [w L, w U ] domain w 1. v L = v 1 <... < v k = v U 2. w L = w 1 <... < w l = w U 3. function u ij := g(v i, w j ) 4. λ ij weight of vertex (i, j) v = λ ij v i w = λ ij w j v u = λ ij u ij 1 = λ ij is SOS3...

79 Special Ordered Sets Implementation & Software Issues Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 Special Ordered Sets of Type 3 SOS3: λ ij = 1 & set condition holds w 1. v = λ ij v i... convex combinations 2. w = λ ij w j 3. u = λ ij u ij {λ 11,..., λ kl } satisfies set condition trangle : {(i, j) : λ ij > 0} i.e. nonzeros in single triangle violates set condn v

80 Special Ordered Sets Implementation & Software Issues Branching on SOS3 λ violates set condition Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 w compute centers: ˆv = λ ij v i & ŵ = λ ij w i find s, t such that v s ˆv < v s+1 & w s ŵ < w s+1 branch on v or w vertical branching: branching: λ ij = 1 T λ ij = 1 L λ ij = 1 B = center of gravity λ ij = 1 R T B v horizontal

81 Special Ordered Sets Implementation & Software Issues Special Ordered Sets of Type 1 Special Ordered Sets of Type 2 Special Ordered Sets of Type 3 Extension to SOS-k Example: electricity transmission network: c(x) = 4x 1 x x 2 x 4 sin(x 3 ) (Martin et al., 2005) extend SOS3 to SOSk models for any k function with p variables on N grid needs N p λ s Alternative (Gatzke, 2005): exploit computational graph automatic differentiation only need SOS2 & SOS3... replace nonconvex parts piece-wise polyhedral approx.

82 Special Ordered Sets Implementation & Software Issues Software Software for Outer Approximation: DICOPT++ (& AIMMS) NLP solvers: CONOPT, MINOS, SNOPT MILP solvers: CPLEX, OSL2 Branch-and-Bound Solvers: SBB & NLP solvers: CONOPT, MINOS, SNOPT & FilterSQP variable & node selection; SOS1 & SOS2 support Global : BARON & MINOPT underestimators & branching CPLEX, MINOS, SNOPT, OSL Online Tools: World, Mac & NEOS World NEOS server www-neos.mcs.anl.gov/

83 Special Ordered Sets Implementation & Software Issues Software COIN-OR COmputational INfrastructure for Operations Research A library of (interoperable) software tools for optimization A development platform for open source projects in the OR community Possibly Relevant Modules: OSI: Open Solver Interface CGL: Cut Generation Library CLP: Coin Linear Programming Toolkit CBC: Coin Branch and Cut IPOPT: Interior Point OPTimizer for NLP NLPAPI: NonLinear Programming API

84 Special Ordered Sets Implementation & Software Issues Software with COIN-OR New implementation of LP/NLP based BB MIP branch-and-cut: CBC & CGL NLPs: IPOPT interior point... OK for NLP(y i ) New hybrid method: solve more NLPs at non-integer y i better outer approximation allow complete MIP at some nodes generate new integer assignment... faster than DICOPT++, SBB simplifies to OA and BB at extremes... less efficient... see Bonami et al. (2005)... coming in 2006.

85 Special Ordered Sets Implementation & Software Issues Software Conclusions rich modeling paradigm most popular solver on NEOS Algorithms for : Branch-and-bound (branch-and-cut) Outer approximation et al. solvers lag 15 years behind MIP solvers many research opportunities!!!

86 Part V References

87 References C. Adjiman, S. Dallwig, C. A. Floudas, and A. Neumaier. A global optimization method, abb, for general twice-differentiable constrained NLPs - I. Theoretical advances. Computers and Chemical Engineering, 22: , I. Akrotirianakis, I. Maros, and B. Rustem. An outer approximation based branch-and-cut algorithm for convex 0-1 problems. Optimization Methods and Software, 16:21 47, E. Balas. Disjunctive programming. In Annals of Discrete Mathematics 5: Discrete Optimization, pages North Holland, E. Balas, S. Ceria, and G. Corneujols. A lift-and-project cutting plane algorithm for mixed 0-1 programs. Mathematical Programming, 58: , E. M. L. Beale. Branch-and-bound methods for mathematical programming systems. Annals of Discrete Mathematics, 5: , P. Bonami, L. Biegler, A. Conn, G. Cornuéjols, I. Grossmann, C. Laird, J. Lee, A. Lodi, F. Margot, N. Saaya, and A. Wächter. An algorithmic framework for convex mixed integer nonlinear programs. Technical report, IBM Research Division, Thomas J. Watson Research Center, B. Borchers and J. E. Mitchell. An improved branch and bound algorithm for Mixed Integer Nonlinear Programming. Computers and Operations Research, 21(4): , R. Borndörfer and R. Weismantel. Set packing relaxations of some integer programs. Mathematical Programming, 88: , M. T. Çezik and G. Iyengar. Cuts for mixed 0-1 conic programming. Mathematical Programming, to appear.

88 References H. Crowder, E. L. Johnson, and M. W. Padberg. Solving large scale zero-one linear programming problems. Operations Research, 31: , D. De Wolf and Y. Smeers. The gas transmission problem solved by an extension of the simplex algorithm. Management Science, 46: , M. Duran and I. E. Grossmann. An outer-approximation algorithm for a class of mixed integer nonlinear programs. Mathematical Programming, 36: , A. M. Geoffrion. Generalized Benders decomposition. Journal of Optimization Theory and Applications, 10: , I. E. Grossmann and R. W. H. Sargent. Optimal design of multipurpose batch plants. Ind. Engng. Chem. Process Des. Dev., 18: , Harjunkoski, I., Westerlund, T., Pörn, R. and Skrifvars, H. Different transformations for solving non-convex trim-loss problems by. European Journal of Opertational Research, 105: , Jain, V. and Grossmann, I.E. Cyclic scheduling of continuous parallel-process units with decaying performance. AIChE Journal, 44: , G. R. Kocis and I. E. Grossmann. Global optimization of nonconvex mixed integer nonlinear programming () problems in process synthesis. Industrial Engineering Chemistry Research, 27: , S. Lee and I. Grossmann. New algorithms for nonlinear disjunctive programming. Computers and Chemical Engineering, 24: , S. Leyffer. Integrating SQP and branch-and-bound for mixed integer nonlinear programming. Computational Optimization & Applications, 18: , 2001.

89 References L. Lovász and A. Schrijver. Cones of matrices and setfunctions, and 0-1 optimization. SIAM Journal on Optimization, 1, H. Marchand and L. Wolsey. The 0-1 knapsack problem with a single continuous variable. Mathematical Programming, 85:15 33, A. Martin, M. Möller, and S. Moritz. Mixed integer models for the stationary case of gas network optimization. Technical report, Darmstadt University of Technology, G. P. McCormick. Computability of global solutions to factorable nonconvex programs: Part I Convex underestimating problems. Mathematical Programming, 10: , Nemhauser, G.L. and Wolsey, L.A. Integer and Combinatorial Optimization. John Wiley, New York, M. Padberg, T. J. Van Roy, and L. Wolsey. Valid linear inequalities for fixed charge problems. Operations Research, 33: , I. Quesada and I. E. Grossmann. An LP/NLP based branch and bound algorithm for convex optimization problems. Computers and Chemical Engineering, 16: , Quist, A.J. Application of Mathematical Optimization Techniques to Nuclear Reactor Reload Pattern Design. PhD thesis, Technische Universiteit Delft, Thomas Stieltjes Institute for Mathematics, The Netherlands, R. Raman and I. E. Grossmann. Modeling and computational techniques for logic based integer programming. Computers and Chemical Engineering, 18: , 1994.

90 References H. D. Sherali and W. P. Adams. A hierarchy of relaxations between the continuous and convex hull representations for zero-one programming problems. SIAM Journal on Discrete Mathematics, 3: , O. Sigmund. A 99 line topology optimization code written in matlab. Structural Multidisciplinary Optimization, 21: , R. Stubbs and S. Mehrohtra. Generating convex polynomial inequalities for mixed 0-1 programs. Journal of Global Optimization, 24: , R. A. Stubbs and S. Mehrotra. A branch and cut method for 0 1 mixed convex programming. Mathematical Programming, 86: , M. Tawarmalani and N. V. Sahinidis. Convexification and Global Optimization in Continuous and Mixed-Integer Nonlinear Programming: Theory, Algorithms, Software, and Applications. Kluwer Academic Publishers, Boston MA, J. Viswanathan and I. E. Grossmann. Optimal feed location and number of trays for distillation columns with multiple feeds. I&EC Research, 32: , Westerlund, T., Isaksson, J. and Harjunkoski, I. Solving a production optimization problem in the paper industry. Report A, Department of Chemical Engineering, Abo Akademi, Abo, Finland, Westerlund, T., Pettersson, F. and Grossmann, I.E. Optimization of pump configurations as problem. Computers & Chemical Engineering, 18(9): , H. P. Williams. Model Solving in Mathematical Programming. John Wiley & Sons Ltd., Chichester, 1993.