Multistage Discrete Optimization Part I: Introduction Ted Ralphs 1 Joint work with Suresh Bolusani 1, Scott DeNegre 3, Menal Güzelsoy 2, Anahita Hassanzadeh 4, Sahar Tahernajad 1 1 COR@L Lab, Department of Industrial and Systems Engineering, Lehigh University 2 SAS Institute, Advanced Analytics, Operations Research R & D 3 The Hospital for Special Surgery 4 Climate Corp Friedrich-Alexander-Universität Erlangen-Nürnberg, 20-21 March 2017
Outline 1 Motivation 2 Simple Example 3 Applications 4 Formal Setting
Attributions Many students and collaborators contributed to development of this material. Current and Former Ph.D Students Suresh Bolusani Scott DeNegre Menal Gúzelsoy Anahita Hassanzadeh Ashutosh Mahajan Sahar Tahernajad Collaborators Stefano Coniglio Andrea Lodi Fabrizio Rossi Stefano Smriglio Gerhard Woeginger Thanks!
Agenda for the Course The workshop will be in 4-5 (unequal) parts: Agenda 1 Introduction 2 Duality 3 Algorithms 4 Software 5 Complexity It is highly unlikely we will get to the fifth part. However, all materials are available on-line here: http://coral.ie.lehigh.edu/~ted/teaching/multistage-optimization There you will also find some background reading material.
Warnings and Caveats The material is fairly technical and there is a lot of notation. I ll try to keep reminding you of the meaning as we go along. Most of it can be understood from first principles, but it will help to have a background in discrete optimization. I m going to be throwing a lot at you fairly quickly, so... Please stop me anytime you have a question!
Outline 1 Motivation 2 Simple Example 3 Applications 4 Formal Setting
What Are We Talking About? In game theory terminology, the problems we address are known as finite extensive-form games, sequential games involving n players. Loose Definition The game is specified on a tree with each node corresponding to a move and the outgoing arcs specifying possible choices. The leaves of the tree have associated payoffs. Each player s goal is to maximize payoff. There may be chance players who play randomly according to a probability distribution and do not have payoffs (stochastic games). Assumption: All players are rational and have perfect information. The problem faced by a player in determining the next move is itself a multilevel/multistage optimization problem. The move must be determined by taking into account the subsequent actions of other players. We are interested in problems where the number of possible moves is enormous, so brute force enumeration is not possible.
Example Game Tree 1: COIN 1 T OR COIN 2 T 2: COIN 2 T OR COIN 3 T COIN 1 TAILS COIN 1 HEADS 1: TRUE 2: UNDETERMINED 1: UNDETERMINED 2: UNDETERMINED COIN 2 TAILS COIN 2 HEADS COIN 2 TAILS COIN 2 HEADS 1: TRUE 2: TRUE 1: TRUE 2: UNDETERMINED 1: TRUE 2: TRUE COIN 3 TAILS COIN 3 HEADS 1: TRUE 2: TRUE 1: TRUE 2: FALSE 1: FALSE 2: UNDETERMINED
Analyzing Games Categories Multi-round vs. single-round Zero sum vs. Non-zero sum Winner take all vs. individual outcomes Goal of analysis Find an equilibrium Determine the optimal first move.
Multilevel and Multistage Games We use the term multilevel for competitive games in which there is no chance player. We use the term multistage for cooperative games in which all players receive the same payoff, but there are chance players. A subgame is the part of a game that remains after some moves have been made. Multilevel Game A game in which n players alternate moves in a fixed sequence (the well-known case of two players is called a Stackelberg game). The goal is to find a subgame perfect Nash equilibrium, i.e., the move by each player that ensures that player s best outcome. Multistage/Recourse Game A cooperative game in which play alternates between cooperating players and chance players. The goal is to find a subgame perfect Markov equilibrium, i.e., the move that ensures the best outcome in a probabilistic sense.
Outline 1 Motivation 2 Simple Example 3 Applications 4 Formal Setting
Example: Coin Flip Game Coin Flip Game k players take turns placing a set of coins heads or tails. In round i, player i places his/her coins. We have one or more logical expression that are of the form COIN 1 is heads OR COIN 2 is tails OR COIN 3 is tails OR... With even (resp. odd) k, even (resp. odd ) players try to make all expressions true, while odd (resp. even) players try to prevent this. Examples k = 1: Player looks for a way to place coins so that all expressions are true. k = 2: The first player tries to flip her coins so that no matter how the second player flips his coins, some expression will be false. k = 3: The first player tries to flip his coins such that the second player cannot flip her coins in a way that will leave the third player without any way to flip his coins to make the expressions true.
Coin Flip Game Tree 1: COIN 1 T OR COIN 2 T 2: COIN 2 T OR COIN 3 T COIN 1 TAILS COIN 1 HEADS 1: TRUE 2: UNDETERMINED 1: UNDETERMINED 2: UNDETERMINED COIN 2 TAILS COIN 2 HEADS COIN 2 TAILS COIN 2 HEADS 1: TRUE 2: TRUE 1: TRUE 2: UNDETERMINED 1: TRUE 2: TRUE COIN 3 TAILS COIN 3 HEADS 1: TRUE 2: TRUE 1: TRUE 2: FALSE 1: FALSE 2: UNDETERMINED
Example: Stochastic Variant The coin flip game can be modified to a recourse problem if we make the even player a chance player. In this variant, there is only one cognizant player (the odd player) who first chooses heads or tails for an initial set of coins. The even player is a chance player who randomly flips some of the remaining coins. Finally, the odd player tries to flip the remaining coins so as to obtain a positive outcome. The objective of the odd player s first move could then be, e.g., to maximize the probability of a positive outcome across all possible scenarios. Note that we need to know what happens in all scenarios in order to make the first move optimally! This is the key observation!
The QBF Problem When expressed in terms of Boolean (TRUE/FALSE) variables, the problem is a special case of the so-called quantified Boolean formula problem (QBF). The case of k = 1 is the well-known Satisfiability Problem. This figure below illustrates the search for solutions to the problem as a tree. The nodes in green represent settings of the truth values that satisfy all the given clauses; red represents non-satisfying truth values. With one player, the solution is any path to one of the green nodes. With two players, the solution is a subtree in which there are no red nodes. The latter requires knowledge of all leaf nodes (important!). x2 = TRUE C1 = TRUE C2 = TRUE C1 = x1 x2 C2 = x2 x3 x1 = TRUE x1 = FALSE C1 = TRUE C2 = x2 x3 C1 = x2 C2 = x2 x3 x2 = FALSE x2 = TRUE x2 = FALSE C1 = TRUE C2 = x3 C1 = TRUE C2 = TRUE x3 = TRUE x3 = FALSE C1 = TRUE C2 = TRUE C1 = TRUE C2 = FALSE C1 = FALSE C2 = x3 Figure 1: Game tree for example SAT game
More Formally More formally, we are given a Boolean formula with variables partitioned into k sets X 1,..., X k. For k odd, the SAT game can be formulated as X 1 X 2 X 3...?X k for even k, we have X 1 X 2 X 3...?X k
Multilevel and Multistage Optimization A (standard) mathematical optimization problem models a (set of) decision(s) to be made simultaneously by a single decision-maker (i.e., with a single objective). Decision problems arising in real-world sequential games can often be formulated as optimization problems, but they involve multiple, independent decision-makers (DMs), sequential/multi-stage decision processes, and/or multiple, possibly conflicting objectives. Modeling frameworks Multiobjective Optimization multiple objectives, single DM Mathematical Optimization with Recourse multiple stages, single DM Multilevel Optimization multiple stages, multiple objectives, multiple DMs Multilevel optimization generalizes standard mathematical optimization by modeling hierarchical decision problems, such as finite extensive-form games. Such models arises in a remarkably wide array of applications.
From QBF to Multilevel Optimization For k = 1, SAT can be formulated as the (feasibility) integer program? x {0, 1} n : i C 0 j x i + i C 1 j (SAT) can be re-formulated as the optimization problem max For k = 2, we then have min x I1 {0,1} I 1 x {0,1} n α s.t. i C 0 j max α x I2 {0,1} I 2 s.t. x i + i C 1 j i C 0 j (1 x i ) 1 j J. (SAT) (1 x i ) α j J x i + i C 1 j (1 x i ) α j J
Game Tree for Optimization Problem Here is a game tree for an optimization reformulation of the game shown in Figure 1. max α s.t. x1 + x2 α x2 + x3 α x1 = 1 x1 = 0 max α s.t. x2 + x3 1 max α s.t. x2 α x2 + x3 α x2 = 1 x2 = 0 x2 = 1 x2 = 0 1 max α s.t. x3 α 1 x3 = 1 x3 = 0 1 0 0 Figure 2: Optimization version of game from Figure 1 This is essentially what we would call a branch-and-bound tree.
How Difficult is the QBF? We focus on solving player one s decision problem, since this subsumes the solution of every other player s problem. No efficient algorithm is known for even the (single player) satisfiability problem. It is not surprising that the k-player satisfiability game is even more difficult (this can be formally proved). The k th player to move is faced with a satisfiability problem. The (k 1) th player is faced with a 2-player subgame in which she must take into account the move of the k th player. And so on... Each player s decision problem appears to be exponentially more difficult than the succeeding player s problem. This complexity is captured formally in the hierarchy of so-called complexity classes known as the polynomial time hierarchy.
Setting for the Rest of the Course We focus on games with two players (one of which may be a chance player) and two cognizant decision stages. Determination of each player s move involves solution of an optimization problem. The optimization problem faced by the first player involves implicitly knowing what the second player s reaction will be to all possible first moves. The need for complete knowledge of the second player s possible reactions is what puts the complexity of these problems beyond that of standard optimization.
Outline 1 Motivation 2 Simple Example 3 Applications 4 Formal Setting
Brief Overview of Practical Applications Hierarchical decision systems Government agencies Large corporations with multiple subsidiaries Markets with a single market-maker. Decision problems with recourse Parties in direct conflict Zero sum games Interdiction problems Modeling robustness : Chance player is external phenomena that cannot be controlled. Weather External market conditions Controlling optimized systems: One of the players is a system that is optimized by its nature. Electrical networks Biological systems
Example: Tunnel Closures [Bruglieri et al., 2008] The EU wishes to close certain international tunnels to trucks in order to increase security. The response of the trucking companies to a given set of closures will be to take the shortest remaining path. Each travel route has a certain risk associated with it and the EU s goal is to minimize the riskiest path used after tunnel closures are taken into account. This is a classical Stackelberg game.
Example: Robust Facility Location [Snyder, 2006]) We wish to locate a set of facilities, but we want our decision to be robust with respect to possible disruptions. The disruptions may come from natural disasters or other external factors that cannot be controlled. Given a set of facilities, we will operate them according to the solution of an associated optimization problem. Under the assumption that at most k of the facilities will be disrupted, we want to know what the worst case scenario is. This is a Stackelberg game in which the leader is not a cognizant DM.
Example: Fibrilation Ablation [Finta and Haines, 2004] Atrial fibrilation is a common form of heart arrhythmia that may be the result of impulse cycling within macroreentrant circuits. AF ablation procedures are intended to block these unwanted impulses from reaching the AV node. This is done by surgically removing some pathways. Since electrical impulses travel via the path of lowest resistance, we can model their flow using a mathematical optimization problem. If we wish to determine the least disruptive strategy for ablation, this is a Stackelberg game. In this case, the follower is not a cognizant DM.
Example: Electricity Network [Bienstock and Verma, 2008] As we know, electricity networks operate according to principles of optimization. Given a network, determining the power flows is an optimization problem. Suppose we wish to know the minimum number of links that need to be removed from the network in order to cause a failure. This, too, can be viewed as a Stackelberg game. Note that neither the leader nor the follower is a cognizant DM in this case.
Outline 1 Motivation 2 Simple Example 3 Applications 4 Formal Setting
Setting: Two-Stage Mixed Integer Optimization We have the following general formulation: 2SMILP { z 2SMILP = min c x + Ξ(x) }, (2SMILP) x P 1 X where P 1 = { x R n1 A 1 x b 1} is the first-stage feasible region, X = Z r1 + R n1 r1 +, A 1 Q m1 n1, and b 1 R m1. X = Z r1 + R n1 r1 + represents first-stage integrality requirements. Ξ is the risk function that represents the impact of future uncertainty. This uncertainty can arise either due to stochasticity or due to the fact that Ξ represents the reaction of a competitor (or both) This risk function is similar to that utilized in the finance literature. The function Ξ, as we define it next, is subadditive (a requirement for coherence in finance) and this has important theoretical implications.
The Risk Function The general class of functions we consider are of the following form. Canonical Risk Function Ξ(x) = E ω Ω [Ξ ω (x)], (RF) Scenario Risk Function Ξ ω (x) = min { d 1 y y argmin{d 2 y y P 2 (A 2 ωx) Y} } (2LRF) ω is a random variable over a finite probability space (Ω, F, P); Realizations of ω are called scenarios; P 2 (β) = {y R n2 + G 2 y A 2 ωx}; Y = Z r2 + R n2 r2 + ; and G 2 Q m2 n2, A 2 ω Q m2 n1 for ω Ω. Note that we drop the usual fixed component b 2 of the second-level right-hand side, but this is w.l.o.g., since adding a first-level constraint x 1 = 1 is equivalent.
Basic Assumptions Linking Variables Definition 1. The set L = ({ i {1,..., n1 } x(a 2 ω) i 0 }), ω Ω is the set of indices of the linking variables. In the above, (A 2 ω) i denotes the i th column of matrix A 2 ω. x L will denote the sub-vector of x R n1 corresponding to the linking variables. The linking variables are those with non-zero coefficients in the second-stage problem for at least one scenario. Assumption 1. L = {1,..., k 1 } for k 1 r 1. Assumption 2. P ω = { (x, y) R n1 n2 x P 1, y P 2 (A 2 ωx) } is bounded for ω Ω.
Rational Reaction Sets Corresponding to each x X, we have the rational reaction set for ω Ω. Rational Reaction Set for scenario ω R ω (x) = argmin{d 2 y y P 2 (A 2 ωx) Y}. For a given x X, R ω (x) set may be empty because either P 2(A 2 ωx) Y is itself empty or there exists r R n 2 + such that G2 r 0 and d 2 r < 0. The latter case cannot occur, since Assumption 2 implies that G 2 r > 0 for all r R n2 +).
Feasible Regions The bilevel feasible region for scenario ω with respect to the first- and second-stage variables in (2LRF) is F ω = {(x, y) X Y x P 1, y R ω (x)} Members of F ω are called bilevel feasible solutions for scenario ω. The second-stage problem should be feasible for all scenarios, so the feasible region with respect to first-level variables only is F 1 = proj x (F ω ). For x R n1, we have that x F 1 x ω Ω ω Ω and we say that x R n1 is feasible if x F 1. proj x (F ω ) R ω (x) ω Ω Ξ(x) <
Feasibility Conditions The feasibility conditions for scenario ω with respect to the first- and second-stage variables in (2LRF) are Feasibility Conditions Feasibility Condition 1. x F 1 Feasibility Condition 2. y ω R ω (x) for all ω Ω We exploit this notion of feasibility later in defining a scheme for branching
Special Case I: Bilevel (Integer) Linear Optimization In bilevel optimization, we have Ω = 1, so (2SMILP) can be re-written in the form Mixed Integer Bilevel Linear Optimization Problem (MIBLP) min { cx + d 1 y x P 1 X, y argmin{d 2 y y P 2 (A 2 x) Y} } (MIBLP) Alternatively, this corresponds to Bilevel Risk Function Ξ(x) = min { d 1 y y argmin{d 2 y y P 2 (A 2 x) Y} } Note that we drop the subscripts associated with the scenario in this case, but notation is otherwise, the same.
Example: Bilevel Optimization Example 1 Moore and Bard [1990] y 5 4 3 2 1 F LP F objective 1 2 3 4 5 6 7 8 x min x Z + x 10y s.t. y argmin { y : 25x + 20y 30 x + 2y 10 2x y 15 2x + 10y 15 y Z + }
Special Case II: Optimization with Recourse Recourse problems are a special case in which d 1 = d 2. In a two-stage stochastic integer optimization problem, we have Stochastic Risk Function [ Ξ(x) = E ω Ω φ(a 2 ω x) ], where ω is the random variable from probability space (Ω, F, P) defined earlier. For each ω Ω, A 2 ω Q m2 is the realization of the input to the second-stage problem for scenario ω. The function φ is the value function of the second-stage MILP. Second-stage Value Function φ(β) = min y P d2 y 2(β) (2S-VF)
Example: Two-stage Stochastic Optimization Example 2 Hassanzadeh and Ralphs [2014] min Ψ(x 1, x 2 ) = min 3x 1 4x 2 + E[φ(ω 2x 1 0.5x 2 )] s.t. x 1 5, x 2 5 x 1, x 2 R +, (Ex.SMP) and ω {6, 12} with a uniform probability distribution. where φ(β) = min 3y 1 + 7 y2 + 3y3 + 6y4 + 7y5 2 s.t. 6y 1 + 5y 2 4y 3 + 2y 4 7y 5 = β y 1, y 2, y 3 Z +, y 4, y 5 R +
Other Special Cases Pure integer. Positive/negative constraint matrix at second stage (inequality constraints). Binary variables at the first and/or second stage. Zero sum and interdiction problems. Mixed Integer Interdiction where max min x P1 INT X y P2 INT (b 2 ) Y dy (MIPINT) P INT 1 = { x X A 1 x b 1} X = B n P INT 2 (b 2 ) = { y Y G 2 y b 2, y u(e x) } Y = Z r + R n r + The case where follower s problem has network structure is called the network interdiction problem and has been well-studied. The model above allows for second-stage systems described by general MILPs.
Value Function Reformulation One way to reformulate (2SMILP) as a single-level problem is as follows. y FLP ω 5 F ω objective 4 3 2 1 min c 1 x + p ωd 1 y ω ω Ω subject to A 1 x b 1 G 2 y ω A 2 ωx ω Ω d 2 y ω φ(a 2 ωx) ω Ω x X y ω Y ω Ω 1 2 3 4 5 6 7 8 x where φ is the value function of the second-stage problem (2S-VF). This is, in principle, a standard mathematical optimization problem. Note that the second-stage variables need to appear in the formulation in order to enforce feasibility.
Risk Function Reformulation A second reformulation can be obtained by projecting out the second-stage variables. min c 1 x + ω Ω p ω z ω subject to z ω Ξ ω (x) x X We can further simplify this reforumlation by noting that Ξ ω can itself be rewritten as Ξ ω (x) = ρ(a 2 ωx) where ρ(β) = min { d 1 y y argmin{d 2 y y P 2 (β) Y} } is the (optimistic) reaction function, so called because it computes the We project out the second-stage variables, as in a traditional Benders algorithm. This leads to a generalized Benders algorithm obtained by constructing approximations of ρ dynamically (Benders cuts ).
Polyhedral Reformulation Convexification considers the following conceptual reformulation. y 5 4 3 2 F ω conv(f ω ) conv(s ω ) min c 1 x + ω Ω p ω d 1 y s.t. (x, y ω ) conv(f ω ) 1 1 2 3 4 5 6 7 8 x where S ω = P ω (X Y) for P ω = {(x, y) R n1 n2 A 1 x b 1, G 2 y A 2 ωx} To get bounds, we ll optimize over a relaxed feasible region. We ll iteratively approximate the true feasible region with linear inequalities.
How Difficult is Multilevel/Multistage Optimization? In some sense, it is counter-intuitive that multilevel optimization is a difficult problem. The search space for (MIBLP) is the same as that for an MILP over the same set of variables. What makes it difficult? One way to see this is consider that a proof of feasibility involves only a single node in the MILP case, but possibly exponentially many nodes in the MIBLP case. max α s.t. x1 + x2 α x2 + x3 α x1 = 1 x1 = 0 max α s.t. x2 + x3 1 max α s.t. x2 α x2 + x3 α x2 = 1 x2 = 0 x2 = 1 x2 = 0 1 max α s.t. x3 α 1 x3 = 1 x3 = 0 1 0 0
Nonconvexity of Feasible Region The figure below is a zero-sum maxi-min MIBLP with P 1 = R n1. F LP and F are the feasible regions of the MIBLPs in which the variables are continuous and discrete, respectively. Even in the continuous case, the feasible region is non-convex. Furthermore, the solution to the continuous problem is integral, but is not the solution to the discrete problem. y 5 4 F LP F objective 3 2 1 1 2 3 4 5 6 7 8 x
Formal Complexity Even in the simplest case in which there are no integer variables, two-stage MILP is an NP-hard problem. Roughly speaking, this is because even a linear program behaves like a combinatorial problem when embedded in the second stage. When the second-stage problem is NP-hard, the two-stage problem is not even in NP. In fact, for even simple cases (knapsack interdiction), these problems are hard for the second level of the polynomial hierarchy. Formally, the decision version of the n-stage problem is complete for Σ p n Σ p 1 = NP. Σ p k is the class of problems that can be solved in non-deterministic polynomial time, given an oracle for Σ p k 1 See Lodi et al. [2014], Caprara et al. [2014].
References I D. Bienstock and A. Verma. The n-k problem in power grids: New models, formulations and computation. Available at http://www.columbia.edu/~dano/papers/nmk.pdf, 2008. M. Bruglieri, R. Maja, G. Marchionni, and P.L. Da Vinci. Safety in hazardous material road transportation: State of the art and emerging problems. In Advanced Technologies and Methodologies for Risk Management in the Global Transport of Dangerous Goods, chapter 6, pages 88 129. IOS Press, 2008. A. Caprara, M. Carvalho, A. Lodi, and G.J. Woeginger. A study on the computational complexity of the bilevel knapsack problem. SIAM Journal on Optimization, 24: 823 838, 2014. B. Finta and D.E Haines. Catheter ablation therapy for atrial fibrillation. Cardiology Clinics, 22(1):127 145, 2004. A Hassanzadeh and T K Ralphs. A generalized benders algorithm for two-stage stochastic program with mixed integer recourse. Technical report, COR@L Laboratory Report 14T-005, Lehigh University, 2014. URL http://coral. ie.lehigh.edu/~ted/files/papers/smilpgenbenders14.pdf.
References II A Lodi, T K Ralphs, and G Woeginger. Bilevel programming and the separation problem. Mathematical Programming, 148:437 458, 2014. doi: 10.1007/s10107-013-0700-x. URL http://coral.ie.lehigh.edu/ ~ted/files/papers/bilevelseparation12.pdf. J.T. Moore and J.F. Bard. The mixed integer linear bilevel programming problem. Operations Research, 38(5):911 921, 1990. L. V. Snyder. Facility location under uncertainty: A review. IIE Transactions, 38(7): 537 554, 2006.