The Dynamic Energy Resource Model

Transcription

1 The Dynamic Energy Resource Model Group Peer Review Committee Lawrence Livermore National Laboratories July, 2007 Warren Powell Alan Lamont Jeffrey Stewart Abraham George 2007 Warren B. Powell, Princeton University 2007 Warren B. Powell Slide 1

2 Dynamic energy resource management Questions:» How will the market evolve in terms of the adoption of competing energy technologies? How many windmills, and where? How much ethanol capacity? How will the capacity of coal, natural gas and oil evolve?» What government policies should be implemented? Carbon tax? Cap and trade? Tax credits for windmills and solar panels? Tax credits for ethanol?» Where should we invest R&D dollars? Ethanol or hydrogen? Batteries or windmills? Hydrogen production, storage or conversion? 2007 Warren B. Powell Slide 2

3 Dynamic energy resource management Uncertainties:» Technology: Carbon sequestration The cost of batteries, fuel cells, solar panels The storage of hydrogen, efficiency of solar panels,» Climate: Global and regional temperatures Changing patterns of snow storage on mountains Wind patterns» Markets: Global supplies of oil and natural gas International consumption patterns Domestic purchasing behaviors (SUV s?) Tax policies The price of oil and natural gas 2007 Warren B. Powell Slide 3

4 Dynamic energy resource management Research challenges:» Making decisions Finding the best decisions (capacity decisions, R&D decisions, government policies) requires solving high-dimensional stochastic, dynamic programs. How do we obtain practical solutions to stochastic, dynamic programs which exhibit state variables with millions of dimensions?» Modeling multiple time scales We have to represent wind, temperature, rain and snow fall, market prices and government policies. This requires modeling hourly, daily, seasonal and yearly dynamics.» Modeling multiple levels of resolution Spatial: We need to represent the location of windmills at state, regional and county levels. Behavioral: We need to capture the differences between travel behavior patterns (long commutes vs. short trips, commercial fleet vehicles vs. personal use), or the difference between light and heavy industrial power use Warren B. Powell Slide 4

5 Dynamic energy resource management Alternative ways of solving large stochastic optimization problems:» Simulation using myopic policies Using rules to determine decisions based on the current state of the system. Rules are hard to design, and decisions now do not consider the impact on the future.» Deterministic optimization Ignores uncertainty (and problems are still very large scale).» Rolling horizon procedures Uses point estimates of what might happen in the future. Will not produce robust behaviors.» Stochastic programming Cannot handle multiple sources of uncertainty over multiple time periods.» Markov decision processes Discrete state, discrete action will not scale ( curse of dimensionality ) 2007 Warren B. Powell Slide 5

6 Dynamic energy resource management Proposed approach: Approximate dynamic programming» Our research combines mathematical programming, simulation and statistics in a dynamic programming framework. Math programming handles high-dimensional decisions. Simulation handles complex dynamics and high-dimensional information processes. Statistical learning is used to improve decisions iteratively. Solution strategy is highly intuitive tends to mimic human behavior.» Features: Scales to very large scale problems. Easily handles complex dynamics and information processes. Rigorous theoretical foundation» Research challenge: Calibrating the model. Designing high quality policies using the tools of approximate dynamic programming. Evaluating the quality of these policies Warren B. Powell Slide 6

7 Outline My experiences A resource allocation model ADP and the post-decision state variable Illustration using blood management Laboratory and theoretical results The dynamic energy resource model 2007 Warren B. Powell Slide 7

8 Outline My experiences A resource allocation model ADP and the post-decision state variable Illustration using blood management Laboratory and theoretical results The dynamic energy resource model 2007 Warren B. Powell Slide 8

9 Yellow Freight System 2004 Warren B. Powell, Princeton University 2007 Warren B. Powell Slide 9

10 2007 Warren B. Powell Slide 10

11 2007 Warren B. Powell Slide 11

12 2007 Warren B. Powell Slide 12

13 2007 Warren B. Powell Slide 13

14 The fractional jet ownership industry 2007 Warren B. Powell Slide 14

15 NetJets Inc Warren B. Powell Slide 15

16 Planning for a risky world Weather Robust design of emergency response networks. Design of financial instruments to hedge against weather emergencies to protect individuals, companies and municipalities. Design of sensor networks and communication systems to manage responses to major weather events. Disease Models of disease propagation for response planning. Management of medical personnel, equipment and vaccines to respond to a disease outbreak. Robust design of supply chains to mitigate the disruption of transportation systems Warren B. Powell Slide 16

17 2007 Warren B. Powell Slide 17

18 2007 Warren B. Powell Slide 18

19 Energy management Energy resource management How to balance investment in ethanol, windmills, nuclear, coalto-hydrogen? When should we make multidecade commitments to evolving technologies? What is the pattern of demands? How will climate change affect adoption patterns? Energy R&D portfolio planning Where should DOE, NSF, invest R&D dollars for new technologies? How do we balance investments in different components of an energy technology pathway? How do we evaluate the probability of a successful R&D program? How do we solve multistage resource allocation problems for R&D problems? 2007 Warren B. Powell Slide 19

20 CASTLE Lab News Part VII - News Thursday, March 2, 1999 New Modeling Language Captures 75 cents Complexities of Real-World Operations! Spans the gap between simulation and optimization. CASTLE Lab announced the development of a powerful new simulation environment for modeling complex operations in transportation and logistics. The dissertation of Dr. Joel Shapiro, it offers the flexibility of simulation environments, but the intelligence of optimization. The modeling language will allow managers to quickly test continued on page Warren B. Powell Slide 20

21 2007 Warren B. Powell Slide 21

22 Outline My experiences A resource allocation model ADP and the post-decision state variable Illustration using blood management Laboratory and theoretical results The dynamic energy resource model 2007 Warren B. Powell Slide 22

23 A resource allocation model Attribute vectors: a = Asset class Time invested Type Location Age Location ETA Home Experience Driving hours Location ETA A/C type Fuel level Home shop Crew Eqpt1 Eqpt Warren B. Powell Slide 23

24 Energy resource modeling The state of a resource: a t Capacity of facilities Location Cost = Carbon output Age Reserves 2007 Warren B. Powell Slide 24

25 A resource allocation model Modeling resources:» The state of a single resource: a = The attributes of a single resource a A The attribute space» The state of multiple resources: R = The number of resources with attribute a ta Rt = Rta The resource state vector a A» The information process: ˆ = The change in the number of resources with R ta ( ) attribute a Warren B. Powell Slide 25

26 A resource allocation model Modeling demands:» The attributes of a single demand: b = The attributes of a demand to be served. b B The attribute space» The demand state vector: D = The number of demands with attribute b tb ( ) Dt = Dtb The demand state vector b B» The information process: ˆ = The change in the number of demands with D tb attribute b Warren B. Powell Slide 26

27 Energy resource modeling The system state: S ( R D ρ ) =,, = System state, where: t t t t R t D t = Resource state (how much capacity, reserves) = Market demands ρ = "system parameters" t State of the technology (costs, performance) Climate, weather (temperature, rainfall, wind) Government policies (tax rebates on solar panels) Market prices (oil, coal) 2007 Warren B. Powell Slide 27

28 Energy resource modeling The decision variable: x t New capacity Retired capacity for each: = Type Location Technology 2007 Warren B. Powell Slide 28

29 Energy resource modeling Exogenous information: W Rˆ t Dˆ t = ( Rˆ Dˆ ˆ ρ ) New information =,, t t t t = Exogenous changes in capacity, reserves = New demands for energy from each source ˆ ρ = Changes in technology (due to R&D) t 2007 Warren B. Powell Slide 29

30 Energy resource modeling The transition function = M t+ 1 t t t+ 1 S S ( S, x, W ) 2007 Warren B. Powell Slide 30

31 A resource allocation model The three states of our system» The state of a single resource/entity at1 at = a t2 a t3» The resource state vector R t R ta1 = Rta 2 Rta 3» The system state vector (,, ) S = R D ρ t t t t 2007 Warren B. Powell Slide 31

32 A resource allocation model Resources Demands 2007 Warren B. Powell Slide 32

33 A resource allocation model t t+1 t Warren B. Powell Slide 33

34 A resource allocation model Optimizing over time t t+1 t+2 Optimizing at a point in time 2007 Warren B. Powell Slide 34

35 Outline My experiences A resource allocation model ADP and the post-decision state variable Illustration using blood management Laboratory and theoretical results The dynamic energy resource model 2007 Warren B. Powell Slide 35

36 Action State Use weather report Do not use weather report Information Forecast rain.1 Forecast cloudy.3 Forecast sunny.6 Schedule game Cancel game State Action Schedule game Cancel game Schedule game Cancel game Schedule game Cancel game Rain.2 -$2000 Clouds.3 $1000 Sun.5 $5000 Rain.2 -$200 Clouds.3 -$200 Sun.5 -$200 Information Rain.8 -$2000 Clouds.2 $1000 Sun.0 $5000 Rain.8 -$200 Clouds.2 -$200 Sun.0 -$200 Rain.1 -$2000 Clouds.5 $1000 Sun.4 $5000 Rain.1 -$200 Clouds.5 -$200 Sun.4 -$200 Rain.1 -$2000 Clouds.2 $1000 Sun.7 $5000 Rain.1 -$200 Clouds.2 -$200 Sun.7 -$200 - Decision nodes - Outcome nodes

37 Laying the foundation Dynamic programming review:»let: S x t t = "State" of our "system" at time t. = "Action" that we take to change the system. CS (, x) = Contribution earned when we take action xfrom state S. t t t» We model system dynamics using: ps ( S, x) = Probability that action x takes us from t+ 1 t t t state S to state S t t Warren B. Powell Slide 37

38 Laying the foundation Bellman s equation:» Standard form: Vt( St) = max x Ct( St, xt) + p( s' St, xt) Vt+ 1( St+ 1 = s') s'» Expectation form: { + 1( + 1 ) } ( ) V ( S ) = max C ( S, x ) + E V S ( S, x ) S t t x t t t t t t t t 2007 Warren B. Powell Slide 38

39 Rain.8 -$2000 Clouds.2 $1000 Sun.0 $5000 Rain.8 -$200 Clouds.2 -$200 Sun.0 -$200 Rain.1 -$2000 Clouds.5 $1000 Sun.4 $5000 Rain.1 -$200 Clouds.5 -$200 Sun.4 -$200 Rain.1 -$2000 Clouds.2 $1000 Sun.7 $5000 Rain.1 -$200 Clouds.2 -$200 Sun.7 -$200 Schedule game Cancel game Schedule game Cancel game Schedule game Cancel game Rain.2 -$2000 Clouds.3 $1000 Sun.5 $5000 Rain.2 -$200 Clouds.3 -$200 Sun.5 -$200 - Decision nodes - Outcome nodes Forecast rain.1 Forecast cloudy.3 Forecast sunny.6 Schedule game Cancel game Use weather report Do not use weather report

40 Schedule game Cancel game Schedule game Cancel game Schedule game Cancel game $2400 -$200 -$1400 -$200 $2300 -$200 $3500 -$200 Forecast rain.1 Forecast cloudy.3 Forecast sunny.6 Schedule game Cancel game Use weather report Do not use weather report

41 -$200 $2300 $3500 $2400 -$200 Forecast rain.1 Forecast cloudy.3 Forecast sunny.6 Schedule game Cancel game Use weather report Do not use weather report

42 $2770 $2400 Use weather report Do not use weather report

43 Bellman s equation We just solved Bellman s equation: { } V ( S ) = max C ( S, x ) + E V ( S ) S t t t t t t+ 1 t+ 1 t x X» We found the value of being in each state by stepping backward through the tree Warren B. Powell Slide 43

44 Bellman s equation The challenge of dynamic programming: { } ( ) V ( S ) = max C ( S, x ) + E V ( S ) S t t t t t t+ 1 t+ 1 t x X Problem: Curse of dimensionality 2007 Warren B. Powell Slide 44

45 The curses of dimensionality What happens if we apply this idea to our blood problem?» State variable is: The supply of each type of blood, along with its age 8 blood types 6 ages = 48 blood types The demand for each type of blood 8 blood types» Decision variable is how much of 48 blood types to supply to 8 demand types dimensional decision vector» Random information Blood donations by week (8 types) New demands for blood (8 types) 2007 Warren B. Powell Slide 45

46 The curses of dimensionality The challenge of dynamic programming: { } ( ) V ( S ) = max C ( S, x ) + E V ( S ) S t t t t t t+ 1 t+ 1 t x X Three curses Problem: Curse of dimensionality State space Outcome space Action space (feasible region) 2007 Warren B. Powell Slide 46

47 The curses of dimensionality The computational challenge: { } ( ) V ( S ) = max C ( S, x ) + E V ( S ) S t t t t t t+ 1 t+ 1 t x X How do we find V ( S )? t+ 1 t+ 1 How do we compute the expectation? How do we find the optimal solution? 2007 Warren B. Powell Slide 47

48 Rain.8 -$2000 Clouds.2 $1000 Sun.0 $5000 Rain.8 -$200 Clouds.2 -$200 Sun.0 -$200 Rain.1 -$2000 Clouds.5 $1000 Sun.4 $5000 Rain.1 -$200 Clouds.5 -$200 Sun.4 -$200 Rain.1 -$2000 Clouds.2 $1000 Sun.7 $5000 Rain.1 -$200 Clouds.2 -$200 S n 7 $200 Schedule game S t S t + 1 Cancel game Forecast rain.1 Schedule game Forecast cloudy.3 Cancel game Forecast sunny.6 Use weather report Schedule game Cancel game Rain 2 -$2000 Do not weath ( { } ) V ( S ) = max C ( S, x ) + E V ( S ) S t t t t t t+ 1 t+ 1 t x X

49 Rain.8 -$2000 Clouds.2 $1000 Sun.0 $5000 Rain.8 -$200 Clouds.2 -$200 Sun.0 -$200 Rain.1 -$2000 Clouds.5 $1000 Sun.4 $5000 Rain.1 -$200 Clouds.5 -$200 Sun.4 -$200 Rain.1 -$2000 Clouds.2 $1000 Sun.7 $5000 Rain.1 -$200 Clouds.2 -$200 Sun.7 -$200 Schedule game Cancel game Schedule game Cancel game Schedule game Cancel game Rain.2 -$2000 Clouds.3 $1000 Sun.5 $5000 Rain.2 -$200 Clouds.3 -$200 Sun.5 -$200 - Decision nodes - Outcome nodes Forecast rain.1 Forecast cloudy.3 Forecast sunny.6 Schedule game Cancel game Use weather report Do not use weather report

50 Pre- and post-decision states New concept:» The pre-decision state variable: St = The information required to make a decision x t Same as a decision node in a decision tree.» The post-decision state variable: x S t = The state of what we know immediately after we make a decision. Same as an outcome node in a decision tree Warren B. Powell Slide 50

51 Pre- and post-decision states Pre-decision, state-action, and post-decision Pre-decision state State Action Post-decision state 9 3 states state-action pairs 9 3 states 2007 Warren B. Powell Slide 51

52 A single, complex entity Pre- and post-decision attributes for our nomadic truck driver: t = 40 Pre-decision t = 40 Decision t = 40 Post-decision t = 50 Pre-decision City ETA Equip Dallas 41.2 Good Chicago - - Chicago 54.7 Good Chicago 56.2 Repair 2007 Warren B. Powell Slide 52

53 Pre- and post-decision states Pre-decision: resources and demands S = ( R, D ) t t t 2007 Warren B. Powell Slide 53

54 Pre- and post-decision states S = S ( S, x ) x M, x t t t 2007 Warren B. Powell Slide 54

55 Pre- and post-decision states x S t S = S ( S, W ) MW, x t+ 1 t t+ 1 W = ( Rˆ, Dˆ ) t+ 1 t+ 1 t Warren B. Powell Slide 55

56 Pre- and post-decision states S t Warren B. Powell Slide 56

57 System dynamics It is traditional to assume you are given the one-step transition matrix: p( S S, x ) = Probability that action x takes us from state S to state S t+ 1 t t t t t+ 1» Computing the transition matrix is impossible for the vast majority of problems. We are going to assume that we are given a transition function: (,, ) S = S S x W M t+ 1 t t t+ 1» This is at the heart of any simulation model.» Often rule-based. Very easy to compute, even for large-scale problems Warren B. Powell Slide 57

58 The transition function Computing the post-decision state:» Method 1 Divide the effect of decisions and information» Method 2 State-action pairs ( Q-learning ) S = ( S x ) x t t t ( ) S = S S, x The pure effect of a decision x M, x t t t x ( ) S = S S, W The effect of the exogenous information MW, t+ 1 t t+ 1, Produces huge post-decision state space» Method 3 Post-decision based on point estimate ( ) S = S S, x, W W is a point-estimate of W at time t. x M t t t t, t+ 1 t, t+ 1 t+ 1 (,, ) S = S S x W M t+ 1 t t t Warren B. Powell Slide 58

59 The transition function Actually, we have three transition functions:» The attribute transition function: ( ) a = a a, x The pure effect of a decision x M, x t t t x ( ) a = a a, W The effect of the exogenous information MW, t+ 1 t t+ 1» The resource transition function ( ) R = R R, x The pure effect of a decision x M, x t t t» The general transition function: ( ) S = S S, x The pure effect of a decision x M, x t t t x ( ) R = R R, W The effect of the exogenous information MW, t+ 1 t t+ 1 x ( ) S = S S, W The effect of the exogenous information MW, t+ 1 t t Warren B. Powell Slide 59

60 Bellman s equations with the post-decision state Bellman s equations broken into stages:» Optimization problem (making the decision): Note: this problem is deterministic! M, x ( ) ( x ) V ( S ) = max C ( S, x ) + V S ( S, x ) t t x t t t t t t t» Simulation problem (the effect of exogenous information):, { } V ( S ) = E V ( S ( S, W )) S x x M W x x t t t t t t 2007 Warren B. Powell Slide 60

61 Bellman s equations with the post-decision state Challenges» For most practical problems, we are not going to be x x able to compute V ( S ). t x x ( ) V ( S ) = max C ( S, x ) + V ( S ) t t x t t t t t» Concept: replace it with an approximation V ( x t St ) and solve» So now we face: What should the approximation look like? How do we estimate it? t x ( ) V ( S ) = max C ( S, x ) + V ( S ) t t x t t t t t 2007 Warren B. Powell Slide 61

62 Approximating the value function Value function approximations:» Linear (in the resource state): x V ( R ) = v R x t t ta ta a A» Piecewise linear, separable: x x Vt ( Rt ) = Vta ( Rta ) a A» Indexed PWL separable: Best when assets are complex, which means that R ta is small (typically 0 or 1). Best when assets are simple, which means that R ta may be larger. ( x ) x Vt ( Rt ) = Vta Rta ( featurest ) a A Helps to capture dependencies. e.g. status of technology, climate, 2007 Warren B. Powell Slide 62

63 Approximating the value function Value function approximations:» Ridge regression (Klabjan and Adelman) ( ) x V ( R ) = V R R = θ R t t tf tf tf fa ta f F a A» Benders cuts (more on this later) f V ( R ) t t x 1 x Warren B. Powell Slide 63

64 Our general algorithm, Step 1: Start with a post-decision state S x n t 1 n Step 2: Obtain Monte Carlo sample of W ( ω ) and compute the next pre-decision state: Step 3: Solve the deterministic optimization using an approximate value function: to obtain. (, ( ) 1 ) S = S S W ω n M, W x, n n t t t 1, (, ))) n n n M x n vˆ = max C ( S, x ) + V ( S ( S x t x t t t t t t n x t Step 4: Update the value function approximation V S V S v n ( x, n ) (1 ) n 1 ( x, n ) n t 1 t 1 = αn 1 t 1 t 1 + αn 1ˆ t Step 5: Find the next post-decision state: x, n M, x n n S = S ( S, x ) t t t t Simulation Optimization Statistics 2007 Warren B. Powell Slide 64

65 Competing updating methods Comparison to other methods:» Classical MDP (value iteration) ( n 1 ) t+ 1 n V ( S) = max C( S, x) + EV ( S ) x» Classical ADP (pre-decision state): n n n n 1 vˆ t = max x Ct( St, xt) + p( s' St, xt) Vt+ 1 ( s' ) s' V ( S ) = (1 α ) V ( S ) + α vˆ vˆ updates V ( S ) n n n 1 n n t t n 1 t t n 1 t, 1» Our method (update V x n around post-decision state):, 1, ( ) n n x n M x n vˆ = max C ( S, x ) + V ( S ( S, x )) t x t t t t t t V ( S ) = (1 α ) V ( S ) + α vˆ n x, n n 1 x, n n t 1 t 1 n 1 t 1 t 1 n 1 t t t t t vˆ updates V ( S ) x t t 1 t Warren B. Powell Slide 65

66 Outline My experiences A resource allocation model ADP and the post-decision state variable Illustration using blood management Laboratory and theoretical results The dynamic energy resource model 2007 Warren B. Powell Slide 66

67 Blood management Managing blood inventories 2007 Warren B. Powell Slide 67

68 Blood management Managing blood inventories over time Week 1 Week 2 Week 2 S 0 Rˆ, Dˆ 1 1 S 1 x 1 S x 1 Rˆ, Dˆ 2 2 S 2 x 2 S x 2 Rˆ, Dˆ 3 3 x S S 3 3 x 3 t=0 t=1 t=2 t= Warren B. Powell Slide 68

69 S t = ( R, ˆ ) t Dt AB+ D ˆt, AB+ x R t AB+,0 R t,( AB+,0) AB+,0 AB- D ˆt, AB AB+,1 R t,( AB+,1) R t,( AB+,2) R R R t,( O,0) t,( O,1) t,( O,2) AB+,1 AB+,2 O-,0 O-,1 O-,2 A+ A- B+ B- O+ O- D ˆt, A+ D ˆt, AB+ D ˆt, AB+ D ˆt, AB+ D ˆt, AB+ D ˆt, AB+ AB+,2 AB+,3 O-,0 O-,1 O-,2 O-,3 Satisfy a demand Hold

70 R t x R t R t + 1 R t,( AB+,0) AB+,0 AB+,0 R ˆt+ 1, AB+ AB+,0 R t,( AB+,1) AB+,1 AB+,1 AB+,1 R t,( AB+,2) AB+,2 AB+,2 AB+,2 AB+,3 AB+,3 R t,( O,0) O-,0 R t,( O,1) O-,1 O-,0 R ˆt+ 1, O O-,0 R t,( O,2) O-,2 O-,1 O-,1 O-,2 O-,2 O-,3 O-,3 Dˆt

71 R t x R t R t,( AB+,0) R t,( AB+,1) R t,( AB+,2) R R R t,( O,0) t,( O,1) t,( O,2) AB+,0 AB+,1 AB+,2 O-,0 O-,1 O-,2 AB+,0 AB+,1 AB+,2 AB+,3 O-,0 O-,1 O-,2 O-,3 Dˆt

72 R t x R t ( ) F R t R t,( AB+,0) R t,( AB+,1) AB+,0 AB+,1 AB+,0 AB+,1 R t,( AB+,2) R R R t,( O,0) t,( O,1) t,( O,2) AB+,2 O-,0 O-,1 O-,2 Solve this as a linear program. AB+,2 AB+,3 O-,0 O-,1 O-,2 O-,3 Dˆt

73 Duals R t x R t ( ) F R t ˆt ν,( AB +,0) AB+,0 AB+,0 ˆt ν,( AB +,1) AB+,1 AB+,1 ˆt ν,( AB +,2) AB+,2 AB+,2 AB+,3 ν ˆt,( O,0) O-,0 ν ˆt,( O,1) O-,1 O-,0 ν ˆt,( O,2) O-,2 O-,1 O-,2 Dual variables give value additional unit of blood.. O-,3 Dˆt

74 Updating the value function approximation Estimate the gradient at n R t ˆn ν t,( AB+,2) ( ) FR t n R t,( AB+,2) 2007 Warren B. Powell Slide 74

75 Updating the value function approximation Update the value function at x, n Rt 1 V ( R ) n 1 x t 1 t 1 ˆn ν t,( AB+,2) ( ) F R t x, n Rt 1 n R t,( AB+,2) 2007 Warren B. Powell Slide 75

76 Updating the value function approximation Update the value function at x, n Rt 1 V ( R ) n 1 x t 1 t 1 ˆn ν t,( AB+,2) x, n Rt Warren B. Powell Slide 76

77 Updating the value function approximation Update the value function at x, n Rt 1 V ( R ) n 1 x t 1 t 1 V ( R ) n x t 1 t 1 x, n Rt Warren B. Powell Slide 77

78 Blood management t 2007 Warren B. Powell Slide 78

79 Blood management 2007 Warren B. Powell Slide 79

80 Blood management 2007 Warren B. Powell Slide 80

81 Blood management 2007 Warren B. Powell Slide 81

82 Blood management Total Shortages (# units) Not Using Value Functions Using Value Functions Iterations 2007 Warren B. Powell Slide 82

83 2007 Warren B. Powell Slide 83

84 2007 Warren B. Powell Slide 84

85 2007 Warren B. Powell Slide 85

86 2007 Warren B. Powell Slide 86

87 2007 Warren B. Powell Slide 87

88 2007 Warren B. Powell Slide 88

89 Implementation metrics Results from the real world: Percent History Model Setouts Swaps Nonpreferred consists Underpowered Overpowered 2007 Warren B. Powell Slide 89

90 Schneider National 2007 Warren B. Powell Slide 90

91 2007 Warren B. Powell Slide 91

92 Case study: truckload trucking 1400 Revenue per WU US_SOLO US_IC US_TEAM Capacity category Revenue per WU U tiliz atio n Historical maximum Simulation Historical minimum Historical Calibrated min model and max Utilization Historical maximum Simulation Historical minimum US_SOLO US_IC US_TEAM Capacity category 2007 Warren B. Powell Slide 92

93 Outline My experiences A resource allocation model ADP and the post-decision state variable Illustration using blood management Laboratory and theoretical results The dynamic energy resource model 2007 Warren B. Powell Slide 93

94 Two-stage optimization Piecewise linear, separable value function approximations: Piecewise linear, separable: Vt ( Rt ) = Vtl ( Rtl ) l L 2007 Warren B. Powell Slide 94

95 Two-stage optimization Benders decomposition: VR ( ) t t x 1 x 0 Multidimensional cuts produce provably convergent, nonseparable value function approximation Warren B. Powell Slide 95

96 The competition Exact solutions using Benders: L-Shaped decomposition (Van Slyke and Wets) V 0 x 0 Stochastic decomposition (Higle and Sen) V 0 x 0 CUPPS (Chen and Powell) V 0 x Warren B. Powell Slide 96

97 The competition 45 Percent over Percent optimal from optimal after iterations iterations Increasing problem size Percent error locations 25 locations 50 locations 100 locations SD L-shaped CUPPS SPAR Benders Separable 2007 Warren B. Powell Slide 97

98 The competition Percent over Percent optimal from optimal after iterations iterations Increasing problem size 35 Percent error locations 25 locations 50 locations 100 locations SD L-shaped CUPPS SPAR Benders Separable 2007 Warren B. Powell Slide 98

99 Multistage problems Deterministic, (integer) multicommodity flow 2007 Warren B. Powell Slide 99

100 Multistage problems Deterministic, (integer) multicommodity flow = optimal continuous relaxation Percent of optimal Base T_30 T_90 I_10 I_40 C_II C_III C_IV R_1 R_5 R_100 R_400 C_1 C_ Warren B. Powell Slide 100

101 Multistage problems Stochastic, (integer) multicommodity flow Rolling horizon ADP Percent of posterior optimal 0 Base I_10 I_40 C_II C_III C_IV R_1 R_5 R_100 R_400 C_1 C_ Warren B. Powell Slide 101

102 Two-stage stochastic programming Properties of separable, piecewise linear value function approximations:» Converges to optimal when we sample all points infinitely often: H. Topaloglu and W. B Powell, OR Letters, 2003.» Provably optimal for two-stage, nonseparable functions with continuously differentiable second stage: R. K.-L. Cheung and W.B. Powell, Operations Research, 2000.» Provably optimal for two-stage, separable problems: Powell, W.B., A. Ruszczynski and H. Topaloglu, Mathematics of Operations Research, 2004.» Near-optimal for two-stage, nonseparable with nondifferentiable second stage: Powell, W.B., A. Ruszczynski and H. Topaloglu, Mathematics of Operations Research, 2004.» Provably optimal for scalar, finite-horizon multistage problems: J. Nascimento and W. B. Powell, (under review, Math of OR) J. Nascimento and W. B. Powell (in preparation) Results apply when we use pure exploitation do not assume points are sampled infinitely often Warren B. Powell Slide 102

103 Outline My experiences A resource allocation model ADP and the post-decision state variable Illustration using blood management Laboratory and theoretical results The dynamic energy resource model 2007 Warren B. Powell Slide 103

104 Energy resource modeling 2008 oil Rt New information 2009 D ˆ oil oil x t Rˆ oil t ˆ oil t ρ t New information oil oil R t + x 1 t + ˆ oil 1R ˆ oil D t + ˆ ρ oil t t + 1 R wind wind t x t ˆ wind Dˆ wind ˆ ρ wind wind wind Rt t t R t + x 1 t + 1R ˆ wind D ˆ wind t + ˆ ρ wind t t + 1 R corn corn t x t ˆ corn Dˆ corn ˆ ρ corn corn Rt t t R corn t + 1 x t + 1R ˆ corn D ˆ corn t + ˆ ρ corn t 1 t + 1 R coal coal t x t ˆ coal Dˆ coal ˆ ρ coal coal coal Rt t t R t + x 1 t + 1R ˆ coal D ˆ coal t + ˆ ρ coal t t Warren B. Powell Slide 104

105 Energy resource modeling We have to allocate resources before we know the demands for different types of energy in the future: 2007 Warren B. Powell Slide 105

106 Energy resource modeling We use value function approximations of the future to make decisions now: 2007 Warren B. Powell Slide 106

107 Energy resource modeling This determines how much capacity to provide: R R xn, t,1 xn, t,2 R R xn, t,3 xn, t,4 R xn, t, Warren B. Powell Slide 107

108 Energy resource modeling Marginal value: n vˆ ( ω ) t,1 R xn, t,1 n vˆ ( ω ) t,2 R xn, t,2 n vˆ ( ω ) t,3 R xn, t,3 n vˆ ( ω ) t,4 R xn, t,4 n vˆ ( ω ) t,5 R xn, t, Warren B. Powell Slide 108

109 Energy resource modeling Using the marginal values, we iteratively estimate piecewise linear functions. V ( R ) x t 1, AB+ t 1, AB+ xn, Rt 1, AB Warren B. Powell Slide 109

110 Energy resource modeling Using the marginal values, we iteratively estimate piecewise linear functions. V ( R ) x t 1, AB+ t 1, AB+ Left derivative k v t Right derivative k v + t xn, Rt 1, AB + R 1t 2007 Warren B. Powell Slide 110

111 Energy resource modeling Using the marginal values, we iteratively estimate piecewise linear functions. V ( R ) x t 1, AB+ t 1, AB+ ( k 1) v + t ( k 1) v + + t xn, Rt 1, AB + R 1t 2007 Warren B. Powell Slide 111

112 Two-stage stochastic programming Linear value function approximations: Linear (in the resource state): V ( R ) = v R t t tl tl l L 2007 Warren B. Powell Slide 112

113 Two-stage stochastic programming Piecewise linear, separable value function approximations: Piecewise linear, separable: Vt ( Rt ) = Vtl ( Rtl ) l L 2007 Warren B. Powell Slide 113

114 Research challenges Approximate dynamic programming:» At the heart of an ADP algorithm is the challenge of finding a value function approximation that works Can be used within commercial LP solvers Can be updated (estimated) easily Is stable Provides high quality solutions» Assessing solution quality Is it realistic? Do we seem to mimic markets and public policy? Is it robust? Do we achieve energy goals under different scenarios? 2007 Warren B. Powell Slide 114

115 Research challenges For the dynamic energy resource model, it is not enough to have a value function that depends purely on the resource vector.» The value of coal plants depends on our ability to sequester carbon.» We need to capture the state of the world in our value function approximations. Strategies: S t» Let be the full system state vector, capturing the cost of technologies, government policies, etc. etc.» Let φ f ( St), f F be a set of features that appear to be important explanatory variables. Identifying features is the art of ADP.» We can then fit value functions that depend on the features. Vt( Rt φ( St)) = Vta( Rta φ( St)) a A 2007 Warren B. Powell Slide 115

116 Research challenges Strategies for fitting Vt( Rt φ( St)) = Vta( Rta φ( St)) a A» Lookup-table Very general, but suffers from curse of dimensionality» Linear regression with low-dimensional polynomials Can work depends on the problem.» Kernel regression Powerful strategy that combines lookup-table with regression models. Use within ADP is surprisingly young. Variety of issues unique to ADP Warren B. Powell Slide 116

117 Research challenges Approximate dynamic programming:» How do we establish that we are getting good solutions? Demonstrate techniques on simpler problems. Compare against other methods for larger problems.» We need algorithms that are fast and stable. Identifying variance reduction methods from the simulation community that work on this problem class. Developing kernel regression techniques for improved fitting of the value function. Finding the best smoothing techniques for recursive updating. Parallel processing for accelerating simulations Warren B. Powell Slide 117

118 Research challenges System modeling» Modeling the evolution of technology using compact representations If we invest in technology, how do we describe the change process?» Modeling physical processes at multiple scales Wind, temperature, rainfall at different levels of discretization.» We need a software architecture that allows a larger community to participate in the modeling We need to tap into various types of domain expertise, such as climate modeling, transportation modeling, 2007 Warren B. Powell Slide 118

119 2007 Warren B. Powell Slide 119

120 Outline R&D for hydrogen fuel cells 2007 Warren B. Powell Slide 120

121 R&D optimization for hydrogen fuel cell We have been testing two methods for solving the hydrogen fuel cell R&D portfolio problem» Brute force Enumerate all decisions Use Monte Carlo sampling to estimate the value of a particular set of technologies Will not scale to large problems» Approximate dynamic programming Replace value function with linear approximation Determine portfolio by solving a knapsack problem using a solver. Scales to large problems, but how large is the error introduced by the linear value function approximation? 2007 Warren B. Powell Slide 121

122 R&D optimization for hydrogen fuel cell Test problems» Smaller dataset 12 projects Must choose 5 to research 792 combinations» Larger dataset 18 projects Must choose 5 to fund 8568 combinations» General First choose projects to research Learn results of research Choose the best technologies for the fuel cell, and evaluate the cost of the fuel cell Warren B. Powell Slide 122

123 R&D optimization for hydrogen fuel cell Elements of our hydrogen fuel cell problem 2007 Warren B. Powell Slide 123

124 R&D optimization for hydrogen fuel cell The optimization problem» Performance of fuel cell depends on parameters» Choose a subset of projects to perform additional research.» Parameters for the chosen project will change in a random way Warren B. Powell Slide 124

125 R&D optimization for hydrogen fuel cell Shape of the cost function 2007 Warren B. Powell Slide 125

126 Marginal value of each research project Top 5 are funded. Projects in common color compete Project 6 drops out of R&D portfolio; project 9 enters 2007 Warren B. Powell Slide 126

127 Results from 792 R&D portfolios 2007 Warren B. Powell Slide 127

128 Confidence interval for the value of the solution resulting from a particular R&D portfolio (five projects). Best estimate 2007 Warren B. Powell Slide 128

129 Optimal solution chosen by ADP Best overall solution (from brute force) 2007 Warren B. Powell Slide 129

130 Results from 8568 R&D portfolios Optimal solution chosen by ADP Best overall solution (from brute force) 2007 Warren B. Powell Slide 130