Inverse Optimization for the Recovery of Market Structure from Market Outcomes

Transcription

1 Inverse Optimization for the Recovery of Market Structure from Market Outcomes John Birge, University of Chicago Ali Hortaçsu, University of Chicago Michael Pavlin, Wilfrid Laurier University April 9, 2014

2 Themes Solutions of optimization models are often observed while the relevant parameters are not. Common examples include problems from network industries such as electricity and physical systems. When the data include constraint coecients, the inverse optimization to discover the parameters is only partially identied. Identication can be possible through norm minimization.

3 A Motivating Problem Economics of Two-Stage Electricity Markets? (Veit et al 2006; Sioshansi, Oren, O'Neill 2010; Botterud et al 2011) Market Power In Electricity Markets? (Cardell, Hitt, Hogan 1996; Jiang, Baldick 2005; Hogan 2012) Results rely on assumptions re participant objectives: Case Study on Wind Producer Objectives in Midwest ISO Two-stages: Day Ahead (DA), Real Time (RT) markets Forecast: Wind producers submit a DA production commitment Stochastic production: shortfall or surplus made up via RT prices Wind DA Revenues: Rev(Q DA ) =Q DA P DA + (Q RT Q DA ) P RT Economic value of intermittent generation depends on forecast quality (Gowrisankaran, Reynolds, Samano 2011; Skea, Anderson )

4 Quick Look at the Data Midwest ISO 2010 Data: Forward Premium E(P DA P RT ) > $2.00 But Under-commitment! Density (wind prod.) Why? Bad forecasting? Risk aversion? Exercise of Market Power? Prerequisite for answer: estimate E( P DA / Q DA ) and E( P RT / Q DA ) Hard with standard econometrics due to endogeneity Avg. Day Ahead Commitment

5 Research Problem: Ideal solution to endogeneity question: An accurate model of price determination process Zonal prices uniform price multi-unit auction (c.f. Reguant 2012; Hortaçsu, Puller 2008; Wolak 2004) However, Locational Marginal Pricing dominate North American Markets (e.g. PJM, Midwest ISO, CAISO, ERCOT) Prices depend on entire network structure Network structure not directly observable Critical Infrastructure Information Act (2002) However, Available information: Midwest ISO: prices, quantities, bids, active transmission constraints Can a useful model of the network be inferred from this information? (Useful for market researchers, participants, designers of CIIA)

6 Roadmap Locational Marginal Price (LMP) Market: Linear model The Estimation Problem... via Inverse Optimization An Algorithm: A sucient explanation Application to Data: Midwest ISO

7 Electricity Dispatch Model Relaxation of the unit commitment problem: Market Participant i {1..N} Produces x i MWh at an announced cost of c i /MWh Lossless network with links (i, j) Transmission between i and j of y (i,j) MWh Topology dened by matrices E, A, and D Network Flow Constraints Ey = x R Physical Constraints Ay = 0 L Transmission Constraints Dy d min c T x s.t. Ey = x Ay = 0 Dy d u x l y 0

8 Locational Marginal Prices Denition: 1. The Locational Marginal Price (LMP) is the immediate cost of supplying one additional MW of power at a particular node. 2. The LMP is the shadow price of the ow constraint min c T x s.t. Ey= x Ay = 0 Dy d u x l y 0 dual variables π σ ρ α, γ

9 LMP Example (Louie, Strunz 2008) C A :$20/MWh C B :$30/MWh Corresponding LP: 2/3 A 1/3 50 B min 20x A + 20x B + 20x C s.t. Ey = x 2y AB y AC = 0 y BC 50 A constraints D constraints C D C C :$25/MWh

10 LMP Example C A :$20 x A <500 C B :$30 x B <100 Corresponding LP: 2/3 A C 1/3 C C :$25 x C < B D x D =-300 min 20x A + 20x B + 20x C s.t. Ey = x 2y AB y AC = 0 y BC 50 0 x A x B x C x D 300 y 0 A constraints D constraints

11 LMP Example C A :20 x A :150 π A :20 C B :30 x B :0 π B :15 A C 1/3 y AB :50 2/3 50 y AC :100 y BC :50 y CD :300 B D Solution: x A = 150 π A = 20 x B = 0 π B = 15 x C = 150 π C = 25 x D = 300 π D = 25 y AB = 50 y AC = 100 y BC = 50 ρ BC = 15 y CD = 300 C C :25 x C :150 π C :25 x D :-300 π D :25

12 Estimation Problem (Single Sample) min c T x s.t. E y = x Ay = 0 Dy d u x l y 0 dual variables π σ ρ α, γ Given data: c, x, u, l, π, ρ Generate a network model: Ē, Ā, D, d Explaining shadow prices π and ρ

13 General Inverse Optimization Given: a partial specication of an optimization model a (partial) specication of an optimal solution Infer missing model parameters such that: Consistency: Known Parameters consistent with optimality Simplicity: Missing parameters minimize a norm Standard form Zhang, Liu 1996,1999 (linear); Ahuja, Orlin 2001 (general): Feasible set known Opt. solution known Cost parameters unknown Minimize L1 or L norm Extensions: Yang, Zhang 2007; Ahmed, Guan 2005; Iyengar, Kang 2005; Wang 2009; Ahuja, Orlin 1998,2002; Cui, Hochbaum 2010 min c T x s.t. Ey=x Ay= 0 Dy d dual variables π σ ρ

14 Our Inverse Optimization Problem Standard form: Feasible set known Opt. solution known Cost parameters unknown Minimize L1 norm Electricity Market Problem: Feasible set unknown Opt. solution partially known Cost parameters known Minimize L1 norm min c T x dual variables min c T x dual variables s.t. Ey=x π s.t. E y = x π Ay= 0 σ Ay = 0 σ Dy d ρ Dy d ρ u x l α, γ u x l α, γ y 0 y 0

15 Our Inverse Optimization Problem Find a simplest Ā, D, d,ē satisfying optimality conditions Minimize 1-Norm Regularize Ā s.t. σ r = 1 Resulting Optimization Problem: min Ā 1 + D 1 + d 1 D l(i,j) ρ (i,j) = π j π i (i, j) ȳ ij > 0 r<rā r(i,j) + l<l Ē ȳ = x Āȳ = 0 Dȳ d 2-Step Algorithm: 1. Find Ē and ȳ: 2. Find Ā, D, d:

16 Step 1: Determine Ē and ȳ Assume no loss Ē k(ij) { 1, 0, 1} limit to ows between sources and sinks Ē i(ij) = 1 only if x i > 0, Ē j(ij) = 1 only if x j < 0, Ē k(ij) = 0 otherwise Minimize requirements on Ā and D to satisfy Ā r(i,j) + D l(i,j) ρ (i,j) = π j π i ȳ ij > 0 r l By solving: min ij ȳ ij max{π i π j, 0} C A :20 x A :150 π A :20 C B :30 x B :0 π B :15 A C C C :25 x C :150 π C :25 B D x D :-300 π D :25 s.t. Ē ȳ = x ȳ ij 0

17 Step 1: Determine Ē and ȳ Assume no loss Ē k(ij) { 1, 0, 1} limit to paths between sources and sinks Ē i(ij) = 1 only if x i > 0, Ē j(ij) = 1 only if x j < 0, Ē k(ij) = 0 otherwise Minimize requirements on Ā and D to satisfy Ā r(i,j) + D l(i,j) ρ (i,j) = π j π i ȳ ij > 0 r l By solving: min y ij max{π i π j, 0} s.t. Ē ȳ = x ȳ ij 0 C A :20 x A :150 π A :20 C B :30 x B :0 π B :15 A C C C :25 x C :150 π C :25 Y AD :150 Y CD :150 B D x D :-300 π D :25

18 Step 2: Determine Ā, D, d Minimize sum of 1-norms subject to optimality constraints Solving: min Ā r(ij) + Dl(ij) + l r,(i,j) Ā r(i,j) + r l r,(i,j) s.t. Āȳ = 0 d l D l(i,j) ρ (i,j) = π j π i ȳ ij > 0 d l 0 C A :20 x A :150 π A :20 C B :30 x B :0 π B :15 A B Y AD :150 C Y CD :150 D C C :25 x C :150 π C :25 x D :-300 π D :25 A and D Constraints: 1 3 ȳad 50

19 Multiple Samples For set of samples 1..S 1. Calculate ȳ s independently 2. Add constraints for each sample min Ā r(ij) + Dl(ij) + d l r,(i,j) r,(i,j) l Ā r(i,j) + r l s.t. Āȳ s = 0 s D l(i,j) ρ s (i,j) = πs j π s i ȳ s ij > 0 d l 0 C A :20 x A :153 π A :20 C B :30 x B :1 π B :15 A C Y AB :1 Y CD :148 Y AD :152 B D C C :25 x C :150 π C :25 x D :-300 π D :25 A and D Constraints: 1 3 (ȳ AD 2ȳ AB ) 50

20 General Implementation Observations for each day and hour x, π, ρ, c vary E, D, A, d constant (approximately) Transmission and generation outages may change d Transmission losses not included Original optimization" may be adjusted for other reasons (e.g., frequency, reliability)

21 Algorithm Observations Extension with multiple observations Step 1 performed independently Step 2 single optimization adding all constraints Algorithm feasible if rows in A greater number of samples Polynomial approximation Step 2: LP standard transformation Each step presents O(n 2 ) variables

22 Results: Application to Midwest ISO 2010/01/01 00:00: Nodes 768 Aggregated Nodes 772 Active Links Transmission bounds per hour Imperfect data Naive implementation 20 ARows 40min Power Flow in Midwest ISO Network

23 Comparison Steps 1. Estimation: Perform estimation of constraints K using a set of identifying samples S K. 2. In-sample testing: (IS) Take bids and oers for s S K determine prices using optimization setup and the estimated constraints. 3. Out-of-sample testing: (OS) Take bids and oers for s / S K where active constraints are a subset of K and determine prices using optimization model. 4. Well-out-of-sample testing: (WOS) Take bids and oers for s / S K where active constraints are not a subset of K and determine prices using optimization model. 5. Assess quality: To test the model quality, we compare the accuracy to prices generated with two benchmark algorithms. The rst benchmark (NC) is a model where we assume that there is no congestion essentially performing a uniform price auction with the observed bids. The second benchmark (UC) produces prices by assuming prices remain unchanged from the rst sample to the other samples.

24 Test Examples ID Hour Active ID Hour Active Test1: In sample Test2: In sample IS :00: IS :00: IS :00: IS :00:00 680:1473 IS :00: Out of sample Out of sample OS :00: OS :00: OS :00: OS :00:00 680:1473 Test3: In sample All Tests: Well out of sample IS :00:00 100:1526:1530 WOS :00: IS :00:00 100:1526:1530 WOS :00:00 100:680:1530 IS :00:00 100:1526:1530 WOS :00:00 3:307:1257:1496:1504 IS :00:00 100:1526:1530 WOS :00:00 100:307:1329:1358:1473:1530 Out of sample OS :00:00 100:1530 OS :00:00 100:1526:1530 Table: Samples for each test case

25 Test: NC Comparison ID All locations and participants Outliers removed cor-coe std-r-err a-a-err mw a-a-err cor-coe std-r-err a-a-err mw a-a-err IS IS OS OS WOS WOS WOS WOS IS IS IS OS OS WOS WOS WOS WOS IS IS IS IS OS OS WOS WOS WOS WOS Table: Benchmark 1 NC: no transmission constraints

26 Price and MW Predictions ID All locations and participants Outliers removed cor-coef st-rg-err av-ab-err mw a-a-err cor-coef st-rg-err av-ab-err mw a-a-err IS IS OS OS WOS WOS WOS WOS IS IS IS OS OS WOS WOS WOS WOS IS IS IS IS OS OS WOS WOS WOS WOS Table: Price and MW predictions from estimated transmission model

27 LMP Predictions (a) In Sample (b) Out of Sample Figure: Predicted vs. True LMPs, estimation using Test 2 with two active constraint

28 Additional Examples Macro-economic models Prices observed in dierent regions Purchase quantities observed Equilibrium model set up as potential optimization Unknown transportation routes and costs to discover Supply chain interactions Prices and quantities observed Unobserved relationships between suppliers and customers Discover relationships and transactions

29 Future Directions Discussed modelling price determination process in LMP based Electricity Markets Next Steps Inverse optimization based formulation/algorithm consistent with dual interpretation of LMPs Predicting market characteristics: price response, congestion costs... Structural estimation: model participant decision making Solution quality: solution robustness to data imperfections Econometrics of general competitive markets: extend to linear market models

30 Conclusions Inverse optimization to discover constraints Needed to determine objective of market participants Many markets include price and quantity observations but not constraints Diculty from bilinear form with constraint and unknown variable values Solution method Results Two-step process Determine consistent primal variables rst Choose constraint coecients with minimum 1-norm Possible to discover simple network congurations Reasonable results with multiple data observations Possible inconsistencies from unknown parameter changes