Distributed demand control in power grids

Transcription

1 Distributed demand control in power grids and ODEs for Markov decision processes Second Conference on the Mathematics of Energy Markets Wolfgang Pauli Institute, Vienna, 4-9 July 217 Ana Bušić Inria Paris Département d Informatique de l ENS Joint work with Prabir Barooah, Yue Chen, and Sean Meyn University of Florida

2 Challenges Challenges of renewable power generation Impact of wind and solar on net-load at CAISO $/MWh Price spike due to high net-load ramping need when solar production ramped out Negative prices due to high mid-day solar production GW GW 27 Load and Net-load Toal Load 4 Renewable Generation 2 Total Wind Total Solar Ramp limitations cause price-spikes Peak ramp Peak 24 hrs Net-load: Toal Load, less Wind and Solar Peak ramp Peak 24 hrs

3 Challenges Challenges of renewable power generation Impact of wind and solar on net-load at CAISO $/MWh Price spike due to high net-load ramping need when solar production ramped out Negative prices due to high mid-day solar production GW GW 27 Load and Net-load Toal Load 4 Renewable Generation 2 Total Wind Total Solar Ramp limitations cause price-spikes Peak ramp Peak 24 hrs Net-load: Toal Load, less Wind and Solar Peak ramp Peak 24 hrs 4 GW (t) = Wind generation in BPA, Jan GW 2 Ramps 1 Jan 1 Jan 2 Jan 3 Jan 4 Jan 5 Jan 6

4 Challenges Challenges of renewable power generation Balancing control loop wind and solar volatility seen as disturbance grid level measurements: scalar function of time (ACE) a linear combination of frequency deviation and the tie-line error (power missmatch between the sceduled and actual power out of the balancing region) compensation G c designed by a balancing authority In many cases control loops are based on standard PI (proportional-integral) control design. Compensation Actuation Disturbances GRID U(t) Y (t) G c H + G p = H a + H b + Measurements P delivered

5 Challenges Challenges of renewable power generation Increasing needs for ancillary services Reference (from Balancing Authority) Σ Balancing Authority Ancillary Services Grid Voltage Frequency Phase t/hour In the past, provided by the generators - high costs!

6 Challenges Tracking Grid Signal with Residential Loads Tracking objective: Reference (from Balancing Authority) Σ Balancing Authority Ancillary Services Grid Voltage Frequency Phase t/hour Prior work Deterministic centralized control: Sanandaji et al. 214 [HICSS], Biegel et al. 213 [IEEE TSG] Randomized control: Mathieu, Koch, Callaway 213 [IEEE TPS] (decisions at the BA) Meyn, Barooah, B., Chen, Ehren 215 [IEEE TAC] (local decisions, restricted load models)

7 Challenges Tracking Grid Signal with Residential Loads Example: 2 pools, 2 kw max load Each pool consumes 1kW when operating 12 hour cleaning cycle each 24 hours Power Deviation: 1 Output deviation Reference 2 pools kw t/hour Input ζt Nearly Perfect Service from Pools Meyn, Barooah, B., Chen, Ehren 215 [IEEE TAC] using an extension/reinterpretation of Todorov 27 [NIPS] (linearly solvable MDPs)

8 Challenges Tracking Grid Signal with Residential Loads Example: 3, pools, 3 MW max load Each pool consumes 1kW when operating 12 hour cleaning cycle each 24 hours Power Deviation: 1 Output deviation Reference 3, pools 5 MW t/hour 3-3 Input ζt Nearly Perfect Service from Pools Meyn, Barooah, B., Chen, Ehren 215 [IEEE TAC] using an extension/reinterpretation of Todorov 27 [NIPS] (linearly solvable MDPs) What About Other Loads?

9 Demand Dispatch Control Goals and Architecture Macro control High-level control layer: BA or a load aggregator. The balancing challenges are of many different categories and time-scales: Automatic Generation Control (AGC); time scales of seconds to 2 minutes. Balancing reserves. In the Bonneville Power Authority, the balancing reserves include both AGC and balancing on timescales of many hours. Balancing on a slower time-scale is achieved through real time markets in some other regions of the U.S. Contingencies (e.g., a generator outage) Peak shaving Smoothing ramps from solar or wind generation

10 Demand Dispatch Control Goals and Architecture Local Control: decision rules designed to respect needs of load and grid Demand Dispatch: Power consumption from loads varies automatically to provide service to the grid, without impacting QoS to the consumer Σ Control C Gas Turbine BP Coal Batteries BP Water Pump BP LOAD BP BP CA Actuator feedback loop Power Grid H Voltage Frequency Phase Grid signal ζ t Local Control Local decision U i t Load i Power deviation Y i t X i t Local feedback loop Min. communication: each load monitors its state and a regulation signal from the grid. Aggregate must be controllable: randomized policies for finite-state loads.

11 Mean Field Model Load Model Controlled Markovian Dynamics Reference (MW) r BA G c ζ Load 1 Load Power Consumption (MW) Load N Discrete time: ith load X i (t) evolves on finite state space X Each load is subject to common controlled Markovian dynamics. Signal ζ = {ζ t } is broadcast to all loads Controlled transition matrix {P ζ : ζ R}: P{Xt+1 i = x Xt i = x, ζ t = ζ} = P ζ (x, x ) Questions How to analyze aggregate of similar loads? Local control design?

12 Aggregate model

13 Mean Field Model How to analyze aggregate? Mean field model N loads running independently, each under the command ζ. Empirical Distributions: µ N t (x) = 1 N N I{X i (t) = x}, i=1 x X U(x) power consumption in state x, y N t = 1 N N µ N t (x)u(x) i=1 U(X i t) = x Mean-field model: via Law of Large Numbers for martingales µ t+1 = µ t P ζt, y t = µ t, U ζ t = f t (y,..., y t ) by design

14

15 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} Myopic Design: P myop ζ (x, x ) := P (x, x ) exp ( ζu(x ) Λ ζ (x) ) with Λ ζ (x) := log( x P (x, x ) exp ( ζu(x ) )) the normalizing constant.

16 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} Myopic Design: P myop ζ (x, x ) := P (x, x ) exp ( ζu(x ) Λ ζ (x) ) with Λ ζ (x) := log( x P (x, x ) exp ( ζu(x ) )) the normalizing constant. Exponential family design: P ζ (x, x ) := P (x, x ) exp ( h ζ (x, x ) Λ hζ (x) ) with h ζ (x, x ) = ζh (x, x ). The choice of H will typically correspond to the linearization of a more advanced design around the value ζ = (or some other fixed value of ζ).

17 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} Individual Perspective Design Consider a finite-time-horizon optimization problem: For a given terminal time T, let p denote the pmf on strings of length T, p (x 1,..., x T ) = T 1 i= P (x i, x i+1 ), where x X is assumed to be given. The scalar ζ R is interpreted as a weighting parameter in the following definition of total welfare. For any pmf p, W T (p) = ζe p [ T t=1 ] U(X t ) D(p p ) where the expectation is with respect to p, and D denotes relative entropy: D(p p ) := x 1,...,x T log ( p(x1,..., x T ) ) p(x 1,..., x T ) p (x 1,..., x T )

18 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} It is easy to check that the myopic design is an optimizer for the horizon T = 1, P myop ζ (x, ) arg max W 1 (p). The infinite-horizon mean welfare is denoted, ηζ 1 = lim T T W T (p T ) Explicit construction via eigenvector problem: P ζ (x, y) = 1 v(y) λ v(x) ˆP ζ (x, y), x, y X, where ˆPζ v = λv, ˆPζ (x, y) = exp(ζu(x))p (x, y) p Extension/reinterpretation of [Todorov 27] + [Kontoyiannis & Meyn 2X]

19 Local Control Design Example: pool pumps How Pools Can Help Regulate The Grid 1,5KW 4V Needs of a single pool. Filtration system circulates and cleans: Average pool pump uses 1.3kW and runs 6-12 hours per day, 7 days per week. Pool owners are oblivious, until they see frogs and algae. Pool owners do not trust anyone: Privacy is a big concern Single pool dynamics: X = {(m, j) : m {, 1}, j {1, 2,..., I}}. On Off I 1 I I I 1

20 Tracking Grid Signal with Residential Loads Example: 2 pools, 2 kw max load Each pool consumes 1kW when operating 12 hour cleaning cycle each 24 hours Power Deviation: 1 Output deviation Reference 2 pools kw t/hour Input ζt Nearly Perfect Service from Pools Meyn et al. 213 [CDC], Meyn et al. 215 [IEEE TAC]

21 Tracking Grid Signal with Residential Loads Example: 3, pools, 3 MW max load Each pool consumes 1kW when operating 12 hour cleaning cycle each 24 hours Power Deviation: 1 Output deviation Reference 3, pools 5 MW t/hour 3-3 Input ζt Nearly Perfect Service from Pools Meyn et al. 213 [CDC], Meyn et al. 215 [IEEE TAC]

22 Range of services provided by pools Example: 1, pools, 1 MW max load 2 12 hr/day cycle Power Deviation Reference MW t/hour ζ

23 Local Design Extending local control design to include exogenous disturbances State space for a load model: X = X u X n. Components X n are not subject to direct control (e.g. impact of the weather on the climate of a building).

24 Local Design Extending local control design to include exogenous disturbances State space for a load model: X = X u X n. Components X n are not subject to direct control (e.g. impact of the weather on the climate of a building). Conditional-independence structure of the local transition matrix P (x, x ) = R(x, x u)q (x, x n), x = (x u, x n) Q models uncontroled load dynamics and exogenous disturbances.

25 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} Nominal model A Markovian model for an individual load, based on its typical behavior. Finite state space X = {x 1,..., x d }; Transition matrix P, with unique invariant pmf π.

26 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} Nominal model A Markovian model for an individual load, based on its typical behavior. Finite state space X = {x 1,..., x d }; Transition matrix P, with unique invariant pmf π. Common structure for design The family of transition matrices used for distributed control is of the form: P ζ (x, x ) := P (x, x ) exp ( h ζ (x, x ) Λ hζ (x) ) with h ζ continuously differentiable in ζ, and the normalizing constant ( Λ hζ (x) := log P (x, x ( ) exp hζ (x, x ) )) x

27 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} Nominal model A Markovian model for an individual load, based on its typical behavior. Finite state space X = {x 1,..., x d }; Transition matrix P, with unique invariant pmf π. Common structure for design The family of transition matrices used for distributed control is of the form: P ζ (x, x ) := P (x, x ) exp ( h ζ (x, x ) Λ hζ (x) ) with h ζ continuously differentiable in ζ, and the normalizing constant ( Λ hζ (x) := log P (x, x ( ) exp hζ (x, x ) )) x Assumption: for all x X, x = (x u, x n) X, h ζ (x, x ) = h ζ (x, x u).

28 Local Design Goal: Construct a family of transition matrices {P ζ : ζ R} Construction of the family of functions {h ζ : ζ R} Step 1: The specification of a function H that takes as input a transition matrix. H = H(P ) is a real-valued function on X X. Step 2: The families {P ζ } and {h ζ } are defined by the solution to the ODE: d dζ h ζ = H(P ζ ), ζ R, in which P ζ is determined by h ζ through: The boundary condition: h. P ζ (x, x ) := P (x, x ) exp ( h ζ (x, x ) Λ hζ (x) )

29 Local Design Extending local control design to include exogenous disturbances For any function H : X R, one can define Then functions {h ζ } satisfy H(x, x u) = x n h ζ (x, x u) = x n Q (x, x n)h (x u, x n) (1) Q (x, x n)h ζ(x u, x n), for some h ζ : X R. Moreover, these functions solve the d-dimensional ODE, with boundary condition h. d dζ h ζ = H (P ζ ), ζ R,

30 Individual Perspective Design Local welfare function: W ζ (x, P ) = ζu(x) D(P P ), where D denotes relative entropy: D(P P ) = x P (x, x ) log ( P (x,x ) P (x,x ) ). 1 T Markov Decision Process: lim sup T T t=1 E[W ζ(x t, P )] Average reward optimization equation (AROE): max P { W ζ (x, P ) + } P (x, x )h ζ(x ) = h ζ(x) + ηζ x where P (x, x ) = R(x, x u)q (x, x n), x = (x u, x n)

31 Individual Perspective Design ODE method for IPD design: Family {P ζ }: P ζ (x, x ) := P (x, x ) exp ( h ζ (x, x ) Λ hζ (x) ) Functions {h ζ }: h ζ (x, x u) = x Q (x, x n)h n ζ (x u, x n), for h ζ : X R solutions of the d-dimensional ODE, with boundary condition h. d dζ h ζ = H (P ζ ), ζ R, H ζ (x) = d dζ h ζ (x) = x [Z ζ(x, x ) Z ζ (x, x )]U(x ), x X, where Z = [I P + 1 π] 1 = n= [P ζ 1 π] n is the fundamental matrix.

32 Example: Thermostatically Controlled Loads refrigerators, water heaters, air-conditioning... TCLs are already equipped with primitive local intelligence based on a deadband (or hysteresis interval) The state process for a TCL at time t: X(t) = (X u (t), X n (t)) = (m(t), Θ(t)), where m(t) {, 1} denotes the power mode ( 1 indicating the unit is on), and Θ(t) the inside temperature of the load Exogenous disturbances: ambient temperature, and usage

33 Example: Thermostatically Controlled Loads The standard ODE model of a water heater is the first-order linear system, d dt Θ(t) = λ[θ(t) Θa (t)] + γm(t) α[θ(t) Θ in (t)]f(t), Θ(t) temperature of the water in the tank Θ in (t) temperature of the cold water entering the tank f(t) flow rate of hot water from the WH m(t) power mode of the WH ( on indicated by m(t) = 1). Deterministic deadband control: Θ(t) [Θ, Θ + ] Nominal model for local control design: based on the specification of two CDFs for the temperature at which the load turns on or turns off 1 F (θ) F (θ) 1 ϱ Θ θ Θ + Θ θ Θ + θ

34 Example: Thermostatically Controlled Loads Discrete-time control. At time instance k, if the water heater is on (i.e., m(k) = 1), then it turns off with probability, p (k + 1) = [F (Θ(k + 1)) F (Θ(k))] + 1 F (Θ(k)) where [x] + := max(, x) for x R; Similarly, if the load is off, then it turns on with probability p (k + 1) = [F (Θ(k)) F (Θ(k + 1))] + F (Θ(k)) The nominal behavior of the power mode can be expressed P{m(k) = 1 θ(k 1), θ(k), m(k 1) = } = p (k) P{m(k) = θ(k 1), θ(k), m(k 1) = 1} = p (k)

35 Example: Thermostatically Controlled Loads Myopic design - exponential tilting of these distributions: p ζ p ζ (k) := P{m(k) = 1 θ(k 1), θ(k), m(k 1) =, ζ(k 1) = ζ} = p (k)e ζ p (k)e ζ + 1 p (k) (k) = P{m(k) = θ(k 1), θ(k), m(k 1) = 1, ζ(k 1) = ζ} = p (k) p (k) + (1 p (k))e ζ If p (k) >, then the probability p ζ (k) is strictly increasing in ζ, approaching 1 as ζ ; it approaches as ζ, if p (k) < 1.

36 Example: Thermostatically Controlled Loads System identification d dt Θ(t) = λ[θ(t) Θa (t)] + γm(t) α[θ(t) Θ in (t)]f(t), Θ(t) temperature of the water in the tank Θ in (t) temperature of the cold water entering the tank f(t) flow rate of hot water from the WH m(t) power mode of the WH ( on indicated by m(t) = 1). Temp. Ranges ODE Pars. Loc. Control Θ + [118, 122] F λ [8, 12.5] 1 6 T s = 15 sec Θ [18, 112] F γ [2.6, 2.8] 1 2 κ = 4 Θ a [68, 72] F α [6.5, 6.7] 1 2 ϱ =.8 Θ in [68, 72] F P on = 4.5 kw θ = Θ Heterogeneous population: 1 WHs simulated by uniform sampling of the values in the table Usage data from Oakridge National Laboratory (35WHs over 5 days)

37 Local Control Design Tracking performance and the controlled dynamics for an individual load Tracking Load On Tracking temp (F) MW 14 Load On BPA Reference: rt Power Deviation 1 temp (F) 8 5 rt 4 MW Typical Load Response 12 1 MW rt 1 MW Load On Nominal power consumption temp (F) 1 MW No reg: rt 1, water-heaters When on, individual load consumes 4, 5 kw With no usage, approx. 2% duty cycle, avg. power consumption 1MW t (hrs) t (hrs)

38 Tracking performance Potential for contingency reserves and ramping Power deviation (MW) Tracking two sawtooth waves with 1, water heaters: average power consumption 8MW Power Deviation ζ ζ Reference t (hrs) ζ

39 Local Control Design Tracking performance and the controlled dynamics for an individual load Heterogeneous setting: 4 loads per experiment; 2 different load types in each case Lower plots show the on/off state for a typical load Open Loop Tracking (MW) Refrigerators 6 15 Fast Electric Water Heaters Slow Electric Water Heaters BPA balancing reserves (filtered/scaled) Power state Mean-field Model Stochastic Output 1 6 hrs 24 hrs Nominal Demand Dispatch 24 hrs

40 Unmodeled dynamics Setting:.1% sampling, and 1 Heterogeneous population of loads 2 Load i overrides when QoS is out of bounds Output deviation Reference 1 N = 3, 1 N = 3, 5 5 MW t/hour Closed-loop tracking t/hour 13 PI control: ζ t = k P e t + k I e I t, et = rt yt, ei t = t s= es.5 opt out %

41 Conclusions and Future Directions Control Architecture Frequency Allocation for Demand Dispatch 2 15 Phase (deg) Magnitude (db) Water Pumping Pool Pumps Chiller Tanks Residential Water Heaters Refrigerators Fans in Commercial Buildings Uncertainty Here Grid Transfer Function Frequency (rad/s) Frequency (rad/s) 1, pools Output deviation Tracking BPA Regulation Signal (MW) Reference (from Bonneville Power Authority) t/hour Bandwidth centered around its natural cycle

42 Conclusions and Future Directions Conclusions Virtual storage from flexible loads Approach: creating Virtual Energy Storage through direct control of flexible loads - helping the grid while respecting user QoS

43 Conclusions and Future Directions Conclusions Virtual storage from flexible loads Approach: creating Virtual Energy Storage through direct control of flexible loads - helping the grid while respecting user QoS Challenges: Stability properties for IPD and myopic design? Information Architecture: ζ t = f(?) Different needs for communication, state estimation and forecast. Capacity estimation (time varying) Network constraints Resource optimization & learning Integrating VES with traditional generation and batteries. Economic issues Contract design, aggregators, markets...

44 Conclusions and Future Directions Conclusions Thank You!

45 Conclusions and Future Directions References: this talk A. Bušić and S. Meyn. Distributed randomized control for demand dispatch. 55th IEEE Conference on Decision and Control, 216. A. Bušić and S. Meyn. Ordinary Differential Equation Methods For Markov Decision Processes and Application to Kullback-Leibler Control Cost. arxiv: v2. Oct 216. S. Meyn, P. Barooah, A. Bušić, Y. Chen, and J. Ehren. Ancillary Service to the Grid Using Intelligent Deferrable Loads. IEEE Trans. Automat. Contr., 6(11): , 215. P. Barooah, A. Bušić, and S. Meyn. Spectral Decomposition of Demand-Side Flexibility for Reliable Ancillary Services in a Smart Grid. 48th Annual Hawaii International Conference on System Sciences (HICSS) A. Bušić and S. Meyn. Passive dynamics in mean field control. 53rd IEEE Conf. on Decision and Control (CDC) 214.

46 Conclusions and Future Directions References: related Demand dispatch: Y. Chen, A. Bušić, and S. Meyn. Individual risk in mean-field control models for decentralized control, with application to automated demand response. 53rd IEEE Conf. on Decision and Control (CDC), 214. Y. Chen, A. Bušić, and S. Meyn. State Estimation and Mean Field Control with Application to Demand Dispatch. 54rd IEEE Conference on Decision and Control (CDC) 215. J. L. Mathieu. Modeling, Analysis, and Control of Demand Response Resources. PhD thesis, Berkeley, 212. J. L. Mathieu, S. Koch, D. S. Callaway, State Estimation and Control of Electric Loads to Manage Real-Time Energy Imbalance, IEEE Transactions on Power Systems, 28(1):43-44, 213. Markov processes: I. Kontoyiannis and S. P. Meyn. Spectral theory and limit theorems for geometrically ergodic Markov processes. Ann. Appl. Probab., 13:34 362, 23. I. Kontoyiannis and S. P. Meyn. Large deviations asymptotics and the spectral theory of multiplicatively regular Markov processes. Electron. J. Probab., 1(3): (electronic), 25. E. Todorov. Linearly-solvable Markov decision problems. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, (19) MIT Press, Cambridge, MA, 27.

47 Conclusions and Future Directions Mean Field Model Linearized Dynamics Mean-field model: µ t+1 = µ t P ζt, y t = µ t, U ζ t = f t (y,..., y t ) Linear state space model: Φ t+1 = AΦ t + Bζ t γ t = CΦ t Interpretations: ζ t is small, and π denotes invariant measure for P. Φ t R X, a column vector with Φ t (x) µ t (x) π(x), x X γ t y t y ; deviation from nominal steady-state A = P T, C = U T, and input dynamics linearized: B T = d dζ πp ζ ζ=