Foresighted Demand Side Management

Transcription

1 Foresghted Demand Sde Management 1 Yuanzhang Xao and Mhaela van der Schaar, Fellow, IEEE Department of Electrcal Engneerng, UCLA. {yxao,mhaela}@ee.ucla.edu. Abstract arxv: v1 [cs.ma] 9 Jan 2014 We consder a smart grd wth an ndependent system operator (ISO), and dstrbuted aggregators who have energy storage and purchase energy from the ISO to serve ts customers. All the enttes n the system are foresghted: each aggregator seeks to mnmze ts own long-term payments for energy purchase and operatonal costs of energy storage by decdng how much energy to buy from the ISO, and the ISO seeks to mnmze the long-term total cost of the system (e.g. energy generaton costs and the aggregators costs) by dspatchng the energy producton among the generators. The decson makng of the foresghted enttes s complcated for two reasons. Frst, the nformaton s decentralzed among the enttes: the ISO does not know the aggregators states (.e. ther energy consumpton requests from customers and the amount of energy n ther storage), and each aggregator does not know the other aggregators states or the ISO s state (.e. the energy generaton costs and the status of the transmsson lnes). Second, the couplng among the aggregators s unknown to them due to ther lmted nformaton. Specfcally, each aggregator s energy purchase affects the prce, and hence the payments of the other aggregators. However, none of them knows how ts decson nfluences the prce because the prce s determned by the ISO based on ts state. We propose a desgn framework n whch the ISO provdes each aggregator wth a conjectured future prce, and each aggregator dstrbutvely mnmzes ts own long-term cost based on ts conjectured prce as well as ts locally-avalable nformaton. The proposed framework can acheve the socal optmum despte beng decentralzed and nvolvng complex couplng among the varous enttes nteractng n the system. Smulaton results show that the proposed foresghted demand sde management acheves sgnfcant reducton n the total cost, compared to the optmal myopc demand sde management (up to 60% reducton), and the foresghted demand sde management based on the Lyapunov optmzaton framework (up to 30% reducton). I. INTRODUCTION The power systems are undergong drastc changes on both the supply sde and the demand sde. On the supply sde, an ncreasng amount of renewable energy (e.g. wnd energy, solar energy) s penetratng the power systems. The adopton of renewable energy reduces the envronmental damage caused by conventonal energy generaton, but also ntroduces hgh fluctuaton

2 2 and uncertanty n energy generaton on the supply sde. To cope wth ths uncertanty n energy generaton, the demand sde s deployng varous solutons, one of whch s the use of energy storage [6]. In ths paper, we study the optmal demand sde management (DSM) strategy n the presence of energy storage, and the correspondng optmal economc dspatch strategy. Specfcally, we consder a power system consstng of energy generators on the supply sde, an ndependent system operator (ISO) that operates the system, and multple aggregators and ther customers on the demand sde. On the supply sde, the ISO receves energy purchase requests from the aggregators as well as reports of (parameterzed) energy generaton cost functons from the generators, and based on these, dspatches the energy generators and determnes the unt energy prces. On the demand sde, the aggregators are located n dfferent geographcal areas and provde energy for resdental customers (e.g. households) or for commercal customers (e.g. an offce buldng) n the neghborhood. In the lterature, the term DSM has been broadly used for dfferent decson problems on the demand sde. For example, some papers (see [1] [4] for representatve papers) focus on the nteracton between one aggregator and ts customers, and refer to DSM as determnng the power consumpton schedules of the users. Some papers [5] [12] focus on how multple aggregators [5] [8] or a sngle aggregator [9] [12] purchases energy from the ISO based on the energy consumpton requests from ther customers. Our paper pertans to the second category of research works. The key feature that sets apart our paper from most exstng works [5] [8] s that all the decson makers n the system are foresghted. Each aggregator seeks to mnmze ts long-term cost, consstng of ts operatonal cost of energy storage and ts payment for energy purchase. In contrast, n most exstng works [5] [8], the aggregators are myopc and seek to mnmzng ther short-term (e.g. one-day or even hourly) cost. In the presence of energy storage, foresghted DSM strateges can acheve much lower costs than myopc DSM strateges because the current decsons of the aggregators wll affect ther future costs. For example, an aggregator can purchase more energy from the ISO than that requested from ts customers, and store the unused energy n the energy storage for future use, f t antcpates that the future energy prce wll be hgh. Hence, the current purchase from the aggregators wll affect how much they wll purchase n the future. In ths case, t s optmal for the enttes to make foresghted decsons, takng nto account the mpact of ther current decsons on the future. Snce the aggregators deploy foresghted DSM strateges, t s also optmal for the ISO to make foresghted economc dspatch,

3 3 n order to mnmze the long-term total cost of the system, consstng of the long-term cost of energy generaton and the aggregators long-term operatonal cost. Note that although some works [9] [12] assume that the aggregator s foresghted, they study the decson problem of a sngle aggregator and do not consder the economc dspatch problem of the ISO. When there are multple aggregators n the system (whch s the case n practce), ths approach neglects the mpact of aggregators decsons on each other, whch leads to suboptmal solutons n terms of mnmzng the total cost of the system. When the ISO and multple aggregators make foresghted decsons, t s dffcult to obtan the optmal foresghted strateges for two reasons. Frst, the nformaton s decentralzed. The total cost depends on the generaton cost functons (e.g. the speed of wnd for wnd energy generaton, the amount of sunshne for solar energy generaton, and so on), the status of the transmsson lnes (e.g. the flow capacty of the transmsson lnes), the amount of electrcty n the energy storage, and the demand from the customers, all of whch may change due to supply and demand uncertanty. However, none of the enttes knows all the above nformaton: the ISO knows only the generaton cost functons and the status of the transmsson lnes, and each aggregator knows only the status of ts own energy storage and the demand from ts own customers. Hence, the DSM strategy needs to be decentralzed, such that each entty can make decsons solely based on ts locally-avalable nformaton. Second, the aggregators are coupled n a complcated way that s unknown to them. Specfcally, each aggregator s purchase affects the prces, and thus the payments of the other aggregators. However, the prce s determned by the ISO based on the generaton cost functons and the status of the transmsson lnes, nether of whch s known to any aggregator. Hence, each aggregator does not know how ts purchase wll nfluence the prce, whch makes t dffcult for the aggregator to make the optmal decson. To overcome the dffculty resultng from nformaton decentralzaton and complcated couplng, we propose a decentralzed DSM strategy based on conjectured prces. Specfcally, each aggregator makes decsons based on ts conjectured prce, and ts local nformaton on the status of ts energy storage and the demand from ts customers. In other words, each aggregator summarzes all the unavalable nformaton nto ts conjectured prce. Note, however, that the prce s determned based on the generaton cost functons and the status of the transmsson lnes, whch s only known to the ISO. Hence, the aggregators conjectured prces are determned by the ISO. We propose a smple onlne algorthm for the ISO to update the conjectured prces

4 4 TABLE I COMPARISONS WITH RELATED WORKS ON DEMAND-SIDE MANAGEMENT. Energy storage Tme horzon Foresghted Aggregators Supply uncertanty Demand Uncertanty [1][2][5] No 1 day No Multple No No [3] No 1 day No Multple Yes No [4] No 1 day No Multple No Yes [6] Yes 1 day No Multple No No [7][8] Yes 1 day No Multple Yes Yes [9][10] Yes Infnte Yes Sngle No Yes [11][12] Yes Infnte Yes Sngle Yes Yes Proposed Yes Infnte Yes Multple Yes Yes based on ts local nformaton, and prove that by usng the algorthm, the ISO obtans the optmal conjectured prces under whch the aggregators (foresghted) best responses mnmze the total cost of the system. A. Related Works on Demand-Sde Management The early works [1] [5] on demand sde management focused on the power consumpton schedulng of the customers n one day. They consdered a power system wth no demand uncertanty (e.g. fxed, nstead of stochastc, demand) and no supply uncertanty (e.g. no renewable energy generaton). In ths smplfed model, they formulated the cost mnmzaton problem [1] or the utlty maxmzaton problem [2][5] as a convex optmzaton problem, and proposed dstrbuted algorthms to acheve the optmal power consumpton schedulng. However, wth the penetraton of renewable energy, there s a hgh degree of uncertanty n the supply. In addton, the demand cannot be the same for every day. Hence, we need to buld an approprate model for the power system that takes nto account the uncertanty n the supply and demand. Some works [3] consdered such a model wth demand and supply uncertantes, but stll focused on the decson problem n one day. Snce the users are myopc and optmze ts one-day utlty, the problem s formulated as a fnte-horzon Markov decson problem (MDP) and a greedy algorthm s proposed to acheve the optmal soluton [3]. Nevertheless, the above works neglected an mportant trend n the future power system: the ncreasng adopton of energy storage on the demand sde to cope wth the uncertan energy

5 5 TABLE II COMPARISONS WITH RELATED MATHEMATICAL FRAMEWORKS. MDP MU-MDP Lyapunov Optmzaton Stochastc Control Stochastc Games [13][14] [9] [12] [15] [17] Ths work Number of decson makers Sngle Multple Sngle Multple Multple Multple Decentralzed nformaton N/A Yes N/A Yes No Yes Couplng among users N/A Weak N/A Strong Strong Strong Optmal Yes Yes Yes No Yes Yes Constructve Yes Yes Yes Yes No Yes supply caused by renewable energy generaton. Some works [6] [8] propose myopc DSM strateges to mnmze the short-term (e.g. one-day or hourly) cost n the presence of energy storage. However, wth energy storage, the strateges that mnmze the long-term cost can greatly outperform the myopc strateges that mnmze the short-term cost. For example, n a myopc strategy, the aggregator may tend to purchase as lttle power as possble as long as the demand s fulflled, n order to mnmze the current operatonal cost of ts energy storage. However, the optmal polcy should take nto consderaton the future prce, and balance the trade-off between the current operatonal cost and the future savng n energy purchase. Some works [9] [12] propose the optmal foresghted DSM strategy n the presence of the energy storage under a sngle-aggregator model. In practce, the power system has many aggregators makng decsons that affect each other s cost. In the practcal system wth many aggregators, t s neffcent for each aggregator to smply adopt the optmal foresghted strategy desgned for a sngle-aggregator model. As we wll show n the smulaton, n the multple-aggregator scenaro, the total cost acheved by such a smple adaptaton of the optmal sngle-aggregator strategy s much hgher (up to 30%) than the proposed optmal soluton. Ths s because the sngle-aggregator strategy ams at achevng ndvdual mnmum cost, nstead of the total cost. Due to the couplng among the aggregators, the outcome n whch ndvdual costs are mnmzed may be very dfferent from the outcome n whch the total cost s mnmzed. In Table I, we summarze the above dscussons on the exstng works n power systems by comparng them n varous aspects. We wll provde a more techncal comparson wth the exstng works n Table VI, after we descrbed the proposed framework.

6 6 B. Related Theoretcal Frameworks Decson makng n a dynamcally changng envronment has been studed and formulated as Markov decson processes (MDPs). Most MDPs solve for sngle-user decson problems. There have been few works [13][14] on mult-user MDPs (MU-MDPs). The works on MU-MDPs [13][14] focus on weakly coupled MU-MDPs, where the term weakly coupled s coned by [13] to denote the assumpton that one user s acton does not drectly affect the others current payoffs. The users are coupled only through some lnkng constrants on ther actons (for example, the sum of ther actons, say the sum data rate, should not exceed some threshold, say the avalable bandwdth). However, once a user chooses ts own acton, ts current payoff and ts state transton are determned and do not depend on the other users actons. In contrast, n ths work, the users are strongly coupled, namely one user s acton drectly affect the others current payoffs. For example, one aggregator s energy purchase affects the unt prce of energy, whch has mpact on the other aggregators payments. There are few works n stochastc control that model the users nteracton as strongly coupled [15]. However, the man focus of [15] s to prove the exstence of a Nash equlbrum (NE) strategy. There s no performance analyss/guarantee of the proposed NE strategy. The nteracton among users wth strong couplng s modeled n the game theory lterature as a stochastc game [17] n game theory lterature. However, n standard stochastc games, the state of the system s known to all the players. Hence, we cannot model the nteracton of enttes n our work as a stochastc game, because dfferent enttes have dfferent prvate states unknown to the others. In addton, the results n [17] are not constructve. They focus on what payoff profles are achevable, but cannot show how to acheve those payoff profles. In contrast, we propose an algorthm to compute the optmal strategy profle. In Table II, we compare our work wth exstng theoretcal frameworks. Note that we wll provde a more techncal comparson wth the Lyapunov optmzaton and MU-MDP frameworks n Table VI, after we descrbed the proposed framework. The rest of the paper s organzed as follows. We ntroduce the system model n Secton II, and then formulate the desgn problem n Secton III. We descrbe the proposed optmal decentralzed DSM strategy n Secton IV. Through smulatons, we valdate our theoretcal results and demonstrate the performance gans of the proposed strategy n Secton V. Fnally, we conclude

7 7 Generator 1 Generator g PMU on lne l ISO Storage Aggregator 1 Storage Aggregator Fg. 1. The system model of the smart grd. The nformaton flow to the ISO s denoted by red dashed lnes, the nformaton flow to the aggregators s denoted by black dotted lnes, and the nformaton flow sent from the ISO s denoted by blue dash-dot lnes. the paper n Secton VI. II. SYSTEM MODEL We frst descrbe the basc model that wll be used n most of the paper, and then dscuss some extensons of the model. A. The Basc Model We consder a smart grd wth one ISO ndexed by 0, G generators ndexed by g = 1, 2,..., G, I aggregators ndexed by = 1, 2,..., I, and L transmsson lnes (see Fg. 1 for an llustraton). The ISO schedules the energy generaton of generators and determnes the unt prces of energy for the aggregators. The generators provde the ISO wth the nformaton of ther energy generaton cost functons, based on whch the ISO can mnmze the total cost of the system. Snce the ISO determnes how much energy each generator should produce, we do not model generators as decson makers n the system; nstead, we abstract them by ther energy generaton cost functons. Each aggregator, equpped wth energy storage, manages the electrcty usage n a small communty of resdental households or a commercal buldng, and determnes how much energy to buy from the ISO. In summary, the decson makers (or the enttes) n the system are the ISO and the I aggregators. We denote the set of aggregators by I = {1,..., I}. In the

8 8 TABLE III INFORMATION AVAILABLE TO EACH ENTITY. Informaton s 0 = (ε, ξ), namely the generaton cost functons and the status of the transmsson lnes s = (d, e ), namely the demand and the amount of energy n storage Known to whom the ISO only Aggregator only followng, we may refer to the ISO or an aggregator generally as entty {0} I, wth entty 0 beng the ISO and entty I beng aggregator. As dscussed before, dfferent enttes have dfferent sets of local nformaton, whch are modeled as ther states. Specfcally, the ISO receves reports of the energy generaton cost functons, denoted by ε = (ε 1,..., ε G ), from the generators, and measures the status of the transmsson lnes such as the phases, denoted by ξ = (ξ 1,..., ξ L ), by usng the phasor measurement unts (PMUs). We summarze the energy generaton cost functons and the status of the transmsson lnes nto the ISO s state s 0 = (ε, ξ) S 0, whch s unknown to the aggregators 1. Each aggregator receves energy consumpton requests from ts customers, and manages ts energy storage. We summarze the aggregate demand d from aggregator s customers and the amount e of energy n aggregator s storage nto aggregator s state s = (d, e ) S, whch s known to aggregator only. We assume that all the sets of states are fnte. We hghlght whch nformaton s avalable to whch entty n Table III. The ISO s acton s how much energy each generator should produce, denoted by a 0 A 0 R G +, where A 0 s the acton set. Each aggregator s acton s how much energy to purchase from the ISO, denoted by a A R +, where A s the acton set. We denote the jont acton profle of the aggregators as a = (a 1,..., a N ), and the jont acton profle of all the aggregators other than as a. We dvde tme nto perods t = 0, 1, 2,..., where the duraton of a perod s determned by how fast the demand or supply changes or how frequently the energy tradng decsons are made. In each perod t, the enttes act as follows (see Fg. 2 for llustraton): The ISO observes ts state s 0. 1 In some systems, the ISO wll provde nformaton about the energy generaton cost and the state of the grd for the aggregators. Our framework stll works for ths case when the ISO s state s 0 s known to the aggregator, as long as the ISO s state s ndependent of the aggregators states.

9 9 The ISO observes ts state: ISO t Aggregator t Aggregator observes ts state: The ISO dspatches the generators (.e. chooses ), and determnes the prces Aggregator makes decsons on energy purchase Aggregator makes payments t+1 t+1 Fg. 2. n one perod. Illustraton of the nteracton between the ISO and aggregator (.e. ther decson makng and nformaton exchange) Each aggregator observes ts state s. Each aggregator chooses ts acton a, namely how much energy to purchase from the ISO, and tells ts amount a of energy purchase to the ISO. Based on ts state s 0 and the aggregators acton profle a, the ISO determnes the prce 2 y (s 0, a) Y of electrcty at each aggregator, and announces t to each aggregator. The ISO also determnes ts acton a 0, namely how much energy each generator should produce. Each aggregator pays y (s 0, a) a to the ISO. 3 The nstantaneous cost of each entty depends on ts current state and ts current acton. Each aggregator s total cost conssts of two parts: the operatonal cost and the payment. Each aggregator s operatonal cost c : S A R s a convex ncreasng functon of ts acton a. An example operatonal cost functon of an aggregator can be c (s, a ) = p 1 {e +a <d } + m (e ), 2 We do not model the prcng as the ISO s acton, because t does not affect the ISO s payoff,.e. the socal welfare (ths s because the payment from the aggregators to the ISO s a monetary transfer wthn the system and does not count n the socal welfare). 3 Snce we consder the nteracton among the ISO and the aggregators only, we neglect the payments from the ISO to the generators, whch are not ncluded n the total cost anyway, because the payments are transferred among the enttes n the system.

10 10 where 1 { } s the ndcator functon, p > 0 s a large postve number that s the penalty when the demand s not fulflled (.e. when e + a < d ), and m (e ) s the mantenance cost of the energy storage that s convex [6]. Then we wrte each aggregator s total cost, whch s the cost aggregator ams to mnmze, as the sum of the operatonal cost and the payment, namely c = c + y (s 0, a, a ) a, where a s the actons of the other aggregators. Note that each aggregator s payments depends on the others actons through the prce. Although each aggregator observes ts realzed prce y, t does not know how ts acton a nfluences the prce y, because the prce depends on the others actons a and the ISO s state s 0, nether of whch s known to aggregagtor. The energy generaton cost of generator g s denoted c g (ε g, a 0,g ), whch s assumed to be convex ncreasng n the energy producton level a 0,g. An example cost functon can be c g (ε g, a 0,g ) = (q 0,g + q 1,g a 0,g + q 2,g a 2 0,g) + q r,g (a 0,g a 0,g) 2, where a 0,g s the producton level n the prevous tme slot. In ths case, the energy generaton cost functon of generator g s a vector ε g = (q 0,g, q 1,g, q 2,g, q r,g, a 0,g). In the cost functon, q 0,g + q 1,g a 0,g + q 2,g a 2 0,g s the quadratc cost of producng a 0 amount of energy [1][2], and q r,g (a 0,g s 0,g ) 2 s the rampng cost of changng the energy producton level. We denote the total generaton cost by c 0 = G g=1 c g. The ISO s cost, denoted c 0, s then the sum of generaton costs and the aggregators costs,.e. c 0 = N =0 c. We assume that each entty s state transton s Markovan, namely ts current state depends only on ts prevous state and ts prevous acton. Under the Markovan assumpton, we denote the transton probablty of entty s state s by ρ (s s, a ). Ths assumpton holds for the ISO for the followng reasons. The ISO s state conssts of the energy generaton cost functons and the status of the transmsson lnes. For renewable energy generaton, the energy generaton cost functon s modeled by the amount of avalable renewable energy sources (e.g. the wnd speed n wnd energy, and the amount of sunshne n solar energy), whch s usually assumed to be..d. [3][7][8]. In our model, we relax the..d. assumpton and allow the amount of avalable renewable energy sources to be correlated across adjacent perods. For conventonal energy generaton, the energy generaton cost functon s usually constant when we do not consder rampng costs. If we consder rampng costs, we can nclude the energy producton level at the prevous perod n the energy generaton cost functon. For the aggregators, the amount of energy

11 11 left n the storage depends only on the amount of energy n the prevous perod and the amount of energy purchases n the current perod. The demand of the aggregator s the total demand of all ts customers. Snce the number of customers s large, the temporal correlaton of each customer s energy demand can be neglected n the total demand. For ths reason, the demand of the aggregator s often assumed to be..d. [11][12]. In our model, we relax the..d. assumpton and allow the demand of the aggregator to be temporally correlated across adjacent perods. We also assume that condtoned on the ISO s acton a 0 and the aggregators acton profle a, each entty s state transton s ndependent of each other. Ths assumpton holds for the ISO, because the energy generaton cost functons and the status of the transmsson lnes depend on the envronments such as weather condtons, and possbly on the prevous energy producton levels when we consder rampng costs, but not on the aggregators demand or ts energy storage. For each aggregator, ts energy storage level depends only on ts own state and acton, but not on the ISO s or the other aggregators states. The demand of each aggregator could potentally depend on the ISO s state, because the ISO s state nfluences the unt prce of energy. However, n practce, consumers are not exposed to real-tme prcng n most cases, and hence are not prceantcpatng (namely they do not determne how much to consume based on ther antcpaton of the real-tme prces). As a result, t s reasonable to assume that the demand of each aggregator s ndependent of the ISO s and the other aggregators states. B. Dscussons and Extensons 1) Non-Statonarty: An mportant concern n smart grds s that the demand s non-statonary, namely the demand s sgnfcantly hgher n peak hours. In ths case, the Markovan assumpton on the aggregators states would not hold f we used the same defnton of state. However, we can augment the state of each aggregator wth a component h that represents the perod n one day. Then the newly-defned state transton s Markovan. Smlarly, the dfference of demand n weekdays and weekends can be captured n the same way, where the addtonal component h makes the state space even larger. Note that the seasonal changes n demand cannot be modeled n ths way, whch would result n a sgnfcant ncrease n the state space and make the model ntractable. However, we can deal wth the non-statonarty n such a large tme scale by adjustng the system parameters and recalculatng the optmal DSM strategy when the system parameters change.

12 12 2) Two-Settlement Markets: Many energy markets (such as the New England market and the PJM market) are two-settlement markets. Specfcally, each aggregator predcts the demand n the next day and submts purchase requests n each perod of the next day n advance. In ths case, each aggregator takes actons at two tme scales: day-ahead and real-tme. We can model the two-settlement market by modelng the day-ahead purchase as a state of the aggregator, f the aggregator predcts the future demand based on hstorcal statstcs. Specfcally, aggregator s state s s = (h, d, d, e ), where h s the perod n the day, d s the amount of energy purchased day-ahead for perod h, and d s the real-tme demand n perod h. The amount d of energy purchased day-ahead for perod h depends on the perod h of the day. In ths case, aggregator only needs to fulfll the resdual demand d d n real tme. Ths approach to model the two-settlement market s also adopted n [23]. C. The DSM Strategy At the begnnng of each perod t, each aggregator chooses an acton based on all the nformaton t has, namely the hstory of ts prvate states and the hstory of ts prces. We wrte each aggregator s hstory n perod t as h t = (s 0, y 0 ; s 1, y 1 ;... ; s t 1 all possble hstores of aggregator n perod t as H t = S t+1, y t 1 ; s t ), and the set of Y t. Hence, each aggregator s strategy can be wrtten as π : t=0h t A. Smlarly, we wrte the ISO s hstory n perod t as h t 0 = (s 0 0, y 0 ; s 1 0, y 1 ;... ; s t 1 0, y t 1 ; s t 0), where y t s the collecton of prces at perod t, and the set of all possble hstores of the ISO n perod t as H t 0 = S t+1 0 N Y t. Then the ISO s strategy can be wrtten as π 0 : t=0h t A. The jont strategy profle of all the enttes s wrtten as π = (π 1,..., π N ). Snce each entty s strategy depends only on ts local nformaton, the strategy π s decentralzed. Among all the decentralzed strateges, we are nterested n statonary decentralzed strateges, n whch the acton to take depends only on the current nformaton, and ths dependence does not change wth tme. Specfcally, entty s statonary strategy s a mappng from ts set of states to ts set of actons, namely π s : S A. Snce we focus on statonary strateges, we drop the superscrpt s, and wrte π as entty s statonary strategy. The jont strategy profle π and the ntal state (s 0 0, s 0 1,..., s 0 N ) nduce a probablty dstrbuton over the sequences of states and prces, and hence a probablty dstrbuton over the sequences of total costs c 0, c 1,.... Takng expectaton wth respect to the sequences of stage-game payoffs,

13 13 we have entty s expected long-term cost gven the ntal state as { ( ) } C (π (s 0 0, s 0 1,..., s 0 I)) = E (1 δ) δt c t, (1) where δ [0, 1) s the dscount factor. t=0 s 0 0,s0 1,...,s0 I III. THE DESIGN PROBLEM The desgner, namely the ISO, ams to maxmze the socal welfare, namely mnmze the longterm total cost n the system. In addton, we need to satsfy the constrants due to the capacty of the transmsson lnes, the supply-demand requrements, and so on. We denote the constrants by f(s 0, a 0, a) 0, where f(s 0, a 0, a) R N wth N beng the number of constrants. We assume that the electrcty flow can be approxmated by the drect current (DC) flow model, n whch case the constrants f(s 0, a 0, a) 0 are lnear n each a. Hence, the desgn problem can be formulated as { mn π C 0 (π (s 0 0, s 0 1,..., s 0 I)) + } C (π (s 0 0, s 0 1,..., s 0 I)) (2) I s.t. f(s 0, π 0 (s 0 ), π 1 (s 1 ),..., π I (s I )) 0, (s 0, s 1,..., s N ). Note that n the above optmzaton problem, we use aggregator s cost C nstead of ts total cost C, because ts payment s transferred to the ISO and s thus canceled n the total cost. Note also that we sum up the socal welfare under all the ntal states. Ths can be consdered as the expected socal welfare when the ntal state s unformly dstrbuted. The optmal statonary strategy profle that maxmzes ths expected socal welfare wll also maxmze the socal welfare gven any ntal state. We wrte the soluton to the desgn problem as π and the optmal value of the desgn problem as C. IV. OPTIMAL FORESIGHTED DEMAND SIDE MANAGEMENT In ths secton, we derve the optmal foresghted DSM strategy assumng that each entty knows ts own state transton probabltes.

14 14 A. The aggregator s Decson Problem and Its Conjectured Prce Contrary to the desgner, each aggregator ams to mnmze ts own long-term total cost C (π (s 0 0, s 0 1,..., s 0 N )). In other words, each aggregator solves the followng problem: π = arg max π C (π, π (s 0 0, s 0 1,..., s 0 N)). Assumng that the aggregator knows all the nformaton, the optmal soluton to the above problem should satsfy the followng: V (s 0, s, s ) = max a A (1 δ) c (s 0, s, a, a )+δ s 0,s,s { ρ 0 (s 0 s 0 ) j N ρ j (s j s j, a j )V (s 0, s, s ) Note that the above equatons would be the Bellman equatons, f the aggregator knew all the nformaton such as the other aggregators strateges π and states s, and the ISO s state s 0. However, such nformaton s never known to the aggregator. Hence, we need to separate the nfluence of the other enttes from each aggregator s decson problem. One way to decouple the nteracton among the aggregators s to endow each aggregator wth a conjectured prce. In general, the conjecture nforms the aggregator of what prce t should antcpate gven ts state and ts acton. However, n the presence of decentralzed nformaton, such a complcated conjecture s hard, f not possble, to form. Specfcally, aggregator s conjectured prce should depend not only on aggregator s acton and state, but also on the ISO s state. Hence, no entty possess all the necessary nformaton to form the conjecture. For ths reason, n ths paper, we propose a smple conjecture, namely the prce does not depend on the aggregator s state and acton. In ths case, the conjectures can be formed by the ISO based on ts local nformaton and then communcated to the aggregators. Denote the conjectured prce as ỹ, we can rewrte aggregator s decson problem as Ṽ ỹ (s ) = max(1 δ) [c (s, a ) + ỹ a ] + δ [ ] ρ (s a A s, a )Ṽ ỹ (s ). Clearly, we can see from the above equatons that gven the conjectured prce ỹ, each aggregator can make decsons based only on ts local nformaton. In Fg. 3, we llustrate the enttes decson makng and nformaton exchange n the desgn framework based on conjectured prces. Comparng Fg. 3 wth Fg. 2 of the system wthout conjectured prces, we can see that n the proposed desgn framework, the ISO sends the conjectured prces to the aggregators before the aggregators make decsons. Ths addtonal s }.

15 15 The ISO observes ts state: ISO t Aggregator t Aggregator observes ts state: The ISO announces the conjectured prces The ISO dspatches the generators (.e. chooses ), and determnes the prces t+1 t+1 Aggregator makes decsons on energy purchase Aggregator makes payments Fg. 3. prces. Illustraton of the enttes decson makng and nformaton exchange n the desgn framework based on conjectured procedure of exchangng conjectured prces allows the ISO to lead the aggregators to the optmal DSM strateges. Note that the conjectured prce s generally not equal to the real prce charged at the end of the perod, and s not equal to the expectaton of the real prce n the future. In ths sense, the conjectured prces can be consdered as control sgnals sent from the ISO to the aggregators, whch can help the aggregators to compute the optmal strateges. In Secton V, we wll compare the conjectured prce wth the expected real prce by smulaton. The remanng queston s how to determne the optmal conjectured prces, such that when each aggregator reacts based on ts conjectured prce, the resultng strategy profle maxmzes the socal welfare. B. The Optmal Decentralzed DSM Strategy The optmal conjectured prces depend on the ISO s state, whch s known to the ISO only. Hence, we propose a dstrbuted algorthm used by the ISO to teratvely update the conjectured prces and by the aggregators to update ther optmal strateges. The algorthm wll converge to the optmal conjectured prces and the optmal strategy profle that acheves the mnmum total system cost C.

16 16 At each teraton k, gven the conjectured prce ỹ (k) Ṽ ỹ(k) (s ) = max(1 δ) a A and obtans the optmal value functon Ṽ ỹ(k) under the current conjectured prce ỹ (k). g [ c (s, a ) + ỹ (k) a ] + δ, each aggregator solves [ s ρ (s s, a )Ṽ ỹ(k) (s ) ], as well as the correspondng optmal strategy πỹ(k) Smlarly, gven the conjectured prces ỹ (k) 0 R G, the ISO solves [ ] Ṽ ỹ(k) 0 0 (s 0 ) = mn (1 δ) c g (s 0, a 0 ) + ỹ (k)t 0 a 0 + δ [ ] ρ 0 (s a A 0 s 0, a 0 )Ṽ ỹ(k) 0 0 (s 0), and obtans the optmal value functon Ṽ ỹ(k) 0 as well as the correspondng optmal strategy πỹ(k) 0 under the current conjectured prce ỹ (k). Then the ISO updates the conjectured prces as follows: where λ (k+1) (s 0 ) R N s calculated as { ( λ (k+1) (s 0 ) = λ (k) (s 0 ) + (k) f ỹ (k+1) = ( λ (k+1) (s 0 ) ) T f(s 0, a) a, s 0, πỹ0(k) 0 (s 0 ), s 1 πỹ1(k) where (k) R ++ s the step sze, and {x} + = max{x, 0}. s 0 1 (s 1 ),..., S 1 s I πỹi(k) I (s I ) S I )} +, Note that n the above update of conjectures, to calculate (the subgradent) λ (k), the ISO (k) s πỹ1 needs to know the average amount of purchase (s ) S from each aggregator. Ths requres addtonal nformaton exchange from the aggregator to the ISO. Moreover, the aggregator may not be wllng to report such nformaton to the ISO. To reduce the amount of nformaton exchange and preserve prvacy, we propose that the ISO calculates the emprcal mean values of the aggregators purchases n the run-tme (whch results n stochastc subgradents). We summarze the algorthm n Table IV, and prove that the algorthm can acheve the optmal socal welfare n the followng theorem. Theorem 1: The algorthm n Table IV converges to the optmal strategy profle, namely { lm k 0 C 0 (πỹ(k) (s 0 0, s 0 1,..., s 0 I)) + } C (πỹ(k) (s 0 0, s 0 1,..., s 0 I)) C I = 0. s 0 0,s0 1,...,s0 I Proof: See the appendx.

17 17 TABLE IV DISTRIBUTED ALGORITHM TO COMPUTE THE OPTIMAL DECENTRALIZED DSM STRATEGY. Input: Each entty s performance loss tolerance ɛ Intalzaton: Set k = 0, ā (0) = 0, I, ỹ (0) = 0, I {0}. repeat Each aggregator solves Ṽ ỹ(k) (s 0 ) (s ) = max a A (1 δ) [ ] c (s, a ) + ỹ (k) (s 0) a + δ [ ] s ρ (s s, a )Ṽ ỹ(k) (s 0 ) (s ) The ISO solves [ ] Ṽ ỹ0(k) 0 (s 0) = mn a A (1 δ) g cg(s0, a0) + ỹ0(k)t a 0 + δ Each aggregator reports ts purchase request πỹ(k) (s 0 ) (s ) The ISO updates ā (k + 1) = ā (k) + πỹ(k) (s 0 ) (s ) for all I The ISO updates the conjectured prces: ( ) T ỹ (k+1) (s 0) = λ(k + 1) T f(s 0,a) a, where (k) = 1 and k+1 { ( )} + λ(k + 1) = λ(k) + (k) f s 0, πỹ0(k) 0 (s 0), ā1(k+1),..., āi (k+1) k+1 k+1 untl Ṽ ỹ(k+1) (s 0 ) Ṽ ỹ(k) (s 0 ) ɛ s 0 [ ] ρ 0(s 0 s 0, a 0)Ṽ ỹ0(k) 0 (s 0) TABLE V INFORMATION NEEDED BY EACH ENTITY TO IMPLEMENT THE ALGORITHM. Entty The ISO Each aggregator Informaton at each step k πỹ(k) (s 0 ) The purchase request (s ) of each aggregator Conjecture on ts prce ỹ (k) (s 0 ) We summarze the nformaton needed by each entty n Table V. We can see that the amount of nformaton exchange at each teraton s small (O(I)), compared to the amount of nformaton unavalable to each entty ( j S states plus the strateges π ). In other words, the algorthm enables the enttes to exchange a small amount (O(I)) of nformaton and reach the optmal DSM strategy that acheves the same performance as when each entty knows the complete nformaton about the system. We brefly dscuss the complexty of mplementng the algorthm n terms of the dmensonalty of the Bellman equatons solved by each entty. For each aggregator, t solves the Bellman equaton that has the the same dmensonalty as the cardnalty of ts state space, namely S.

18 18 For each ISO, the dmensonalty of ts state space s large, because the generaton cost functons ε are a vector of length G and the status of the transmsson lnes s a vector of length L. However, the ISO s decson problem can be decomposed due to the followng observaton. Note that the generators energy generaton cost functons are ndependent of each other. Then we have the followng theorem. Theorem 2: Gven the conjectured prce ỹ 0 (k), the ISO s value functon Ṽ ỹ0(k) 0 can be calculated by Ṽ ỹ0(k) 0 (s 0 ) = G g=1 Ṽ ỹ0,g(k) 0,g (ε g ), where Ṽ ỹ0,g(k) 0,g solves Ṽ ỹ0,g(k) 0,g (ε g ) = mn a 0,g (1 δ) [ c g (ε g, a 0,g ) + ỹ 0,g (k) T a 0,g ] + δ Proof: The proof follows drectly from Lemma 1 n the appendx. ε g [ ] ρ 0 (ε g ε g, a 0,g )Ṽ ỹ0,g(k) 0,g (ε g). From the above proposton, we know that the dmensonalty of the ISO s decson problem s G g=1 E g, where E g s the cardnalty of the set of generator g s generaton cost functons. The dmensonalty ncreases lnearly wth the number of generators, nstead of exponentally wth the number of generators and transmsson lnes wthout decomposton. C. Learnng Unknown Dynamcs In practce, each entty may not know the dynamcs of ts own states (.e., ts own state transton probabltes) or even the set of ts own states. When the state dynamcs are not known a pror, each entty cannot solve ther decson problems usng the dstrbuted algorthm n Table IV. In ths case, we can adapt the onlne learnng algorthm based on post-decson state (PDS) n [20], whch was orgnally proposed for wreless vdeo transmssons, to our case. The man dea of the PDS-based onlne learnng s to learn the post-decson value functon, nstead of the normal value functon. Each aggregator s post-decson value functon s defned as U ( d, ẽ ), where ( d, ẽ ) s the post-decson state. The dfference from the normal state s that the PDS ( d, ẽ ) descrbes the status of the system after the purchase acton s made but before the demand n the next perod arrves. Hence, the relatonshp between the PDS and the normal state s d = d, ẽ = e + (a d ). Then the post-decson value functon can be expressed n terms of the normal value functon

19 19 as follows: U ( d, ẽ ) = d ρ (d, ẽ (a d ) d, ẽ ) V (d, ẽ (a d )). In PDS-based onlne learnng, the normal value functon and the post-decson value functon are updated n the followng way: V (k+1) U (k+1) (d (k), e (k) (d (k), e (k) ) = max(1 δ) c (d (k), e (k), a ) + δ U (k) a ) = (1 α (k) )U (k) (d (k), e (k) ) + α (k) V (k) (d (k) (d (k), e (k), e (k) + (a d (k) )), (a d (k) )). We can see that the above updates do not requre any knowledge about the state dynamcs. It s proved n [20] that the PDS-based onlne learnng wll converge to the optmal value functon. D. Detaled Comparsons wth Exstng Frameworks Snce we have ntroduced our proposed framework, we can provde a detaled comparson wth the exstng theoretcal framework. The comparson s summarzed n Table VI. Frst, the proposed framework reduces to the myopc optmzaton framework when we set the dscount factor δ = 0. In ths case, the problem reduces to the classc economc dspatch problem. Second, the Lyapunov optmzaton framework s closely related to the PDS-based onlne learnng. In fact, t could be consdered as a specal case of the PDS-based onlne learnng when we set the post-decson value functon as U (s ) = c (s, a ) + (e + a ) 2 e 2, and choose the acton that mnmzes the post-decson value functon n the run-tme. However, the Lyapunov drft n the above post-decson value functon depends only on the status of the energy storage, but not on the demand. In contrast, n our PDS-based onlne learnng, we explctly consders the mpact of the demand when updatng the normal and post-decson value functons. Fnally, the key dfference between our proposed framework and the framework for MU- MDP [13][14] s how we penalze the constrants f(s 0, a 0, a). In partcular, the framework n [13][14], f drectly appled n our model, would defne only one Lagrangan multpler for all the constrants under dfferent states s 0. Ths leads to performance loss n general [14]. In contrast, we defne dfferent Lagrangan multplers to penalze the constrants under dfferent states s 0, and potentally enable the proposed framework to acheve the optmalty (whch s ndeed the case as have been proved n Theorem 1).

20 20 TABLE VI RELATIONSHIP BETWEEN THE PROPOSED AND EXISTING THEORETICAL FRAMEWORKS. Framework Relatonshp Representatve works Myopc δ = 0 [3] Lyapunov optmzaton Aggregator s post-decson value functon U (s ) = c (s, a ) + (e + a d ) 2 e 2 [11][12] MU-MDP Lagrangan multpler λ(s 0) = λ for all s 0 [13][14] V. SIMULATION RESULTS In ths secton, we valdate our theoretcal results and compare aganst exstng DSM strateges through extensve smulatons. We use the wdely-used IEEE test power systems wth the data (e.g. the topology, the admttances and capacty lmts of transmsson lnes) provded by Unversty of Washngton Power System Test Case Archve [21]. We descrbe the other system parameters as follows (these system parameters are used by default; any changes n certan scenaros wll be specfed): One perod s one hour. The dscount factor s δ = The demand of aggregator at perod t s unformly dstrbuted among the nterval [d (t mod 24) d (t mod 24), d (t mod 24) + d (t mod 24)]. In other words, the dstrbuton of demand s tme-varyng across a day. We let the peak hours for all the aggregators to be from 17:00 to 22:00. The mean value d (t mod 24) and the range d (t mod 24) of aggregator s demand are descrbed as follows (values are adapted from [22]): 50 + ( 1) 0.5 MW f t mod 24 [17, 22] d (t mod 24) = 25 + ( 1) 0.5 MW otherwse and 5 MW f t mod 24 [17, 22] d (t mod 24) = 2 MW otherwse All the aggregators have energy storage of the same capacty 25 MW. All the aggregators have the same lnear energy storage cost functon [6]: c (s, a ) = 2 (a d ) +, (3) (4)

21 21 namely the mantenance cost grows lnearly wth the remanng energy level and s ndependent of the amount of charge and dscharge. We ndex the energy generators startng from the renewable energy generators. All the renewable energy generators have lnear energy generaton cost functons: [22] c g (a 0,g ) = g a 0,g, where the unt energy generaton cost has the same value as the ndex of the generator (these values are adapted from [22], whch cted that the unt energy generaton cost ranges from $0.19/MWh to $10/MWh). Although the energy generaton cost functon s determnstc, the maxmum amount of energy producton s stochastc (due to wnd speed, the amount of sunshne, and so on). The maxmum amounts of energy producton of all the renewable energy generators follow the same unform dstrbuton n the range of [90, 110] MW. The rest of energy generators are conventonal energy generators that use coal, all of whch have the same energy generaton cost functon: [6] c g (a 0,g ) = 1 2 (a 0,g) }{{} 10 (a 0,g a 0,g) 2. }{{} generaton cost rampng cost In other words, the conventonal energy generators have fxed (.e. not stochastc) generaton cost functons. The status of the transmsson lnes s ther capacty lmts. The nomnal values of the capacty lmts are the same as specfed n the data provded by [21]. In each perod, we randomly select a lne wth equal probablty, and decrease ts capacty lmt by 10%. We compare the proposed DSM strateges wth the followng schemes. Centralzed optmal strateges ( Centralzed ): We assume that there s a central controller who knows everythng about the system and solves the long-term cost mnmzaton problem as a sngle-user MDP. Ths scheme serves as the benchmark optmum. Myopc strateges ( Myopc ) [1] [8]: In each perod t, the aggregators myopcally mnmzes ther current costs, and based on ther actons, the ISO mnmzes the current total generaton cost. Sngle-user Lyapunov optmzaton ( Lyapunov ) [9] [12]: We let each aggregator adopt the stochastc optmzaton technque proposed n [9] [12]. Based on the aggregators purchases, the ISO mnmzes the current total generaton cost.

22 Total Cost Normalzed By Number of Buses ($) Centralzed Proposed MUMDP Lyapunov Myopc Capacty of Energy Storage (MW) Fg. 4. The normalzed total cost per hour versus the capacty of the energy storage n the IEEE 14-bus system. A. Performance Evaluaton Now we evaluate the performance of the proposed DSM strategy n varous scenaros. 1) Impact of the energy storage: Frst, we study the mpact of the energy storage on the performance of dfferent schemes. We assume that all the generators are conventonal energy generators usng fossl fuel, n order to rule out the mpact of the uncertanty n renewable energy generaton (whch wll be examned next). The performance crteron s the total cost per hour normalzed by the number of buses n the system. We compare the normalzed total cost acheved by dfferent schemes when the capacty of the energy storage ncreases from 5 MW to 45 MW. Fg. 4 6 show the normalzed total cost acheved by dfferent schemes under IEEE 14-bus system, IEEE 30-bus system, and IEEE 118-bus system, respectvely. Note that we do not show the performance of the centralzed optmal strategy under IEEE 118-bus system, because the number of states n the centralzed MDP s so large that t s ntractable to compute the optmal soluton. Ths also shows the computatonal tractablty and the scalablty of the proposed dstrbuted algorthm. Under IEEE 14-bus and 30-bus systems, we can see that the proposed DSM strategy acheves almost the same performance as the centralzed optmal strategy. The slght optmalty gap comes from the performance loss experenced durng the convergence process of the conjectured prces. Compared to the DSM strategy based on sngle-user Lyapunov optmzaton, our proposed strategy can reduce the total cost by around 30% n most cases. Compared to the myopc DSM strategy, our reducton n the total cost s even larger and ncreases wth the capacty of the energy storage (up to 60%).

23 Total Cost Normalzed By Number of Buses ($) Centralzed Proposed MUMDP Lyapunov Myopc Capacty of Energy Storage (MW) Fg. 5. The normalzed total cost per hour versus the capacty of the energy storage n the IEEE 30-bus system Total Cost Normalzed By Number of Buses ($) Proposed MUMDP Lyapunov Myopc Capacty of Energy Storage (MW) Fg. 6. The normalzed total cost per hour versus the capacty of the energy storage n the IEEE 118-bus system. 2) Impact of the uncertanty n renewable energy generaton: Now we examne the mpact of the uncertanty n renewable energy generaton. For a gven test system, we let half of the generators to be renewable energy generators. Recall that the maxmum amounts of energy producton of the renewable energy generators are stochastc and follow the same unform dstrbuton. We set the mean value of the maxmum amount of energy producton to be 100 MW, and vary the range of the unform dstrbuton. A wder range ndcates a hgher uncertanty n renewable energy producton. Hence, we defne the uncertanty n renewable energy generaton as the maxmum devaton from the mean value n the unform dstrbuton. Fg. 7 shows the normalzed total cost under dfferent degrees of uncertanty n renewable energy generaton. Agan, the proposed strategy acheves the performance of the centralzed optmal strategy n the IEEE 14-bus system. We can see that the costs acheved by all the schemes ncrease wth the uncertanty n renewable energy generaton. Ths happens for the

24 24 Total Cost Normalzed By Number of Buses ($) Centralzed Proposed MUMDP Lyapunov Myopc Total Cost Normalzed By Number of Buses ($) Centralzed Proposed MUMDP Lyapunov Myopc Uncertanty n Renewable Energy Generaton (MW) Uncertanty n Renewable Energy Generaton (MW) Fg. 7. The normalzed total cost per hour versus the uncertanty n renewable energy generaton n the IEEE 14-bus system. The aggregators have energy storage of capacty 25 MW and 50 MW, respectvely. followng reasons. Snce the renewable energy s cheaper, the ISO wll dspatch renewable energy whenever possble, and dspatch conventonal energy for the resdual demand. However, when the renewable energy generaton has larger uncertanty, the varaton n the resdual demand s hgher, whch results n a hgher varaton n the conventonal energy dspatched and thus a larger rampng cost. To reduce the rampng cost, the ISO needs to be more conservatve n dspatchng the renewable energy, whch results n a hgher total cost. However, we can also see from the smulaton that when the aggregators have larger capacty to store energy, the ncrease of the total cost wth the uncertanty s smaller. Ths s because the energy storage can smooth the demand, n order to mtgate the mpact of uncertanty n the renewable energy generaton. Ths shows the value of energy storage to reduce the cost. 3) Farness: Now we nvestgate how the ndvdual costs of the aggregators are nfluenced by the capacty of ther energy storage. In partcular, we are nterested n whether some aggregators are affected by havng smaller energy storage. We assume that half of the aggregators have energy storage of capacty 50 MW, whle the other half have energy storage of much smaller capacty 10 MW. In Fg. 8, we compare the average ndvdual cost of the aggregators wth smaller energy storage and that of the aggregators wth larger energy storage. We can see that the average cost of the aggregators wth smaller energy storage does ncrease wth the uncertanty n renewable energy generaton. Hence, the aggregators wth hgher energy storage have an advantage over those wth smaller energy storage, because they have hgh flexblty n copng wth the prce