Modeling and Design of Real-Time Pricing Systems Based on Markov Decision Processes

Transcription

1 Appled Mathematcs, 04, 5, Publshed Onlne June 04 n ScRes. Modelng and Desgn of Real-Tme Prcng Systems Based on Markov Decson Processes Koch Kobayash *, Ichro Maruta, Kazunor Sakurama 3, Shun-ch Azuma School of Informaton Scence, Japan Advanced Insttute of Scence and Technology, Ishkawa, Japan Graduate School of Informatcs, Kyoto Unversty, Kyoto, Japan 3 Graduate School of Engneerng, Tottor Unversty, Tottor, Japan Emal: * k-kobaya@jast.ac.jp Receved Aprl 04; revsed May 04; accepted 9 May 04 Copyrght 04 by authors and Scentfc Research Publshng Inc. Ths work s lcensed under the Creatve Commons Attrbuton Internatonal Lcense (CC BY). Abstract A real-tme prcng system of electrcty s a system that charges dfferent electrcty prces for dfferent hours of the day and for dfferent days, and s effectve for reducng the peak and flattenng the load curve. In ths paper, usng a Markov decson process (MDP), we propose a modelng method and an optmal control method for real-tme prcng systems. Frst, the outlne of real-tme prcng systems s explaned. Next, a model of a set of customers s derved as a mult-agent MDP. Furthermore, the optmal control problem s formulated, and s reduced to a uadratc programmng problem. Fnally, a numercal smulaton s presented. Keywords Markov Decson Process, Optmal Control, Real-Tme Prcng System. Introducton In recent years, there has been growng nterest n energy and the envronment. For problems on energy and the envronment such as energy savng, several approaches have been studed (see, e.g., [] []). In ths paper, we focus on real-tme prcng systems of electrcty. A real-tme prcng system of electrcty s a system that charges dfferent electrcty prces for dfferent hours of the day and for dfferent days, and s effectve for reducng the peak and flattenng the load curve (see, e.g., [3]-[6]). In general, a real-tme prcng system conssts of one controller decdng the prce at each tme and multple electrc customers such as commercal facltes * Correspondng author. How to cte ths paper: Kobayash, K., et al. (04) Modelng and Desgn of Real-Tme Prcng Systems Based on Markov Decson Processes. Appled Mathematcs, 5,

2 K. Kobayash et al. and homes. If electrcty conservaton s needed, then the prce s set to a hgh value. Snce the economc load becomes hgh, customers conserve electrcty. Thus, electrcty conservaton s acheved. In the exstng methods, the prce at each tme s gven by a smple functon wth respect to power consumptons and voltage devatons and so on (see, e.g., [6]). In order to realze more precsely prcng, t s necessary to use a mathematcal model of customers. In ths paper, usng a Markov decson process (MDP), we propose a mathematcal model of real-tme prcng systems. Snce n many cases, the status of electrcty conservaton of customers s dscrete and stochastc, t s approprate to use an MDP. Then, a set of electrcty customers s modeled by a mult-agent MDP. Furthermore, we consder the fnte-tme optmal control problem. By approprately settng the cost functon, t s acheved that customers conserve electrcty actvely. Ths problem can be used for the model predctve control method, whch s a control method that the fnte-tme optmal control problem s solved at each tme. In addton, the fnte-tme optmal control problem can be reduced to a uadratc programmng problem. The proposed approach provdes us wth a basc of real-tme prcng systems. Ths paper s organzed as follows. In Secton, the outlne of real-tme prcng systems s explaned. In Secton 3, a model of electrcty customers s derved. In Secton 4, the optmal control problem s formulated, and ts soluton method s derved. In Secton 5, a numercal smulaton s shown. In Secton 6, we conclude ths paper. Notaton: Let denote the set of real numbers. Let I n, 0 m n denote the n n dentty matrx, the m n zero matrx, respectvely. For smplcty, we sometmes use the symbol 0 nstead of 0 m n, and the symbol I nstead of I n. For two events AB,, let E AB denote the condtonal expected value of A under the event B.. Outlne of Real-Tme Prcng Systems In ths secton, we explan the outlne of real-tme prcng systems studed n ths paper. Fgure shows an llustraton of real-tme prcng systems studed n ths paper. Ths system conssts of one controller and multple electrc customers such as commercal facltes and homes. For an electrc customer, we suppose that each customer can montor the status of electrcty conservaton of other customers. In other words, the status of some customer affects that of other customers. For example, n commercal facltes, we suppose that the status of rval commercal facltes can be checked by lghtng, Blog, Twtter, and so on. Dependng on power consumpton,.e., the status of electrcty conservaton, the controller determnes the prce at each tme. If electrcty conservaton s needed, then the prce s set to a hgh value. Snce the economc load becomes hgh, customers conserve electrcty. Thus, electrcty conservaton s acheved. In ths paper, the status of electrcty conservaton of each customer s modeled by a Markov decson process (MDP). Then a set of customers s modeled by a mult-agent MDP (MA-MDP). Furthermore, by usng the obtaned MA-MDP model, we consder the optmal control problem and ts soluton method. Fgure. Illustraton of real-tme prcng systems. 486

3 K. Kobayash et al. 3. Model of Customers Frst, consder modelng the dynamcs of each customer by a one-dmensonal MDP. The value of the state x 0,,, n 0,,, n expresses the status of s randomly chosen among the fnte set { }. The element of { } electrcty conservaton, and 0 mples the status that a customer conserves electrcty maxmally, n mples the status that a customer does not conserve electrcty. Then the MDP of a customer s gven by π ( t ) P( u ) π + = () where u s the control nput, and corresponds to the prce. The vector T = [ ] denotes the probablty dstrbuton, that s, [ 0,] 0 0, n n π mples π π π π the probablty that the state s at tme t. In addton, the ntal probablty dstrbuton must satsfy the followng condton: n = 0 The transton probablty matrx P( u( t )) s gven by ( ) P u t π 0 =. n a + bu t a + bu t a + bu t an + bu t =. a n + bu n ann + bu n The control nput s determned under the condton for each element: and the condton for each column: 0 a + bu t () n = j aj + bu t =. (3) Next, consder modelng the dynamcs of a set of customers by an MA-MDP. The number of customers s x 0,,, n, and from (), the MDP model s gven by. For the customer, the state s gven by { } gven by π ( + ) = π t P u t t. Then, we suppose that the MA-MDP model expressng the dynamcs of a set of customers s gven by ( ) λp u t λi λ ( I π t ) + π π ( t ) λi λp ( u ) λ I + π =, π ( t ) + λ ( ) I λi λp u t π where λ j expresses the effect of couplngs between customers, and s a constant satsfyng the followng condton: λ =, =,,,. (5) j j= For smplcty of dscusson, couplng terms are gven by some condton correspondng to (5). λ I j (4), but may be replaced wth matrces satsfyng 487

4 K. Kobayash et al. 4. Optmal Control 4.. Problem Formulaton Consder the followng problem. Problem. Suppose that for the MA-MDP model (4) expressng the dynamcs of customers, the ntal state x 0 = x, =,,,, the desred state x d, and the predcton horzon N are gven. Then, fnd a control 0 nput seuence u t, =,,,, t = 0,,, N mnmzng the followng cost functon subject to the followng constrant: ( ) ( 0) N d 0 t= 0 = (6) J = E Q x t x + R u t x = x ( ) where f ( ) s a gven lnear functon, M s a gven vector., Hereafter, for smplcty of notaton, the condton x ( 0) x0 By usng the constrant (7), the nput constrant such as mn adjustng Q, R, several specfcatons such that the state x desred state x can be consdered. 4.. Soluton Method d f u t, u t,, u t M, (7) Q R are gven weghts. = n the cost functon (6) s omtted. u u t u can be mposed. In addton, by max t must converges to the neghborhood of the We derve a soluton method for Problem. Frst, consder the MDP model (). The MDP model s a class of nonlnear systems. However, n ths case, t can be transformed nto a lnear system. The MDP model () can be rewrtten as ( π + + πn ) ( π + + πn ) bu t t t bu t t t π ( t+ ) = Aπ +, bu n ( π + + πn ) where a a a n a a a n A =. an an ann By the property of the probablty dstrbuton, the relaton π + + πn = holds. From ths fact, the MDP model () can be euvalently transformed nto the followng lnear system: where [ ] T π ( t ) Aπ Bu + = +, (8) B = b b bn. Next, by usng the lnear system (8), consder representng the MA-MDP model (4) as a lnear system. The lnear system for the customer s denoted by π ( t+ ) = Aπ + Bu. Then, the MA-MDP model (4) can be euvalently transformed nto the followng lnear system: ( + ) π t π t u t = A + B, (9) π ( t+ ) π u 488

5 K. Kobayash et al. where λa λ I λb 0 A =, B =. λ I λ A 0 λb Fnally, consder the cost functon (6). Defne Then we can obtan Therefore, the cost functon (6) can be rewrtten as N t= 0 = L [ n ] L : = 0, ( n ) = : 0. π = E x t L t, ( ) π E x t = L t. ( ) J = E Q x t xd + R u t N ( ) = QE x t Qx E x t + Qx + R u t d d t= 0 = (0) N Q( L xdl) π R u Qx d. t= 0 = = + + From the above dscusson, Problem s euvalent to the followng problem. Problem fnd u t, =,,, t = 0,,, N, mn Cost functon 0, subject to System 9, Ineualty constrant, 7, Eualty constrant 3. Problem s reduced to a uadratc programmng (QP) problem, and can be solved by a sutable solver such as MATLAB and IBM ILOG CPLEX. In addton, f R = 0, then Problem s reduced to a lnear program- mng (LP) problem (we remark that Qx n the cost functon (0) s a constant). See [7] for further detals. 5. Numercal Example d Snce t s dffcult to use data n real systems, we present an artfcal example. The state s chosen among the 0,,,3. The number of consumers s gven by = 5. The coeffcent matrces A, B n the lnear fnte set { } system for the consumer are gven by A =, A =, A3 =,

6 K. Kobayash et al. The parameters The parameters N, From =, =, = A4 A5 B λ j n (9) are gven by λ λ λ3 λ4 λ λ λ λ3 λ4 λ λ3 λ3 λ33 λ34 λ 35 = λ4 λ4 λ43 λ44 λ λ5 λ5 λ53 λ54 λ x d, Q, and R are gven by N = 0, x d = 0, Q =, and R = 0, respectvely. x 0 = In R =, Problem s reduced to an LP problem. The ntal state s gven by [ ] T addton, the nput constrant u 0 s mposed. In ths numercal example, we consder the followng two cases: u t = u t = = u t holds). The prce for each customer s the same (.e., The prce for each customer s dfferent. Case () s the conventonal case n real-tme prcng systems. In Case (), we suppose that the dfference n the prce s covered by usng local concurrences such as the Eco-pont pont system [8]. The Eco-money system [9] n Japan were ntroduced to stmulate the economy and rase awareness of global warmng. In the Eco-pont pont system, many ponts, whch correspond to money n a local concurrency, are gven for the products that are effectve from the vewponts of electrcty conservaton and the envronment. Such a system for energy management systems has been dscussed n [0]. Next, we present the computatonal results. Frst, the computatonal result n Case () s explaned. Fgures -6 π t ncreases and show the probablty dstrbuton for each customer. From these fgures, we see that 0 π decreases. Thus, the state converges to 0, whch corresponds to the status that a customer conserves 3 electrcty maxmally, wth a certan probablty. Furthermore, the optmal value of the cost functon s , and the optmal control nput s derved as u 0 = 0.5, u = 0.65, u = , u 3 = , u 4 = 0.568, u 5 = 0.55, u 6 = 0.584, u 7 = 0.589, u 8 = 0.59, u 9 = Fgure. π (t) n Case (). 490

7 K. Kobayash et al. Fgure 3. π (t) n Case (). Fgure 4. π 3 (t) n Case (). Fgure 5. π 4 (t) n Case (). Next, the computatonal result n Case () s explaned. Fgures 7- show the probablty dstrbuton for π t are each customer. Comparng Fgures -6 wth Fgures 7-, we see that transent responses of 0 5 mproved n Case (). In partcular, for the customer 5, the steady state of π s also mproved (see 0 t 49

8 K. Kobayash et al. Fgure 6. π 5 (t) n Case (). Fgure 7. π (t) n Case (). Fgure 8. π (t) n Case (). Fgure 6 and Fgure ). Furthermore, the optmal value of the cost functon s , and we see that the optmal value of the cost functon s mproved. The optmal control nput u t = u t u t u t u t u t s derved as T 49

9 K. Kobayash et al. Fgure 9. π 3 (t) n Case (). Fgure 0. π 4 (t) n Case () u( 0 ) = 0.5, u( ) = , u( ) = 0.608, u( 3 ) = 0.549, u( 4) = , u( 5 ) = , u( 6 ) = , u( 7 ) = , u( 8) = , u ( 9 ) = From these values, we see that n the steady state, u ( t ) s wdely dfferent to system consdered here, t s approprate to use a local concurrency. 6. Conclusons u t, =,,3,4. Thus, n the In ths paper, we have proposed a modelng method and an optmal control method of real-tme prcng systems usng the MDP-based approach. In many cases, the status of electrcty conservaton of customers s dscrete and 493

10 K. Kobayash et al. Fgure. π 5 (t) n Case (). stochastc, and the use of the MDP model s effectve. A real-tme prcng system s modeled by mult-agent MDPs, and the optmal control problem s reduced to a QP problem. Furthermore, a numercal smulaton has been shown. The proposed method provdes us wth a new method n real-tme prcng of electrcty. There are several open problems. Frst, t s mportant to develop the dentfcaton method of the MA-MDP model based on the exstng result (see, e.g., []) for MDPs. Snce the effect of couplngs between customers was smplfed, t s also mportant to consder modelng t more precsely. Next, the optmal control problem s reduced to a QP problem or an LP problem. These problems can be solved faster than a combnatoral optmzaton problem such as a mxed nteger programmng problem. However, for large-scale systems, the computaton tme for solvng the optmal control problem wll be long. Then, t s mportant to develop a dstrbuted algorthm. Acknowledgements Ths research was partly supported by JST, CREST. References [] Camacho, E.F., Samad, T., Garca-Sanz, M. and Hskens, I. (0) Control for Renewable Energy and Smart Grds. In: Samad, T. and Annaswamy, A.M., Eds., The Impacy of Control Technology, IEEE Control Systems Socety, New York. [] Ruhua, Z., Yume, D. and Yuhong, L. (00) New Challenges to Power System Plannng and Operaton of Smart Grd Development n Chna. Proceedngs of the 00 Internatonal Conference on Power System Technology, Hangzhou, 4-8 October 00, -8. [3] Borensten, S., Jaske, M. and Rosenfeld, A. (00) Dynamc Prcng, Advanced Meterng, and Demand Response n Electrcty Markets. Center for the Study of Energy Markets, Unversty of Calforna, Berkeley. [4] Roozbehan, M., Dahleh, M. and Mtter, S. (00) On the Stablty of Wholesale Electrcty Markets under Real-Tme Prcng. Proceedngs of the 49th IEEE Conference on Decson and Control, Atlanta, 5-7 December 00, [5] Samad, P., Mohsenan-Rad, A.-H., Schober, R., Wong, V.W.S. and Jatskevch, J. (00) Optmal Real-Tme Prcng Algorthm Based on Utlty Maxmzaton for Smart Grd. Proceedngs of the st IEEE Internatonal Conference on Smart Grd Communcatons, Gathersburg, 4-6 October 00, [6] Vvekananthan, C., Mshra, Y. and Ledwch, G. (03) A Novel Real Tme Prcng Scheme for Demand Response n Resdental Dstrbuton Systems. Proceedngs of the 38th Annual Conference of the IEEE Industral Electroncs Socety, Monteral, 5-8 October 0, [7] Bello, D. and Rano, G. (006) Lnear Programmng Solvers for Markov Decson Processes. Proceedngs of the 006 IEEE Systems and Informaton Engneerng Desgn Symposum, Charlottesvlle, 8 Aprl 006, [8] Eco-Pont System for Housng

11 K. Kobayash et al. [9] Kyoto Eco Money. [0] Sawashma, K., Kubota, Y., Lu, H., Takemae, T., Yoshda, K. and Wan, Y. (0) Soco-Personal Energy Management System. Keo ALPS0 Group K Fnal Report. [] Rust, J. (994) Structural Estmaton of Markov Decson Processes. In: Handbook of Econometrcs, Elsever, Amsterdam, Vol. IV, Chapter 5,