Pricing for Coordination in Open Loop Differential Games

Transcription

1 Preprnts of the 19th World Congress The Internatonal Federaton of Automatc Control Prcng for Coordnaton n Open Loop Dfferental Games Danel Calderone Lllan J. Ratlff S. Shankar Sastry Department of Electrcal Engneerng and Computer Scences, Unversty of Calforna, Berkeley, Berkeley, CA USA {danjc, ratlffl, sastry} at eecs.berkeley.edu Abstract: We provde a method for desgnng a quadratc prcng scheme to nduce a desred local Nash equlbrum n an open-loop dfferental game wth nonlnear dynamcs. In addton, we present condtons when the nduced equlbrum s a global equlbrum. The results are appled to the problem of nducng network managers to nvest n securty n a mult-network wth an epdemc model of the spread of malware. Keywords: Game Theory, prcng desgn, nonlnear systems 1. INTRODUCTION When resources are scarce, competton develops between self nterested agents. Game theory s an establshed technque for modelng ths nteracton, and t has emerged as an engneerng tool for analyss and synthess of systems comprsed of dynamcally coupled decson makng agents possessng competng nterests (Coogan et al. (2013),L and Marden (2011), Ratlff et al. (2012)). In such scenaros, the strateges chosen by the selfsh agents result n a soluton that s often neffcent from a socetal pont of vew. Ths motvates the desgn of coordnatng mechansms that nduce agents to play a Nash equlbrum wth desrable propertes, namely an equlbrum that s socally optmal. Engneerng problems n whch there are decson makng agents, ether wth competng nterests or dfferent nformaton sets, as well as a socal planner who s tasked wth coordnatng the agents are appearng more frequently n the lterature as technology s ntegrated nto nfrastructure (Oldewurtel et al. (2010)). It s mportant to accurately model these systems and develop control strateges accountng for the nterests of all the partcpatng agents whle meetng the organzatonal objectve whch may represent socal welfare or the common good. The nteracton of selfsh agents n mult-agent system may be cast as a dfferental game by modelng dynamcally coupled agents as strategc players. The coordnaton problem may then be cast as an optmzaton problem whereby a socal planner determnes a coordnaton mechansm ensurng the agents play the desred equlbrum. The coordnaton mechansms modfy the agents nomnal utlty functons thereby allowng for a socal planner to shape the strateges of the agents n order to meet a desred global objectve. In Ratlff et al. (2012); Calderone et al. (2013); Coogan et al. (2013) the problem of fndng prces to nduce a socally optmal Nash equlbrum n a lnear quadratc game s solved. In ths paper, we show that the result can be generalzed to open loop dfferental games wth non lnear dynamcs and possbly non convex costs that are separable n the state and the control. In partcular, we formulate the problem of desgnng prces to nduce a desred (socally optmal) equlbrum for games wth non lnear dynamcs and non convex costs as a feasblty problem. We show that f ths feasblty problem has a soluton, then the desred equlbrum s a local Nash equlbrum of the game resultng from mposton of prces on the players. Further, f the desred equlbrum s the unque equlbrum of the prcng nduced game, the desgned prces cause the agents to play the socally optmal soluton. If n addton the dynamcs are convex, then we provde an extended feasblty problem to desgn prces that force the socally optmal soluton to be the unque Nash equlbrum of the prcng nduced game. Fnally, we apply the theory to the problem of securty n mult networks whch s a rsng problem n the study of cyber physcal systems (Cardenas et al., 2008; Bloem et al., 2009; Alpcan and Başar, 2010). In ths example, we add an objectve functon and addtonal constrants to the prcng desgn feasblty problem to ensure a budget balanced soluton. The rest of the paper s organzed as follows. In Secton 2, we formulate the game. In Secton 3, we defne the prcng optmzaton problem and state our man results. n Secton 4, we consder the applcaton of desgnng prces to nduce nvestment n securty n a mult network system wth an epdemc model for the spread of malware. The prcng scheme guarantees that the network dynamcs remans stable meanng the spread of malware does not destablze the networks. In Secton 5, we summarze the contrbutons and dscuss future drectons. 2. AGENT GAME Consder a dynamc game wth n agents where the system dynamcs are gven by the general nonlnear ordnary dfferental equaton ẋ = f(t, x, u), x 0 (t) = x 0 (1) where Copyrght 2014 IFAC 9001

2 x(t) = [x 1 (t) T... x n (t) T ] T (2) and u(t) = [u 1 (t) T... u n (t) T ] T. (3) Each x (t) R n where n s the dmenson. The th agent has control over control nput u (t) R m where m s the dmenson of nput u (t) and has nomnal cost gven by J (x(t 0 ), u) = 1 q (t, x) + r (t, u) dt + q (t f, x) (4) 2 t 0 We suppress the dependence of the cost J on the ntal condton x(t 0 ) when t s clear from context. By an abuse of notaton, we let u denote the strategy over the horzon [t 0, t f ] and we drop the dependence on t n the control acton u (t) where t s clear from context. Each player s nterested n mnmzng ther cost J (u, u ) wth respect to ther choce varable u and where = {1,..., 1, + 1,..., n}. We restrct each agent s choce u to be an open loop control and we denote the space of open loop control strateges for agent by Γ. Gven the cost J (u, u ), the Hamltonan for the th player s gven by H (t, x, p, u, u ) = q (t, x) + r (t, u) + p (t) T f(t, x, u) (5) and the optmzed Hamltonan for the th player s gven by H (t, x, p ) = mn H (t, x, p, u, u ). (6) u Γ Note that the co state for each player s a vector p (t) R n for each t [t 0, t f ]. Let {u }n =1 be an open loop Nash equlbrum (ether local or global). Then, n equlbrum, the optmalty condtons for agent s optmzaton problem are ẋ(t) = H p (t, x, p, u ) (7) ṗ (t) T = H x (t, x, p, u ) (8) 0 = H (t, x, p, u ) (9) for all t [t 0, t f ] where and x(t 0 ) = x 0 and p (t f ) = q x (x (t f )). (10) H p (t, x, p, u ) = f(t, x, u ), (11) H x (t, x, p, u ) = q x (t, x ) + p T and H f x (t, x, u ), (12) (t, x, p, u ) = r (t, u ) + p T f (t, x, u ). (13) Equaton (7) s the state equaton, Equaton (8) s the co state equaton, and Equaton (9) s the nput statonarty condton. Defnton 1. A Nash equlbrum n the open loop dfferental game defned by the costs (4) and dynamcs (1) s a set of strateges {u [t 0, t f ]} n =1 such that for each {1,..., n} J (u, u ) J (u, u ) u Γ. (14) The above defnton can be nterpreted as sayng a set of strateges {u }n =1 s a Nash equlbrum f no player can decrease ther cost by unlaterally devatng from ther strategy n {u [t 0, t f ]} n =1. The problem of fndng an open loop Nash equlbrum for the game defned by (1) and (4) s to fnd a set of control strateges {u [t 0, t f ]} n =1 such that the nequalty (14) s satsfed for each {1,..., n}. In general, fndng open loop Nash equlbra of dynamc games s a dffcult problem. Some recent work has explored the characterzaton and computaton of local equlbra n contnuous games ncludng open loop dfferental games Ratlff et al. (2013). In Secton 4 we wll use the technques ntroduced n Ratlff et al. (2013) to compute the Nash equlbrum under the prcng scheme n order to valdate that the prcng scheme results n the desred behavor modfcaton. 3. PRICING DESIGN The prcng desgn problem s defned to be the optmzaton problem solved by the socal planner n whch she desgns prcng mechansms to nduce agents to use the desred equlbrum {u (t) = K (t)} n =1. We wll restrct ourselves to modfyng each player s cost by addng a prcng mechansm, P (t, u), that s composed of a quadratc term and a lnear term n the control nputs. Namely, we defne P (t, u) = u T (t)r (t)u(t) + c u(t) dt (15) t 0 where R (t) = R (t) T 0. Thus each player s new cost wth prcng s gven by J (u) = t 0 q (t, x) + r (u, t) + u T R (t)u + c u dt + q (t f, x(t f )). (16) For convenence, we wll partton R (t) and c (t) as follows: R (t) := [ R 1 (t) R n (t) ] and c (t) := [ c 1 (t) c n (t) ]. (17) The Hamltonan for the th agent generated under the prcng scheme s gven by H (t, x, p, u) = q (t, x) + r (t, u) + u(t) T R (t)u(t) + c (t)u + p T f(t, x, u) (18) and the optmzed Hamltonan under prcng s gven by H (t, x, p ) = mn H (t, x, p, u, u ). (19) u Γ The optmalty condtons under prcng are ẋ(t) = H p (t, x, p, u ) = f(t, x, u ) (20) ṗ T = H x (t, x, p, u ) = q x (t, x ) p T H (t, x, p, u ) = (R (t)) T u + c (t) + p T f x (t, x, u ) (21) f (t, x, u ) + r (t, u ) = 0. (22) Note that snce the costs are separable n the states and controls and snce we do not allow the prces to depend 9002

3 on the state, both the state equaton and the co state equaton are ndependent of the prcng mechansm. Thus, gven a set of desred controls, we can solve for the correspondng state trajectory, x(t) and co state trajectory p (t) on the tme nterval [t 0, t f ]. Then, use the state and co state to choose prces that satsfy the nput statonarty condton. Defne the desred equlbrum u = K(t) = [K 1 (t) K n (t)] T. (23) Further, let x (t) and p (t) denote the state and co state at tme t under the desred equlbrum control K(t) respectvely. At the desred equlbrum, the nput statonarty condton s r (t, K(t)) + R u (t) T K(t) + c (t) + p (t) T f (t, x (t), K(t)) = 0 (24) Rearrangng (24), we defne α (t) as follows: α (t) = R(t) T K(t) + c (t) = p (t) T f (t, x (t), K(t)) r (t, K(t)) (25) for each {1,..., n}. Note that α (t) s completely known. Thus, we want to fnd R (t) and c (t) to satsfy (25). Thus desgnng prces to make K(t) a local Nash equlbrum amounts to solvng the feasblty problem defned below n Equaton (26). R(t)K(t) + c(t) = α(t) t (26) where R(t) = [R1(t) 1 Rn(t)] n T (27) c(t) = [c 1 1(t) c n n(t)] T (28) α(t) = [α 1 (t) α n (t)] T. (29) We wll use the notaton R[t 0, t f ] and c[t 0, t f ] to denote R(t) and c(t) for each t [t 0, t f ]. We can summarze the above results n the followng theorem. Theorem 1. Consder the game defned by nomnal agent costs (4) and dynamcs (1). Let {u [t 0, t f ]} n =1 be the desred Nash equlbrum. If there exsts a soluton (R[t 0, t f ], c[t 0, t f ]) (30) to the feasblty problem defned n Equaton (26), then the desred soluton {u [t 0, t f ]} n =1 s a local Nash equlbrum to the prcng nduced game defned by costs (16) and dynamcs (1). In the case that the desred soluton s the unque Nash equlbrum to the nduced game, we get the followng result. Corollary 1. Consder the game defned by nomnal agent costs (4) and dynamcs (1). Let {u [t 0, t f ]} n =1 be the desred Nash equlbrum. If there exsts a soluton (R[t 0, t f ], c[t 0, t f ]) (31) to the feasblty problem defned n Equaton (26) and the desred Nash equlbrum s unque n the resultng game defned by costs (16) and dynamcs (1), then the prces (R[t 0, t f ], c[t 0, t f ]) nduce the agents to play {u [t 0, t f ]} n = Dynamcs Convex n the State and Control In the case that the desred equlbrum s unque, the prcng mechansms are guaranteed to enforce the desred equlbrum strateges. Let us recall the noton of strct dagonal convexty ntroduced by Rosen n Rosen (1965) and then extended to the nfnte dmensonal case n Haure and Moresno (2001). Defnton 2. A functon L (x, u, t, p ) s dagonally strctly convex n (x, u) f for all ū, û, x, and ˆx we have (ū û ) T Å L (x, ū, t, p ) L (x, û, t, p ) ( x ˆx ) T Å L x ( x, u, t, p ) L x (ˆx, u, t, p ) ã ã > 0 (32) The followng lemma and theorem provde condtons under whch a Nash equlbrum of an open loop dfferental game s unque. Lemma 1. Assume that L (x, u) s convex n (x, u) and assume that the total runnng cost L(x, u) = L (x, u) (33) s dagonally strctly convex n (x, u). Further, assume that the dynamcs f(t, x, u) are convex n x and u. Then, the combned Hamltonan H(t, x, p) = H (t, x, p ) (34) s dagonally strctly convex n x and concave n p. Theorem 2. If the combned Hamltonan H(t, x, p) s dagonally strctly convex n x and concave n p, then the open loop Nash equlbrum s unque. Lemma 1 s a modfed verson of Lemma 2.1 n Haure and Moresno (2001). Theorem 2 s stated n Haure and Moresno (2001) and ts proof can be found n Carlson and Haure (1996) usng our verson of the lemma. If the costs under prcng satsfy the assumptons of Lemma 1, the desred equlbrum s the unque Nash equlbrum of the open loop dfferental game nduced through prcng. Defne the total runnng cost L (x, u, t, p ) (35) where L (x, u, t, p ) = q (x, t) + r (u, t) + u T R (t)u + c (t)u(t). (36) Assumpton 1. q (x, t) + r (u, t) s dagonally strctly convex n (x, u). Provded Assumpton 1, by Lemma 1 and Theorem 2, we only need to ensure that the prcng mechansm s dagonally strctly convex n u,.e. we need to enforce (û(t) ū(t)) T R(t)(û(t) ū(t)) > 0 (37) for all û and ū and for each t [t 0, t f ]. Thus n order to ensure dagonal strct convexty n u, we smply need to ensure that the symmetrc component of R(t),.e. R(t) + R(t) T, s postve defnte for all t. 9003

4 ß R(t)K(t) + c(t) = α(t) t R(t) + R(t) T 0 t (38) We have the followng result. Theorem 3. Consder the game defned by nomnal agent costs (4) and dynamcs (1). Suppose that f(t, x, u) s convex n x and u. Let {u [t 0, t f ]} n =1 be the desred Nash equlbrum. Then, f there exsts a soluton (R[t 0, t f ], c[t 0, t f ]) (39) to the feasblty problem defned n Equaton (38), then the prces (R[t 0, t f ], c[t 0, t f ]) nduce the agents to play {u [t 0, t f ]} n =1 to the game defned by (16) and dynamcs (1). Further, the desred Nash equlbrum s the unque equlbrum n the prcng nduced game. 4. PRICING IN MULTI NETWORKS We use the epdemc model for the spread of malware n a mult network ntroduced n Bloem et al. (2009). Selfspreadng attacks on computer networks are expensve owng to the damage they cause and the securty nvestment requred to defend aganst them. The socal planner s goal s to desgn prcng mechansms that coordnate the networks so that the overall mult-network s stablzed. Suppose that we have n networks wth N nodes n the th network and let x (t) R denote the number of nfected hosts n a network where hosts can be fractonally nfected. Let u (t) be the malware removal rate for network, α be the cross network parwse rate of nfecton, and β be the parwse rate of nfecton wthn networks. In general, computers wthn a network are more lkely to communcate wth one another than across networks; hence, we assume β > α. The spread of malware s then captured n the followng epdemc model: ẋ (t) = β(n x (t))x (t)+ α(n x (t))x j (t) u (t) j=1,j (40) Each network ndependently tres to choose u so that the th network s stablzed. For each network n the mult network, we consder a cost that s quadratc n the state,.e. the number of nfected hosts, and quadratc n the control,.e. the patchng rate. The nomnal cost for network s J (u) = 0 x T Q x + u T M u dt. (41) where Q and M are the cost of an nfected network host and the cost of the mplemented patchng response respectvely. The socal planner desgns prcng mechansms to coordnate the networks by nducng them to choose a desred control acton whch stablzes the entre multnetwork. We consder a group of sx networks wth N 1 = 3500, N 2 = 500, N 3 = 2000, N 4 = 1000, N 5 = 500, and N 6 = For each network, we take Q to be a dagonal matrx wth random postve entres where the (, )th element of Q s larger than the others. The M matrces are chosen to be 0 except for the (, )th element whch s 1. The rato between Q (, ) and M (, ) s 10 to 1. We scale the tme horzon to be over the nterval [0, 1] and take the ntal number of nfected nodes n each network to be half of the total number of nodes. As n Bloem et al. (2009), we take β = We set α = 2 3 β. Usng the dscretzaton scheme for optmal control problems descrbed n Chapter 4 of Polak (1997), we compute a centralzed soluton usng the sum of all the agents costs and standard nonlnear programmng technques. In general, ths only gves us a local optmum to the centralzed problem, but we wll see that t does mprove the performance of the system as compared to the Nash equlbrum. From the centralzed soluton, we determne a desred set of controls, K(t) = [K 1 (t) K n (t)] T. In order to desgn prces, the socal planners solves a modfed verson of the feasblty problem outlned n (38) at each tme step. Snce the nomnal costs are quadratc n the control, we replace each M wth R so that the new cost for each player becomes J (u) = 0 x T Q x + u T R (t)u + c (t)u dt. (42) In ths case, Equaton (25) becomes ᾱ (t) = R(t) T K(t) + c (t) = p (t) T f (t, x (t), K(t)) In addton, we add an objectve and several more constrants to make the problem budget balanced. Our fnal optmzaton problem s gven by mn {R (t),c (t)} n =1 R (t) M + c (t) (43) =1 subject to: R(t)K(t) + c(t) = ᾱ(t) (44) R(t) + R(t) T 0 (45) R (t) M 0 (46) =1 =1 R (t) = R (t) T 0, (47) at each tme step. Recall that R(t) = [R 1 1(t) R n n(t)] T. Equaton (46) forces the sum of the quadratc components of the prces to be greater than the sum of the nomnal quadratc components. Gven ths constrant, (43) seeks to make the costs wth prces as close as possble to the nomnal costs. Usng the same dscretzaton scheme as n the centralzed problem, we numercally approxmate local Nash equlbra under both the nomnal costs and the costs wth prcng usng the steepest descent algorthm presented n Ratlff et al. (2013). Snce the dynamcs are not convex n the state, we cannot guarantee global unqueness of the nduced equlbrum; however by checkng the 2nd-order suffcent condtons presented n Ratlff et al. (2013), we can show that the Nash under prcng s an solated local Nash. Thus control sgnals ntalzed close to the equlbrum wll converge to t under the steepest descent algorthm. By solvng the prcng problem, we make the centralzed soluton a local Nash equlbrum of the game wth prces. Moreover, we fnd that a wde varety of ntalzatons for the controls actually converge to the desred equlbrum. Computaton of basns of attracton of equlbra for ths problem and other general nonlnear problems s left as future work. 9004

5 It should be noted that the 2nd-order suffcent condtons for solated local Nash equlbra are only applcable to fnte dmensonal problems. Thus we can not guarantee that we nduce an solated equlbra of the actual nfnte dmensonal optmzaton problem but only of the fntedmensonal dscretzed problem. Fgure 1 shows the control nputs and Fgure 2 shows the state trajectores for the nomnal Nash equlbrum, the centralzed optmal soluton, and the Nash equlbrum under prcng. centralzed problem at each tme step. Each ndvdual player s runnng cost wth prces s not guaranteed to be equal to ther porton of the socal cost, however. As an example n Fgure 4, we plot the runnng costs of players 1 and 6. Fg. 3. Sum of ndvdual runnng costs at the nomnal Nash, socal optmum, and prcng nduced Nash. The fact that the socal optmum cost and prcng nduced Nash cost are equal means that we can acheve budget balance. Fg. 1. Control sgnals at the nomnal Nash equlbrum, the centralzed optmum, and prcng nduced Nash equlbrum. Fg. 2. Number of nfected nodes n each network over tme. Note that the nomnal Nash strateges do not elmnate all nfected nodes where as the socally optmal strateges and prcng nduced Nash strateges do. Fgure 3 compares the sum of the runnng costs. The centralzed soluton as well as the Nash under prcng reduces the total cost to the system by 8.2%. We also see that we are able to force the sum of all the runnng costs wth prces to be equal to the runnng cost of the Fg. 4. Indvdual runnng costs for players 1 and 6. Though the sum of all the runnng costs s the same as the socal optmum under prcng, each ndvdual player s runnng cost s dfferent from ther porton of the socal cost. 5. CONCLUSION In summary, we have formulated a feasblty problem for fndng quadratc and lnear prces to nduce a socally optmal local Nash equlbra n the context of open-loop dfferental games wth non-lnear dynamcs and general nomnal costs. In addton, we have shown that under specal condtons on the dynamcs and prces, the nduced 9005

6 equlbrum s the global equlbrum of the game wth prces. We apply these technques to the problem of ncentvzng network managers to nvest n securty to prevent the spread of epdemcs. In ths partcular example, we are able to desgn budget balanced prces that make the socally optmal controls an solated local Nash equlbra. We are currently nvestgatng computaton of basns of attracton for the equlbra of the nduced game. We are also studyng the stablty of the equlbra as well as condtons to ensure unqueness of equlbra n non-convex games. REFERENCES Alpcan, T. and Başar, T. (2010). Network securty: A decson and game-theoretc approach. Cambrdge Unversty Press. Bloem, M., Alpcan, T., and Başar, T. (2009). Optmal and robust epdemc response for multple networks. Control Engneerng Practce, 17(5), Calderone, D., Ratlff, L.J., and Sastry, S.S. (2013). Prcng desgn for robustness n lnear-quadratc dynamc games. In In the Proceedngs of the 52rd Annual IEEE Conference on Decson and Control. Cardenas, A., Amn, S., and Sastry, S. (2008). Secure control: Towards survvable cyber-physcal systems. In Dstrbuted Computng Systems Workshops, ICDCS th Internatonal Conference on, Carlson, D. and Haure, A. (1996). A turnpke theory for nfnte-horzon open-loop compettve processes. SIAM Journal on Control and Optmzaton, 34(4), Coogan, S., Ratlff, L., Calderone, D., Tomln, C., and Sastry, S.S. (2013). Energy management va prcng n LQ dynamc games. In Amercan Control Conference. Haure, A. and Moresno, F. (2001). Computaton of S- adapted Equlbra n Pecewse Determnstc Games va Stochastc Programmng Methods. Brkhäuser Boston. L, N. and Marden, J.R. (2011). Desgnng games for dstrbuted optmzaton. In Proceedngs of the 50th IEEE Conference on Decson and Control and European Control Conference, Oldewurtel, F., Ulbg, A., Parso, A., Andersson, G., and Morar, M. (2010). Reducng peak electrcty demand n buldng clmate control usng real-tme prcng and model predctve control. In Proceedngs of the 49th IEEE Conference on Decson and Control, Polak, E. (1997). Optmzaton: algorthms and consstent approxmatons. Sprnger New York. Ratlff, L.J., Burden, S.A., and Sastry, S.S. (2013). Characterzaton and computaton of local nash equlbra n contnuous games. In Proceedngs of the 51st Annual Allerton Conference on Communcaton, Control, and Computng. Ratlff, L., Coogan, S., Calderone, D., and Sastry, S. (2012). Prcng n lnear-quadratc dynamc games. In Proceedngs of the 50th Annual Allerton Conference on Communcaton, Control, and Computng, Rosen, J.B. (1965). Exstence and unqueness of equlbrum ponts for concave n-person games. Econometrca, 33(3),