Socially-aware Multiagent Learning towards Socially Optimal Outcomes

Transcription

1 Soally-aware Multagent Learnng towards Soally Optmal Outomes Xaohong L and hengwe Zhang and Janye Hao and Karl Tuyls and Sq hen 3 Abstrat. In multagent envronments, the apablty of learnng s mportant for an agent to behave approprately n fae of unknown opponents and dynam envronment. From the system desgner s perspetve, t s desrable f the agents an learn to oordnate towards soally optmal outomes, whle also avodng beng exploted by selfsh opponents. To ths end, we propose a novel gradent asent based algorthm SA-IGA whh augments the bas gradentasent algorthm by norporatng soal awareness nto the poly update proess. We theoretally analyze the learnng dynams of SA-IGA usng dynamal system theory and SA-IGA s shown to have lnear dynams for a wde range of games nludng symmetr games. The learnng dynams of two representatve games the prsoner s dlemma game and oordnaton game are analyzed n detals. Based on the dea of SA-IGA, we further propose a pratal multagent learnng algorthm, alled SA-PGA, based on Q-learnng update rule. Smulaton results show that SA-PGA agent an aheve hgher soal welfare than prevous soal-optmalty orented ondtonal Jont Aton Learner JAL and also s robust aganst ndvdually ratonal opponents by reahng Nash equlbrum solutons. Introduton In multagent systems, the ablty of learnng s mportant for an a- gent to adaptvely adjust ts behavors n response to oexstng a- gents and unknown envronments n order to optmze ts performane. Multagent learnng algorthms have reeved extensve nvestgaton n the lterature, and lots of learnng strateges [5, 3, 3] have been proposed to faltate oordnaton among agents. The mult-agent learnng rtera proposed n [4] requre that an agent should be able to onverge to a statonary poly aganst some lass of opponents onvergene and the best-response poly a- ganst any statonary opponent ratonalty. If both agents adopt a ratonal learnng strategy n the ontext of repeated games and also ther strateges onverge, then they wll onverge to a Nash equlbrum of the stage game. Indeed, onvergene to Nash equlbrum has been the most ommonly aepted goal to pursue n multagent learnng lterature. Untl now, a number of gradent-asent based multagent learnng algorthms [7, 4,, ] have been sequentally proposed towards onvergng to Nash equlbrum wth mproved onvergene performane and more relaxed assumptons less nformaton s requred. Under the same dreton, another well-studed famly of multagent learnng strateges s based on renforement learnng e.g., Q-learnng []. Representatve examples nlude ds- Tanjn Unversty, hna. Emal:{xaohongl, henvy@tju.edu.n, janye.hao@tju.edu.n Unversty of Lverpool, UK. Emal: ktuyls@lverpool.a.uk 3 Southwest Unversty, hna. Emal: sqhen@swu.edu.n trbuted Q-learnng n ooperatve games [], mnmax Q-learnng n zero-sum games [], Nash Q-learnng n general-sum games [9], and other extensons [, 5], to name just a few. Agent s 3/3 /5 5/ / Table : The Prsoner s lemma Game All the aforementoned learnng strateges pursue onvergng to Nash equlbrum under self-play, however, Nash equlbrum soluton may be undesrable n many senaros. One well-known example s the prsoner s dlemma P game shown n Table. By onvergng to the Nash equlbrum,, both agents obtan the payoff of, whle they ould have obtaned a muh hgher payoff of 3 by oordnatng on the non-equlbrum outome,. In stuatons lke the P game, onvergng to the soally optmal outome under self-play would be more preferred. To address ths ssue, one natural modfaton for a gradent-asent learner s to update ts poly along the dreton of maxmzng the sum of all agents expeted payoff nstead of ts own. However, n an open envronment, the agents are usually desgned by dfferent partes and may have not the nentve to follow the strategy we desgn. The above way of updatng strategy would be easly exploted and taken advantage by equlbrumdrven self-nterested agents. Thus t would be hghly desrable f an agent an onverge to soally optmal outomes under self-play and Nash equlbrum aganst self-nterested agents to avod beng exploted. In ths paper, we frst propose a new gradent-asent based algorthm SA-IGA whh augments the bas gradent asent algorthm by norporatng soal awareness nto the poly update proess. A SA-IGA agent holds a soal atttude to reflet ts soally-aware degree, whh an be adjusted adaptvely based on the relatve performane between ts own and ts opponent. A SA-IGA agent seeks to update ts poly n the dreton of nreasng ts overall payoff whh s defned as the average of ts ndvdual and the soal payoff weghted by ts soally-aware degree. We theoretally show that for a wde range of games e.g., symmetr games, the dynams of SA-IGAs under self-play exhbts lnear haratersts. For generalsum games, t may exhbt non-lnear dynams whh an stll be analyzed numerally. The learnng dynams of two representatve games P game and oordnaton game are analyzed n detals. Lke prevous theoretal multagent learnng algorthms, SA-IGA

2 also requres addtonal assumpton of knowng the opponent s poly and the game struture. To relax the above assumpton, we then propose a pratal gradent asent based multagent learnng strategy, alled Soally-aware Poly Gradent Asent SA-PGA. SA-PGA relaxes the above assumptons by estmatng the performane of ts own and the opponent usng Q-learnng tehnques. We emprally evaluate ts performane n dfferent types of benhmark games and smulaton results show that SA-PGA agent outperforms prevous learnng strateges n terms of maxmzng the soal welfare and Nash produt of the agents. Besdes, SA-PGA s also shown to be robust aganst ndvdually ratonal opponents and onverges to Nash equlbrum solutons. The remander of the paper s organzed as follows. Seton revews normal-form game and the bas gradent asent approah. Seton 3 ntrodues the SA-IGA algorthm and analyzes ts learnng dynams theoretally. Seton 4 presents the pratal multagent learnng algorthm SA-PGA n detals. In Seton 5, we extensvely evaluate the performane of SA-PGA under varous benhmark games. Lastly we onlude the paper and pont out future dretons n Seton 6. Bakground. Normal-form games In a two-player, two-aton, general-sum normal-form game, the payoffs for eah player {r, an be spefed by a matrx as follows, [ ] r R = r r r Eah player smultaneously selets an aton from ts aton set A = {,, and the payoff of eah player s determned by ther jont. For example, f player r selets the pure strategy of aton whle player selets the pure strategy of aton, then player r reeves a payoff of r r and player reeves the payoff of r. Apart from pure strateges, eah player an also employ a mxed strategy to make desons. A mxed strategy an be represented as a probablty dstrbuton over the aton set and a pure strategy s a speal ase of mxed strateges. Let p r [, ] and p [, ] denote the probablty of hoosng aton by player and player r respetvely. Gven a jont mxed strategy p r, p, the expeted payoffs of player and player r an be spefed as follows, V r p r, p =r r p r p + r p r p + r r p r p + r r p r p V p r, p =r p rp + r p r p + r p r p + r p r p respetvely. A jont strategy s alled a Nash Equlbrum NE, f no player an get a better expeted payoff by hangng ts urrent strategy u- nlaterally. Formally, p r, p [, ] s a NE, ff V r p r, p V r p r, p and V p r, p V p r, p for any p r, p [, ].. Gradent Asent GA When a game s repeatedly played, an ndvdually ratonal player updates ts strategy towards maxmzng ts expeted payoffs. A player employng GA-based algorthms updates ts poly towards the dreton of ts expeted reward gradent, whh an be shown n the followng equatons. p t+ V p t η p t+ Π [,] p t p + p t+ 3 where parameter η s the gradent step sze, and Π [,] s the projeton funton mappng the nput value to the vald probablty range of [, ], used to prevent the gradent movng the strategy out of the vald probablty spae. Formally, we have, Π [,] x = argmn z [,] x z 4 To smplfy the notatons, let us denote u = r +r r r, = r r and d = r r. For the two-player ase, the above way of GA-based updatng n Equaton and 3 an be represented as follows, p t+ r Π [,] p t r + η u rp t + r 5 p t+ Π [,] p t + η u p t r + d In the ase of nfntesmal gradent step sze η, the learnng dynams of the players an be modeled as a system of dfferental equatons and analyzed usng dynam system theory [7]. It s proved that the agents wll onverge to a Nash equlbrum, or f the strateges themselves do not onverge, then ther average payoffs wll nevertheless onverge to the average payoffs of a Nash equlbrum [7]. Followng [7], varous GA-based algorthms have been proposed to mprove the onvergene performane towards Nash equlbra and representatve examples nlude IGA-WoLF Wn or Learn Fast [4], Weghted Poly Learner PWL [] and Gradent Asent Wth Poly Predton IGA-PP []. In ontrast, n ths work, we seek to norporate the soal awareness nto GA-based strategy update and am at mprovng the soal welfare of the players under self-play rather than pursung Nash equlbrum solutons. Meanwhle, ndvdually ratonal behavor s employed when playng aganst a selfsh agent. Smlar dea of adaptvely behavng dfferently aganst dfferent opponents was also employed n prevous algorthms [, 8, 4, 6]. However, all the exstng works fous on maxmzng an agent s ndvdual payoff aganst dfferent opponents n dfferent types of games, but do not dretly take nto onsderaton the goal of maxmzng soal welfare e.g., ooperate n the prsoner s dlemma game. 3 Soally-aware Infntesmal Gradent Asent SA-IGA In our daly lfe, people usually do not always behave as a purely ndvdually ratonal entty and seeks to aheve Nash equlbrum solutons. For example, when two person subjets play a P game, reahng mutual ooperaton may be observed frequently. Smlar phenomenon have also been observed n extensve human-subjet based experments n games suh as the Publ Good game and Ultmatum game, n whh human subjets are usually found to obtan muh hgher payoff by mutual ooperaton rather than pursung Nash equlbrum solutons. If the above phenomenon s transformed nto omputatonal models, t ndates that an agent may not only update ts poly n the dreton of maxmzng ts own payoff, but also take nto onsderaton other s payoff. We all ths type of agents as soally-aware agents. In ths paper, we norporate the soal awareness nto the gradent-asent based learnng algorthm. In ths way, apart from 6

3 learnng to maxmzng ts ndvdual payoff, an agent s also e- qupped wth the soal awareness suh that t an reah mutually ooperatve solutons faed wth another soally-aware opponent self-play; behave n a purely ndvdually ratonal manner aganst a purely ratonal opponent. Spefally, for eah agent {r,,we dstngush two types of expeted payoffs, namely V dv and V so. The payoff V dv p r, p and V so p r, p represent the ndvdual and soal payoff the average payoff of both players that agent pereves under the jont strategy p r, p respetvely. The payoff V dv p r, p follows the same defnton as Equaton and the payoff V so p r, p an be defned as follows, V so p r, p = dv [Vr p r, p + V dv p r, p ], {r, 7 Eah agent adopts a soal atttude w to reflet ts soallyaware degree. The soal atttude ntutvely models an agent s soally frendly degree towards ts partner. Spefally, t s used as the weghtng fator to adjust the relatve mportane between V dv and V so, and agent s overall expeted payoff s defned as follows, V p r, p = w V dv p r, p + w V so p r, p, {r, 8 Eah agent updates ts strategy n the dreton of maxmzng the value of V. Formally we have, p η p V p r, p p p Π [,] p + p where parameter η p s the gradent step sze of p. If w =, t means that the agent seeks to maxmze ts ndvdual payoff only, whh s redued to the ase of tradtonal gradent-asent updatng; f w =, t means that the agent seeks to maxmze the sum of the payoffs of both players. Fnally, eah agent s soally-aware degree s adaptvely adjusted n response to the relatve value of V dv and V so as follows. urng eah round, f player s own expeted payoff V dv exeeds the value of V so, then player nreases ts soal atttude w,.e., t beomes more soal-frendly beause t pereves tself to be earnng more than the average. onversely, f V dv s less than V so t, then the agent tends to are more about ts own nterest by dereasng the value of w. Formally we have, w = { Π[,] w + w f V dv Π [,] w w f V dv where w s the adjustment step sze of w. > V so < V so 3. Theoretal Modelng and Analyss of SA-IGA 9 An mportant aspet of understandng the behavor of a multagent learnng algorthm s theoretally modelng and analyzng ts underlyng dynams [9, 5, 3]. In ths seton, we frst show that the learnng dynams of SA-IGA under self-play an be modeled as a system of dfferental equatons. Based on the adjustment rules n Eq 9 and, the learnng dynams of a SA-IGA agent an be modeled as a set of equatons n. For ease of exposton, we onentrate on a unonstraned update equatons by removng the poly projeton funton whh does not affet our qualtatve analytal results. Any trajetory wth lnear non-lnear haraterst wthout onstrants s stll lnear non-lnear when a boundary s enfored. p t+ w t+ p t+ w t+ η p V p t r p, p t η w V dv V so p t w t + p t+ + w t+ Substtutng V dv and V so by ther defntons Eq. and 7, the learnng dynams of two SA-IGA agents an be expressed as follows, [ p t+ r = η p u r + u u r p t+ w t+ r = w t r [ = η p u + u r u w t p t + ] r wr t + r p t r + d ] r d w t + d η w [ ur u p t rp t + r p t r + d d r p t + e ] w t+ = η w [ ur u p t rp t + r p t r + d d r p t + e ] where u = r + r r r, = r r,d = r r, and e = r r r wth {r,. As η p and η w, t s straghtforward to show that the above equatons beome dfferental. Thus the unonstraned dynams of the strategy par and soal atttudes as a funton of tme s modeled by the followng system of dfferental equatons: ṗ r = u r + u u r w r p + r w r + r ṗ = u + u r u w p r + d r d w + d ẇ r = ε [u r u p rp + r p r + d d r p + e] ẇ = ε [u r u p rp + r p r + d d r p + e] 3 where ε = η w η p >. Based on the above theoretal modelng, next we analyze the learnng dynams of SA-IGA qualtatvely as follows. Theorem SA-IGA has non-lnear dynams when u r u. Proof From the system of dfferental equatons n 3, t s s- traghtforward to verfy that the dynams of SA-IGA learners are non-lnear when u r u due to the exstene of w rp, w p r or p r p n all equatons. Sne SA-IGA s dynams are non-lnear when u r u, n general we annot obtan a losed-form soluton, but we an stll resort to solve the equatons numerally to obtan useful nsght of the system s dynams. Moreover, a wde range of of mportant games fall nto the ategory of u r = u, n whh the system of equatons beome lnear. Therefore, t allows us to use dynam system theory to systematally analyze the underlyng dynams of SA-IGA. Theorem SA-IGA has lnear dynams when the game tself s symmetr.

4 Agent s aton aton aton a/a /d aton d/ b/b Table : The General Form of a Symmetr Game Proof A two-player two-aton symmetr game an be represented n Table n general. It s obvous to hek that t satsfes the onstrant of u r = u, gven that u = r + r r r, {r,. Thus the theorem holds. 3. ynams Analyss of SA-IGA Prevous seton manly analyzed the dynams of SA-IGA n a qualtatve manner. In ths seton, we move to provde detaled analyss of SA-IGA s learnng dynams n two representatve games: the Prsoner s lemma game Table 3 as a symmetr game example and oordnaton game Table 4 as an asymmetr game example. Spefally we analyze the SA-IGA s learnng dynams by dentfyng the exstng equlbrum ponts, whh provdes useful nsghts nto understandng of SA-IGA s dynams. Theorem 3 The dynams of SA-IGA algorthm under Prsoner s lemma P game have three types of equlbrum ponts: {.,, wr, w, where wr, w < mn T R P S, ;.,, wr, w, where wr, w > max ; 3. p, p, w, w, others { T R, P S The frst and seond types of equlbrum ponts are stable, whle the last s not. We say an equlbrum pont s stable f one the strategy starts lose enough to the equlbrum wthn a dstane δ from t, t wll reman lose enough to the equlbrum pont forever. Agent s R/R S/T T/S P/P Table 3: The Prsoner s lemma Gamewhere T > R > P > S Proof 3 Followng the system of dfferental equatons n Equatons 3, we an express the dynams of SA-IGA n P game as follows: ṗ r = u p + T S w r + S P ṗ = u p r + T S w + S P ẇ r = ε S T p r p ẇ = ε S T p r p 4 where ε = ηw η p >,u = R + P S T. We start wth provng the last type of equlbrum ponts: If there ext an equlbrum eq = p r, p, w r, w T, 4, then we have ṗ eq = and ẇ eq =, {r,. By solvng the above equatons, we have p r = p = S T u w + P S and w = w u r = w. Sne p r, p,, then we have, { T R w r, w > mn T S w r, w < max P S, T S { T R P S, T S T S Then eq = p r, p, wr, w T s an equlbrum. The stablty of eq an be verfed usng theores of non-lnear dynams[6]. By expressng the unonstraned update dfferental equatons n the form of ẋ = Ax + B, we have u T S A = u T S ε S T ε T S ε T S ε S T After alulatng matrx A s egenvalue, then we have λ =, λ = u, λ 3 = u + k and λ 4 = u k, where k s a onstant. Sne there exst an egenvalue λ >, the equlbrum eq s not stable. Next we turn to prove the frst type of equlbrum. In ths ase, we need to put the projeton funton bak sne we are dealng wth boundary ases. If p =, {r,, aordng to the known ondtons, we have w r, w < mn. ombned wth { T R, P S the unonstraned update dfferental equatons, we have lm p p <, then p remans unhanged. And beause p r = p =, then for w [, ], ẇ,, wr, w =, then,, wr, w s an equlbrum. Beause w r, w < mn { T R, P S, there exst a δ >, and a set U eq, δ = { x [, ] 4 x eq < δ, that for x U eq, δ, lm p p <. Thus p wll stablze on the pont of. Also, beause lm w = S T lm p r p = S T lm = t t t then w s also stable, and thus the equlbrum eq s stable. The seond type of equlbrum an be proved smlarly, whh s omtted here. Intutvely, for a P game, from Theorem 3, we know that f both SA-IGA players are ntally suffently soal-frendly the value of w s large than a ertan threshold, then they wll always onverge to mutual ooperaton of,. In other words, gven that the value of w exeeds ertan threshold, the strategy pont of, or, n the strategy spae s asymptotally stable. If both players start wth a low soally-aware degree w s smaller than ertan threshold, then they wll always onverge to mutual defeton of, eventually. For the rest of ases, there exst nfnte number of equlbrum ponts n-between the above two extreme ases, all of whh are not stable. Next we turn to analyze the dynams of SA-IGA playng oordnaton game by dentfyng all equlbrum ponts. The general form of a oordnaton game s shown n Table 4. Intutvely, both Nash equlbra, and, an be part of the equlbrum ponts dependng on the agents soal-aware degrees. Formally we have, Theorem 4 The dynams of SA-IGA algorthm under a oordnaton game have three types of equlbrum ponts:.,, wr, w, wth wr = w = when P > p > s; wr = w = when T < P < p; and s S w r < P S T t w < p t when P = p;

5 Agent s R/r S/s T/t P/p Table 4: The General Form of a oordnaton Game where R > T P > S and r > s p > t.,, wr, w, wth wr = w = when R > r > t; wr = w = when T < R < r; and T t w r < R T S s w < r s when R = r; 3. others non-boundary equlbrum ponts p r, p, wr, w The frst and seond types of equlbrum ponts are stable, whle the last non-boundary equlbrum ponts are not. The defnton of a stable equlbrum pont s the same as Theorem 3. Proof 4 Followng the system of dfferental equatons n Equatons 3, we an express the dynams of SA-IGA n oordnaton game as follows: ṗ r = u r + u u r w r p + r w r + r ṗ = u + u r u w p r + d r d w + d 5 ẇ r = ε [u r u p rp + r p r + d d r p + e] ẇ = ẇ r where ε = ηw η p >,u r = R+P S T >, u = r+p s t >, r = S P, = s p, d r = T P, d = t p, and e = P p. We an see that the dynam of oordnaton game s nonlnear when u r u. We start wth provng the last type of equlbrum ponts frst: If there ext a equlbrum eq = p r, p, wr, w T, 4, then there have ṗ eq = and ẇ eq =, {r,. By lnearzng the unonstraned update dfferental equatons nto the form of ẋ = Ax + B n pont eq = p r, p, wr, w T, we have u r a 3 A = u a 4 εa 3 εa 4 εa 3 εa 4 where u r = u r + u u r w r, u = u + u r u w, r = r w r + r, and d = d r d w + d. The parameters a j are represented as funtons of p r, p, wr and w. Wthout loss of generalty, we set u r u. Beause of u r u >, and wr, w [, ], we have u r [ u+ur,u r ] and u [u, u+ur ], whh means u > u >. After alulatng matrx A s egenvalue n Matlab, we have an egenvalue λ =, a egenvalue λ wth ts real part Re λ >, an egenvalue λ 3 wth Re λ 3 < and an egenvalue λ 4 lose to. Sne there exsts an egenvalue λ >, the equlbrum eq s not stable[6]. Next we turn to prove the frst type of equlbrum. In ths ase, we need to put the projeton funton bak sne we are dealng wth boundary ases. For the ase P > p > s, we have V dv eq > V so eq, thus ẇ r eq > and ẇ eq <, whh means w r and w wll keeps w r = and w =. Beause ṗ r eq = s p+s P < and ṗ eq = t p <, then p r and p wll keeps p r = and p =. Aordng to the ontnuty theorem of dfferental equatons [7],,,, s a stable equlbrum. The ase p > P > T an be proved smlarly, whh s omtted here. For the ase P = p, we have V dv = V so, then ẇ r eq = ẇ eq = ε Vr dv Vr so =. Beause T t w < p t, we have ṗ r = T t w + t p <. Beause s S w r < P S, we have ṗ = s S w + S P <. Aordng to the ontnuty theorem of dfferental equatons,,, wr, w s a stable equlbrum. The stablty of the seond type of equlbrum ponts an be proved smlarly, whh s omtted here. 4 A Pratal Algorthm In SA-IGA, eah agent needs to know the poly of ts opponent and the payoff matrx, whh are usually not avalable before a repeated game starts. Based on the dea of SA-IGA, we relax the above assumptons and propose a pratal multagent learnng algorthm alled Soally-Aware Poly Gradent Asent SA-PGA. The overall flow of SA-PGA s shown n Algorthm. In SA-PGA, eah agent only needs to observe the payoffs of both agents by the end of eah round. Algorthm SA-PGA for player : Let α, and δ p, δ w, be learnng rates. : Intalze Q dv a, Q op a,q a, w.5, π a. A 3: repeat 4: Selet aton a A aordng to mxed strategy π wth sutable exploraton. 5: Observng reward r and ts opponent s reward r, Q dv a α Q dv a + αr, Q op a α Q op a + αr, 6: Q a w Q dv a + w Qop a, 7: Average payoff V = a A π aq a 8: for eah aton a A do 9: π a π a + δ p Q a V s : end for : π Π [π ] : V dv = a A π a Q dv a 3: V op = a A π a Q op a 4: V so = V dv + V op 5: w w + δ w V dv V so 6: untl the repeated game ends In SA-IGA, we know that agent s poly the probablty of seleton eah aton s updated based on the partal dervatve of the expeted value V, whle the soal atttude w s adjusted aordng to the relatve value of V dv and V so. Here n SA-PGA, we frst estmate the value of V dv and V op usng Q-values, whh are updated based on the mmedate payoffs reeved durng repeated nter. Spefally, eah agent keeps a reord of the Q-value of eah aton for both ts own and ts opponent Q dv and Q op Lne. Both Q-values are updated followng Q-learnng update rules aordngly by the end of eah round Lne 5. The overall Q-value of eah agent s alulated as the weghted average of Q dv and Q op weghted by ts soal atttude w Lne 6. Based on the Q-values, we estmate the value of V n SA-IGA as the expeted Q-value over all gven the urrent poly Lne 7. However, V s smply an estmated value nstead of a funton whh annot be dfferentated.

6 To obtan the dervatve of V wth respet to dfferent, we estmate t as the dfferene between eah aton s Q-value and the expeted Q-value over all the value of V Lne 9. Agent s probablty of seletng an aton s updated n the dreton of the estmated dervatve of the aton s expeted value Lne 8-. After that, agent s poly s mapped bak to the vald probablty spae Lne. Smlarly, the expeted ndvdual payoff and ts opponent s payoff when agent plays poly π are estmated based on ts urrent poly and Q-values Lne -3. The value of V so s alulated as the average between V dv and V op Lne 4. Fnally, the soal atttude of agent s updated n the same way as we ntrodued n SA-IGA based on the estmated V -values Lne 5. The updatng dreton of w s estmated as the dfferene between V dv and V so. 5 Expermental Evaluaton We start the performane evaluaton wth analyzng the learnng performane of SA-PGA under two-player two-aton repeated games. In general a two-player two-aton game an be lassfed nto three ategores[8]: ategory : r r r r r r r r > or r r r r >. In ths ase, eah player has a domnant strategy and thus the game only has one pure strategy NE. ategory : r r r r r r r r < and r r r r < and r r r r r r >. In ths ase, there are two pure strategy NEs and one mxed strategy NE. ategory 3: r r r r r r r r < and r r r r < and r r r r r r <. In ths ase, there only exsts one one mxed strategy NE. where r r j and r r j are payoffs of player r and player respetvely when player r takes aton whle player takes aton j. We selet one representatve game for eah ategory for llustraton. 5. ategory For ategory, we onsder the P game as shown n Table. In ths game, both players have one domnant strategy, and, s the only pure strategy NE, whle there also exsts one soally optmal outome, under whh both players an obtan hgher payoffs. Fgure a show the learnng dynams of the pratal SA-PGA algorthm playng the P game. The x-axs p represents player s probablty of playng aton and the y-axs p represents player s probablty of playng aton. We randomly seleted ntal poly ponts as the startng pont for the SA-PGA agents. We an observe that the SA-PGA agents are able to onverge to the mutual ooperaton equlbrum pont startng from dfferent ntal poles. Fgure b llustrates the learnng dynams predted by the theoretal SA-IGA approah. Smlar to the settng n Fgure a, the same set of ntal poly ponts are seleted and we plot all the learnng urves aordngly. We an see that for eah startng poly pont, the learnng dynams predted from the theoretal SA-IGA s well onsstent wth the learnng urves from smulaton. Ths ndates that we an better understand and predt the dynams of SA-PGA algorthm usng ts orrespondng theoretal SA-IGA model. 5. ategory For ategory, we onsder the G game as shown n Table 5. In ths game, there exst two pure strategy Nash equlbra, and,, and both of them are also soally optmal. Fgure a llustrates the learnng dynams of the pratal SA- PGA algorthm playng a G game. The x-axs p represents player s probablty of playng aton and the y-axs p represents player s probablty of playng aton. Smlar to the ase of P game, ntal poly ponts are randomly seleted as the startng ponts. We an see that the SA-PGA agents an onverge to ether of the aforementoned two equlbrum ponts dependng on the ntal poles they start wth. Fgure b shows the learnng dynams predted by the theoretal SA-IGA approah. Smlar to the settng n Fgure a, we adopt the same set of ntal poly ponts for omparson purpose. All the learnng urves startng from these poly ponts are drawn aordngly. We an observe that for eah startng poly pont, the learnng dynams predted from the theoretal SA-IGA s well onsstent wth the learnng urves obtaned from smulaton. Therefore, the theoretal model an faltate better understandng and predtng the dynams of SA-PGA algorthm. 5.3 ategory 3 Agent s 3/4 / / 4/3 Table 5: oordnaton game ategory The game we use n ategory 3 s shown n Table 6. In ths game, there only exst one mxed strategy Nash equlbrum, whle the pure strategy outome, s soally optmal. Fgure 3a llustrates the learnng dynams of the pratal SA- PGA algorthm playng the game n Table 6. The x-axs p and y-axs p represent player s probablty of playng aton and player s probablty of playng aton respetvely. Smlar to the prevous ases, ntal poly ponts are randomly seleted as the startng ponts. From Fgure 3a, we an see that the SA-PGA agents an always onverge to the soally optmal outome, no matter where the ntal poles start wth. Fgure 3b presents the learnng dynams of agents predted by the theoretal SA-IGA approah. Smlar to the settng n Fgure 3a, we adopt the same set of ntal poly ponts for omparson purpose, and the orrespondng learnng urves are drawn aordngly. From Fgure 3b, we an observe that for eah startng poly pont, the theoretal SA-IGA model an well predt the smulaton results of SA-PGA algorthm. Therefore, better understandng and nsghts of the dynams of SA-PGA algorthm an be obtaned through analyzng ts orrespondng theoretal model. Agent s 3/ 4/4 /3 5/ Table 6: An example game of ategory 3

7 p.. p a SA-PGA n P game p.. p b SA-IGA n P game Fgure : The Learnng ynams of SA-IGA and SA-PGA n P game parameter w r = w = 5, δ p ε =. =., α = and p.. p a SA-PGA n G p.. p b SA-IGA n G Fgure : The Learnng ynams of SA-IGA and SA-PGA n oordnaton game parameter w r = w = 5, δ p =., α = and ε =. p.. p a SA-PGA for the game wth one mx NE p.. p b SA-IGA for the game wth one mx NE Fgure 3: The Learnng ynams of SA-IGA and SA-PGA n game wth one mx NE parameter w r = w = 5, δ p =., α = and ε =.

8 5.4 Performane n General-sum Games In ths seton we turn to evaluate the performane of SA-PGA wth prevous representatve learnng strateges JAL [] and WoLF-PH [4] n two-player s repeated games under self-play. JAL s seleted sne ths algorthm s spefally desgned to enable agents to aheve mutual ooperaton.e., maxmzng soal welfare nstead of neffent NE for games lke prsoner s dlemma. WoLF-PH s seleted as one representatve NE-orented algorthm for baselne omparson purpose. For all prevous strateges the same parameter settngs used n ther orgnal papers are adopted. We use all possble struturally dstnt two-player, two-aton onflt games as a testbed for SA-PGA. In eah game, eah player ranks the four possble outomes from to 4.We use the rank of an outome as the payoff to that player for any outome. We perform the evaluaton under randomly generated games wth strt ordnal payoffs. We perform, nter for eah run and the results are averaged over runs for eah game. We ompare ther performane based on the the followng two rtera: utltaran soal welfare and Nash soal welfare. Utltaran soal welfare s the sum of the payoffs obtaned by the two players n ther onverged state, averaged over randomly generated games. Nash soal welfare s the produt of the payoffs obtaned by two players n ther onverged state, averaged over randomly generated games. Both rtera reflet the system-level effeny of dfferent learnng strateges n terms of the total payoffs reeved for the agents. Besdes, Nash soal welfare also partally reflets the farness n terms of how equal the agents payoffs are. The overall omparson results are summarzed n Table 7. We an see that SA-IGA outperforms the prevous JAL strategy under both rtera. The WoLF-PH strategy s desgned to aheve NE and thus an only aheve the same level of performane as adoptng NE solutons. If a learnng agent s fang selfsh agents that attempt to explot others, one reasonable hoe for an effetve algorthm s to learn a Nash equlbrum. In ths seton, we evaluate the ablty of SA- PGA aganst selfsh opponents. We adopt the same three representatve games used n prevous setons as the testbed and the results are gven n Fgure 4, 5 and 6 respetvely. We an observe that for the P and oordnaton games, the SA-PGA agent an suessfully aheve the orrespondng NE soluton. Ths property s desrable sne t prevents the SA-PGA agent from beng taken advantage by selfsh opponents. The results also show how the soally-aware degree w of SA-PGA agent hanges, whh vares dependng on the game struture. For P and oordnaton game, a SA-PGA agent eventually behaves as a purely ndvdually ratonal entty and one pure strategy NE s eventually onverged to. In ontrast, for the thrd type of game Table 6, a SA-PGA agent behaves as a purely soally ratonal agent and ooperate wth the selfsh agent towards the soally optmal outome, wthout fully explotng the opponent. Ths ndates the leverness of SA-PGA sne hgher ndvdual payoff an be aheved under the outome, than pursung Nash equlbrum,. p. 5 5 N Fgure 4: SA-PGA aganst a selfsh agent for n P gamew r =, p r =. and p = p. 3 4 N Fgure 5: SA-PGA aganst a selfsh agent for n oordnaton gamew r =, p r =. and p = p p w p w p Table 7: Performane omparson wth JAL and WoLF-PH Utltaran Soal Nash Produt Welfare SA-PGA our strategy 7.4 ±.3.76 ±.5 w r = w = 5 JAL [] 6.54 ±.3 87 ± WoLF-IGA [4] ± ± 5 p. w p p 5.5 Aganst Selfsh Agents 3 4 N Fgure 6: SA-PGA aganst a selfsh agent for the game wth only one mx NEw r =, p r =. and p = 6 onluson and Future Work In ths paper, we proposed a novel way of norporatng soal awareness nto tradtonal gradent-asent algorthm to faltate reahng mutually benefal solutons e.g.,, n P game. We frst present a theoretal gradent-asent based poly updatng approah SA-IGA and analyzed ts learnng dynams usng dynamal system theory. For P games, we show that mutual ooperaton, s stable equlbrum pont as long as both agents are strongly soallyaware. For A games, ether of the Nash equlbra, and, an be a stable equlbrum pont dependng on the agents soallyaware degrees. Followng that, we proposed a pratal learnng algorthm SA-PGA relaxng the mpratal assumptons of SA-IGA. Expermental results show that a SA-PGA agent an aheve hgher

9 soal welfare than prevous algorthms under self-play and also s robust aganst ndvdually ratonal opponents. As future work, more testbed senaros e.g., populaton of agents wll be appled to further evaluate the performane of SA-PGA. Another nterestng dreton s to nvestgate how to further mprove the onvergene rate of SA-PGA. REFERENES [] Sheref Abdallah and Vtor Lesser, A multagent renforement learnng algorthm wth non-lnear dynams, Journal of Artfal Intellgene Researh, 5 549, 8. []. Banerjee and S. Sen, Reahng pareto optmalty n prsoner s dlemma usng ondtonal jont aton learnng, AAMAS 7, 9 8, 7. [3] aan Bloembergen, Karl Tuyls, anel Hennes, and Mhael Kasers, Evolutonary dynams of mult-agent learnng: a survey, Journal of Artfal Intellgene Researh, , 5. [4] M. H. Bowlng and M. M. Veloso, Multagent learnng usng a varable learnng rate, Artfal Intellgene, 5 5, 3. [5] Luan Busonu, Robert Babuska, and Bart e Shutter, A omprehensve survey of multagent renforement learnng, Systems, Man, and ybernets, Part : Applatons and Revews, IEEE Trans on, 38, 56 7, 8. [6] oran hakraborty and Peter Stone, Multagent learnng n the presene of memory-bounded agents, Autonomous agents and mult-agent systems, 8, 8 3, 4. [7] Levnson N oddngton E A, Theory of ordnary dfferental equatons, MGraw-Hll, 955. [8] Vnent ontzer and Tuomas Sandholm, Awesome: A general multagent learnng algorthm that onverges n self-play and learns a best response aganst statonary opponents, Mahne Learnng, 67-, 3 43, 7. [9] Junlng Hu and Mhael P Wellman, Nash q-learnng for generalsum stohast games, The Journal of Mahne Learnng Researh, 4, 39 69, 3. [] M. Lauer and M. Renmller, An algorthm for dstrbuted renforement learnng n ooperatve mult-agent systems, n IML, pp ,. [] M. Lttman, Markov games as a framework for mult-agent renforement learnng, n Proeedngs of the th nternatonal onferene on mahne learnng, pp. 3 38, 994. [] Mhael L Lttman, Frend-or-foe q-learnng n general-sum games, n IML, volume, pp. 3 38,. [3] Laetta Matgnon, Gullaume J Laurent, and Nadne Le Fort-Pat, Independent renforement learners n ooperatve markov games: a survey regardng oordnaton problems, The Knowledge Engneerng Revew, 7, 3,. [4] Rob Powers and Yoav Shoham, Learnng aganst opponents wth bounded memory., n IJAI, volume 5, pp. 87 8, 5. [5] Eduardo Rodrgues Gomes and Ryszard Kowalzyk, ynam analyss of multagent q-learnng wth ε-greedy exploraton, n Proeedngs of the 6th Annual Internatonal onferene on Mahne Learnng, pp AM, 9. [6] Leond P Shlnkov, Andrey L Shlnkov, mtry V Turaev, and Leon O hua, Methods of qualtatve theory n nonlnear dynams, volume 5, World Sentf,. [7] Satnder Sngh, Mhael Kearns, and Yshay Mansour, Nash onvergene of gradent dynams n general-sum games, n Proeedngs of the Sxteenth onferene on Unertanty n artfal ntellgene, pp ,. [8] Karl Tuyls, Peter JanT Hoen, and Bram Vanshoenwnkel, An evolutonary dynamal analyss of mult-agent learnng n terated games, Autonomous Agents and Mult-Agent Systems,, 5 53, 6. [9] Karl Tuyls, Katja Verbeek, and Tom Lenaerts, A seleton-mutaton model for q-learnng n mult-agent systems, n Proeedngs of the seond nternatonal jont onferene on Autonomous agents and multagent systems, pp AM, 3. []. J.. H. Watkns and P.. ayan, Q-learnng, Mahne Learnng, 79 9, 99. [] hongje Zhang and Vtor R Lesser, Mult-agent learnng wth poly predton, n Proeedngs of the Twenty-Fourth AAAI onferene on Artfal Intellgene, pp ,.