Rock-Paper-Scissors: An Example of a Multi-agent Dynamical System

Transcription

1 Rock-Paper-Scssors: A Example of a Mult-aget Dyamcal System Krst Lu Departmet of Mathematcs Krst.lu@gmal.com Abstract: I order to study ad uderstad mult-aget systems, I create smulatos of agets playg the game Rock-Paper-Scssors for oe ad two agets cases. Usg the dfferetal equatos from "Stablty ad Dversty Collectve Adaptato" explore how rewards from the evromet ad agets' memory loss affect the probablty dstrbuto of choosg a ew acto for each aget.

2 Itroducto I wated to study a mult-aget system because of the complex dyamcal behavor that arose from the seemly smple two aget system. Computers today ot oly compute for a perso thousads of teratos of a system a few secods but also allow teracto betwee a system ad a perso a way that paper ad pecl could ot accomplsh. My ambtous goal was to observe the dyamcal behavor of two agets four-dmesos ad I saw vsual pytho as a perfect tool to complete such a task. Not oly would the computer create ad graph trajectores a four-dmesoal space for me but the computer would also allow me to test dfferet tal codtos statly for comparso. Workg from the paper Stablty ad Dversty a Collaboratve Dyamc System, I created smulatos that volved a sgle aget ad two agets playg a game of Rock-Paper-Scssors. The smulatos allowed oe to chage the value of parameters ad also exame varous tal codtos. The sgle aget case behaved as expected by covergg to oe of three possble fxed pots depedg o the parameters. O the other had, the smulato volvg two agets was extremely sestve to tal codtos ad chages parameters. Graphg the trajectores four dmesos was abadoed favor of explorg the dfferet lmt cycles ad geeral behavor of the two aget system. Backgroud Mult-aget systems are systems whch there are several depedet agets workg collaboratvely or agast each other order to satsfy a goal. Real world examples of mult-aget systems clude flocks of brds ad sect swarms. The agets my smulatos are defed as autoomous agets. These agets modfy ther behavor based o formato they gather from the evromet ad from other agets. I ths way, agets are ot explctly aware of the evromet they are ad must lear about the evromet through rewards the agets receve from teractos. Agets playg Rock-Paper-Scssors ft the codtos of a mult-aget system sce the agets play depedet of each other ad modfy ther actos (playg ether rock, paper, or scssors as a move) based o the rewards they receve from the evromet ad teractos wth other agets. Whle each acto of the dvdual aget s terestg, vewg the group of agets as a whole allows oe to see other complex behavor ad patters. Ths s terestg because the agets are t workg uso but rather are modfyg ther behavors depedetly based o local teractos wth each other. Dyamcal System The two systems I vestgated were the sgle aget ad two agets systems of agets playg the game Rock-Paper-Scssors. The reaso for studyg the sgle aget case s to ga a better uderstadg of how a aget s affected by dfferet parameters wthout the added complcato of aother aget. For a aget, I looked at the chage the aget's probablty dstrbuto wth respect to tme. To do ths I treated each

3 probablty dstrbuto as a vector wth three compoets sce the game there are three possble actos. The sum of a aget's probablty dstrbuto must equal oe. As a result, the trajectory of the aget's probablty dstrbuto as t chages remas wth a two-dmesoal tragle called a smplex (Fgure ). The rewards a aget receves s also a vector R = ( ε,,) where the rewards are ε for tes, - for loses, ad for ws. Fgure : The tragular smplex for a aget wth three compoets (actos). The equato for a sgle aget s: x = β( R R) + α( H H) x R s the reward gve for the partcular acto (... N) ad N R = xr whch s the et reforcemet averaged over the aget's possble actos. The dfferece betwee the rewards tells the aget how the reward for the curret acto compares to the average of the other rewards. H = log( x ) s the self-formato or degree of surprse whe N acto s take ad H = x H, the Shao etropy. The dfferece serves to make = the probablty dstrbuto uform. The two parameters β ad α cotrol the two dffereces for the rewards ad the self-formato of each acto. The adaptato rate s β 0,. If β > 0, the aget wll crease the probablty of choosg the cotrolled by [ ) acto that led to the hghest reward. I the specal case of β = 0, the aget's choce dstrbuto s uaffected by curret reforcemets ad chooses actos radomly. The α 0,. Whe α > 0, memory has less of a aget's memory loss rate s cotrolled by [ ) effect o the aget's behavor by gvg past reforcemets less effect tha curret reforcemets. I geeral, the probablty dstrbuto becomes more uform. I the specal case of α =, there s total memory loss ad the probablty dstrbuto s uform ad coverges to the Nash equlbrum of,,. The Nash equlbrum represets the pot at whch the aget chooses actos that yeld the hghest rewards for all partes whch are the sgle aget ad the evromet. I the secod specal case of α = 0, the aget has perfect memory ad the aget's behavor wll be govered by β ( R R). The equatos for the two aget case are: x x y y = β [ R R ] + α [ H H ] j Y Y Y Y Y j Y j j = β [ R R ] + α [ H H ] =

4 Here the equatos are the same as the sgle aget case. Oe dfferece s the rewards scheme for two agets. There are two matrces, oe for each aget, whch ε descrbe the rewards based o the agets' teracto wth each other. ad εy B = εy ε Y A = ε ε where ε ad ε Y are tes for Aget ad Aget 2 respectvely. Addtoally, aother dfferece s how the R's ad H's are calculated ad those equatos ca be foud the Sato paper (equatos.). For two agets, the ma dyamcal behavor occurs four-dmesoal space sce the state space for the two agets s twodmesoal; the product of the state spaces gves four dmesos. Methods I used vsual pytho to smulate the chage probablty dstrbuto for a aget (sgle-aget case) or agets (two-aget case). For all smulatos, I used the fourth order Ruge-Kutta tegrato method to solve the dfferetal equatos. I the sgle-aget case, I allowed the user to vary the alpha ad beta values durg the smulato. I beleve the stat feedback ads the user's uderstadg of how those parameters affect where the dstrbutos coverge. The smulato for two-agets also allows the user to vary the alpha ad beta parameters for Aget ad Aget 2. Eve though the epslos caot be vared, addtoal slders ca be added to the smulato. A clear butto s avalable so that the traset trajectores ca be cleared from the scree. Traset trajectores are cluded the dsplay because the tal behavor of the agets s also terestg to watch. Sce there are so may parameter values to choose from, I decded to use specfc parameters from the paper Sato, et al. so that the system observed would be chaotc. These parameters areα = 0.098, β =.0, ε = 0.5, α Y = 0.0, β Y =.0, ad ε Y = 0.3. Wth these values, the smulato's ma fucto was to show the trajectores of the two agets wth varyg tal codtos. Results There were three ma outcomes from the sgle aget case whch were covergece at the vertex, er smplex, ad ceter of the smplex. The trajectory coverged to a vertex of the smplex whe α = 0 ad β > 0. The vertces of the tragle are pots at whch the aget s playg oly oe acto 00% of the tme. Covergg to oe of these vertces meas that the aget s sestve to the curret rewards t receves ad modfes the probablty dstrbuto to maxmze the curret rewards (Fgure 2). Whe the trajectory coverged to a pot sde the smplex, the aget had some memory loss sceα > 0 (Fgure 3). Ths meas that the aget's probablty dstrbuto was becomg uform but rewards from the evromet preveted the trajectory from covergg to the Nash equlbrum. Fally, there was total memory loss whe α = ad the trajectores became uform ad coverged to the ceter of the smplex (Fgure 4). I addto to settg the parameter values before rug the smulato, the user ca

5 chage the values durg the ru also. The result s a fuy lookg curve (Fgure 5) but whle chagg the parameters the user ca see ad uderstad how covergece s affected by β ad α. Fgure 2: Covergece at the smplex's vertex shows the aget reactg strogly to curret rewards ad gog towards a probablty dstrbuto of playg oe acto 00% of the tme. Fgure 4: The aget coverges to the ceter of the smplex due to complete memory loss. Fgure 3: Covergece sde the smplex dcates some memory loss ( α > 0 ) Fgure 5: The trajectory ca be mapulated durg the smulato by chagg the parameter values. I the two agets smulato, covergece was the form of lmt cycles for both agets. Lmt cycles were usually the form of a tragle that was wth the smplex whle other cycles were trefol patters or almost crcular orbts that occurred the mddle of the smplex. The traset phase of the trajectores would sometmes look radom but after usg the "clear" butto the smulato, the true behavor would be revealed. Trajectores were foud to ed a lmt cycle of ether a tragle or a rregular "sprographc" shape that creates a teror space (Fgures 6 ad 7). It s ot kow what other lmt cycle shapes there are f ay others exst. Fgure 6: A tal codto that started chaotc (left) ad eded a tragular lmt cycle (rght). Fgure 7: A dfferet tal codto wth radom behavor (left) but the trajectory settles dow to a "sprographc" cycle (rght) Other codtos start wth o the lmt cycles such as the rregular "sprographc" shape (Fgure 8), the tragular orbt (Fgure 9) ad a almost crcular orbt (Fgure 0). The rregular "sprographc" shape repeats deftely the teror of the smplex, s ot perfectly crcular, ad has a tragular teror space. The tragular orbt seems to develop from tal codtos that le ear the edges of the smplex. Fally the almost crcular orbt appears whe the tal codtos are ear the ceter of the smplex ad are exactly the same value for both agets. I foud that whe the agets had the same

6 value for ther tal codtos, the trajectores would slowly spral to the almost crcular orbt sde the smplex. Fgure 8: Trajectores that started the rregular "sprograph" (left). A teror tragular space ca be see the close-up of oe of the trajectores (rght). Fgure 9: A tragular orbt. Fgure 0: A almost crcular orbt sde the smplex. Aother set of terestg tal codtos are those that started o oe of the smplex's edges (Fgure ). I ths case, the trajectores stay mostly o the edges of the smplex showg that the agets are playg a sub-game by usg oly two of the three actos. However, after a traset perod, the trajectores settle to a tragular lmt cycle wth the smplex. Fgure : Ital codtos o a edge of the smplexes. The trajectores travel from oe edge to aother (left), the the trajectores become radom (mddle), ad fally the trajectores settle to a tragular orbt (rght). Cocluso Although smple, the smulato for the sgle aget case allowed oe to explore how the parameters β ad α affect the dyamcal behavor of a aget eablg the user to uderstad how memory loss ad rewards affect covergece. For the two agets case, eve though the parameters were't vared, as the oe aget case, partal uderstadg of the complcated system was acheved by observg dfferet tal codtos. By chagg the tal codtos, several dfferet lmt cycles were foud. Also the use of the "clear" butto allowed oe to look beyod the terestg traset state to the actual lmt cycles. The ed lmt cycles of the agets show that wth some memory loss, the agets settle to a patter of swtchg betwee the three actos whch results the tragular shapes of the cycles. A more systematc ad mathematcal approach could be used to classfy ad predct the lmt cycles ad explorato of how varyg the parameters whle the smulato s rug would yeld more formato about how agets adapt to oe aother.

7 Bblography Y. Sato, E. Akyama, ad J. P. Crutchfeld, "Stablty ad Dversty Collectve Adaptato," Joural of Theoretcal Bology (2004) submtted.