Collective Evolution of Turn-taking Norm in Dispersion Games

Transcription

1 ollective Evolution of Turn-taking Norm in ispersion Games Akira NAMATAME ept. of omputer Science National efense Academy Yokosuka, ,Japan http//

2 Outline of the Talk ollective construction of norms Majority games vs. ispersion games Repeated games on networks Synchronization in games on networks Agents movements for better synchronization 2 - MASH-Brest

3 Social Norm Social norms are rules that are understood by members of a society, that guide and constrain social behavior without the force of laws. Social norms emerge out of social interaction with others; they may or may not be stated explicitly. Any sanctions for deviating from them come from social networks, not the legal system. Social norm Agent Internalisation of social norm Agent behavioural rule 3 - MASH-Brest

4 Norm evelopment Emergence and sustainability of social norms are core issues for the research on social sciences Studies on norm development (Sherif, 1936; Jacobs, 1961) - Social/cultural learning as a general engine for norm development But, we do not have a reasonable theory of norm development. A more fundamental question: -why some rules are sustained as norms, while others are not. Evolutionary perspective - Rules that help us acquire desirable behavior in a given social environment are more likely to be transmitted culturally and maintained as a social norm. 4 - MASH-Brest

5 Research Agenda o human systems inherently exhibit emergent behaviors? - If so, are the behaviors desirable, undesirable, or mixed? - an we devise effective decentralized mechanisms to elicit desired emergent behaviors? Resurgence of collective constructs social norms and conventions - How can we study these collective constructs fruitfully? - Explore adaptive and evolutionary perspective Repeated game as a useful research tool 5 - MASH-Brest

6 Research Map Internal enforcement Preference/Belief Behavioural rule/norm Random network Lattice networks Small-world network AGENT Network topology R1 External enforcement Social interaction Repeated games with strategic complementarity or substitutability R2 R3 6 - MASH-Brest

7 Evolutionary Perspective Evolutionary game is suited for examining norm developments Evolutionary games focus on whether the interaction among individual strategies lead to a stable collective state ( evolutionary equilibrium ) - Represents various behavioral properties of individuals as strategies in an underlying game. - Examines how each strategy performs in the game against other strategies in terms of fitness. - More fit strategies proliferate in the population gradually via social learning (imitation). 7 - MASH-Brest

8 Evolutionary Perspective (cont.) Evolutionary dynamics converge to Nash equilibria of the underlying games, not necessarily desirable outcomes. Nash equilibrium is the phrase of the day, but is it a good solution? an we do better than Nash Equilibrium? Perhaps we want to just learn some good policy in a decentralized manner. Then what? ollective evolution 8 - MASH-Brest

9 <The Folk Theorem> Repeated Games and The Folk Theorem In a repeated interaction, any mutually beneficial outcome can be sustained in an equilibrium. <riticism of The Folk Theorem> The theory of repeated games does not provide a criterion for equilibrium selection The vast of majority of works on repeated games are symmetric games, and the theory of asymmetric games is rather silent 9 - MASH-Brest

10 lassification of yadic Games The simplest strategic games in which each player has just two strategies: dyadic games istinct 12 symmetric 2 2 games [Rapaport and Guyer] After excluding strategically equivalent games,there remain five archetypal games Prisoner s ilemma, oordination game Hawk-ove game, Battle of sexes (hicken) game Leader games I II interaction R T R S S P T P 10 - MASH-Brest

11 Majority Games Prisoner s dilemma oordination Efficiency and equity are achieved at symmetric equilibrium with pure strategy, (, ). Majority games: In N-person situations agents move toward to taking the same action 11 - MASH-Brest

12 ispersion Games Hawk-ove Battle of the Sexes hicken Leader ual asymmetric Nash equilibria with pure strategy, (, ) (,) One symmetric Nash equilibrium with mixed strategy Pareto-efficiency is achieved by alternating taking or. ispersion games: In N-person situations, agents tend to split into two groups at the mixed Nash equilibrium 12 - MASH-Brest

13 Repeated ispersion Games Hawk-ove game Leader game Battle of the Sexes chicken game: I II I II oordinated alternating reciprocity, (, ) (,) is at least better than joint cooperation (, ) I II MASH-Brest

14 Variant Prisoner s ilemma Game I II I II Prisoner s ilemma Variant Prisoner s ilemma T > R > P > S : is dominated by. T + S > 2R : They are better off if one agent cooperates and the other defect MASH-Brest

15 Summary: Turn-Taking is Mutually Benefit in ispersion Games I II R T R S T + S > 2R T S P P More complicated and subtle challenge for turn-taking behavior The players need to evolve a form of alternating reciprocity Initial state (,) (,) (,) What rules will implement alternating reciprocity efficiently? First cooperate while co-player defects (,), and then defects while co-player cooperates (,), and so on MASH-Brest

16 Repeated Games on Networks Types of pair-wise interactions <Majority games> Prisoner s dilemma game oordination game <ispersion games> Hawk-dove game hicken game (battle-of-sexes game) Variant Prisoner s dilemma game Local model Small-world model Random model 16 - MASH-Brest

17 lassification of Social Interactions Strategic complementarity Agents are better off if they take the same action. This type of social interaction is formulated as coordination (majority) games Strategic substitutability Agents are better off if they take the distinct actions. This type of social interaction is formulated as dispersion (minority) games MASH-Brest

18 riteria for esirable Emergence Stability:Nash equilibrium Need to be equilibrium of underlying games. Efficiency:Pareto-optimality Need to be efficient equilibrium Fairness (equity): Need to be equitable equilibrium majority games: there is no conflict among three criteria minority game: there is conflict between efficiency and equity 18 - MASH-Brest

19 onventional Evolutionary ynamics All agents interact each others with fixed rules, and more fitter rules survive in proportion to their success Well-known rules used for prisoner s dilemma game ALL- ALL- RANOM TFT TF2T FRIEMAN JOSS PER- PER MASH-Brest

20 Learning with Some Memories Reactionary action: Memory A t (action at time t) Reactionary action with memory one: ( h t-1 ) A t <Example: Tit-for-Tat > (., efect ) efect (., ooperate ) ooperate Finite Memories: { ( h t-n,, h t-2, h t-1 ) } A t Is a short memory enough or a longer memory may be necessary in more complex games? 20 - MASH-Brest

21 Learning of oupling Rules oupling rule: Strategy choice is driven by joint actions past strategy Own Opp 0 0 # 0 1 # 1 0 # 1 1 # Own: own strategy Opp: opponent s strategy strategy at t Agents decide the next strategy based on the interaction rule which is the combination of the previous strategies of own and opponent. Agents learns strategies marked by # in coupling rule MASH-Brest

22 Previous Research I II <Browning & olman (2004)> Three-move histories Global model: Each agent interact with all others Natural selection: more fitter rules survive The mean payoff rose to MASH-Brest

23 23 - MASH-Brest

24 Results (cont.) Three-move histories xyz, x, y,z : arbitrary payoffs P, R, S, T After hundreds of generations of evolution, terminating in R in the Prisoner s ilemma and hicken simulations: xyr terminating in S or T in the battle of sexes and leader games 24 - MASH-Brest

25 ollective Evolution bit past strategy Own Opp strategy at t # # # # bit : 0 : 1 #: 0 or 1 First Owns Hand Memory of past history Own Strategy One-move history Local model Mutual learning 25 - MASH-Brest

26 Learning Better Rules from Each Other Rule 1 Rule 2 Hellow Rule 1 Rule 2 Rules are updated by imitating over the successful neighbor MASH-Brest

27 oupling Rules with Memory of Previous Round Type 1: (ALL-) Type 9: Type 2: Type 10: Type 3: Type 11: (TFT) Type 4: Type 12: Type 5: Type 13: Type 6: Type 14: Type 7: (PAVLOV) Type 15: (FRIEMAN) Type 8: Type 16: (ALL-) Investigate which rules will prevail through collective evolution 27 - MASH-Brest

28 Simulation Results (1) (1) ilemma game (2) oordination game (3) Hawk-ove game The average payoff per generation Lattice model SW model R model Lattice model SW model R model Lattice model SW model R model All agents get the same payoff at Pareto-efficiency 28 - MASH-Brest

29 oupling Rules Evolved : ilemma, oordination, Hawk-ove Games The rules of 2,500 agents were aggregated into four types These four rules also share common values Bit RuleType Initial Strategy choice Number of agents All evolved rules: 00##1, #:0 or 1 Friedman style rule : (TFT : 00101) 29 - MASH-Brest

30 State Transitions of Rules Evolved (1) ilemma game (2) oordination game (3) Hawk-ove game The phase diagram of the strategy choices between two agents Initial (00) (01) (10) (11) bit 0 0 # # 1 Initial strategy MASH-Brest

31 Simulation Results: ispersion Games (1) Symmetric game (2) Asymmetric game (3) Variant ilemma game The average payoff per generation Lattice model SW model R model Lattice model SW model R model Lattice model SW model R model The average payoff is better than the value at Nash equilibrium, but lower than the value at maximum efficiency 31 - MASH-Brest

32 oupling Rules Evolved: Symmetric ispersion Game The percentages are evenly distributed in all agents at start After hundreds of generations of evolution, The coupling rules of 2,500 agents were aggregated into four types These four rules also share some commonality Bit Rule Type Initial Strategy choice strategy # # # # Number of agents MASH-Brest

33 State Transitions of Evolved Rules :Symmetric ispersion Game <Evolved rules> # # 0 1 # Agents repeat the same strategy if they succeed to gain a higher payoff (case 1) (case 2) Phase diagram of two agents with the same rule Phase diagram of two agents with different rules initial initial MASH-Brest

34 oupling Rules Evolved : Asymmetric ispersion Games The coupling rules of 2,500 agents were aggregated into four types These four rules also share some commonality Bit RuleType initial Strategy choice # Number of agents MASH-Brest

35 oupling Rules Evolved : Asymmetric ispersion Games (cont.) <Evolved rules> # # 1 0 # Agents change their strategy if they succeed to gain a higher payoff (case 1) Phase diagram of two agents with the same rule (case 2) initial initial Phase diagram of two agents with different rules 35 - MASH-Brest

36 Summary: Rule Evolution in Repeated Games Evolution of norms depends on strategic structures Majority games: prisoner s dilemma, coordination game, and hawk-dove game: Rapid evolution of Friedman style rule, rather that TFT, that realize symmetric cooperation ispersion games: symmetric, asymmetric dispersion game, variant prisoner s dilemma game oordinated alternating rule that realizes turn-taking behavior 36 - MASH-Brest

37 Synchronization: oherent motion among the constituent oscillators Prevalent appearance in physics, chemistry, and biology, (synchronously flashing fireflies, pacemaker cells in the heart, neurons in the brain, laser, chemical reactions) 37 - MASH-Brest

38 Synchronization in omplex Networks Synchronization is not a state but a process of adjusting rhythms due to interaction. When subsystems (e.g. people, animals, cells, neurons) synchronize, they also can communicate. There are rich synchronization phenomena in complex networks (self-organized structure formation) hierarchical transitions 38 - MASH-Brest

39 Synchronization in omplex Networks (cont.) 39 - MASH-Brest

40 Algebraic onnectivity of Networks Network A Network B λ 2 = λ 2 = k1 {0, 1} Laplacian matrix Laplacian matrix = egree Adjacency matrix k 2 O {0, 1} k n λ 1 = 0 is always an eigenvalue of a Laplacian matrix λ 2 is called the algebraic connectivity, and is a good measure of synchronization MASH-Brest

41 Synchronization in omplex Networks 41 - MASH-Brest

42 Synchronization in Globally onnected Networks Observation: No matter how large the network is, a globally coupled network will synchronize if its coupling strength is sufficiently strong Good if synchronization is useful 42 - MASH-Brest

43 Synchronization in Locally onnected Networks Observation: No matter how strong the coupling strength is, a locally coupled network will not synchronize if its size is sufficiently large Good - if synchronization is harmful 43 - MASH-Brest

44 Synchronization in Small-World Networks Start from a nearest neighbor coupled network small-world network Add a link, with probability p, between a pair of nodes Good news: A small-world network is easy to synchronize! X.F.Wang and G.R.hen: Int. J. Bifurcation & haos (2001) 44 - MASH-Brest

45 Synchronization in Networked Agents Synchronization is a process of adjusting interactions moving toward for better evolution. With synchronization, they converge to Nash equilibrium, but the payoff is close to Paretooptimal Hellow or or 45 - MASH-Brest

46 Synchronization in Games on Networks What is the impact of the interaction structure on synchronization in games on networks? The lattice networks (locally coupled agents) where the connectivity among agents is mostly reserved foster to promote to synchronization Local model Small-world model Random model 46 - MASH-Brest

47 Agents Movement Internal enforcement AGENT External enforcement Rule A Hellow Rule B Network topology R1 R2 R3 It is not what you know, it s who you know that account. It is important to interact with the right peoples MASH-Brest

48 Why agents move? Rule type 1: 1,1,0,1,0 Rule type 2: 0,1,0,1,1 Rule type 3: 1,1,0,0,1 Rule type 4: 0,1,0,0,1 Achieved payoff: 1.2 Payoff at Pareto-optimal:1.5 issatisfied agents change their sites Location of agents with each rule type The agents of having the same rule cluster together with lower payoff 48 - MASH-Brest

49 esirable Outcomes with Agents Movements Threshold for movement θ= The payoff at Pareto optimal ρ ( 0 <ρ 1 ) Rule of agent movement (1) the payoff at τ < θ move at the next generation (2) the payoff at τ > θ stay at the same site generation τ generation τ MASH-Brest

50 How to etermine Threshold for Movement? threshold θ θ= Payoff at Pareto-optimal ρ (0 <ρ 1 ) average payoff Payoff at Pareto-optimal :( 4 1)/2 = 1.5 The optimal threshold is around 1.45 (ρ=0.85) 50 - MASH-Brest

51 Simulation Results under Agent Mobility Variant P Symmetric game Asymmetric game Pareto optimal:2.0 Rule evolution phase 0.8 <ρ< 0.9 Agent movement phase Pareto optimal:1.5 Pareto optimal:1.0 Rule type 1: 1,1,0,1,0 Rule type 2: 0,1,0,1,1 Rule type 3: 1,1,0,0,1 Rule type 4: 0,1,0,0,1 Location of agents with different rules 51 - MASH-Brest

52 oncluding Remarks (1): Micro-Macro Macro ynamics of Mind and Norms How a particular psychological functioning becomes self-sustaining? sustaining? It becomes self-sustaining sustaining when having the belief is adaptive and this adaptive process leads to better outcomes than the outcome of having an alternative belief. Norms, conventions Interaction Internal models 52 - MASH-Brest

53 New book: World Scientific, MASH-Brest

54 oncluding Remarks(3): Internalization of Norms Social norms emerge out of social interaction, and they are internalized in networked agents. Social norm Inverse problem Emergent behavior Forward problem Repeated social interactions 54 - MASH-Brest

55 Learning Agenda in Other Learners Strategic uncertainty Reinforcement Learning Evolutionary games, Learning in games, ollective evolution Environment uncertainty ecision Theory Game Theory Single-agent Multiple agents 55 - MASH-Brest