Coevolutionary Modeling in Networks 1/39

Transcription

1 Coevolutionary Modeling in Networks Jeff S. Shamma joint work with Ibrahim Al-Shyoukh & Georgios Chasparis & IMA Workshop on Analysis and Control of Network Dynamics October 19 23, 2015 Jeff S. Shamma Coevolutionary Modeling in Networks 1/39

2 Motivation: Networked decision architectures Systems characterized by multiple interacting components Applications: Engineered/Societal/Hybrid Autonomous vehicle teams Swarm robotics Transportation networks Power networks Sensor networks Social networks Network notions: Physical connectivity Information flow Strategic dependency etc Jeff S. Shamma Coevolutionary Modeling in Networks 2/39

3 Coevolutionary modeling x + = F(x, α) α + = G(α, x) (nodal dynamics) (network topology) Jeff S. Shamma Coevolutionary Modeling in Networks 3/39

4 Illustration: Host-Virus Interaction J. Weitz, Quantitative Viral Ecology, 2015 Chapter 5: Coevolutionary Dynamics of Viruses and Microbes Jeff S. Shamma Coevolutionary Modeling in Networks 4/39

5 SIS dynamics under social distancing Fixed contact graph neighbor set: N i Jeff S. Shamma Coevolutionary Modeling in Networks 5/39

6 SIS dynamics under social distancing Fixed contact graph neighbor set: N i Node states: x i (t) {0, 1} Infection rate: Recovery rate: p 01 = 1 j N i (1 β i x j ) p 10 = δ Paarporn, Eksin, JSS & Weitz (2015), Epidemic spread over networks with agent awareness and social distancing. Jeff S. Shamma Coevolutionary Modeling in Networks 5/39

7 SIS dynamics under social distancing Fixed contact graph neighbor set: N i Node states: x i (t) {0, 1} Infection rate: Recovery rate: Social distancing: p 01 = 1 j N i (1 β i x j ) p 10 = δ β i (t) depends on (subset of...) x(t) Paarporn, Eksin, JSS & Weitz (2015), Epidemic spread over networks with agent awareness and social distancing. Jeff S. Shamma Coevolutionary Modeling in Networks 5/39

8 SIS dynamics under social distancing Fixed contact graph neighbor set: N i Node states: x i (t) {0, 1} Infection rate: Recovery rate: Social distancing: p 01 = 1 j N i (1 β i x j ) p 10 = δ β i (t) depends on (subset of...) x(t) Coevolution: Fixed graph with state dependent weights Paarporn, Eksin, JSS & Weitz (2015), Epidemic spread over networks with agent awareness and social distancing. Jeff S. Shamma Coevolutionary Modeling in Networks 5/39

9 Illustration: Averaging dynamics x + i = θx i + (1 θ)avg[x Ni ] Jeff S. Shamma Coevolutionary Modeling in Networks 6/39

10 Illustration: Averaging dynamics x + i = θx i + (1 θ)avg[x Ni ] Exogenous (random) N i (t) Jeff S. Shamma Coevolutionary Modeling in Networks 6/39

11 Illustration: Averaging dynamics x + i = θx i + (1 θ)avg[x Ni ] Exogenous (random) N i (t) State dependent (proximity) N i (t) { } N i (t) = j x i (t) x j (t) ρ Jeff S. Shamma Coevolutionary Modeling in Networks 6/39

12 Time scale separation x + = F(x, α eq (x)) (fast network) Jeff S. Shamma Coevolutionary Modeling in Networks 7/39

13 Time scale separation x + = F(x, α eq (x)) (fast network) α + = G(α, x eq (α)) (fast nodes) Jeff S. Shamma Coevolutionary Modeling in Networks 7/39

14 Outline Network formation games Latent benefits Dynamic reinforcement Establishment costs Jeff S. Shamma Coevolutionary Modeling in Networks 8/39

15 Outline Network formation games Latent benefits Dynamic reinforcement Establishment costs Main message: Coevolution vs equilibrium analysis Jeff S. Shamma Coevolutionary Modeling in Networks 8/39

16 Network formation games Setup: Agents create & sever links with neighbors Links provide benefits but are costly Assume: Unilateral link decisions Arrows indicate flow of benefits Issues: Characterize emergent network topologies Incentivize desirable network topologies Shaping benefit/cost models Adaptation rules Jeff S. Shamma Coevolutionary Modeling in Networks 9/39

17 Setup Elements: Collection of agents/nodes: I Agent i links restricted to neighbors: N i Action set α i A i = 2 N i Jeff S. Shamma Coevolutionary Modeling in Networks 10/39

18 Setup Elements: Collection of agents/nodes: I Agent i links restricted to neighbors: N i Action set α i A i = 2 N i Induced networks: Collection of link choices α = (α 1,..., α I ) defines a graph, G(α) (i j): A path in G(α) (i j) : Length of path (i j) dist G(α) (i, j) = min (i j) G(α) (i j) : Minimum distance from i to j Jeff S. Shamma Coevolutionary Modeling in Networks 10/39

19 Setup Elements: Collection of agents/nodes: I Agent i links restricted to neighbors: N i Action set α i A i = 2 N i Induced networks: Collection of link choices α = (α 1,..., α I ) defines a graph, G(α) (i j): A path in G(α) (i j) : Length of path (i j) dist G(α) (i, j) = min (i j) G(α) (i j) : Minimum distance from i to j Connectivity: (i j) G(α) for all i, j Assume: Connectivity is feasible through N i Jeff S. Shamma Coevolutionary Modeling in Networks 10/39

20 Benefit/cost models Connections model: R i (α) = (s i) G(α) δ dist G(α)(s,i) Reward for all accessed agents direct or indirect Reward may diminish with distance (δ < 1) Jeff S. Shamma Coevolutionary Modeling in Networks 11/39

21 Benefit/cost models Connections model: R i (α) = (s i) G(α) δ dist G(α)(s,i) Reward for all accessed agents direct or indirect Reward may diminish with distance (δ < 1) Truncated favorites model: R i (α) = s N i χ { distg(α) (s, i) K } K is specified benefit radius May lack coordination structure (as with δ < 1) Jeff S. Shamma Coevolutionary Modeling in Networks 11/39

22 Benefit/cost models Connections model: R i (α) = (s i) G(α) δ dist G(α)(s,i) Reward for all accessed agents direct or indirect Reward may diminish with distance (δ < 1) Truncated favorites model: R i (α) = s N i χ { distg(α) (s, i) K } K is specified benefit radius May lack coordination structure (as with δ < 1) Maintenance cost: Number of links C i (α) = κ 0 α i Jeff S. Shamma Coevolutionary Modeling in Networks 11/39

23 Nash Networks Agent utility: u i (α) = u i (α i, α i ) = R i (α) C i (α) Jeff S. Shamma Coevolutionary Modeling in Networks 12/39

24 Nash Networks Agent utility: Best reply set: u i (α) = u i (α i, α i ) = R i (α) C i (α) BR i (α i ) = { } α i ui (α i, α i ) u i (α i, α i ) Jeff S. Shamma Coevolutionary Modeling in Networks 12/39

25 Nash Networks Agent utility: Best reply set: u i (α) = u i (α i, α i ) = R i (α) C i (α) BR i (α i ) = Nash network: α = (α 1,..., α n) satisfies { } α i ui (α i, α i ) u i (α i, α i ) α i BR i (α i) for all i = 1, 2,..., n Jeff S. Shamma Coevolutionary Modeling in Networks 12/39

26 Nash network illustrations: Connections model u i (α i, α i ) = s χ {(s i) G(α)} κ 0 α i (δ = 1) Jeff S. Shamma Coevolutionary Modeling in Networks 13/39

27 Network formation games and dynamics Dynamics: Agents update links in reaction to choices of others Jeff S. Shamma Coevolutionary Modeling in Networks 14/39

28 Network formation games and dynamics Dynamics: Agents update links in reaction to choices of others Best response dynamics: At stage t, random agent i is activated For agent i: α i (t) BR i (α i (t 1)) For remaining agents: α i (t) = α i (t 1) Jeff S. Shamma Coevolutionary Modeling in Networks 14/39

29 Network formation games and dynamics Dynamics: Agents update links in reaction to choices of others Best response dynamics: At stage t, random agent i is activated For agent i: α i (t) BR i (α i (t 1)) For remaining agents: Issues: Convergence? Selection? α i (t) = α i (t 1) Jeff S. Shamma Coevolutionary Modeling in Networks 14/39

30 Context: Learning/evolutionary games Shift of focus: Away from solution concept Nash equilibrium Towards how players might arrive to solution i.e., dynamics Jeff S. Shamma Coevolutionary Modeling in Networks 15/39

31 Context: Learning/evolutionary games Shift of focus: Away from solution concept Nash equilibrium Towards how players might arrive to solution i.e., dynamics The attainment of equilibrium requires a disequilibrium process. Arrow, The explanatory significance of the equilibrium concept depends on the underlying dynamics. Skyrms, Jeff S. Shamma Coevolutionary Modeling in Networks 15/39

32 Literature Monographs: Weibull, Evolutionary Game Theory, Young, Individual Strategy and Social Structure, Fudenberg & Levine, The Theory of Learning in Games, Samuelson, Evolutionary Games and Equilibrium Selection, Young, Strategic Learning and Its Limits, Sandholm, Population Dynamics and Evolutionary Games, Expositions: Hart, Adaptive heuristics, Econometrica, Fudenberg & Levine, Learning and equilibrium, Annual Review of Economics, Marden & JSS, Game theory and distributed control, Handbook of Game Theory, Jeff S. Shamma Coevolutionary Modeling in Networks 16/39

33 Outline Network formation games Latent benefits Dynamic reinforcement Establishment costs Jeff S. Shamma Coevolutionary Modeling in Networks 17/39

34 Latent benefit flow: Motivation Fast network: x + i = θx i + (1 θ)avg j: xj x i ρ[x j (t)] Jeff S. Shamma Coevolutionary Modeling in Networks 18/39

35 Latent benefit flow: Motivation Fast network: x + i = θx i + (1 θ)avg j: xj x i ρ[x j (t)] Fast nodes: Best response dynamics: corresponds to α i (t) BR i (α i (t 1)) α + = G(α, x eq (α)) i.e., immediate realization of benefits/costs (missing x). Jeff S. Shamma Coevolutionary Modeling in Networks 18/39

36 Latent benefit flow: Motivation Fast network: x + i = θx i + (1 θ)avg j: xj x i ρ[x j (t)] Fast nodes: Best response dynamics: corresponds to α i (t) BR i (α i (t 1)) α + = G(α, x eq (α)) i.e., immediate realization of benefits/costs (missing x). Latent benefits: Benefits slow to react to changes in network topology Agent decisions based on myopic considerations Jeff S. Shamma Coevolutionary Modeling in Networks 18/39

37 Latent benefit flow: Dynamics Shorthand: Model: d ij = dist G(α) (j i) b ij (t) = f (b ij (t 1), α i, α i ) { α dij b ij (t 1) + (1 α dij )δ d ij, δ d ij b ij (t 1), = β dij b ij (t 1) + (1 β dij )δ d ij, δ d ij < b ij (t 1) 0 α 1 α 2... < 1 1 > β 1 β Features: Benefit increases lag behind network connectivity Longer distances mean slower response Benefit decreases lag behind network connectivity Longer distances mean faster response Steady state benefits = Static benefits Jeff S. Shamma Coevolutionary Modeling in Networks 19/39

38 Latent benefit flow: Myopic decisions Myopic best response dynamics: α i (t) = arg max f (b ij (t 1), α i, α i (t 1)) κ 0 α i (t) α i j = arg max α i u i (α i, α i (t 1); b i (t 1)) Jeff S. Shamma Coevolutionary Modeling in Networks 20/39

39 Latent benefit flow: Myopic decisions Myopic best response dynamics: α i (t) = arg max f (b ij (t 1), α i, α i (t 1)) κ 0 α i (t) α i j = arg max α i u i (α i, α i (t 1); b i (t 1)) Interpretation: State dependent utility cf: Marden (2012), State Based Potential Games, Automatica Jeff S. Shamma Coevolutionary Modeling in Networks 20/39

40 Latent benefit flow: Myopic decisions Myopic best response dynamics: α i (t) = arg max f (b ij (t 1), α i, α i (t 1)) κ 0 α i (t) α i j = arg max α i u i (α i, α i (t 1); b i (t 1)) Interpretation: State dependent utility cf: Marden (2012), State Based Potential Games, Automatica In case of time-scale separation, reduces to standard best response Jeff S. Shamma Coevolutionary Modeling in Networks 20/39

41 Main results Strict equilibrium: (α i, α i ; B ) u i (α i, α i; b i ) > u i (α i, α i; b i ) Proposition: (α, B ) is strict if for all i and α i α i, (1 α d ij )(δ d ij δ d j S ij )+ (1 β d ij )(δ d ij δ d ij )+κ0 ( α i α i ) > 0. j S + Jeff S. Shamma Coevolutionary Modeling in Networks 21/39

42 Main results Strict equilibrium: (α i, α i ; B ) u i (α i, α i; b i ) > u i (α i, α i; b i ) Proposition: (α, B ) is strict if for all i and α i α i, (1 α d ij )(δ d ij δ d j S ij )+ (1 β d ij )(δ d ij δ d ij )+κ0 ( α i α i ) > 0. j S + Proposition: Strict equilibria are local attractors. Proposition: There exist equilibria of myopic best response dynamics that are not equilibria of the static game for (1 β )δ > κ 0 > (1 α 1 )(δ δ 2 ) i.e., immediate cost outweighs long term benefit. Al-Shyoukh and JSS (2014), A coevolutionary model of strategic network formation. in P. Contucci et al. (eds): Jeff S. Shamma Coevolutionary Modeling in Networks 21/39

43 Latent benefits: Illustration Non-Nash equilibrium: ( ) δ δ 2 α = (1, 0), (1, 1), (0, 1), B α = δ δ δ 2 δ Jeff S. Shamma Coevolutionary Modeling in Networks 22/39

44 Latent benefits: Illustration Non-Nash equilibrium: ( ) δ δ 2 α = (1, 0), (1, 1), (0, 1), B α = δ δ δ 2 δ Sample run of convergence to non-nash: t =1 4 t =2 4 t =3 4 t =4 4 t = t =6 4 t =7 4 t =8 4 t =9 4 t =10 Jeff S. Shamma Coevolutionary Modeling in Networks 22/39

45 Outline Network formation games Latent benefits Dynamic reinforcement Establishment costs Jeff S. Shamma Coevolutionary Modeling in Networks 23/39

46 Stability & multi-agent learning Caution! Single agent learning Multiagent learning Sato, Akiyama, & Farmer, Chaos in a simple two-person game, PNAS, Piliouras & JSS, Optimization despite chaos: Convex relaxations to complete limit sets via Poincare recurrence, SODA, Jeff S. Shamma Coevolutionary Modeling in Networks 24/39

47 Instability and equilibrium selection Different dynamics lead to different outcomes Modified reinforcement learning (trend based) leads to convergence Can modified reinforcement learning lead to non-convergence? Arslan & JSS, Anticipatory learning in general evolutionary games, Jeff S. Shamma Coevolutionary Modeling in Networks 25/39

48 Selective instability Reinforcement learning: x i = action propensities x i (t + 1) = x i (t) + δ(t)(a i (t) x i (t)), δ(t) = u i(a(t)) t + 1 p i (t) = (1 ε)x i (t) + ε N 1 δ std (t) = u i (a(t)) 1 T U i (t) + u i (a(t)) Interpretation: Increased probability of utilized action. Dynamic reinforcement learning: Introduce running average y i (t + 1) = y i (t) + 1 t + 1 (x i(t) y i (t)) p i (t) = (1 ε)π x i (t) + γ(x i (t) y i (t)) + ε }{{} N 1 new term Jeff S. Shamma Coevolutionary Modeling in Networks 26/39

49 Marginal foresight & instability Proposition: The pure NE a has positive probability of convergence iff 0 < γ i < u i(a i, a i) u i (a i, a i ) + 1 u i (a i, a i ), a i a i (as opposed to all pure NE) Proof: ODE method of stochastic approximation. Jeff S. Shamma Coevolutionary Modeling in Networks 27/39

50 Marginal foresight & instability Proposition: The pure NE a has positive probability of convergence iff 0 < γ i < u i(a i, a i) u i (a i, a i ) + 1 u i (a i, a i ), a i a i (as opposed to all pure NE) Proof: ODE method of stochastic approximation. Implication: Introduction of forward looking agent can destabilize equilibria Surviving equilibria = equilibrium selection Jeff S. Shamma Coevolutionary Modeling in Networks 27/39

51 Risk, payoff, and foresight dominance Payoff vs Risk vs Foresight dominance: (2 2 symmetric coordination games): RD & not PD foresight dominance RD & PD & Identical interest foresight dominance RD & PD together foresight dominance Jeff S. Shamma Coevolutionary Modeling in Networks 28/39

52 Risk, payoff, and foresight dominance Payoff vs Risk vs Foresight dominance: (2 2 symmetric coordination games): RD & not PD foresight dominance RD & PD & Identical interest foresight dominance RD & PD together foresight dominance Proposition: The wheel network is foresight dominant. Jeff S. Shamma Coevolutionary Modeling in Networks 28/39

53 Outline Network formation games Latent benefits Dynamic reinforcement Establishment costs Jeff S. Shamma Coevolutionary Modeling in Networks 29/39

54 Establishment costs: Setup As before: Benefit: Maintenance cost: j i δ dist G (j i) κ 0 α i Familiarity vector: x i (A i ) Induced neighbor familiarity, ψ(x i ) Unspecified: How x i evolves... Establishment cost of action α i : κ 1 ψ(α i ) T( ) 1 ψ(x i ) κ 1 penalizes links to unfamiliar neighbors κ 1 encourages mixing G.C. Chasparis and JSS (2013), Network formation: Neighborhood structures, establishment costs, and distributed learning?. Jeff S. Shamma Coevolutionary Modeling in Networks 30/39

55 Establishment cost example Neighbor set: Action set: Action: Subset familiarity: Induced neighbor familiarity: {1, 2, 3} A = {1, 2, 3, 12, 13, 23, 123} α i = 12 ( 0, 0, 0, 1, 0, 0, 0 ) x i = ( 1/2, 0,..., 0, 1/2 ) ψ(α i ) = ( 1, 1, 0 ) ψ(x i ) = ( 1, 1/2, 1/2 ) Establishment cost: κ 1ψ(α i ) T ( ) 1 1 (1 ψ(x i )) = κ /2 1 1/2 ( ) 0 = κ /2 1/2 Jeff S. Shamma Coevolutionary Modeling in Networks 31/39

56 Framework: State-Dependent Games Connections with establishment cost model (C): v i (α, x i ) = δ distg(α)(s,i) ( κ 0 α i + κ 1 ψ(α i ) T (1 ψ(x i )) ) (s i) G(α) Indexed collection of utility functions If κ 1 = 0, then same as before Evolution of x i left unspecified Jeff S. Shamma Coevolutionary Modeling in Networks 32/39

57 Nash Networks, revisited Better reply: α i BR i (α) iff v i ((α i, α i ), α i ) > v i ((α i, α i ), α i ) Presumes familiarity with current action, α i Nash network: BR i (α ) = for all i I v i ((α i, α i), α i ) v i ((α i, α i), α i ) for all alternatives α i α i Strict Nash network: v i ((α i, α i), α i ) > v i ((α i, α i), α i ) for all alternatives α i α i Jeff S. Shamma Coevolutionary Modeling in Networks 33/39

58 Coordination Property Improvement step: IS i (α) BR i (α): v i ((α i, α i ), α i ) > v i ((α i, α i ), α i ) and v i ((α i, α i ), α i) > v i ((α i, α i ), α i ) A better reply that continues to be a better reply with regained familiarity Coordination property: There exists φ : A R such that if α is not a NE, there exists an i I, such that for some α i IS i (α) φ(α i, α i ) > φ(α i, α i ) Implication: Existence of Nash Networks Compare: Resembles weakly-acyclic game Jeff S. Shamma Coevolutionary Modeling in Networks 34/39

59 Structure of Nash Networks Proposition: For κ 1 0, C is a coordination game for δ = 1 and κ 0 + κ 1 < 1 Proof: φ(α) = i I v i(α, α i ) Consequences: C admits Nash networks If α is a Nash network in C, then G(α ) is connected Jeff S. Shamma Coevolutionary Modeling in Networks 35/39

60 Effect of establishment cost Proposition: Compare κ 1 = 0 vs small κ 1 > 0: non-nash network = non-nash network Nash network = strict Nash network Proposition: Compare κ 1 = 0 vs small κ 1 < 0: non-nash network = non-nash network non-strict Nash network = non-nash network strict Nash network = strict Nash network Implication: κ 1 < 0 can eliminate inefficient Nash networks Jeff S. Shamma Coevolutionary Modeling in Networks 36/39

61 Learning Dynamics: Adaptive Play Familiarity: (degree of freedom) Update rule: α i (t) = x i (t + 1) = 1 M M 1 τ=0 α i (t τ) { α i (t 1) if BR i (α(t 1); x i (t)) = ; α i otherwise where α i(t) { α i (t 1) with probability p; BR i (α(t 1); x i (t)), with probability 1 p Jeff S. Shamma Coevolutionary Modeling in Networks 37/39

62 Learning Dynamics: Adaptive Play Familiarity: (degree of freedom) Update rule: α i (t) = x i (t + 1) = 1 M M 1 τ=0 α i (t τ) { α i (t 1) if BR i (α(t 1); x i (t)) = ; α i otherwise where α i(t) { α i (t 1) with probability p; BR i (α(t 1); x i (t)), with probability 1 p Proposition: Converges to Nash network for C. Jeff S. Shamma Coevolutionary Modeling in Networks 37/39

63 In progress: Coevolutionary preferential attachment Standard: New arrival randomly links to node i k i Variation: Reputed degree lags actual degree c i (t + 1) = αc i (t) + (1 α)k i (t) Note: Clock updates upon new arrival to node i Simulations: Increased super hubs Coevolutionary PA Standard PA Model P(k) k Al-Shyoukh, Chasparis, and JSS (2014), Coevolutionary modeling in network formation Jeff S. Shamma Coevolutionary Modeling in Networks 38/39

64 Concluding remarks Network formation games: Latent benefits vs steady state benefits Dynamic reinforcement vs quasi-static reinforcement Establishment costs vs maintenance costs Main message: Coevolution vs equilibrium analysis Jeff S. Shamma Coevolutionary Modeling in Networks 39/39