Dynamics at the Boundary of Game Theory and Distributed Computing

Transcription

1 Dynamics at the Boundary of Game Theory and Distributed Computing Aaron D. Jaggard 1, Neil Lutz 2, Michael Schapira 3, and Rebecca N. Wright 4 1 U.S. Naval Research Laboratory, Washington, DC 20375, USA. aaron.jaggard@nrl.navy.mil 2 Rutgers University, Piscataway, NJ 08854, USA. njlutz@cs.rutgers.edu 3 Hebrew University of Jerusalem, Jerusalem 91904, Israel. schapiram@huji.ac.il 4 Rutgers University, Piscataway, NJ 08854, USA. rebecca.wright@rutgers.edu Abstract. We use ideas from distributed computing and game theory to study dynamic and decentralized environments in which computational nodes, or decision makers, interact strategically and with limited information. In such environments, which arise in many realworld settings, the participants act as both economic and computational entities. We exhibit a general non-convergence result for a broad class of dynamics in asynchronous settings. We consider implications of our result across a wide variety of interesting and timely applications: circuit design, social networks, Internet routing, and congestion control. We also study the computational and communication complexity of testing the convergence of asynchronous dynamics, as well as the effects of limited asynchrony. For uncoupled game dynamics, in which preferences are private inputs, we give new bounds on the recall necessary for self stabilization to an equilibrium. Our work opens a new avenue for research at the intersection of distributed computing and game theory. 1 Introduction Dynamic environments where decision makers repeatedly interact arise in a variety of settings, such as Internet protocols, large-scale markets, social networks, and multi-processor computer architectures. Study of these environments lies at the boundary of game theory and distributed computing. The decision makers are simultaneously strategic entities with individual economic preferences and computational entities with limited resources, working in a decentralized and uncertain environment. To understand the global behaviors that result from these interactions the dynamics of these systems we draw on ideas from both disciplines. The notion of self stabilization to a legitimate state in a distributed system parallels that of convergence to an equilibrium in a game. The foci, however, are very different. In game theory, there is extensive research on dynamics that result from what is perceived as natural strategic decision making (e.g., best- or better-response dynamics, fictitious play, or regret minimization). Even simple heuristics that require little information or computational resources can yield sophisticated behavior, such as the convergence of best-response dynamics to equilibrium points (see [Hart 2005] and references therein). These positive results for simple game dynamics are, with few exceptions (see Sec. 2), based on the sometimes implicit and often unrealistic premise of a controlled environment in which actions are synchronously coordinated. Distributed computing research emphasizes the environmental uncertainty that results from decentralization, but has no notion of natural rules of behavior. It has long been known that environmental uncertainty in the form of both asynchrony [Fischer et al. 1985; Lynch 1989] and arbitrary initialization [Dolev 2000] introduces Preliminary versions of some of this work appeared in Innovations in Computer Science ICS 2011 [Jaggard et al. 2011] and Symposium on Algorithmic Game Theory SAGT 2014 [Jaggard et al. 2014].

2 substantial difficulties for protocol termination in distributed systems. Our work bridges the gap between these two approaches by initiating the study of game dynamics in distributed computing settings. We take the first steps of this research agenda, focusing primarily on systems in which the decision makers, or computational nodes, are deterministic and have bounded recall, meaning that their behavior is based only on the recent history of system interaction. Our main contribution is a general impossibility result (Thm. 4.1) showing that a large and natural class of boundedrecall dynamics can fail to converge in asynchronous environments whenever there are at least two equilibrium points of the dynamics. We apply this result to several domains: convergence of game dynamics to pure Nash equilibria, stabilization of asynchronous circuits, convergence to a stable routing tree of the Border Gateway Protocol (BGP), and more. We begin in Sec. 2 with an overview of related work. Before giving our general model of asynchronous interaction, we describe in Sec. 3 a restricted model in which nodes behaviors are historyless, depending only on the current state of the system. We prove the main non-convergence result separately and directly for this special case, and this version of the result is sufficient for many of our applications. In Sec. 4 we present our full model and give the general version of our main theorem, which we show is essentially tight. The implications of this result for game dynamics are discussed in Sec. 5; we show that the historyless version of the main theorem is equivalent to the non-convergence of asynchronous best-response dynamics in the presence of multiple equilibria. In Sec. 6, we describe applications of our main theorem to asynchronous circuit design, social networks, interdomain routing protocols such as BGP, and congestion control in networks. We explore the impact of asynchrony that is either randomized (rather than adversarial) or bounded on convergence guarantees in Sec. 7. In Sec. 8, we present both communication and computational complexity hardness results for checking whether an asynchronous system will always converge: we show that it is PSPACE-hard and requires exponential communication, even in the historyless case. We consider the case where the decision makers are given private inputs that determine their behavior (i.e., the dynamics are uncoupled [Hart and Mas-Colell 2003]) in Sec. 9. We give both positive and negative results for self stabilization of game dynamics in this environment. In Sec. 10, we prove our main result by using a valency argument (a now-standard technique in distributed computing theory [Lynch 1989; Fich and Ruppert 2003]) to show that decision protocols cannot satisfy a condition we call independence of decisions. We discuss the connections of this work to consensus protocols in Sec. 11, showing that the famous result of Fischer et al. [1985] on the impossibility of resilient consensus similarly follows from the fact that such protocols must, yet cannot, satisfy independence of decisions. Finally, we outline interesting directions for future research in Sec Related Work Our work relates to many ideas in game theory and in distributed computing. Here, we discuss gametheoretic work on the dynamics of simple strategies and on asynchrony, distributed computing work on fault tolerance and self stabilization, and other connections between game theory and computer science (for more, see [Halpern 2003]). We also highlight the application areas we consider. Algorithmic game theory Since our work draws on both game theory and computer science, it may be considered part of the broader research program of algorithmic game theory (AGT), which merges concepts and ideas from those two fields. The three main areas of study in AGT have been algorithmic mechanism design, which applies concepts from computer science to economic mechanism 2

3 design; algorithmic and complexity research on the computation of equilibria; and studying the price of anarchy, which describes the efficiency of equilibria and draws on approximability research [Nisan et al. 2007]. Our work does not fall into any of these categories but creates a new link between game theory and computer science by exploring game dynamics in distributed computing settings. Adaptive heuristics. Much work in game theory and economics deals with adaptive heuristics (see [Hart 2005] and references therein). Generally speaking, this long line of research explores the convergence of simple and myopic rules of behavior (e.g., best-response/fictitious-play/no-regret dynamics) to an equilibrium. However, with few exceptions (see below), such analysis has so far primarily concentrated on synchronous environments in which steps take place simultaneously or in some other predetermined order. In this work, we explore dynamics of this type in asynchronous environments, which are more realistic for many applications. Game-theoretic work on asynchronous environments. Some game-theoretic work on repeated games considers asynchronous moves 5 (see [Lagunoff and Matsui 1997; Yoon 2004], among others, and references therein). Such work does not explore the behavior of dynamics, but has other research goals (e.g., characterizing equilibria, establishing folk theorems). To the best of our knowledge, we are the first to study the effects of asynchrony (in the broad distributed computing sense) on the convergence of game dynamics to equilibria. Fault-tolerant computation. We use ideas and techniques from work in distributed computing on protocol termination in asynchronous computational environments where nodes and communication channels are possibly faulty. Protocol termination in such environments, initially motivated by multiprocessor computer architectures, has been extensively studied in the past three decades [Fischer et al. 1985; Ben-Or 1986; Dolev et al. 1987; Borowsky and Gafni 1993; Herlihy and Shavit 1999; Saks and Zaharoglou 2000], as nicely surveyed in [Lynch 1989; Fich and Ruppert 2003]. Fischer, Lynch and Paterson [1985] showed, in a landmark paper, that a broad class of failure-resilient consensus protocols cannot provably terminate. Intuitively, the risk of protocol non-termination in that work stems from the possibility of failures; a computational node cannot tell whether another node is silent due to a failure or is simply taking a long time to react. Our non-convergence result, by contrast, applies to failure-free environments. In game-theoretic work that incorporated fault tolerance concepts, Abraham et al. [2006] studied equilibria that are robust to defection and collusion. Self stabilization. The concept of self stabilization is fundamental to distributed computing and dates back to Dijkstra [1974] (see [Dolev 2000] and references therein). Convergence of dynamics to an equilibrium in our model can be viewed as the self stabilization of such dynamics (where the equilibrium points are the legitimate configurations). Our formulation draws ideas from work in distributed computing (e.g., Burns distributed daemon model [1987]) and in networking research [Griffin et al. 2002] on self stabilization. Uncoupled dynamics. Hart and Mas-Colell [2003] introduced the concept of uncoupled game dynamics and gave a non-convergence result in a continuous-time setting. In a later paper [2006], they showed that this result held in a discrete-time setting, even in the presence of randomness, and addressed convergence to mixed Nash equilibria by bounded-recall uncoupled dynamics. Babichenko [2010; 2012] investigated the situation when the uncoupled nodes are finite-state automata, as well as completely uncoupled dynamics, in which each node can see only the history of its own actions and payoffs. Young [2009] and Pradelski and Young [2012] gave completely uncoupled 5 Often, the term asynchrony merely indicates that players are not all activated at each time step, and thus is used to describe environments where only one player is activated at a time ( alternating moves ), or where there is a probability distribution that determines who is activated when. 3

4 dynamics that achieve an equilibrium in a high proportion of steps but do not necessarily converge. Hart and Mansour [2010] analyzed the time to convergence for uncoupled dynamics. Applications. We discuss the implications of our non-convergence result across a wide variety of applications, that have previously been studied: convergence of game dynamics (see, e.g., [Hart and Mas-Colell 2003, 2006]); asynchronous circuits (see, e.g., [Davis and Nowick 1997]); diffusion of innovations, behaviors, etc., in social networks (see [Morris 2000] and [Immorlica et al. 2007]); interdomain routing [Griffin et al. 2002; Sami et al. 2009]; and congestion control [Godfrey et al. 2010]. 3 Historyless Dynamic Interaction We begin our exploration of dynamics with historyless dynamics, in which computational nodes react only to the current state of the system. While seemingly very restricted, historyless dynamics capture the prominent and extensively studied best- and better-response dynamics from game theory (as we discuss in Sec. 5). We show that historyless dynamics also encompass a host of other applications of interest, ranging from Internet protocols to the adoption of technologies in social network (as we explain in Sec. 6). In this section we formally define our model of historyless dynamic interaction, and we prove our main theorem in this special case. We present historyless dynamics separately from our general model of asynchronous interaction as an accessible and intuitive introduction to the structures we consider in this work. When we prove the general version of our main theorem (Thm. 4.1) in Sec. 10, we do so in a way that illuminates its kinship to consensus results, but this requires introducing more notation and an additional layer of abstraction. The direct proof of the historyless case given in this section contains many of the core ideas of that proof despite being more straightforward and less burdened by notation. 3.1 Dynamics of Historyless Systems Intuitively, a historyless interaction system consists of a collection of computational nodes, each capable of selecting actions that are visible to the other nodes. The state of the system at any time consists of each node s current action. Each node has a deterministic reaction function that maps states to actions. Definition 3.1. A historyless interaction system (or just historyless system) is characterized by a tuple (n, A, f): The system has n Z + computational nodes, labeled 1,..., n. A = A 1... A n, where each A i is a set called the action space of node i. A is called the state space of the system, and a state is an n-tuple a = (a 1,..., a n ) A. f : A A is a function given by f(a) = (f 1 (a),..., f n (a)), where f i : A A i is called node i s reaction function. We now describe the dynamics of our model: the ways that a system s state can evolve due to interactions between nodes. Informally, there is some initial state, and, in each discrete time step 1, 2, 3,..., a schedule allows some of the nodes to react to the current state. Definition 3.2. A schedule is a function σ : Z + 2 [n] that maps each t to a (possibly empty) subset of the computational nodes. 6 If i σ(t), then we say that node i is activated at time t. 6 [n] denotes {1,..., n}, and for any set S, 2 S is the set of all subsets of S. 4

5 The nodes that are activated in a given timestep react simultaneously; each applies its reaction function to the current state to choose a new action. This updated action is immediately observable to all other nodes. 7 Since the reaction functions are deterministic, an initial state and a schedule completely determine the resulting infinite state sequence; we call these state sequences trajectories. Definition 3.3. Let a 0 A be a state, and let σ be a schedule. The (a 0, σ)-trajectory of the system is the infinite state sequence a 0, a 1, a 2,... such that for every time t Z + and node i [n], { a t fi (a i = t 1 ) if i σ(t) a t 1 i otherwise. (1) We visualize a historyless interaction system as a directed multigraph whose vertices are the states. Each state a has outgoing edges for every possible activation set; the terminus of the edge labeled with a set S [n] is the state that would be reached by starting at a and activating the nodes in S. An example is given in Fig. 1. A trajectory corresponds to a path in this graph. a f 1(a) f 2(a) αα α α αβ β α βα α β ββ β β, {1}, {2}, {1, 2} αα {2} αβ {1, 2} {1} {1} {1, 2} βα ββ, {1}, {2}, {1, 2} {2} Fig. 1. Example of a historyless interaction system with two computational nodes, 1 and 2. Each node has the action space {α, β}, and the reaction functions are given in the table on the left. The stable states are αα and ββ. Our main theorem is an impossibility result, in which we show that an adversarially chosen initial state and schedule can prevent desirable system behavior. But an arbitrary schedule might never let some or all nodes react, or might stop activating them after some time. We limit this adversarial power (thereby strengthening our impossibility result) by restricting our attention to fair schedules, which never permanently stop activating any node. Definition 3.4. A schedule σ is fair if each node i is activated infinitely many times in σ, i.e., {t Z + : i σ(t)} is an infinite set. A trajectory is fair if it is the (a, σ)-trajectory for some state a and fair schedule σ. 7 This model has perfect monitoring. While this is clearly unrealistic in some important real-life contexts (e.g., some of the environments considered in Sec. 6), this restriction only strengthens our main results, which are impossibility results. 5

6 We are especially interested in whether a system s fair trajectories always converge, eventually remaining at a single stable state forever. Definition 3.5. A trajectory a 0, a 1, a 2,... converges to a state b if there exists some T Z + such that, for all t > T, a t = b. Notice that a historyless system can only converge to state that is a fixed point of the function f. We call such a state a stable state of the system; if the system reaches a stable state, then it converges to that state. In fact, convergence to a particular stable state may also be guaranteed by reaching some other states. We say that such states are committed to the stable state. Definition 3.6. A state a is committed to a state b if every fair trajectory from a converges to b. That is, if for every fair schedule σ, the (a, σ)-trajectory converges to b. An uncommitted state is one that is not committed to any state. 3.2 Non-convergence Result for Historyless Systems Our main theorem concerns systems in which the reaction functions are self-independent, meaning that each node ignores its own current action when reacting to the system s state. In discussing self-independent reaction functions, it is convenient to use the notation A i = A 1... A i 1 A i+1... A n, the state space of the system when i is ignored. Similarly, for a state a, a i A i denotes (a 1,..., a i 1, a i+1,..., a n ), i.e., a with i s action omitted. Using this notation, we formally define self independence in historyless systems. Definition 3.7. A reaction function f i in a historyless interaction system is self-independent if there exists a function g i : A i A i such that f i (a) = g i (a i ) for every state a A. In proving the historyless case of the main theorem, we will make repeated use of the following consequence of self independence. Lemma 3.1. In a historyless interaction system with self-independent reaction functions, if two committed states differ in only one node s action, then they are committed to the same stable state. Proof. To see this, let a and a be committed states that differ only in their i th coordinate, for some i [n], as in Fig. 2. Let σ be any fair schedule such that σ(1) = {i}, and consider the (a, σ)- and (a, σ)-trajectories. When node i is activated in the first timestep, it will choose the same action regardless of whether the initial state was a or a, by self independence. This means that both these trajectories have the same second state and are therefore identical after the first timestep. Thus since a and a are both committed, they must be committed to the same stable state. Theorem 3.1 (Special case of main theorem: historyless case). No historyless interaction system with self-independent reaction functions and more than one stable state is convergent. 6

7 a = (a 1,..., a i,..., a n) a = (a 1,..., a i,..., a n) {i} {i} Fig. 2. Activating {i} from two states that differ only in the ith coordinate will have the same outcome. To prove this theorem, we show that if such a system was convergent, then there would be a fair trajectory that never reaches a committed state, contradicting the assumption of convergence. The first step (Lem. 3.2) is to show that an uncommitted state exists. We then show (Lem. 3.3) that from any uncommitted state, it is always possible to activate any given node without visiting a committed state. This means that committed states can be avoided forever on a trajectory that activates every node infinitely many times, i.e., a fair trajectory. Lemma 3.2. Every convergent historyless system with self-independent reaction functions and more than one stable state has at least one uncommitted state. Proof. Suppose instead that every state in the system is committed. Then by inductively applying Lem. 3.1, it would follow that all states must be committed to the same stable state. This is impossible, though, since each stable state is committed to itself. Hence there must be an uncommitted state. Lemma 3.3. In every convergent historyless system with self-independent reaction functions and more than one stable state, for any uncommitted state a and any node i, there is a path from a that activates i and visits no committed state. Proof. Assume for contradiction that there is no such path, that every i-activating path from a reaches a committed state. Let σ be any schedule, and consider the subgraph shown in Fig. 3. The (a, σ)-trajectory is a, u 1, u 2,.... For each time t, v t is the state reached by following the schedule σ for t 1 steps, then activating σ(t) {i}. By self independence, activating {i} from u t or v t leads to same state, which we call w t. For every t Z +, the state v t is reached from a by a path that activates i, so by our assumption v t is a committed state. Let b be the state to which v 1 is committed. We now show that every v t is committed to b. Fix t 2, and suppose that v t 1 is committed to b. Since w t 1 is reachable from v t 1, it is also committed to b. Consider all states that result from activating a set containing i at u t. By assumption, all these states must be committed. As before, Lem. 3.1 can be applied inductively to show that v t must be committed to the same stable state as w t 1, namely b. If σ is a fair schedule, then there is some time t for which i σ(t). For this t, we have u t = v t, so u t is committed to b. Thus for any fair schedule σ, the (a, σ)-trajectory converges to b, contradicting the assumption that a is uncommitted. We conclude that there is an i-activating path from a to another uncommitted state. It follows immediately from these two lemmas that, in a convergent system satisfying the hypotheses of Thm. 3.1, it is be possible to activate each node infinitely many times without ever visiting a committed state. This means that there is a fair trajectory that never reaches a committed state. Since stable states are committed, this contradicts the assumption of convergence, proving the theorem. 7

8 σ(1) a u 1 σ(2) u 2 σ(3) u 3 σ(1) {i} {i} σ(2) {i} {i} σ(3) {i} {i}... v 1 w 1 {i} v 2 w 2 {i} v 3 w 3 {i} Fig. 3. The states in the bottom row are all committed to the same stable state. 4 Interaction systems with history In this section, we present our full model of asynchronous interaction, in which nodes reaction functions may take the system s past states into account in addition the current state. We also state our main theorem, a general impossibility result for convergence of nodes actions under boundedrecall dynamics in asynchronous, distributed computational environments. This theorem is proved in Sec Computational Nodes Interacting Our model for asynchronous interaction with history generalizes the historyless interaction systems of Sec. 3: at every discrete timestep, each node activated by a schedule simultaneously applies its deterministic reaction function to select a new action, which is immediately visible to all other nodes. But whereas the domain of reaction functions in a historyless system is the state space, we now consider reaction functions whose domain is the set of all histories. Definition 4.1. Given a state space A, a history is a nonempty finite sequence of states, H A l, for some l Z +. The set of all histories is A + = l Z + A l. We can now formally define general interaction systems. This definition differs from that of a historyless interaction system only in the domain of the reaction functions. Definition 4.2. An interaction system is characterized by a tuple (n, A, f): The system has n Z + computational nodes, labeled 1,..., n. A = A 1... A n, where each A i is a set called the action space of node i. A is called the state space of the system, and a state is an n-tuple a = (a 1,..., a n ) A. f : A + A is a function given by f(h) = (f 1 (H),..., f n (H)), where f i : A + A i is called node i s reaction function. Our definition of a trajectory needs to be adjusted for these systems. In this context, a trajectory begins with an initial history of arbitrary length rather than at a single initial state. Notice that the initial history might be erroneous; we do not require that it could have emerged from system interaction. Definition 4.3. Let H = (a 0,..., a l 1 ) be a history, for some l Z +, and let σ be a schedule. Then the (H, σ)-trajectory of the system is the infinite sequence a 0, a 1, a 2,... extending H such that for every t l and i [n], { fi (a 0,..., a t 1 ) if i σ(t) a t i = a t 1 i otherwise. (2) 8

9 The definitions of fairness and convergence are as in Defs. 3.4 and 3.5. However, fair trajectories in general interaction systems can converge to a larger class of states than in historyless systems. We call any such state a steady state. Definition 4.4. A state a is a steady state if there exists a history H and a fair schedule σ such that the (H, σ) trajectory converges to a. Unlike stable states in historyless systems, it is possible for the system to visit a steady state without converging to that state. In this sense steady states are not necessarily stable in general interaction systems; the convergence may be fragile and sensitive to small changes in the activation schedule. 4.2 Informational Restrictions on Reaction Functions This framework allows for very powerful reaction functions. We now present several possible restrictions on the information they may use. These are illustrated in Fig. 4. Informally, a node s reaction function is self-independent if it ignores the node s own past and current actions; this generalizes the notion of self independence given in Def Definition 4.5. A reaction function f i is self-independent if node i s own (past and present) actions do not affect the outcome of f i. For a history H = (a 0,..., a l 1 ), we write H i for (a 0 i,..., al 1 i ). Using this notation, f i is self-independent if there exists a function g i : A + i A i such that f i (H) = g i (H i ) for every history H A +. A reaction function has bounded recall if it ignores the distant past. Definition 4.6. Given k Z + and a history H = (a 0,..., a t 1 ) A t with t k, the k-history at H is H k := (a t k,..., a t 1 ), the k-tuple of most recent states. A reaction function f i has k-recall if it only depends on the k-history and the time counter, i.e., there exists a function g i : A k Z + A i such that f i (H) = g i (H k, t) for every time t k and history H A t. A reaction function with bounded recall is stationary if it also ignores the time counter. When it is clear that all reaction functions in a system are stationary and have k-recall, we may refer to a k-history as a history, since it captures all information that is relevant to the nodes future behavior. Definition 4.7. We say that a k-recall reaction function is stationary if the time counter t is of no importance. That is, if there exists a function g i : A k A i such that f i (H) = g i (H k ) for every time t k and history H A t. A reaction function f i is historyless if f i is both 1-recall and stationary. That is, if f i only depends on the nodes most recent actions. Naturally, a system in which every reaction function is historyless may be viewed as a historyless system in the sense of Def For all of these restrictions, we may slightly abuse notation by referring to the restricted-domain function g i, rather than f i, as the node s reaction function. 4.3 General Non-convergence Result We now present a general impossibility result for convergence of nodes actions under bounded-recall dynamics in asynchronous, distributed computational environments. 9

10 a a t 2 1 a t 1 1 a a t 2 1 a t 1 1 a a t 2 1 a t 1 1 a a t 2 2 a t 1 2 a a t 2 2 a t 1 2 a a t 2 2 a t 1 2 a a t 2 3 a t 1 3 a a t 2 3 a t 1 3 a a t 2 3 a t 1 3 Fig. 4. Shading shows the information about past and current actions available to node 3 at time t given different reaction function restrictions. Left: self-independent. Node 3 can see the entire record of other nodes past actions, but not its own. The length of this record gives the current timestamp t. Center: 2-recall. Node 3 can see only the two most recent states. Unless the reaction function is stationary, it may also use the value of the current timestamp. Right: self-independent and historyless. Node 3 can only see other nodes most recent actions and cannot even see the value of the current timestamp. Theorem 4.1 (Main theorem). In an interaction system where every reaction function is selfindependent and has bounded recall, the existence of multiple steady states implies that the system is not convergent. We note that this result holds even if nodes reaction functions are not stationary. A direct proof of Thm. 4.1 would be similar in structure to the proof of Thm In Sec. 10, we prove a more general result, Thm. 10.2, which states that no decision protocol satisfies a condition we call independence of decisions (IoD). We also show that a convergent system with multiple steady states and self-independent, bounded-recall reaction functions would be an IoD-satisfying decision protocol, so Thm. 4.1 is a consequence of Thm An advantage of this approach is that it illuminates the connection between our result and the seminal result of Fischer, Lynch, and Paterson [1985] on the impossibility of resilient consensus. We show in Sec. 11 that a resilient consensus protocol in their model would also be an IoD-satisfying decision protocol, so their result then also follows from Thm Note that system convergence is closely related to self stabilization, which is a guarantee that the system will reach and remaining within a set of legitimate states. For a set L A, we say that a system self-stabilizes to L if for every fair trajectory a 0, a 1, a 2,..., there is some T Z + such that, for every t > T, a t L. Theorem 4.1 precludes self stabilization to any singleton set in systems satisfying its hypotheses. 4.4 Neither Bounded Recall nor Self-independence Alone Implies Non-convergence We show that the statement of Thm. 4.1 does not hold if either the bounded-recall restriction or the self independence restriction is removed. Example 4.1. The bounded-recall restriction cannot be removed. Consider a system with two nodes, 1 and 2, each with the action space {α, β}. The self-independent reaction functions of the nodes are as follows: node 2 always chooses node 1 s action; node 1 will choose β if node 2 s action changed from α to β in the past, and α otherwise. Observe that node 1 s reaction function has unbounded recall: it depends on the entire history of interaction. We make the observations that the system is convergent and has two steady states. Observe that if node 1 chooses β at some point in time due to the fact that node 2 s action changed from α to β, then it will continue to do so thereafter; if, on the other hand, 1 never does so, then from some point in time 10

11 onwards, node 1 s action is constantly α. In both cases, node 2 will have the same action as node 1 eventually, and thus convergence to one of the two steady states, (α, α) and (β, β), is guaranteed. Hence, two steady states exist and the system is convergent nonetheless. Example 4.2. The self independence restriction cannot be removed. Consider a system with two nodes, 1 and 2, each with action space {α, β}. Each node i has a historyless reaction function f i, defined as follows: f i (α, α) = β, and in all other cases the node always (re)selects its own current action (e.g., f 1 (α, β) = α, f 2 (α, β) = β). Observe that the system has three steady states, namely all action profiles but (α, α), yet can easily be seen to be convergent. 5 Asynchronous Game Dynamics Traditionally, work in game theory on game dynamics (e.g., best-response dynamics) relies on the explicit or implicit premise that players behavior is somehow synchronized (in some contexts play is sequential, while in others it is simultaneous). In Sec. 9, we discuss simultaneous play when information about the game is distributed. Here, we consider the realistic scenario that there is no computational center than can synchronize players selection of strategies. We describe these dynamics in the setting of this work and exhibit an impossibility result for best-response, and more general, dynamics. A game is characterized by a triple (n, S, u). There are n players, 1,..., n. Each player i has a strategy set S i. S = S 1... S n is the space of strategy profiles s = (s 1,..., s n ). Each player i has a utility function u i : S R, where u = (u 1... u n ). Intuitively, player i prefers states for which u i is higher. Informally, a player is best-responding when it has no incentive to unilaterally change its strategy. Definition 5.1. In a game U = (n, S, u), player i is best-responding at s S if u i (s) u i (s ) for every s S such that s i = s i. We write s i BRi U (s). A strategy profile s S is a pure Nash equilibrium ( PNE) if every player is best-responding at s. There is a natural relationship between games and the interaction systems described in Sec. 4. A player with a strategy set corresponds directly to a node with an action space, and a strategy profile may be viewed as a state. These correspondences are so immediate that we often use these terms interchangeably. Consider the case of best-response dynamics for a game in which best responses are unique (a generic game): starting from some arbitrary strategy profile, each players chooses its unique best response to other players strategies when activated. Convergence to pure Nash equilibria under best-response dynamics is the subject of extensive research in game theory and economics, and both positive [Rosenthal 1973; Monderer and Shapley 1996] and negative [Hart and Mas-Colell 2003, 2006] results are known. If we view each player i in a game (n, S, u) as a node in an interaction system, then under best-response dynamics its utility function u i effectively induces a self-independent historyless reaction function f i : S i S i, as long as best responses are unique. Formally, f i (a i ) = arg max α S i u i (a 1,..., α,..., a n ). Conversely, any system with historyless and self-independent reaction functions can be described as following best-response dynamics for a game with unique best responses. Given reaction functions 11

12 f 1,..., f n, consider the game where each player i s utility function is given by { 1 if fi (a) = a u i (a) = i 0 otherwise. Best-response dynamics on this game replicate the dynamics induced by those reaction functions. Thus historyless and self-independent dynamics are exactly equivalent to best-response dynamics. Since pure Nash equilibria are fixed points of these dynamics, Thm. 3.1 may be restated in the following form. Theorem 5.1. If there are two or more pure Nash equilibria in a game with unique best responses, then asynchronous best-response dynamics can potentially oscillate indefinitely. In fact, best-response dynamics are just one way to derive reaction functions from utility functions, i.e., to translate preferences into behaviors. In general, a game dynamics protocol is a mapping from games to systems that makes this translation. Given a game (n, S, u) as input, the protocol selects reaction functions f = (f 1,..., f n ), and returns an interaction system (n, S, f). The above non-convergence result holds for a large class of these protocols. In particular, it holds for boundedrecall and self-independent game dynamics, whenever pure Nash equilibria are steady states. When cast into game-theoretic terminology, Thm. 4.1 says that if players choices of strategies are not synchronized, then the existence of two (or more) pure Nash equilibria implies that this broad class of game dynamics are not guaranteed to reach a pure Nash equilibrium. This result should be contrasted with positive results for such dynamics in the traditional synchronous game-theoretic environments. In particular, this result applies to best-response dynamics with bounded recall and consistent tie-breaking rules (studied by Zapechelnyuk [2008]). Theorem 5.2. If there are two or more pure Nash equilibria in a game with unique best responses, then all bounded-recall self-independent dynamics for which those equilibria are fixed points can fail to converge in asynchronous environments. 6 Applications: Circuits, Social Networks, and Routing We present implications of our impossibility result in Sec. 4 for several well-studied environments: circuit design, social networks and Internet protocols. 6.1 Asynchronous Circuits The implications of asynchrony for circuit design have been extensively studied in computer architecture research [Davis and Nowick 1997]. By regarding each logic gate as a node executing an inherently historyless and self-independent reaction function, we show that an impossibility result for stabilization of asynchronous circuits follows from Thm In this setting there is a Boolean circuit, represented as a directed graph G, in which the vertices represent the circuit s inputs and logic gates, and the edges represent the circuit s connections. The activation of the logic gates is asynchronous. That is, the gates outputs are initialized in some arbitrary way, and then the update of each gate s output, given its inputs, is uncoordinated and unsynchronized. A stable Boolean assignment in this framework is an assignment of Boolean values to the circuit inputs and the logic gates that is consistent with each gate s truth table. We say that 12

13 a Boolean circuit is inherently stable if it is guaranteed to converge to a stable Boolean assignment regardless of the initial Boolean assignment. To show how Thm. 3.1 applies to this setting, we model an asynchronous circuit with a fixed input as a historyless interaction system with self-independent reaction functions. Every node in the system has action space {0, 1}. There is a node for each input vertex, which has a constant (and thus self-independent) reaction function. For each logic gate, the system includes a node whose reaction function implements the logic gate on the actions of the nodes corresponding to its inputs. If any gate takes its own output directly as an input, we model this using an additional identity gate; this means that all reaction functions are self-independent. Since every stable Boolean assignment corresponds to a stable state of this system, this instability result follows from Thm. 3.1: Theorem 6.1. If two or more stable Boolean assignments exist for an asynchronous Boolean circuit with a given input, then that asynchronous circuit is not inherently stable on that input. 6.2 Diffusion of Technologies in Social Networks Understanding the ways in which innovations, ideas, technologies, and practices disseminate through social networks is fundamental to the social sciences. We consider the classic economic setting [Morris 2000] (which has lately also been approached by computer scientists [Immorlica et al. 2007]) where each decision maker has two technologies {X, Y } to choose from and wishes to have the same technology as the majority of its friends (neighboring nodes in the social network). We exhibit a general asynchronous instability result for this environment. In this setting there is a social network of users, represented by a connected graph in which users are the vertices and edges correspond to friendship relationships. There are two competing technologies, X and Y. Each user will repeatedly reassess his choice of technology, at timesteps separated by arbitrary finite intervals. When this happens, the user will select X if at least half of his friends are using X and otherwise select Y. A stable global state is a fixed point of these choice functions, meaning that no user will ever again switch technologies. Observe that if every user has chosen X or every user has chosen Y, then the system is in a stable global state. The dynamics of this diffusion can be described as asynchronous best-response dynamics for the game in which each player s utility is 1 if his choice of technology is consistent with the majority (with ties broken in favor of X) and 0 otherwise. This game has unique best responses, and the strategy profiles (X,..., X) and (Y,..., Y ) are both pure Nash equilibria for this game. Thus Thm. 5.1 implies the following result. Theorem 6.2. In every social network with at least one edge, the diffusion of technologies can potentially oscillate indefinitely. 6.3 Interdomain Routing Interdomain routing is the task of establishing routes between the smaller networks, or autonomous systems (ASes), that make up the Internet. It is handled by the Border Gateway Protocol (BGP). We abstract a recent result of Sami et al. [2009] concerning BGP non-convergence and show that this result extends to several BGP-based multipath routing protocols that have been proposed in the past few years. In the standard model for analyzing BGP dynamics [Griffin et al. 2002], there is a network of source ASes that wish to send traffic to a unique destination AS d. Each AS i has a ranking 13

14 function < i that specifies i s strict preferences over all simple (loop-free) routes leading from i to d. Each AS also has an export policy that specifies which routes it is willing to make available to each neighboring AS. Under BGP, each AS constantly selects the best route that is available to it (see [Griffin et al. 2002] for more details). BGP safety, i.e., guaranteed convergence to a stable routing outcome, is a fundamental desideratum that has been the subject of extensive work in both the networking and the standards communities. We now cast interdomain routing into the terminology of Sec. 3 to obtain a non-termination results for BGP as a corollary of Thm Each AS acts as a computational node. The action space of each node i is the set of all simple routes from i to the destination d, together with the empty route. For every state (R 1,..., R n ) of the system, i s reaction function f i considers the set of routes S = {(i, j)r j : j is i s neighbor and R j is exportable to i}. If S is empty, f i returns. Otherwise, f i selects the route in S that is optimal with respect to < i. Observe that this reaction function is deterministic, self-independent, and historyless, and that a stable routing tree is a stable state of this system. Thus Thm. 3.1 implies the following result of Sami et al. Theorem 6.3. [Sami et al. 2009] If there are multiple stable routing trees in a network, then BGP is not safe on that network. Importantly, the asynchronous model of Sec. 3 is significantly more restrictive than that of Sami et al., so the result implied by Thm. 3.1 is stronger. Thm. 4.1 implies an even more general non-safety result for routing protocols that depend on ASes that act self-independently and with bounded recall. In particular, this includes recent proposals for BGP-based multi-path routing protocols that allow each AS to send traffic along multiple routes, e.g., R-BGP [Kushman et al. 2007] and Neighbor- Specific BGP (NS-BGP) [Wang et al. 2009]. 6.4 Congestion Control We now consider the fundamental task of congestion control in communication networks, which is achieved through a combination of mechanisms on end-hosts (e.g., TCP) and on switches/routers (e.g., RED and WFQ). We briefly describe the model of congestion control studied by Godfrey et al. [2010]. There is a network of routers, represented by a directed graph G, in which vertices represent routers, and edges represent communication links. Each edge e has capacity c e. There are n sourcetarget pairs of vertices (s i, t i ), termed connections, that represent communicating pairs of endhosts. Each source-target pair (s i, t i ) is connected via some fixed route, R i. Each source s i transmits at a constant rate γ i > 0. 8 For each of a router s outgoing edges, the router has a queue management, or queuing, policy, that dictates how the edge s capacity should be allocated between the connections whose routes traverse that edge. The network is asynchronous, so routers queuing decisions can be made simultaneously. An equilibrium of the network is a global configuration of edges capacity allocation such that the incoming and outgoing flows on each edge are consistent with the queuing policy for that edge. Godfrey et al. show that, while one might expect flow to be received at a constant rate whenever it is transmitted at a constant rate, this is not necessarily the case. Indeed, 8 This is modeled via the addition of an edge e = (u, s i) to G, such that c e = γ i, and u has no incoming edges. 14

15 Godfrey et al. present examples in which connections throughputs can potentially fluctuate ad infinitum, never converging an equilibrium. We now model such a network as a historyless interaction system to show that every network with multiple equilibria can oscillate indefinitely. The computational nodes of the system are the edges. The action space of each edge e intuitively consists of all possible ways to divide traffic going through e between the connections whose routes traverse e. More formally, for every edge e, let N(e) be the number connections whose paths go through e. Then e s action space is A e = {( x 1,..., x N(e) ) : each xi 0 and Σ i x i c e }. Edge e s reaction function f e models the queuing policy according to which e s capacity is shared: for every N(e)-tuple of nonnegative incoming flows (w 1,..., w N(e) ), f e outputs an action (x 1,..., x N(e) ) A e such that for every i [N(e)], x i w i a connection s flow leaving the edge cannot exceed its flow entering the edge. These reaction functions are historyless and selfindependent, and an equilibrium of the network is a stable state of this system. Using Theorem 3.1, then, we can obtain the following impossibility result. Theorem 6.4. If there are multiple capacity-allocation equilibria in the network, then dynamics of congestion control can potentially oscillate indefinitely. 7 r-convergence and Randomness We now consider the implications for convergence of two natural restrictions on schedules: r- fairness and randomization. Recall from Sec. 3 that a fair schedule is one in which every node is activated infinitely many times, but without any restriction on the delay between these activations. An r-fair schedule is one in which no node ever has to wait more than r timesteps before its next activation. This allows us to define a weaker form of system convergence, r-convergence. Definition 7.1. For r Z +, a schedule σ is r-fair if every node is activated at least once in every r consecutive timesteps, i.e., for every t Z + and every node i, i σ(t)... σ(t + r 1). A trajectory is r-fair if it results from some r-fair schedule, and a system is r-convergent if every r-fair trajectory converges. 7.1 Snakes in Boxes and r-convergence. Theorem 3.1 deals with convergence and not r-convergence, and thus does not impose restrictions on the number of consecutive time steps in which a node can be nonactive. What happens if there is an upper bound r on this number? We now show that if r < n 1, then sometimes convergence of historyless and self-independent dynamics is achievable, even in the presence of multiple steady states (and so our impossibility result breaks). Example 7.1. A system that is r-convergent if and only if r < n 1. Consider a system with n 2 nodes, 1,..., n, each with the action space {α, β}. Nodes historyless and self-independent reaction functions are as follows. For every i [n], f i (a) = α if a i = α n 1 and β otherwise. Observe that there exist two stable states: α n and β n. 15

16 Observe that if r n 1, then the following oscillation is possible. Initially, only node 1 s action is β and all other nodes actions are α. Then, nodes 1 and 2 are activated and, consequently, node 1 s action becomes α and node 2 s action becomes β. Next, nodes 2 and 3 are activated, and thus 2 s action becomes α and 3 s action becomes β. Then nodes 3 and 4 are activated, then 4 and 5, and so on, traversing all nodes over and over again in cyclic order. This goes on indefinitely, never reaching one of the two stable states. Observe that each node is activated at least once within every sequence of n 1 consecutive time steps. If r < n 1, however, then convergence is guaranteed. To see this, observe that if at some point in time there are at least two nodes whose action is β, then convergence to β n is guaranteed. Clearly, if each node s action is α, then convergence to α n is guaranteed. Thus, an oscillation is possible only if, in each time step, exactly one node s action is β. Observe that, given our definition of nodes reaction functions, this can only be if the activation sequence is (essentially) as described above, i.e., exactly two nodes are activated at a time. Observe also that this kind of activation sequence is impossible for r < n 1. What about r > n? We use a classical combinatorial result due to Abbott and Katchalski regarding the size of a snake-in-the-box in a hypercube to address this question. Definition 7.2. A chordless path in a hypercube Q n is a path P = (v 0,..., v k ) such that for each v i, v j on P, if v i and v j are neighbors in Q n then v j {v i 1, v i+1 }. A snake in a hypercube is a simple chordless cycle. Theorem 7.1. [Abbott and Katchalski 1991] Let t Z + be sufficiently large. Then, the size S of a maximal snake in the z-hypercube Q z is at least λ 2 z for some λ 0.3. Using this theorem, we show that some systems are r-convergent for exponentially large r, but are not convergent in general. Theorem 7.2. Let n Z + be sufficiently large. There exists an integer r = Ω(2 n ) and a interaction system G with n nodes, in which each node i has two possible actions and each f i is historyless and self-independent, such that G is r-convergent but not (r + 1)-convergent. Proof. Let the action space of each of the n nodes be {0, 1}. Consider the possible action profiles of nodes 3,..., n, i.e., the set {0, 1} n 2. Observe that this set of actions can be regarded as the (n 2)-hypercube Q n 2, and thus can be visualized as the graph whose vertices are indexed by the binary (n 2)-tuples and such that two vertices are adjacent iff the corresponding (n 2)-tuples differ in exactly one coordinate. Hence, the size of a maximal snake in the Q n 2 hypercube is Ω(2 n ). Let S be a such a snake. Without loss of generality, we can assume that 0 n 2 is on S (otherwise we can rename nodes actions so as to achieve this). For nodes i = 1 and 2, the historyless and self-independent reaction functions are f i (a) = 0 when a i = 0 n 1 and 1 otherwise. Before giving the reaction functions for nodes 3,..., n, we describe an orientation of the edges in Q n 2, on which those functions will be based. Orient the edges in S to form a cycle. For any edge that has exactly one endpoint in S, orient the edge toward S. An example is given in Fig. 5. Orient all other edges arbitrarily. For each i = 3,..., n, this orientation defines a function g i : Q n 2 {0, 1}, and the historyless and self-independent reaction functions are defined as follows, for every state a = (a 1,..., a n ). { 1 if a1 = a f i (a) = 2 = 1 g i (a 3,..., a n ) otherwise. 16