Solving Extensive Form Games

Transcription

1 Chapter 8 Solving Extensive Form Games 8.1 The Extensive Form of a Game The extensive form of a game contains the following information: (1) the set of players (2) the order of moves (that is, who moves when) (3) the players payo s as a function of the moves that were made (4) the players sets of actions for each move they have to make (5) the information of each player afore each move he has to make (6) probability distributions over any exogenous events Extensive Form Presentation of a Game, Some Terminology A game in extensive form starts at an initial decision node at which player i makes a decision. Each of the possible choices by player i is represented by a branch. At the end of each branch is another decision node at which player j has to make a choice; again each choice is represented by a branch. This can be repeated until no more choices are made. 119

2 120 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES We then reach the end of the game, represented by terminal nodes. At each terminal node, we list the players payo s arising from the sequence of moves leading to that terminal node. We are in a game of perfect information if all players know in which decision node they are when making a choice. The set of all decision nodes between which a player cannot distinguish are called an information set. Hence, in games of perfect information each information set contains exactly one decision node. Note: Any simultaneous move game can be represented as a game in extensive form with imperfect information. Example: (The Battle of Sexes) The battle of the sexes with normal form representation 2 L R 1 U 8,2 0,0 D 0,0 2,8 can alternatively be represented as a game in extensive form Example: "Macho" game

3 8.1. THE EXTENSIVE FORM OF A GAME 121 Take the same payo vectors. Assume that player 1 moves rst. Normal form presentation of this game in extensive form: s 1 2 fu; Dg s 2 2 fl j U ^ L j D, L j U ^ R j D; R j U ^ L j D, R j U ^ R j Dg Denote LL: L j U ^ L j D LR: L j U ^ R j D RL: R j U ^ L j D RR: R j U ^ R j D The normal form presentation of the game is then LL LR RL RR U 8,2 8,2 0,0 0,0 D 0,0 2,8 0,0 2,8 where Player 1 is the row player and Player 2 is the column player De nition of a game in extensive form A game in extensive form consists of the following items:

4 122 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES (1) a nite set of nodes X, a nite set of actions A and a nite set of players N (2) a function p : X! X [? assigning a single immediate predecessor of each node x: [ p(x) is non-empty for all x 2 X except for one node, designated the initial node x 0 ; p(x 0 ) =?. The immediate successor of x is then s(x) = p 1 (x): s(x) is empty i node x is a terminal node (s(x) =?), s(x) is nonempty i node x is a decision node. The set of all predecessors and successor of node x is found by iterating p and s until p(~x) = x 0 and s(e~x) =? where ~x is a predecessor of x and e~x a successor of x ] (3) a function : X n fx 0 g! A giving the action that leads to any noninitial node x from its immediate predecessor p(x) with the property that if x 0 ; x 00 2 s(x) and x 0 6= x 00 then (x 0 ) 6= (x 00 ): The choice set available at decision node x is c(x) = fa 2 A j a = (x 0 ) for x 0 2 s(x)g (4) A collection of information sets H and a function H : X! H assigning each decision node x to an information set H(x) 2 H. That is, the information sets in H form a partition of X. The function H satis es that, if H(x) = H(x 0 ) then c(x) = c(x 0 ). In words, all decision nodes assigned to a single information set have the same choices available. Choices available at information set H are written as C(H) = fa 2 A j a 2 c(x) for x 2 Hg :

5 8.1. THE EXTENSIVE FORM OF A GAME 123 (5) A function : H! N [ f0g assigning each information set in H to the player (or to Nature, denoted as player 0) who moves at the decision nodes in that set.the collection of player i information is denoted by H i = fh 2 H j i = (H)g : (6) A function : H 0 A! [0; 1] assigning probabilities to actions at information sets where Nature moves and satisfying (H; a) = 0 if a =2 C(H) and H 2 H 0 and P (H; a) = 1 for all H 2 H 0 : a2c(h) A collection of payo functions u = fu 1 (:); :::; u n (:)g assigning utilities to the players for each terminal node that can be reached, u i : T! R where T is the set of terminal nodes A game in extensive form is speci ed by T = fx 2 X j s(x) =?g E = fx ; A; N; p(:); (:); H; H(:); (:); (:); (u i (:)) i g " predecessor [ Finite number of actions, players and moves only for expositional convenience.] A strategy for player i is a function s i : H i! A such that s i (H) 2 C(H) for all H 2 H i " choices available at information set H In words, a strategy of player i de nes an action for each information set of player i. A strategy is a complete contingent plan. It speci es actions at information sets that may not be reached during the actual play of the game.

6 124 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES 8.2 Nash Equilibrium Note that each strategy pro le (s i (:); :::; s n (:)) leads to a terminal node (if Nature does not move) or a probability distribution over terminal nodes if Nature moves. We can thus rewrite (expected) utility functions to depend on strategy pro- les, denoted by U. We can then de ne a Nash equilibrium in the game E as a strategy pro le (s i (:); :::; s n(:)) such that for all i U i (s i (:); s i(:)) U i (s i (:); s i (:)) for all s i (H) 2 C(H) for all H 2 H i : We can as well consider mixed strategies i (:), which are randomizations over pure strategies. The notion of Nash equilibrium then generalizes similar to Chapter 5. Example:(The Battle of the Sexes) We already analyzed the NE of this game. To nd Nash equilibria it is convenient to work with the normal form. Example: ("The Macho Game") LL LR RL RR U 8,2 8,2 0,0 0,0 D 0,0 2,8 0,0 2,8 To support DR as an equilibrium player 2 has to play R in the information set of player 2 that would be reached if player 1 played U. This is the threat of player 2 to choose the bad outcome (0,0). This threat is not credible because after the choice U a rational player should choose L. Hence, we can reject the equilibrium (D,RR) [and also the equilibrium (U,LL)] as containing threats that are not credible. We may therefore want to consider a stronger equilibrium concept that excludes threats that are not credible.

7 8.3. SUBGAME PERFECT NE, BACKWARD INDUCTION Subgame Perfect Nash Equilibrium, Backward Induction De nition 1 A subgame of an extensive form game game with the following properties: E is a subset of the A subgame starts with a single decision node. It contains exactly this decision node and all of its successors. If a decision node x is in the subgame, then all x 0 2 H(x) are also in the subgame. Remark: The game as a whole is a subgame. Example 7: (The Battle of the Sexes) This game has only one subgame, the whole game. Remark: Any simultaneous move game has only one subgame. Example ("Macho game"): This game has three subgames: the whole game

8 126 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES the game starting at x 0 the game starting at x 00 Remark: If a game E is a game of perfect information there exists a subgame for each x 2 X that starts at x: The general idea to solve for a nite game is to exclude Non-Nash behavior, that is, we require players to play a Nash equilibrium in each subgame. The solution procedure for a nite game (of perfect information) is called backward induction. With perfect information only sequential rationality is required. For this, we start at decision nodes that have only terminal nodes as successors. If player i decides at this node x then he chooses action a i that maximizes his payo at this decision node. All other branches a i that leave the decision node x can be eliminated. We can then move the tree upward to eliminate more branches. Example: Sequential rationality implies that players choose UL along the equilibrium path. Proposition 2 (Zermelo s Theorem) Every nite game of perfect information has a pure strategy equilibrium that can be derived through backward

9 8.3. SUBGAME PERFECT NE, BACKWARD INDUCTION 127 induction (equivalent to subgame perfect NE as de ned below). If no player has the same payo s at any two terminal nodes, there exists a unique NE that can be derived through backward induction. Example: (on backward induction) Unique solution by backward induction. For nite games (with perfect or imperfect information) we can generalize the procedure by considering Nash equilibria. The generalized backward induction procedure works as follows: 1. Identify all NE of the nal subgames (those that have no other subgames nested within). 2. Select one NE in each those subgames and derive the reduced extensive form in which the subgames are replaced by the equilibrium payo s in the selected NE of those subgames. 3. Repeat stages 1 and 2 for the reduced game until this procedure provides a path of play from the initial node to a terminal node. This procedure selects a subset of the NE of the game in extensive form E. It selects all subgame perfect NE (formal de nition later).

10 128 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES Example: subgame at x 1 Normal form: Player 2 F A N Player 1 F 0,2-1,-2 4,3 A -4,0 3,1 5,0 Out F if In Out A if In In F if In In A if In F A N 2,2 2,2 2,2 2,2 2,2 2,2 0,2-1,-2 4,3-4,0 3,1 5,0 3 Nash equilibria:

11 8.3. SUBGAME PERFECT NE, BACKWARD INDUCTION 129 (Out ^ F if In,F) (Out ^ A if In,F) (In ^ A if In,F) Only (In ^ A if In,A) is subgame perfect. NE at x 1 : (A; A) Reduced "game": Player 1 chooses In in the reduced game! [Playing F in the subgame that starts at x 1 is not credible ] Note that in the above example we require more than sequential rationality. Players anticipate that only NE will be played in the following subgames. De nition 3 A strategy pro le s in an n-player extensive form game E is a subgame perfect Nash equilibrium (SPNE) if it induces a Nash equilibrium in every subgame of E. Observations: If the only subgame is the game as a whole, every NE is a SPNE. (=)a NE of a simultaneous move game is always subgame perfect)

12 130 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES A SPNE induces a SPNE in every subgame of E: Formal justi cation to use the generalized backward induction procedure: Proposition 4 Consider an extensive form game E and some subgame S E of E. Suppose that strategy pro le s s S is an SPNE in subgame E. Let ^E be the reduced game formed by replacing subgame S E by a terminal node with payo s equal to those arising from play of s s. Then (1) In any SPNE s of E in which s s S is an SPNE of E, players reduced strategies ^s of s in the reduced game ^E constitute an SPNE of ^E. (2) If ^s is an SPNE of ^E, then the strategy pro le s that speci es the moves in s s at information sets belonging to S E and that speci es the moves as described by ^s at information sets not belonging to S E is an SPNE of E. Remarks: Remark 1: Special class of dynamic games E are nitely repeated simultaneous move t t games N. If N has a unique Nash equilibrium t, then there exists a unique subgame perfect equilibrium of the multistage game E. It consists of each player playing t i in stage t regardless of previous play.

13 8.3. SUBGAME PERFECT NE, BACKWARD INDUCTION 131 Remark 2: If certain variables x can be changed faster than some other variables y then, modelling such a situation, the choice of x comes at a later stage than the choice of variables y. Remark 3: In games of perfect information with a nite upper bound of moves on a path and continuous action spaces, one nds subgame perfect equilibria by substituting the last player s choice through its best response thus reducing the game by one stage and by repeating this process until one reaches the initial node. Example: Stackelberg duopoly with quantity choice P (q) = 1 q zero marginal costs of production we can consider quantity choices q i 2 [0; 1] Stage 1: Player 1 chooses q 1 Stage 2: Player 2 chooses q 2 1 (q 1 ; q 2 ) = q 1 (1 q 1 q 2 ) 2 (q 1 ; q 2 ) = q 2 (1 q 1 q 2 ) At stage 2, player 2 chooses its best response given q 1 r 2 (q 1 ) = arg max q 2 2 (q 1 ; q 2 )

14 132 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES 1 q 1 2q 2 = 0 () r 2 (q 1 ) = 1 q 1 2 Anticipating the reaction of player 2 at stage 2, player 1 maximizes 1 (q 1 ; r 2 (q 1 )) with respect to q 1 1 q 1 max q 1 1 q 1 q 1 2[0;1] 2 1 q 1 1 (q 1 ; r 2 (q 1 )) = q in subgame perfect equilibrium f.o.c.: 1 2 q 1 = 0 =) q 1 = 1 2 q 2 = 1 4 p = = 1 8 ; 2 = 1 16 Leader obtains higher pro ts than follower (Cournot competition). 8.4 Bargaining Games Rubinstein s Sequential Bargaining Game 1 Euro to be split between 2 players, player 1 and player 2. Players make alternate o ers. Player 1 starts, proposes to split 1 Euro, (s 1 ; 1 s 1 ). Player 2 accepts or rejects. If he accepts payo s are paid out. Otherwise we enter period 2. Player 2 now makes a proposal (s 2 ; 1 s 2 ) and player 1 can accept or reject. If he accepts, payo s (s 2 ; 1 s 2 ) are paid out otherwise we enter period 3 where it is again the turn of player 1. This game continues until an agreement is reached, possibly for an in nite number of periods.

15 8.4. BARGAINING GAMES 133 Players are impatient. They rather prefer a payo in earlier period. This is captured by a discount factor ; 0 < < 1: Game of Perfect Information Consider rst a bargaining game with a nite number of periods: player makes an o er, player 2 can make a countero er or accept. If player 1 does not accept, the countero er payo s (s; 1 s) will be paid out. This game we can solve by backward induction. Consider 3 periods. In period 3 we assume that player 1 can receive s which is worth s in period 2. Hence, if player 2 makes an o er in period 2 of less than s player 1 will reject. =) s 2 s Hence, player 2 proposes (s; 1 s) : [note that 1 s > (1 s)]

16 134 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES With this player 2 can guarantee a payo of 1 s in period 2 for himself. This payo is worth (1 s) in period 1. Hence, in period 1 player 1 has to o er at least (1 s) to player 2 to make him accept the o er. 1 s 1 (1 s) Hence, player 1 proposes (1 (1 s); (1 s)) : In the unique subgame perfect equilibrium of this game player 1 proposes (1 (1 s); (1 s)) in period 1 and player 2 accepts. Consider now Rubinstein s in nite bargaining game. Player 1 makes a proposal in odd periods, player 2 in even periods. Observation: If we reach any odd period which is not t = 1; then the subgame starting at this point is identical to the game starting at t = 1 (with payo s valued at the corresponding point in time). [stationarity of the game] Changing the idea of the players this holds for all periods. s 1 denotes the largest payo that player 1 gets in any SPNE. This is the largest amount player 2 can expect for himself in period 2: s 2 = s 1 (valued at period 1). =) the smallest payo that player 1 gets in any SPNE is Claim: s 1 = 1 s 1 () 1 = s (*) s s 1 1 s 1 (**) Reason: The amount 1 is the smallest payo player 2 can get in period 2. s Hence, player 2 only accepts if he receives at least 1 in period 2 which leaves s s 1 1 s 1 for player 1. =) s 1 1 s 1 = s 1 + s 1 s 1 (**) (*), s 1 s 1 s 1 s 1 =) s 1 = s 1 =) Player 1 s payo in SPNE is uniquely determined, denote it by s 1: From (*) we know that s 1 = 1 s 1 () s 1 = 1 1+ =) s 2 = 1+

17 8.5. INFINITELY REPEATED GAMES AND FOLK THEOREMS 135 Hence, we have the following result Proposition 5 The Rubinstein bargaining game has a unique SPNE. In equilibrium, player 1 receives payo 1 and player 2 receives payo Remark: As players become more and more patient, each player receives 1. 2 lim!1 u i () = In nitely Repeated Games with Observable Actions and Folk Theorems In a repeated game we distinguish between the payo that accrues to the stage game and the overall payo that is obtained when playing the repeated game. The stage-game payo function is denoted by g i : A! R where A = i2n A i. Denote the space of probability distributions over A. A mixed action of player i is denoted by i. Let a t (a t 1; :::; a t n) be the actions that are played in period t, where t = 0; 1; 2; :::. In a game with observable actions players observe the realized action at the end of each period (in applications this may mean that they can make the correct inferences although they may not observe the other players actions directly); this is common knowledge. Hence in period t 1 the game has a history h t = (a 0 ; a 1 ; :::; a t 1 ) which is common knowledge among players. In period t = 0 the game has started; the null history is denoted by h 0. Let H t be the space of all possible histories in period t. (Note that players only observe actual choices; they do not observe mixed actions.) A pure strategy s i for player i in the repeated game is a sequence of maps, fs t ig t=0;1;2;:::. Here s t i map possible period-t histories h t to actions a i, s t i : H t! A i. A mixed (behavior) strategy i in the repeated game is a sequence of maps t i from H t to i. Interpretation of in nite horizon: in each period players think that the game continues with positive probability. Speci cation of payo function for the in nitely repeated game: use discount factor and consider discounted sum: 1X u i = E (1 ) t g i ( t (h t )) t=0

18 136 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES where 1 is a normalization factor. In each period begins a subgame. For any strategy pro le and history in period t, h t, compute each player s expected discounted payo at period t (measured in period-t units), E (1 ) 1X t g i ( (h )) =t This is called the continuation payo. Remark: If is a Nash equilibrium of the stage game, then the stategies each player i plays i in each period independent of the history are a Nash equilibrium. Implication: Repeated play of a game does not decrease the set of equilibrium payo s. We will do two things. First, we look at NE of the in nitely repeated game. Then we look at SPNE. Let us rst consider what individually rational and feasible payo in the in nitely repeated game are. With respect to individual rationality we de ne player i s reservation utility or minmax value as v i = min i max i g i ( i ; i ) This is the payo player i obtains when the other players minimize player i s payo, while player i maximizes his payo given this behavior. Clearly, player i s payo is at least v i in any Nash equilibrium of the stage game. Remark: Player i s payo is at least v i in any Nash equilibrium of the in nitely repeated game, irrespective of the level of the discount factor. A strategy pro le that gives payo s less than the minmax value for at least one player violates individual rationality for some players and thus can be ruled out on a priori reasons as an equilibrium. Next consider feasible payo s. We have to careful since the set of feasible payo s in the stage game (and thus in the repeated game for small discount factors) need not be convex. The reason is that many convex combinations of pure-strategy payo s cannot be obtained by independent randomizations but require strategies to be correlated (as we have seen earlier in our analysis of correlated equilibrium, e.g. in the battle of the sexes ). As we say earlier, we can use a public randomization device to obtain a convex set of payo s. In :

19 8.5. INFINITELY REPEATED GAMES AND FOLK THEOREMS 137 the repeated game we need such a randomization device in each period. Let the randomization device in the in nitely repeated game thus be a sequence f! 0 ;! 1 ;! 2 :::g of independent draws from uniform distribution of the [0; 1]- interval whose realization! t is observed at the beginning of period t. The history in period t is then h t = (a 0 ; a 1 ; :::; a t 1 ;! 0 ;! 1 ; :::;! t ) and the space of all histories in period t is denoted by H t. A pure strategy s i is then a sequence of maps s t i : H t! A i. Denote the set of feasible payo s by V = convex hull fvj9a 2 A : g(a) = vg We then obtain our rst folk theorem: Proposition 6 For every vector v 2 V with v i v i for all players i, there exists a < 1 such that for all 2 (; 1) there is a Nash equilibrium of the in nitely repeated game with payo s v. Idea of the proof: punish any deviation by using the minmax stategy. However, in the subgame that starts after such a deviation such behavior does not constritute a Nash equilibrium. Hence, the equilibrium is not subgame perfect. We now turn to the analysis of the SPNE of the in nitely repeated game. Instead of using minmax punishment players can use the weaker punishment in which in response to a deviation all players choose Nash equilibrium actions of the stage game. This gives rise to the following result. Proposition 7 (Friedman, Review of Economic Studies 1971) Let be a Nash equilibrium of the stage game with payo v. Then for any v 2 V with v i v i for all players i, there exists a < 1 such that for all 2 (; 1) there is a subgame perfect Nash equilibrium of the in nitely repeated game with payo s v. Example: In nitely repeated prisoner s dilemma; discount factor 2 L R 1 U -2,-2-10,-1 D -1,-10-6,-6

20 138 CHAPTER 8. SOLVING EXTENSIVE FORM GAMES For su ciently high discount rate the outcome (-2,-2) can be supported as a SPNE. The following strategies will do the job: play always "Don t Confess" (U or L, respectively) unless there has been a deviation in the past by any player, in that case play always "Confess" (D,R) [we use the payo matrix of the prisoner s dilemma; the story cannot be told as nicely as in the one-shot version] such strategies are called "grim trigger strategies" How does it work? 1 Following these strategies gives payo s ( 2) for both players. 1 A deviation gives payo ( 6) 1 1 ( 2) >? ( 6) () 2 > (1 ) 6 () 1 > 5 () > 1 5 Example: Bertrand duopoly (homogeneous goods)[at equal prices demand split equally] For 1 2 any price p 2 [c; pm ] can be the price along the equilibrium path in an SPNE. A di erent type of strategy consists of rewarding the players who punish a deviator by using a minmax stategy for a number of periods and then to reward the punishers. This works if rewarding the punishers does not reward the deviator; this requires the set of feasible payo s to be of su cient dimension. Denote dim V the dimension of the set V. Proposition 8 (Fudenberg and Maskin, Econometrica 1986) Suppose dim V = N. For every vector v 2 V with v i v i for all players i, there exists a < 1 such that for all 2 (; 1) there is a subgame perfect Nash equilibrium of the in nitely repeated game with payo s v. For pure strategies the proof is easy to follow. For mixed strategies it is more involved. Note that the dimensionality condition can be weakened to dim V = N 1.