Lecture 3: Computing Markov Perfect Equilibria

Transcription

1 Lecture 3: Computing Markov Perfect Equilibria April 22, / 19

2 Numerical solution: Introduction The Ericson-Pakes framework can generate rich patterns of industry dynamics and firm heterogeneity. The cost of this richness is analytical intractability. To use this framewor we should be able to solve the model numerically. Following the definition of MPE we need to find values V ( ) and policies P( ) at each ω S s.t. given P( ), the values V ( ) solve the Bellman equations, given value functions V ( ), the policies P( ) satisfy the optimality conditions. Below we discuss how to solve the model numerically for given values of the parameter vector. 2 / 19

3 Numerical solution: Overview The solution algorithm follows Pakes and McGuire (1994) Proposed algorithm is a Gaussian method We want to compute the following functions defined on a set of discrete points V (ω i, ω i ) incumbents value functions V e (ω) entrants value function r(ω i, ω i ) probability of staying for an incumbent r e (ω) entry probability x(ω i, ω i ) investment policy of incumbent firms x e (ω) investment policy of potential entrants 3 / 19

4 Numerical solution: Overview The algorithm stores in memory current expected value and policy functions for each incumbent firm and each potential entrant in each state ω S o. Each iteration circles through the states in some fixed order, updating all policies and expected values for each state every time it visits it. Iterations continue until the changes in the values and policies from one iteration to the next exceed some tolerance criterion. For the method we describe Doraszelski and Pakes (2007) suggest using a unit-free sup norm defined as V l V l V l = max V l (ω) V l 1 (ω) ω S o 1 + V l (ω) and similar for policies. 4 / 19

5 Numerical solution: Values and policies Recall our definition of value functions V (ω i, ω i ) =π(ω i, ω i ) + (1 r(ω i, ω i ))E[φ χ(ω i, ω i, φ) = 0] ( + r(ω i, ω i ) β ) W (ν ω i, ω i )p(ν x(ω i, ω i )) x(ω i, ω i ) ν where W (ν ω i, ω i ) = ω i,η V (ω i + ν η, ω i)q(ω i ω i, ω i )p(η) For this discussion let s use Pr(ν = 1 x i ) = αx i 1 + αx i, ν {0, 1} 5 / 19

6 Numerical solution: Initial guess Set initial values and policies for all states in S o : V 0 (ω i, ω i ) = π(ω i, ω i ) 1 β r 0 (ω i, ω i ) = 1 x 0 (ω i, ω i ) = 0 V e,0 (ω) = 0 r e,0 (ω) = 0 x e,0 (ω) = 0 Note: if you have some information that gives you better initial conditions, you would use them. Iteration determines from V l ( ), x l ( ), r l ( ), V e,l ( ), x e,l ( ), r e,l ( ) V l 1 ( ), x l 1 ( ), r l 1 ( ), V e,l 1 ( ), x e,l 1 ( ), r e,l 1 ( ) 6 / 19

7 Numerical solution: Iterations Go through the following steps (1) Compute {W l ( )} by using policies and expected values from l 1: 1 r l 1 (ω i, ω i ) if ω i = ø, p l 1 (ω i ω i, ω i, η) = r l 1 (ω i, ω i )p(ν i x l 1 (ω i, ω i )) if ω i = ω i + 1 η r l 1 (ω i, ω i )(1 p(ν i x l 1 (ω i, ω i ))) if ω i = ω i η Probability that E entrants enter is ( ) r e,l 1 (E E [ ] E E, ω) E r e,l 1 (ω) E 1 r e,l 1 (ω) The firm s perceived probabilities of next periods values of its competitors state is q l (ω i ω i, ω i, η) = j i p l 1 (ω j ω j, ω j, η) p(ν j x e,l 1 i (ω))r e,l 1 (E, ω). E j=1 7 / 19

8 Numerical solution: Iterations Compute W l (ν ω i, ω i ) = ω i,η V l 1 (ω i + ν η, ω i)q l (ω i ω i, ω i, η)p(η) (2) Calculate x l (ω i, ω i ), r l (ω i, ω i ), and V l (ω i, ω i ): recall optimality ( x i β ) W (ν ω i, ω i ) p(ν x i) 1 = 0 x i 0 x i ν which for the simple case ν {0, 1} is [ ] ) x i (β W l (1 ω i, ω i ) W l p(1 xi ) (0 ω i, ω i ) 1 x i = 0 x i 0 8 / 19

9 Numerical solution: Iterations recall that continuation policy is given by ( χ(ω i, ω i, φ) = arg max(1 χ)φ+χ χ {0,1} β ν W (ν ω i, ω i )p(ν x(ω i, ω i )) x(ω i, ω i ) by using this policy we define the probability that incumbent firm i draws a φ that induces it to remain in the industry as ( ) r l (ω i, ω i ) = F β ν W l (ν ω i, ω i )p(ν x l (ω i, ω i )) x l (ω i, ω i ) ) finally, calculate V l (ω i, ω i ) defined as before by V (ω i, ω i ) =π(ω i, ω i ) + (1 r(ω i, ω i ))E[φ χ(ω i, ω i, φ) = 0] ( + r(ω i, ω i ) β ) W (ν ω i, ω i )p(ν x(ω i, ω i )) x(ω i, ω i ) ν 9 / 19

10 Numerical solution: Iterations (3) Compute V e,l (ω), x e,l (ω), and r e,l (ω) by following similar steps as for the incumbents (you should use W e,l (ν ω) instead of W l (ν ω i, ω i )). Implement procedures (1) - (3) for all ω S o. This completes one iteration. Repeat iterations while the changes in the values and policies from one iteration to the next exceed your tolerance criterion. 10 / 19

11 Numerical solution: Discussion In general, for this class of models, we do not know if this is a converging process. What if not? Perhaps, need to find a fixed point using some non-linear solver. Recall the problem of multiple equilibria. This is another reason why the full solution approach is difficult to apply for estimation. 11 / 19

12 Numerical solution: Computational burden Computational burden is the product of (i) number of points evaluated at each iteration, #S o ; (ii) time per point evaluated; (iii) number of iterations. Number of states With n firms, one variable per firm having k possible values, the number of distinct grid points is k n. Due to symmetry and anonymity state space does not grow exponentially in n. Result (Pakes and McGuire (1994)): The number of distinct vectors (ω 1,..., ω n) with ω i ω i 1 and ω i {1,..., K} is given by ( ) K + n 1 = n (K + n 1)! (K 1)! n! 12 / 19

13 Numerical solution: Computational burden Time per state consists of computing the summation to get continuation values, i.e., relevant elements of {W ( )} and then solving for optimal policies and the probabilities of entry and exit given {W ( )}. For example if there are two firms, one state variable per firm each of which can move up or down by at most 1, then each firm can move to one of K = 3 2 = 9 elements in the next period. If there are n firms continuing from that state and E potential entrants, the total number of states that need to be summed over is K n+e 1 (in case of unordered states). Number of iterations. We don t know much about this. This depends on the application. It may not grow fast in #S o, but grows pretty fast with β, particularly as it approaches / 19

14 Numerical solution: Alleviating computational burden There are several ways to do this Change underlying assumptions of the model (a) Continuous time models, e.g., Doraszelski and Judd (2012), Arcidiacono et al. (2012), or some sort of sequential moves mechanism. (b) Restricting the set of strategies, e.g., Abbring and Campbell (2010) and Abbring et al. (2010) assume that no firm will produce after an older rival exits. Approximating rather than finding exact MPE. (a) Stochastic approximation algorithm by Pakes and McGuire (2001) (b) Oblivious equilibrium by Weintraub et al. (2010) 14 / 19

15 Numerical solution: Stochastic approximation algorithm Focus on a recurrent class of points: do not obtain accurate policies on the entire state space, just on the recurrent set of points. Do not calculate integrals over possible future values but approximate them using averages of their past outcomes. For simplicity assume φ and φ e are fixed numbers and there is only one potential entrant. Incumbent s value function { ( )} V (ω i, ω i W ) = π(ω i, ω i )+max φ, max x i x i + β ν W (ν ω i, ω i )p(ν x i ) Entrant s value function { V (ω e, ω W ) = max 0, φ e + max x e ( x e + β ν W (ν ω e, ω)p(ν x e ) )} With W (ν ω i, ω i ) = ω i,η V (ω i + ν η, ω i W )q(ω i, ω, η)p(η) 15 / 19

16 Numerical solution: Stochastic approximation algorithm Note that once we know {W ( )}, we know equilibrium policies and value functions. Let W (ω) refer to the vector of values of W (ν ω i, ω i ), W = {W (ω) : ω S o }, and let l index iterations. The algorithm circles through the following steps 1. Substitute W l (ω l ) into the incumbents and entrant s Bellman equations and compute optimal policies x(ω l ), χ(ωi l, ω i), l and χ e (ω l ), which would determine the distribution of the next location. 2. Obtain new location ω l+1 using the calculated policy and random draws, i.e., ω l+1 i = { ω l i + ν l+1 i ω e + ν l+1 e η l+1 for incumbents, η l+1 for entrants where ν l+1 i and νe l+1 are draws from p(ν xi l ) and p(ν xe) l and η l+1 is drawn from p(η). Compute V (ω l i + ν η l+1, ω l+1 i W l ) 16 / 19

17 Numerical solution: Stochastic approximation algorithm 3. Update W (ω l ). Let α(ω, l) 0 denote the number of times the state ω = ω l had been visited prior to iteration l and compute W l+1 (ν ωi l, ω i) l 1 = 1 + α(ω, l + 1) V (ωl i + ν η l+1, ω l+1 i W l ) + α(ω, l) 1 + α(ω, l + 1) W l (ν ω i, ω i ) Note that the algorithm repeatedly visits only points in a recurrent class, i.e., we should not expect that it generates precise estimates of the equilibrium W (ω) s associated with ω s that are outside the recurrent class. 17 / 19

18 Numerical solution: Stochastic approximation algorithm Final remarks: The location of the stochastic algorithm will eventually wander into a recurrent class of points and stay there forever, i.e., we will visit each point in R S o more often with the increased number of iterations. The accuracy of the estimates at a point depends on the number of visits to the point. Unlike other algorithms it updates the values and policies only in the recurrent class. In some cases, recurrent class grows linearly in the number of state variables, i.e., stochastic algorithm helps to alleviate curse of dimensionality. It does not require a summation over possible future states to update the information in memory at each point visited. Unlike deterministic updates the MC estimate of W (ω) contains the variance induced by the simulated draws. 18 / 19

19 Abbring, J. and Campbell, J. (2010). Last-in first-out oligopoly dynamics. Econometrica, 78(5): Abbring, J. H., Campbell, J. R., and Yang, N. (2010). Simple markov-perfect industry dynamics. Working paper WP , Federal Reserve Bank of Chicago. Arcidiacono, P., Bayer, P., Blevins, J. R., and Ellickson, P. B. (2012). Estimation of dynamic discrete choice models in continuous time. Unpublished manuscript. Doraszelski, U. and Judd, K. (2012). Avoiding the curse of dimensionality in dynamic stochastic games. Quantitative Economics, 3(1): Doraszelski, U. and Pakes, A. (2007). A framework for applied dynamic analysis in IO. In Armstrong, M. and Porter, R., editors, Handbook of Industrial Organization, volume 3, chapter 30, pages Elsevier B.V. Pakes, A. and McGuire, P. (1994). Computing markov-perfect nash equilibria: numerical implications of a dynamic differentiated product model. RAND Journal of Economics, 25(4): Pakes, A. and McGuire, P. (2001). Stochastic algorithms, symmetric Markov perfect equilibrium, and the curse of dimensionality. Econometrica, 69(5): Weintraub, G. Y., Benkard, L., and Roy, B. V. (2010). Computational methods for oblivious equilibrium. Operations Research, 58(4): / 19