Learning in Mean Field Games

Transcription

1 Learning in Mean Field Games P. Cardaliaguet Paris-Dauphine Joint work with S. Hadikhanloo (Dauphine) Nonlinear PDEs: Optimal Control, Asymptotic Problems and Mean Field Games Padova, February 25-26, In the occasion of Martino s Bardi 60th birthday P. Cardaliaguet (Paris-Dauphine) Learning in MFG 1 / 17

2 Description of the MFG equilibrium The MFG system is a Nash equilibrium for a nonatomic game in continuous time where the payoff is of optimal control type. Each tiny agent knows the initial density m 0 of all agents and forecasts that this probability density will be (m(t, )) t 0 in the future. He solves his optimal control problem accordingly. When all the agents play optimality, their probability density ( m(t, )) t 0 evolves in time. Nash equilibrium means that the original forecast was correct : m(t, ) = m(t, ) for all t. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 2 / 17

3 This heuristic yields to the MFG system : where (MFG) (i) t u u + H(x, Du) = f (x, m(t)) in [0, T ] R d (ii) t m m div(md ph(x, Du)) = 0 in [0, T ] R d (iii) m(0, ) = m 0, u(t, x) = g(x, m(t )) in R d H : R d R d R is a smooth Hamilonian, convex in the second variable, The coupling functions f, g : R d P(R d ) R are smooth and regularizing. Some references : Early work by Lasry and Lions (2006) and Huang, Caines and Malhame (2006) Similar models in the economic literature (Aiyagari ( 94), Krusell, Smith ( 98)) P. Cardaliaguet (Paris-Dauphine) Learning in MFG 3 / 17

4 Basic results of the MFG system Existence of MFG equilibria : under general conditions if the maps f and g are regularizing" (Lasry-Lions) Uniqueness : under monotonicity conditions on f and g (Lasry-Lions) : (f (x, m) f (x, m ))d(m m ) 0, (g(x, m) g(x, m ))d(m m ) 0. R d R d Link with differential games with finitely many players. from the MFG system to the N player differential games Many contributions (Huang-Caines-Malahmé, Carmona-Delarue,...) from Nash equilibria of N player differential games to the MFG system. LQ differential games (Bardi, Bardi-Priuli) Open loop NE (Fischer, Lacker), Closed loop NE (C.-Delarue-Lasry-Lions). P. Cardaliaguet (Paris-Dauphine) Learning in MFG 4 / 17

5 In the talk, we address the question : How come the players actually play an MFG equilibrium? Related to the question of learning in classical game theory. (see, e.g., Fudenberg-Levine ( 98)) Basic learning procedure : the fictitious play (Brown ( 51), Shapley ( 64), Monderer-Shapley ( 96)) We prove the convergence of this learning procedure for potential MFG (in the spirit of potential games", Monderer-Shapley ( 96)) Throughout the talk, we work with periodic boundary conditions, i.e., in T d = R d /Z d. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 5 / 17

6 In the talk, we address the question : How come the players actually play an MFG equilibrium? Related to the question of learning in classical game theory. (see, e.g., Fudenberg-Levine ( 98)) Basic learning procedure : the fictitious play (Brown ( 51), Shapley ( 64), Monderer-Shapley ( 96)) We prove the convergence of this learning procedure for potential MFG (in the spirit of potential games", Monderer-Shapley ( 96)) Throughout the talk, we work with periodic boundary conditions, i.e., in T d = R d /Z d. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 5 / 17

7 Potential MFG Outline 1 Potential MFG 2 Fictitious play in MFG P. Cardaliaguet (Paris-Dauphine) Learning in MFG 6 / 17

8 Potential MFG Derivatives in the space of measures Let P(T d ) be the set of Borel probability measures on T d (endowed with the weak convergence). Definition Let U : P(T d ) R be a map. Directional derivative : It is the map δu δm : P(Td ) T d R s.t., for m, m P(T d ), 1 U(m ) U(m) = 0 T d δu δm ((1 s)m + sm, y)d(m m)(y). Intrinsic derivative : it is the map D mu : P(T d ) T d R d defined by δu D mu(m, y) := D y (m, y) δm For instance, if U(m) = g(x)dm(x), then δu (m, y) = g(y) + cst and DmU(m, y) = Dg(y). T d δm P. Cardaliaguet (Paris-Dauphine) Learning in MFG 7 / 17

9 Potential MFG Potential Mean Field Game A Mean Field Game is called a Potential Mean Field Game if the instantaneous and final cost functions f, g : T d P(T d ) R derive from potentials, i.e., there exists F, G : P(T d ) R such that f (x, m) = δf δg (x, m), g(x, m) = (x, m). δm δm Remark : On can show that a C 1 map f : T d P(T d ) R derives from a potential, if and only if, Reminiscent of Schwarz Theorem. δf δf (y, m, x) = δm δm (x, m, y) x, y Td, m P(T d ). P. Cardaliaguet (Paris-Dauphine) Learning in MFG 8 / 17

10 Potential MFG MFG equilibria as minimizers of a potential We assume that f and g derive from C 1 potential maps F : P(T d ) R and G : P(T d ) R. We consider the optimal control of the Kolmogorov equation : Given an initial condition (t 0, m 0 ) [0, T ] P(T d ), let T T U(t 0, m 0 ) := inf L (x, α(t, x)) m(t, x) dxdt + F (m(t))dt + G(m(T )) (m,α) t 0 T d t 0 where L(x, α) = sup p p, α H(x, p) and where the pair (m, α) solves the Kolmogorov equation t m m div ( αm ) = 0 in [0, T ] T d, m(t 0 ) = m 0 in T d. Proposition (Lasry-Lions) If (m, α) is a minimum for U(t 0, m 0 ), then there exists u such that (u, m) is a solution of the MFG system with ( ) α(t, x) = D ph(x, Du(t, x)). Conversely, assume that f and g are monotone. If (u, m) is the solution of the MFG system and α is given by ( ), then (m, α) is a minimum for U(t 0, m 0 ). P. Cardaliaguet (Paris-Dauphine) Learning in MFG 9 / 17

11 Potential MFG Theorem (C.-Delarue-Lasry-Lions) The map U := U(t, m) is C 1 and solves the Hamilton-Jacobi equation t U(t, m) + H (y, D mu(t, m, y)) dm(y) div [D mu] (t, m, y)dm(y) = F (m) T d T d in [0, T ] P(T d ), U(T, m) = G(m) in P(T d ), Moreover, D mu(t, x, m) = D x U(t, x, m) where U is the classical solution to the master equation, t U x U + H(x, D x U) div y [D mu] dm(y) + D mu D ph(y, D x U) dm(y) = f (x, m) R d R d in [0, T ] R d P(R d ) U(T, x, m) = g(x, m) in R d P(R d ) Remark : (m, α) is optimal for U(t 0, m 0 ) if and only if α = D ph(x, U(t, m)) and t m m div ( md ph(x, U(t, m) ) = 0 in [0, T ] T d, m(t 0 ) = m 0 in T d. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 10 / 17

12 Fictitious play in MFG Outline 1 Potential MFG 2 Fictitious play in MFG P. Cardaliaguet (Paris-Dauphine) Learning in MFG 11 / 17

13 Fictitious play in MFG The fictitious play It runs as follows : The agents share the same initial belief (m 0 (t)) t [0,T ] on the evolution of the population density. If the game has been played n times, then : At the beginning of stage n + 1, the players have observed the same past and share the same belief (m n (t)) t [0,T ] on the evolving density of the population. They compute their corresponding optimal control with value function u n+1 accordingly. When all players actually implement their optimal strategy, the population density evolves in time and the players observe the resulting evolution (m n+1 (t)) t [0,T ]. At the end of stage n + 1 the players update their belief according to the rule (the same for all the players), which consists in computing the average of their observation up to time n + 1. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 12 / 17

14 Fictitious play in MFG This yields to a sequence (u n, m n, m n ) satisfying : t u n+1 u n+1 + H(x, Du n+1 (t, x)) = f (x, m n (t)), t m n+1 m n+1 div(m n+1 D ph(x, Du n+1 )) = 0, m n+1 (0) = m 0, u n+1 (x, T ) = g(x, m n (T )) where m n = 1 n m k. n k=1 and the smooth initial belief (m 0 (t)) t [0,T ] is given. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 13 / 17

15 Fictitious play in MFG Main result Assume H is C 2 with C 1 I d D 2 pp H(x, p) CI d and D x H(x, p), p C ( p ). The maps f and g sufficiently smooth and derive from potentials. The initial measure m 0 has a smooth density. Theorem [C.-Hadikhanloo] Under the previous assumptions : The family {(u n, m n )} n N is uniformly continuous and any cluster point is a solution to the MFG system. In particular, if f and g are monotone, then the whole sequence {(u n, m n )} n N converges to the unique solution of the MFG system. Remarks : The drawback of the result is that players have to know the mechanisms f and g. The structure of proof is inspired by ideas of Monderer-Shapley ( 96) and relies on the potential structure of the game. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 14 / 17

16 Fictitious play in MFG Other results The fictitious play works also for first order Mean Field Games : (MFG) (i) t u + H(x, Du) = f (x, m(t)) in [0, T ] R d (ii) t m div(md ph(x, Du)) = 0 in [0, T ] R d (iii) m(0, ) = m 0, u(t, x) = g(x, m(t )) in R d Because the lack of regularity of the solutions, statements and proofs are completely different. The same procedure can be adapted to N players : any cluster point is then an approximate MFG equilibrium when N is large. P. Cardaliaguet (Paris-Dauphine) Learning in MFG 15 / 17

17 Fictitious play in MFG Conclusion So far, we have explained that the fictitious play mechanism also works for a large class of Mean field games. Open problems Convergence rate. More general models of learning. Learning in Mean field games with common noise (not potential). P. Cardaliaguet (Paris-Dauphine) Learning in MFG 16 / 17

18 Fictitious play in MFG Happy birthday, Martino! P. Cardaliaguet (Paris-Dauphine) Learning in MFG 17 / 17