Explicit solutions of some Linear-Quadratic Mean Field Games

Transcription

1 Explicit solutions of some Linear-Quadratic Mean Field Games Martino Bardi To cite this version: Martino Bardi. Explicit solutions of some Linear-Quadratic Mean Field Games. Networks and Heterogeneous Media, AIMS-American Institute of Mathematical Sciences, 1, 7 (), pp <1.3934/nhm >. <hal-66444> HAL Id: hal Submitted on 3 Jan 1 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

2 Explicit solutions of some Linear-Quadratic Mean Field Games Martino Bardi Dipartimento di Matematica P. e A. Università di Padova via Trieste, 63; I-3511 Padova, Italy bardi@math.unipd.it 7th July 11 Abstract We consider N-person differential games involving linear systems affected by white noise, running cost quadratic in the control and in the displacement of the state from a reference position, and with long-time-average integral cost functional. We solve an associated system of Hamilton-Jacobi-Bellman and Kolmogorov-Fokker-Plank equations and find explicit Nash equilibria in the form of linear feedbacks. Next we compute the limit as the number N of players goes to infinity, assuming they are almost identical and with suitable scalings of the parameters. This provides a quadratic-gaussian solution to a system of two differential equations of the kind introduced by Lasry and Lions in the theory of Mean Field Games [19. Under a natural normalization the uniqueness of this solution depends on the sign of a single parameter. We also discuss some singular limits, such as vanishing noise, cheap control, vanishing discount. Finally, we compare the L-Q model with other Mean Field models of population distribution. 1 Introduction Consider a system of linear stochastic differential equations dx i t = (A i X i t α i t)dt + σ i dw i t, X i = x i, i = 1,..., N, (1) where Wt i is a Brownian motion and αt i is the control of the i-th player of the differential game that we now describe. For each initial vector of positions X = (x 1,..., x N ) we consider for the i-th player the long-time-average cost functional with quadratic running cost [ J i (X, α 1,..., α N 1 T ) := lim inf T + T E R i (αi t) + ( ) T X t X i Q i ( ) X t X i dt, where R i >, Q i is a symmetric matrix, and X i is a given reference position. We wish to study the Nash equilibrium strategies of this N-person game and understand the limit behavior as N + within the theory of Mean Field Games initiated by Lasry and Lions [19,, 1. We recall that this theory is intimately connected to the modeling of economic equilibrium with rational anticipations, following the fundamental contribution of Aumann [4. We refer 1

3 to the recent survey [13 for several applications of Mean Field Games to economics and mathematical finance. In this paper we limit ourselves to the case of 1-dimensional state space for each player, because our goal is to give solutions as explicit as possible to the systems of Hamilton-Jacobi- Bellman (briefly, HJB) and Kolmogorov-Fokker-Plank equations arising in the theory. Thus A i is a given scalar here. However the case of general d-dimensional X i t can be studied similarly via the solution of suitable matrix Riccati equations and will be treated in a forthcoming paper. In Section we define the admissible strategies and introduce the system of N HJB and KFP equations associated to the N-person game, as in [19, 1. Under a generic condition we find explicit quadratic solutions v i for the HJB equations and Gaussian solutions m i for the KFP equations, and affine feedback strategies that give a Nash equilibrium of the game. In Section 3 we introduce the assumption that the running cost of the i-th player is symmetric with respect to the positions of any two other players. It leads to reducing the (N + 1)N/ coefficients of Q i to just four parameters: the primary costs of self-displacement, q i >, and cross-displacement, β i, and the secondary costs of self and cross-displacement, η i and γ i ; also the N entries of the reference position X i reduce to two, the preferred value h i is for the i-th player and his reference value r i for the other agents. Next we assume the players are almost identical, i.e., they have the same parameters in the dynamical system and cost functional, except possibly the secondary costs of displacement. Then there is a unique identically distributed quadratic-gaussian solution of the N HJB-KFP equations, i.e., such that all v i are equal and so are all m i. Section 4 is devoted to the limit as N + for parameters such that q N q, β N β/n, η N i η/n, γ N i γ/n Then the identically distributed solution v N, m N, λ N of the preceding N system converges to a solution of the Mean Field system of two equations νv xx + (vx) R Axv x + λ = V [m(x) in, νm xx ( v xr m Axm ) x = [ min v(x) RAx =, Here V [m is the non-local operator in, m(x)dx = 1, m > in. V [m(x) := q(x h) + β(x h) (y r) dm(y) ( + γ (y r) dm(y)) + η (y r) dm(y). (3) Such solution is explicit and unique among quadratic-gaussian ones, except for one critical value of β. Moreover it is the unique solution of () if β, by a monotonicity argument of Lasry and Lions [19, 1. Note, however, that the normalization condition on v in () is different from the null-average condition of the periodic case [19, 1. ()

4 The results of these three sections parallel those of the seminal papers [19, 1 on games with ergodic cost criterion with the following main differences. Lasry and Lions consider system (1) in dimension d 1 with A i =, running costs of the form L i (X i t, α i t)+f i (X 1 t,..., X N t ) with L i superlinear in α and F i Z d -periodic in each entry, so their state space for each agent is a torus. No explicit formulas can be expected for these general costs and the proofs rely on some hard estimates for the HJB equations. In our Linear-Quadratic (briefly, L-Q) case the explicit quadratic-gaussian formulas for solutions allow rather elementary calculations; on the other hand the unboundedness of data and solutions requires some additional care in the proof of the verification theorem. In Section 5 we exploit the formulas for solutions to study several singular limits. For vanishing noise σ i we show that the distributions m i become Dirac masses, the Nash equilibrium feedback remains the same for the limit deterministic game, a fact known for finite horizon problems [6, 9, and the vanishing viscosity limit commutes with N +. For the cheap control limit, that is, R i, the distributions m i become again Dirac masses and the limit commutes with N +. After solving in quadratic-gaussian form the discounted infinite horizon problem, we show that for vanishing discount there is convergence to the long-time-average cost problem, and also this limit commutes with N +. Finally we study the scaling 1/N = o(β N ) that is related to a singular perturbation of (). Section 6 discusses an interpretation of the L-Q Mean Field Game as a model of the distribution of a population. We compare it to the Mean Field model with local log utility studied by Guéant [1, 11 and reported in [13, where explicit quadratic-gaussian solutions are also found. The important parameters of V (3) in this discussion are q and β, because β > means that it is costly for an individual to imitate his/her peers, whereas for β < resembling the others is rewarding as in the log model of [1, 11. We conclude this introduction with some bibliographical remarks. Huang, Caines, and Malhamé studied L-Q stochastic games with discounted cost and large number of players motivated by several engineering applications [14, 16. They also developed their approach to encompass nonlinear systems and more general costs [15, 17 independently of the Lasry-Lions theory. Discrete Mean Field Games were studied by Gomes et al. [1, numerical methods by Achdou and Capuzzo-Dolcetta [, see also [18 and [1. For the background on N-person differential games we refer to the books [6, 9 and for the ergodic stochastic case to [7, see also the references therein. Games with N players and ergodic payoff The notations of the paper are chosen to be consistent with those of [19, 1. We assume that each player controls a 1-dimensional state variable with linear dynamics, that is, dx i t = (A i X i t α i t)dt + σ i dw i t, X i = x i, i = 1,..., N (4) where A i, σ i are given, σ i, (Wt 1,..., Wt N ) are N independent Brownian motions, the control α i : [, + ) of the i-th player is a bounded process adapted to Wt i. For each initial vector of positions X = (x 1,..., x N ) N we consider the long-time-average cost functional [ J i (X, α 1,..., α N 1 T ) := lim inf T + T E R i (αi t) + F i (Xt 1,..., Xt N )dt, R i >, 3

5 and we assume that F i is quadratic in the following sense. For each player i there is a reference position of the whole state vector X i, and F i is a quadratic form in X X i, i.e., for a symmetric matrix Q i, F i (x 1,..., x N ) := ( X X i ) T Q i ( X X i ) = N j,k=1 q i jk (xj x j i )(xk x k i ), q i ii >, where the condition qii i > means that xi i is a preferred position for the i-th player. We are interested in Nash equilibrium points, that is, vectors of admissible control strategies α = (α 1,..., α N ) such that J i (X, α) = min J i (X, α 1,..., α i 1, α i, α i+1,..., α N ) i = 1,..., N. (5) α i For the current cost functional it is natural to choose as admissible strategy, or admissible control function, for the i-th player any bounded process α i adapted to Wt i such that the corresponding solution Xt i of (4) satisfies, for some C >, E[X i t C, E[(X i t) C, t >, (6) and is ergodic in the following sense: there exists a probability measure m α i such that xdm α i(x), x dm α i(x) < +, and for any polynomial g of degree at most two 1 lim T + T E [ T g ( Xt i ) dt = g(x)dm α i(x), (7) locally uniformly with respect to the initial position x i of X i t. Note that for a N-vector α of such controls the liminf in the cost functionals is a lim and J i (X, α) do not depend on the initial position X. Important examples of admissible strategies are the affine feedbacks whose trajectory is ergodic, as made precise by the next Lemma. Lemma.1. For the feedback consider the process α i t := α i (X i t) where X i t solves Then α i is admissible. α i (x) = K i x + c i, x, K i > A i, (8) dx i t = [(A i K i )X i t c i dt + σ i dw i t. Proof. The explicit solution of the linear equation for Xt i satisfies (6) and it is also known to be ergodic with Gaussian invariant measure m α i of mean c i /(K i A i ) and variance ν i /(K i A i ). Then (7) holds for any bounded and continuous g. To get the conclusion it s enough to check it for g(x) = x and g(x) = x and this is easily done by integrating on [, T the explicit expressions for E[Xt i and E[(Xt) i. In order to write the system of Hamilton-Jacobi-Bellman equations and Kolmogorov- Fokker-Plank equations associated to the game as in [19, 1 we observe that the i-th Hamiltonian is H i (x, p) = p A i xp, x, p, R i 4

6 and for a N-vector of probability measures on (m 1,..., m N ) we denote f i (x; m 1,..., m N ) := F i (x 1,..., x i 1, x, x i+1,..., x N ) dm j (x j ), ν i := (σi ) N 1. j i We want to solve the system ν i vxx i + (vi x) R i A i xvx i + λ i = f i (x; m 1,..., m N ) in, i = 1,..., N ( ν i (m i ) xx v i x R i m i A i xm i = in, i = 1,..., N )x m i(x)dx = 1, m i > in, where with a slight abuse of notation we are denoting with m i a measure as well as its density. Since we are not in the periodic setting of [19, 1 the solutions v i are expected to be unbounded and cannot be normalized by prescribing the value of their average. In the next result we produce solutions with v i a quadratic polynomial and m i Gaussian, namely, v i (x) = (x µ i) + R ia i x, m i (x) = s i 1 πsi ν i R i exp ( (x µ i) s i ν i R i (9) ), (1) for two vectors µ = (µ 1,..., µ N ) and s = (s 1,..., s N ) with s i > that we will compute explicitly. We define the N N matrix B by B ii := q i ii + R i (A i ), B ij := q i ij i j. Theorem.1. Under the above conditions, if det B then i) there exists a unique triple (λ 1,..., λ N ) N, µ N, s (, + ) N, such that (1) solves (9), and ii) the affine feedback s i = ( q i iir i + (R i A i ) ) 1/, (11) µ = B 1 p, p i := q i iix i i + j i q i ijx j i, (1) α i (x) = x µ i s i R i + A i x, x, i = 1,..., N, (13) is a Nash equilibrium point for all initial positions X N among the admissible strategies and J i (X, α) = λ i for all X and i. Proof. For any v i C 1 (), the i-th equation of the second group on N equations in (9) can be integrated to get ( ( 1 Ri A i x )) m i (x) = c i exp ν i v i (x). R i Therefore we are left with the first group of N equations and we plug into them v i of the form (1) to get ( ) 1 ν i + R i A i + 1 ( (x µi ) ) + (R i A i x) R i (A i x) + λ i = f i. (14) s i R i s i 5

7 Next we compute f i = f i (x; m 1,..., m N ) using that m i is the distribution of a Gaussian random variable N (µ i, s i ν i R i ): f i (x; m 1,..., m N ) = qii(x i x i i) + (x x i i) j i b i := q i ij(µ j x j i ) + b i j,k i, j k q i jk (µ j x j i )(µ k x k i ) + j i q i jj(s j ν j R j + (µ j x j i ) ). Then (14) is an equality between two quadratic polynomials. By equating the coefficients of x we get 1 R i s R i(a i ) = qii i (15) i that gives (11). By equating the coefficients of x we get µ i R i s i = q i iix i i + j i qij(µ i j x j i ), i = 1,..., N, (16) and using (15) we get the matrix equation Bµ = p with p given by (1). Finally, by equating the remaining terms we obtain λ i = ν i ( 1 s i + R i A i ) µ i R i s i + qii(x i i i) x i i qij(µ i j x j i ) + b i. (17) This completes the proof of i). Consider the feedback α = (α 1,..., α N ) given by (13) and note it is admissible by Lemma.1. Let α i be an admissible strategy for the i-th player. By Dynkin s formula and the first equation in (9) E [ v i (XT i ) v i (x i ) [ T ( = E ν i vxx i + A i xv x αtv i x i ) (X i t ) dt [ ( T ( ) v E ν i vxx i + A i i ) xv x x (X i R t) Ri i (αi t) dt [ T = λ i T E (f i (X it) ) + Ri (αi t) where the inequality is an equality if α i = α i. We divide both sides by T and let T +. The left hand side vanishes because v i is a quadratic polynomial and the estimates (6) hold. Then λ i lim inf T + j i [ 1 T (f T E i (X it; ) m 1,..., m N ) + Ri (αi t) dt with equality if α i = α i. We claim that the right hand side is J i (X, α 1,..., α i 1, α i, α i+1,..., α N ). Then λ i = J i (X, α) and (5) holds. To prove the claim we consider each term of the running cost F i. We begin with the terms with j = k, j i. Then 1 lim T + T E [ T (X j t xj i ) dt = (x j x j i ) dm j (x j ) = s j ν j R j + (µ j x j i ) 6 dt,

8 by (7) and the fact that the invariant measure m j of the process X j t corresponding to the control α j is a Gaussian N (µ j, s j ν j R j ). Next, we consider the terms with j k, j, k i. Then the ergodicity of (X j t, Xk t ) gives [ 1 T lim T + T E (X j t xj i )(Xk t x k i ) dt = (x j x j i )(xk x k i )dm j (x j )dm k (x k ) = (µ j x j i )(µ k x k i ). The remaining terms are those involving Xt i and X j t with j i. By the definition of the admissible controls (7) Xt i is ergodic with invariant measure m α i. Then, for µ := x dm α i(x), [ 1 T lim T + T E (Xt i x i i)(µ j x j i ) dt = ( µ x i i)(µ j x j i ). On the other hand the ergodicity of (X j t, Xi t) gives [ 1 T lim T + T E (X j t xj i )(Xi t x i i) dt = This completes the proof of the claim and of the theorem. (x j x j i )(xi x i i)dm j (x j )dm α i(x i ) = (µ j x j i )( µ xi i). Remark.1. The solution associated to the optimal feedback α i is the Ornstein-Uhlenbeck process dxt i = Xi t µ i dt + σ i dwt i s i R i and therefore it is mean-reverting with constant mean µ i. Note also that the Nash equilibrium α does not depend on the noise intensities σ i. Remark.. The minimal assumption on qii i for the validity of the preceding theorem is qii i > R i(a i ) / because s i remains well-defined by (11). Remark.3. The proof of Theorem.1 ii) shows that any solution (ṽ 1,..., ṽ N ), ( m 1,..., m N ), (λ 1,..., λ N ) of (9) defines a Nash equilibrium feedback α i (x) := ṽi x(x) R i provided that α is an admissible strategy and ṽ i (x) C(1 + x ) for some constant C and all x. If the condition det B of Theorem.1 fails there may be either infinitely many solutions or none, as the next example shows. Example.1. For N = we consider almost identical players, as we will do for all N later on, namely, q11 1 = q =: q, R 1 = R =: R, A 1 = A =: A, q1 1 = q 1 =: β/, x1 1 = x =: h, x 1 = x 1 =: r. Then det B = (q + RA ) β. For β = q + RA all vectors µ = (µ 1, µ ) satisfying µ 1 + µ = r + qh/β solve the equation Bµ = p with p defined by (1). Then there are infinitely many Gaussian solutions. In the case β = q RA, instead, there are no solutions of Bµ = p (and therefore no Gaussian solutions) unless qh = βr, and in this last case there are again infinitely many (all µ such that µ 1 = µ ). 7

9 3 Symmetric and almost identical players A natural assumption that we will use in the rest of the paper is the following condition saying that the i-th player is influenced in the same way by any two other players. Symmetry Assumption: the cost F i of the i-player is symmetric with respect to the position of any two other players, i.e., F i (x 1,..., x j,..., x k,..., x N ) = F i (x 1,..., x k,..., x j,..., x N ) j, k i. (18) Lemma 3.1. The Symmetry Assumption holds if and only if q i ij = q i ik =: β i, qi jj = q i kk =: η i, x j i = xk i =: r i, j, k i, q i lj = qi kl = qi km =: γ i l, j, k, m i, l j, k l, k m. Proof. The sufficiency is trivial. For the necessity note that the Symmetry Assumption is an identity between two second degree polynomials. By equating their coefficients one easily gets the conclusions. If we also set the positional cost F i takes the simpler form q i := q i ii, h i := x i i i = 1,..., N, F i (x 1,..., x N ) = q i (y i ) + β i y i j i y j + γ i j,k i, j k y j y k + η i (y j ), where y i = x i h i and y j = x j r i for j i. The parameters involved in the running cost of the i-th player are only six, besides the control cost R i, and they can be called h i = preferred own position (happy state), r i = reference position of the other players, q i = primary cost of self-displacement, η i = secondary cost of self-displacement, β i = primary cost of cross-displacement, γ i = secondary cost of cross-displacement. The only sign condition on these parameters is q i >, that can be relaxed if A i, see Remark.. Under the Symmetry Assumption the formulas for the Gaussian solution of the system (9) simplify a bit. For instance the cost corresponding to the Nash equilibrium becomes λ i = νi + ν i R i A i µ i s i ( qi + R i (A i ) ) + q i h i h i β i (µ j r i ) + γ i j,k i, j k j i j i (µ j r i )(µ k r i ) + η i (s j ν j R j + (µ j r i ) ). (19) j i 8

10 A more important consequence of the Symmetry Assumption is that the positional cost F i can be written in the form arising in Mean Field Games, that is, F i (x 1,..., x N ) = V i 1 δ xj (x i ) i = 1,..., N, () N 1 where δ xj is the Dirac measure on concentrated at x j, and the operator V maps probability measures on to quadratic polynomials and is given by V i [m(x) := q i (x h i ) + β i (x h i )(N 1) j i (y r i ) dm(y) + γ i ((N 1) (y r i ) dm(y)) + (η i γ i )(N 1) (y r i ) dm(y). (1) This is easy to check using the identity ( ) j,k i, j k yj y k = j i yj j i (yj ). Remark 3.1. The Symmetry Assumption is essentially necessary for representing F i as in () with V i [m(x) depending only on x and on K l(x, y) dm(y) for a finite number of smooth kernels K l. In fact, imposing such a form to each of the four terms of F i leads to the conditions of Lemma 3.1. Definition 3.1. We say that the players are almost identical if F i satisfies the Symmetry Assumption (18) and the players have the same - control system, i.e., A i = A and σ i = σ (and hence ν i = ν > ) for all i, - cost of the control, i.e., R i = R > for all i, - reference positions, i.e., h i = h and r i = r for all i, - primary costs of displacement, i.e., q i = q > and β i = β for all i. The term almost identical is motivated by the independence on i of four of the parameters appearing in the operator V i, whereas the two secondary costs of displacement γ i, η i are still allowed to depend on i. Note also that the reference state vectors X i are all different if h r. For almost identical players we produce solutions of (9) that are Gaussian and also identically distributed. Theorem 3.1. Assume the players are almost identical and q + RA β(1 N). () Then i) there exist unique µ, s >, such that v i (x) = v(x) := (x µ) s + RAx, m i (x) = m(x) := 1 exp ( πsνr ) (x µ) sνr (3) solve (9) for some (λ 1,..., λ N ) N, moreover s = ( qr + R A ) 1/, µ = qh + rβ(n 1) q + β(n 1) + RA, (4) 9

11 λ i = ν µ + νra s Rs + qh hβ(n 1)(µ r) + γ i (N 1)(N )(µ r) + η i (N 1)(sνR + (µ r) ), i = 1,..., N; (5) ii) the affine feedback α i (x) = x µ sr + Ax, x, i = 1,..., N, is a Nash equilibrium point for all initial positions X N among the admissible strategies (7), and J i (X, α) = λ i for all X and i. Proof. We plug solutions of the form (3) into (9) and we arrive, as in the proof of Theorem.1, at the equations (14), where now all terms on the left hand side are independent of i, but λ i, and the right hand side f i is given by f i (x; m 1,..., m N ) = q(x h) + (x h)β(n 1)(µ r) + b i b i := γ i (N 1)(N )(µ r) + η i (sνr + (µ r) ). The equality of the coefficients of x gives the expression for s in (4), as in the proof of Theorem.1. Next, by equating the coefficients of x the system (16) decouples and reduces to µ = qh + β(n 1)(µ r), Rs that is solvable by () and gives qh + rβ(n 1) µ = Rs Rs β(n 1) + 1 as well as (4). Finally, by equating the remaining terms we obtain (5). The proof of ii) is the same as in Theorem.1. Remark 3.. The assumption () of the last theorem is weaker than the one of Theorem.1, namely det B. In fact, q +RA = β(1 N) implies det B =. On the other hand, if N = and β = q + RA, () is satisfied and det B =. In this case we saw in Example 4.1 that there are infinitely many Gaussian solutions to (9) and only one is identically distributed. Remark 3.3. If the drift A = and β the expected value µ of the distribution m is a weighted average of the two reference states h and r, and it coincides with the preferred state h if the cost of cross-displacement β vanishes. Remark 3.4. If A we can allow any q > RA / in the Theorem. 4 The limit as N + In this section we study the limit as the number of players N goes to +. For simplicity we assume that the control system, the cost per unit control, and the reference positions remain the same, i.e., A, ν, R, h, r are independent of N. To underline the dependence of all the other quantities on N we add a superscript N to them. The natural scaling of the coefficients q N, β N, γi N, ηi N involved in the running cost is the following lim N q N = q, lim N β N (N 1) = β, lim N γ N i (N 1) = γ, lim N η N i (N 1) = η i. (6) 1

12 We also denote with Vi N measure m on, the operator defined by (1) and note that, for any probability V N i [m(x) V [m(x) as N +, locally uniformly in x, where V [m(x) := q(x h) + β(x h) (y r) dm(y) ( + γ (y r) dm(y)) + η (y r) dm(y). (7) Finally we denote with v N, m N, λ N i the Gaussian identically distributed solution of (9) produced in Theorem 3.1. As in [19, 1 we expect the limit of these solution to satisfy the system of two Mean Field equations νv xx + (vx) R Axv x + λ = V [m(x) in, νm xx ( v xr m Axm ) x = in, (8) [ min v(x) RAx =, m(x)dx = 1, m > in. Note the normalization condition on v that replaces the null-average condition of the periodic case in [19, 1. Theorem 4.1. Assume the players are almost identical, (6) holds, and q + RA >, q + RA β. (9) Then i) system (8) has exactly one solution v, m, λ of the quadratic-gaussian form ) (x µ) v(x) := + RAx 1 (x µ), m(x) := exp (, (3) s πsνr sνr given by λ = ν s s = ( qr + R A ) 1/, µ = qh + rβ β + q + RA, (31) + νra µ Rs + qh hβ(µ r) + (γ + η)(µ r) + ηsνr; (3) ii) as N +, v N v in Cloc 1 () with second derivative converging uniformly in, m N m in C k () for all k, and λ N i λ for all i; iii) if in addition β, then v, m, λ given in i) is the unique solution of (8). Proof. i) We plug a solution of the form (3) into (8) and get ν s νra + 1 ( ) (x µ) R s + (RAx) RA x + λ = q(x h) + β(x h)(µ r) + (γ + η)(µ r) + ηsνr. 11

13 By equating the coefficients of x and x on both sides we get (31), whereas the remaining terms give (3). ii) Note that (9) and (6) imply q N > RA / and () for N large enough. Then Remark 3.4, the explicit formulas (3), (4), (5), and the assumption (6) give immediately the stated convergence. iii) To prove uniqueness let us first check the monotonicity of V in L if β. For two probability measures with densities m, n ( ) V [m V [n (x) (m n)(x) dx = β (x h) β x (y r)(m n)(y) dy (m n)(x) dx = y(m n)(y) dy (m n)(x) dx = ( β x(m n)(x) dx). Now we follow the method of [19, 1. Let u, n, λ 1 be another solution of (8). Multiply the first equation in (8) by (m n) and subtract the same expression computed on u, n, λ 1 ; next multiply the second equation in (8) by (m n) and subtract the same expression computed on u, n, λ 1 ; subtract the second identity from the first and integrate on N. By using m(x)dx = n(x)dx we arrive at ( V [m V [n ) (x) (m n)(x) dx + + m(x) R n(x) R ( (u x ) ( (v x ) (v x) (u x) v x (u x v x ) u x (v x u x ) ) ) dx dx =. Since each of the three terms is non-negative, it must vanish. Then m > and p / strictly convex imply u x v x. Next, the condition min [ v(x) RAx / = min [ u(x) RAx / gives u u. Therefore w(x) := m(x) n(x) solves ( vx νw xx R )x w Axw = in, w(x)dx = and by direct integration it is easy to see that w. Finally λ 1 = λ by the first equation in (8). Remark 4.1. In the proof we showed the operator V defined by (7) is monotone in L, as defined in [19, 1, if and only if β, and strictly monotone if and only if β >. Remark 4.. The case left out of the theorem above is β = q RA. If qh βr then there is no solution of the Gaussian form (3). If instead qh = βr there is a continuum of solutions, because for every µ the functions (3) with s = ( qr + R A ) 1/ and the constant (3) solve (8). This is also an example that statement iii) of the last theorem concerning uniqueness may not hold if β <. A different example of Mean Field system with infinitely many Gaussian solutions was given by Guéant [1, see Section 6 for a discussion. 1

14 Remark 4.3. The parameter β is the signed cost per unit time and per unit of displacement of the single player from h and of the average player from r. If β > there is a positive cost if both displacements are in the same direction, i.e., both to the right or both to the left, and a negative cost if they are in opposite directions. If β < the reverse situation occurs. Therefore we can say that imitation among players is costly if β > and rewarding if β <. The statement iii) of Theorem 4.1 says that there cannot be multiple solutions to (8) unless imitation is rewarding. Remark 4.4. The assumption β = β N does not imply uniqueness for the system (9) associated to N players, different from the case (8) describing infinitely many players. In fact, for N = and β = q + RA there are infinitely many Gaussian solutions, see Remark 3.. Remark 4.5. If A negative values of q are allowed. In this case h is not a preferred positions as it is rewarding to stay far from it. Remark 4.6. If we have expansions of the parameters in powers of 1/N, such as q N = q + q 1 N + q +..., β N = β N + β +..., etc., we can easily get expansions of the solution N v N, m N, λ N i in powers of 1/N. Note also that we can assume the parameters A, ν, R, h, r depend on N, provided they converge as N. Example 4.1. In [14, 16 the authors considered infinite horizon discounted functionals, as in Section 5.4 below, with positional cost F i (x 1,..., x N ) = x i b 1 x j + c. N The case c requires some minor modifications to our previous calculations, but for c = F i is a quadratic form in X with symmetric and almost identical players and the parameters are qi N = 1, βi N = b N, γn i = ηi N = b, i, N. N Then the scaling assumption (6) holds, with and the limit positional cost is j i q = 1, β = b, γ = b, η =, V [m(x) = ( x b y dm(y)). Therefore the quadratic-gaussian solution of Theorem 4.1 has mean µ =, s = ( R + R A ) 1/, and cost λ = ν ( R + R A ) 1/ + νra. 5 Other limiting cases 5.1 The small noise or vanishing viscosity limit We consider the limit as the noise coefficient σ j tends to in the dynamics (4) of the j-th player. This is a vanishing viscosity limit ν j for the j-th HJB equation and the j-th 13

15 KFP equation in (9). The limit of the Gaussian solutions (1) found in Theorem.1 is easy: the function v j in (1) does not change because s j and µ j do not depend on ν j, whereas m j converge to the Dirac mass δ µj in the sense of distributions. This limit satisfies the system (9) with ν j =, although the j-th KFP equation is verified only in the sense of distributions by the measure δ µj. The affine feedbacks α i (13) still define a Nash equilibrium point, since the proof of Theorem.1 ii) holds unchanged. If σ i = for all i = 1,..., N we have therefore found a Nash equilbrium for a deterministic N-person differential game. The fact that the equilibrium feedback is the same for the limit deterministic game as for all positive noise intensities is remarkable, and it was known for finite horizon L-Q problems [6, 9. Next we perform the vanishing viscosity limit ν in the Mean Field system of equations (8). As before v is unchanged and m δ µ in the sense of distributions. Then v(x) = (x µ) s + RAx, m(x) = δ µ, λ = qh µ Rs hβ(µ r)+(γ +η)(µ r), (33) with s, µ given by (31), solve the first order Mean Field system (v x) R Axv x + λ = V [m(x) in, ( v xr m Axm ) x = in, [ min v(x) RAx =, m(x)dx = 1, m > in. with V [m defined by (7) and the second equation verified in the sense of distributions. It is easy to see that s and µ are uniquely determined by the form of v and m in (33). Moreover (33) is the unique solution of (34) if β, by the same proof as part iii) of Theorem 4.1. Finally note that the limits N + and ν commute: if we assume the N players be almost identical, v N, m N, λ N i be the limit as ν i = ν of the Gaussian identically distributed solution of (9), and (6), then as N + v N v in C loc (), mn m in the sense of distributions, and λ N i λ for all i, where v, m, λ are given by (33). We refer to [1 for an example where the small noise limit does not commute with N Cheap control We investigate the limit as R i and for simplicity we limit ourselves to the case of almost identical players. Note that the equations in (9) become very degenerate for R i =, but we can use the explicit formulas of Theorem 3.1. We see that s + but sr as R, whereas qh + rβ(n 1) µ =: µ N. q + β(n 1) Then v(x) in Cloc () and m δ µ N in the sense of distributions. Therefore the Gaussian solution converges to a Dirac mass as in the small noise limit, but here v vanishes. Moreover (34) λ i (µn ) 4q + qh hβ(n 1)(µ N r) + γ i (N 1)(N )(µ N r) + η i (N 1)(µ N r) =: λ N (35) for all i = 1,..., N. 14

16 A very similar result holds for the Gaussian solutions of the Mean Field equations (8) of Theorem 4.1: v(x) in C loc (), m δ µ in the sense of distributions, where µ qh + rβ q + β =: µ, and λ µ 4q + qh hβ( µ r) + γ( µ r) + η( µ r) =: λ. Note that both µ N and µ are weighted averages of the two reference states h and r, if β. Finally we remark that also the cheap control limit R commutes with N + (of course under the assumption (6)). In fact µ N µ and therefore δ µ N δ µ in the sense of distributions and λ N λ. 5.3 Large cost of cross-displacement If the parameters of the cost functional scale in a different way from (6) the convergence of the Gaussian identically distributed solution v N, m N, λ N i is much harder. A case that we find interesting is a decay of β N slower than 1/N as N, or even no decay at all. Therefore we assume lim N βn (N 1) = + (36) instead of the second condition in (6) and keep the other three assumptions on the behavior at infinity of q N, γi N, ηi N. Then the second term in V N i [m N (x) diverges for x h, unless y dmn (y) r, and we wonder about the validity of a Mean Field system of the form (8) for some new limit operator V. We pass to the limit in the formulas (3) (4) (5) for v N, m N, λ N i and get (3) with s as in (31) and µ = r, and λ = ν s + νra r Rs + qh h(qh qr rra ) + ηsνr. Therefore the mean value µ of the limit distribution is just the reference state r instead of a linear combination of h and r as in all cases studied before. Moreover the limits v, µ, λ solve the Mean Field system (8) with the limit operator V given by (7) with the second term β (y r) dm(y)(x h) replaced by ( q(h r) rra ) (x h). The same limit is obtained if we let the cost of cross-displacement β tend to + or. This is a singular perturbation or penalization problem for the Mean Field system (7) (8). 5.4 Discounted problems and the vanishing discount limit Consider cost functionals with infinite horizon and discounted running cost, that is, for some ρ > and the same quadratic F i J i = E [ + ( ) Ri (αi t) + F i (Xt 1,..., Xt N ) e ρt dt. 15

17 The associated system of N HJB and KFP equations is the same as (9) with λ i replaced by ρv i. By the same calculations as in Theorem.1 one finds Gaussian solutions with m i as in (1) and vρ(x) i = (x µ i) + R ia i x + c i, s i under the condition that det(b ρd), where D is the diagonal matrix with D ii = R i A i. It is easy to write explicit formulas for the 3N unknown parameters s ρ i, µρ i, cρ i ; in particular, ρc ρ i = λ i ρ(µρ i ) s ρ, i where λ i is given by (17). The vanishing discount limit consists of letting ρ. We easily get that s ρ i s i, µ ρ i µ i, c ρ i, where the limits are given in Theorem.1, so vρ i diverges but ρvρ i λ i and vρ(x) i vρ() i + (µ ρ i ) /s ρ i vi (x) given by (1). Thus in the limit ρ we recover the unique Gaussian solution of (9). This link between discounted infinite horizon problems and ergodic control is well known for problems with a single player [5 or two-person zero-sum games [3, see also the references therein. Next for ρ > fixed we let N + under the assumption of almost identical players (3.1) and with the scaling (6) of the parameters. Denote with vρ N, m N ρ the identically distributed Gaussian solutions of the discounted N-player problem. By the usual method we see that if q + RA + ρra β there is exactly one quadratic v ρ and Gaussian m ρ solving the Mean Field system for the discounted problem νv xx + (vx) R Axv x + ρv = V [m(x) in, νm xx ( v xr m Axm ) x = [ min v(x) RAx =, in, m(x)dx = 1, m > in. Moreover one checks on the explicit formulas that v N ρ v ρ and m N ρ m ρ as N +. If we let now ρ in (37) we get the Gaussian solution of the Mean Field system (8) for the ergodic problem found in Theorem 4.1, and therefore the limits N + and ρ commute. 6 Models of population distribution The Mean Field equations (8) can be interpreted as modeling the stationary states of a population formed by a continuum of identical individuals who move around seeking to minimize their own cost functional depending on the distribution function m of the entire population. The form and the parameters of the cost functional describe the preferences of the individuals. The archetype model of this kind has the discounted cost functional [ + ( α i E t ) + q Xt i h log m(x i t) e ρt dt, q, (38) 16 (37)

18 that was studied in depth by Guéant [1, 11 in many space dimensions and with drift A =, see also the survey paper [13. This model is not derived as the limit as N + of a N- person game; in fact the right hand side of the H-J-B equation in (37) is q x h log m(x) instead of a non-local integral operator V [m(x), although it can be recovered as a limit of such operators, as argued in [1. We want to compare some of the results by Guéant [1, 11 with Theorem 4.1 on the L-Q model (7), (8) (and the same can be done for the discounted problem (37) with similar results). For simplicity we restrict to the case A =, as in [1, 11, and h = r, so there is only one reference position in the state space, as in (38). The choice of the strictly decreasing running cost log m in (38) aims at modeling a population whose agents wish to resemble their peers as much as possible. On the other hand, a consequence of this monotonicity assumption is that no uniqueness is guaranteed for the corresponding Mean Field equations for any value of the parameters. For this reason an interesting stability analysis is performed in [1, 11, 13. In our model, the term of V that describes the willingness to resemble the other individuals is β(x h) (y h) dm(y). The term q(x h) is the same as in the log model (38) and the other two terms are irrelevant for the present discussion. The parameter β is the signed cost per unit time and per unit of displacement of the single player and of the average player from h. If β > there is a positive cost if both displacements are in the same direction, i.e., both to the right or both to the left, and a negative cost if they are in opposite directions. If β < the reverse situation occurs. Therefore we can say that imitation among players is costly if β > and rewarding if β <. The statement iii) of Theorem 4.1 says that there is uniqueness of solution to (8) unless imitation is rewarding. We continue the comparison in the range β <, so that both models describe a preference for imitation, although in different forms. One of the results of [1, 11, 13 for the log model is the existence of a unique Gaussian solution for any q >, with mean h, whereas for q = there is a continuum of Gaussian solutions, with arbitrary mean µ. In the L-Q model (with A = and h = r) there is existence and uniqueness of a Gaussian solution for any q > and q β/, with mean h, and for q = β/ there is a continuum of Gaussian solutions with arbitrary mean, cfr. Remark 4.. So there is the same bifurcation phenomenon, but for a different value of q. On the other hand, in our model as q the variance of m goes to + : the distribution tries to become uniform on, so this limit is singular. Finally we recall that two other models of population distribution are proposed in Section 5 of the survey [13 besides (38). They involve the following operators ( V (1) [m(x) = b x y dm(y)), V () [m(x) = b (x y) dm(y). Note that both are special cases of V given by (7): it is enough to take h = r = and q = β/ = b in both, γ = b and η = for V (1), γ = and η = b for V (). Acknowledgments. I wish to thank Yves Achdou for sending me the notes of his beautiful lectures on Mean Field Games and Italo Capuzzo Dolcetta for the encouragement to complete this research. This work was partially supported by the Fondazione CaRiPaRo Project Nonlinear Partial Differential Equations: models, analysis, and control-theoretic problems. 17

19 References [1 Y. Achdou, F. Camilli, I. Capuzzo-Dolcetta: Mean field games: numerical methods for the planning problem, preprint 1. [ Y. Achdou, I. Capuzzo-Dolcetta: Mean field games: numerical methods. SIAM J. Numer. Anal. 48 (1), [3 O. Alvarez, M. Bardi: Ergodic problems in differential games, in Advances in dynamic game theory, vol. 9 of Ann. Internat. Soc. Dynam. Games, p Birkhäuser Boston, Boston, MA, 7. [4 R.J. Aumann: Markets with a continuum of traders. Econometrica 3 (1964), [5 M. Bardi, I. Capuzzo Dolcetta: Optimal control and viscosity solutions of Hamilton- Jacobi-Bellman equations. Birkhäuser Boston Inc., Boston, MA, [6 T. Basar, G.J. Olsder: Dynamic noncooperative game theory. Second edition. Academic Press, Ltd., London, [7 A. Bensoussan, J. Frehse: Regularity results for nonlinear elliptic systems and applications. Springer-Verlag, Berlin,. [8 P. Cardaliaguet: Notes on Mean Field Games (from P.-L. Lions lectures at Collège de France), 1. [9 J.C. Engwerda: Linear quadratic dynamic optimization and differential games. Wiley, Chichester, 5. [1 D.A. Gomes, J. Mohr, R.R. Souza: Discrete time, finite state space mean field games. J. Math. Pures Appl. (9) 93 (1), [11 O. Guéant: Mean field games and applications to economics, Ph.D. Thesis 9, Université Paris-Dauphine. [1 O. Guéant: A reference case for mean field games models. J. Math. Pures Appl. (9) 9 (9), [13 O. Guéant, J.-M. Lasry, P.-L. Lions: Mean field games and applications, in Paris- Princeton Lectures on Mathematical Finance 1, Springer, to appear. [14 M. Huang, P.E. Caines, R.P. Malhamé: Individual and mass behaviour in large population stochastic wireless power control problems: centralized and Nash equilibrium solutions. Proc. the 4nd IEEE Conference on Decision and Control, Maui, Hawaii, pp , December 3. [15 M. Huang, P.E. Caines, R.P. Malhamé: Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle. Commun. Inf. Syst. 6 (6), 1 51 [16 M. Huang, P.E. Caines, R.P. Malhamé: Large-population cost-coupled LQG problems with nonuniform agents: individual-mass behavior and decentralized ɛ-nash equilibria. IEEE Trans. Automat. Control 5 (7),

20 [17 M. Huang, P.E. Caines, R.P. Malhamé: An invariance principle in large population stochastic dynamic games. J. Syst. Sci. Complex. (7), [18 A. Lachapelle, J. Salomon, G. Turinici: Computation of mean field equilibria in economics. Math. Models Methods Appl. Sci. (1), [19 J.-M. Lasry, P.-L. Lions: Jeux à champ moyen. I. Le cas stationnaire. C. R. Acad. Sci. Paris 343 (6), [ J.-M. Lasry, P.-L. Lions: Jeux à champ moyen. II. Horizon fini et contrôle optimal. C. R. Acad. Sci. Paris 343 (6), [1 J.-M. Lasry, P.-L. Lions: Mean field games. Jpn. J. Math. (7),