Mean Field Games: Basic theory and generalizations

Transcription

1 Mean Field Games: Basic theory and generalizations School of Mathematics and Statistics Carleton University Ottawa, Canada University of Southern California, Los Angeles, Mar 2018

2 Outline of talk Mean field games (MFGs) Historical background of noncooperative games Simple examples of MFG; nonlinear diffusion dynamics: interacting particle modeling Fundamental approaches Generalizations Three concrete MFG models Competitive binary Markov decision process (MDP) Stochastic growth model LQ mean field social optimization with a major player

3 Historical background of noncooperative games MFG: basic theory Generalizations of modeling A mean field game is a situation of stochastic (dynamic) decision making where each agent interacts with the aggregate effect of all other agents; agents are non-cooperative. Example: Hybrid electric vehicle recharging control interaction through price

4 Historical background of noncooperative games MFG: basic theory Generalizations of modeling Applications of MFG: Economic theory Finance Communication networks Social opinion formation Power systems Electric vehicle recharging control Public health (vaccination games) Cyber-security problems (botnets)...

5 Historical background of noncooperative games MFG: basic theory Generalizations of modeling Historical Background of Noncooperative Games 1. Cournot duopoly Cournot equilibrium, person zero-sum von Neumann s minimax theorem, N-person nonzero-sum Nash equilibrium, 1950 Antoine Cournot John von Neumann John Nash ( ) ( ) ( )

6 Historical background of noncooperative games MFG: basic theory Generalizations of modeling Two directions for generalizations: Include dynamics L. Shapley (1953), MDP model; Rufus Isaacs (1965), Differential games, Wiley, 1965 (work of 1950s). Consider large populations of players Technical challenges arise... the curse of dimensionality when the number of players is large Example: 50 players, each having 2 states, 2 actions. Need a system configuration: 4 50 =

7 Historical background of noncooperative games MFG: basic theory Generalizations of modeling von Neumann and Morgenstern s version on games with a large number of players... When the number of participants becomes really great, some hope emerges that the influence of every particular participant will become negligible, and that the above difficulties may recede and a more conventional theory becomes possible.... In all fairness to the traditional point of view this much ought to be said: It is a well known phenomenon in many branches of the exact and physical sciences that very great numbers are often easier to handle than those of medium size. An almost exact theory of a gas, containing about freely moving particles, is incomparably easier than that of the solar system, made up of 9 major bodies... This is, of course, due to the excellent possibility of applying the laws of statistics and probability in the first case. J. von Neumann and O. Morgenstern (1944), Theory of games and economic behavior, Princeton University Press, p. 12.

8 Historical background of noncooperative games MFG: basic theory Generalizations of modeling Early efforts to deal with large populations J. W. Milnor and L.S. Shapley (1978); R. J. Aumann (1975). This is for cooperative games E. J. Green (1984). Non-cooperative games M. Ali Khan and Y. Sun (2002). Survey on Non-cooperative games with many players. Static models. B. Jovanovic and R. W. Rosenthal (1988). Anonymous sequential games. It considers distributional strategies for an infinite population. However, individual behaviour is not addressed....

9 Historical background of noncooperative games MFG: basic theory Generalizations of modeling Modeling of mean field games Mass influence zi ui i Play against mass m(t) Model: Each player interacts with the empirical state distribution of N players via dynamics and/or costs (see examples) δ (N) z Individual state and control (zi, u i ), δ (N) z(t) = 1 N N i=1 δ z i (t) Objective: Overcome dimensionality difficulty

10 Historical background of noncooperative games MFG: basic theory Generalizations of modeling MFG Example 1: Continuous time LQ dxt i = (A θi Xt i + But i + GX (N) )dt + DdW i J i (u i, u i ) = E 0 t e ρt { X i t (ΓX (N) t To relate to the empirical distribution, denote δ (N) X (t) = 1 N X (N) t = 1 N N i=1 δ X i (t) N Xt i = i=1 R t, 1 i N + η) 2 Q + (u i t) T Ru i t yδ (N) X (t) (dy) } dt

11 Historical background of noncooperative games MFG: basic theory Generalizations of modeling MFG Example 2: Discrete time Markov decision processes (MDPs) N MDPs (x i t, a i t), 1 i N, where state xt i has transitions affected by action at i but not by (at 1,..., at i 1, at i+1,..., at N ). Player i has cost J i = E T t=0 ρ t l(x i t, x (N) t, a i t), ρ (0, 1]

12 Historical background of noncooperative games MFG: basic theory Generalizations of modeling MFG Example 3: Diffusion model of a Nash game of N players: dx i = 1 N N f (x i, u i, x j )dt + σdw i, 1 i N, t 0, j=1 J i (u i, u i ) = E T 0 [ 1 N N j=1 ] L(x i, u i, x j ) dt, T <. Mean field coupling in dynamics and costs Denote δ (N) x(t) = 1 N N i=1 δ x i (t). Then 1 N N f (x i, u i, x j ) = f (x i, u i, y)δ (N) x(t) (dy) R n j=1

13 Historical background of noncooperative games MFG: basic theory Generalizations of modeling The traditional approach: infeasibility Rewrite vector mean field dynamics (controlled diffusion): dx(t) = f N (x(t), u 1 (t),..., u N (t))dt + σ N dw (t). Cost of agent i {1,..., N}: J i (u i, u i ) = E T 0 l i(x(t), u i, u i )dt where u i is the set of controls of all other agents Dynamic programming (N coupled HJB equations): { [ ] 0 = v i t + min u i fn T v i x Tr( 2 v i x σ 2 N σn T ) + l i, v i (T, x) = 0, 1 i N Need too much information since the HJBs give an individual strategy of the form u i (t, x 1,..., x N ). Computation is heavy, or impossible in nonlinear systems. Need a new methodology: mean field stochastic control theory!

14 Historical background of noncooperative games MFG: basic theory Generalizations of modeling Problem 3 : player Nash game states: ( 1 ) L strategies: ( L 1 ) costs: L ( ) 1 route 1 A large-scale coupled equation system Example: 1 coupled dynamic programming equations Performance route 2 route 1 Problem 3 : Optimal control of a single player state: control: mean field µ W is fixed and not controlled by L L L route 2 MFG equation system: 1 equation of optimal control; 1 equation of mean field dynamics (for ) µ W Example: HJB PDE; FPK PDE (or McKean-Vlasov SDE) Figure : The fundamental diagram of MFG theory The red route: top-down (impose consistency in the mean field approximations). See Huang, Malhame, and Caines (2006; nonlinear case), Huang, Caines, and Malhame (2003, 2007; LQ case) The blue route: bottom-up. See Lasry and Lions (2006, 2007; nonlinear case, restricted to decentralized information)

15 Solution of Example 1: Problem P 0 dxt i = (A θ i Xt i + Bui t J i (u i, u i ) = E 0 Riccati equation: (N) + GX t )dt + DdWt i, Historical background of noncooperative games MFG: basic theory Generalizations of modeling 1 i N { e ρt Xt i (ΓX (N) t + η) 2 Q + (ui t) T Rut i } dt ρπ = ΠA + A T Π ΠBR 1 B T Π + Q The MFG solution equation system: { d Xt dt = (A BR 1 B T Π + G) X t BR 1 B T s t, I.C. X 0 ρs t = ds t dt + (AT ΠBR 1 B T )s t + ΠG X t Q(Γ X t + η) The set of decentralized strategies for the N players: Results: û i t = R 1 B T (ΠX i t + s t ), 1 i N. i) Existence for ODE via fixed point argument. ii) The set of strategies is an ϵ-nash equilibrium

16 Historical background of noncooperative games MFG: basic theory Generalizations of modeling The fundamental diagram with diffusion dynamics: P 0 Game with N players; Example dx i = f (x i, u i, δ x (N) )dt + σ( )dw i J i (u i, u i ) = E T 0 l(x i, u i, δ x (N) )dt δ x (N) : empirical distribution of (x j ) N j=1 solution HJBs coupled via densities p N i,t, 1 i N +N Fokker-Planck-Kolmogorov equations u i adapted to σ(w i (s), s t) (i.e., restrict to decentralized info for N players); so giving u N i (t, x i ) construct performance? (subseq. convergence) N P Limiting problem, 1 player dx i = f (x i, u i, µ t)dt + σ( )dw i J i (u i ) = E T 0 l(x i, u i, µ t)dt Freeze µ t, as approx. of δ (N) x solution û i (t, x i ) : optimal response HJB (with v(t, ) given) : v t = inf ui (f T v xi + l Tr[σσT v xi x i ]) Fokker-Planck-Kolmogorov : p t = div(fp) + (( σσt 2 ) jk p) x j i xk i Coupled via µ t (w. density p t; p 0 given) The consistency based approach (red) is more popular; related to ideas in statistical physics (McKean-Vlasov eqn); FPK can be replaced by an MV-SDE

17 Historical background of noncooperative games MFG: basic theory Generalizations of modeling A brief note: Due to time constraint, this talk did not cover the so-called mean field type control (with a single decision maker) One such example is mean-variance portfolio optimization, where the objective contains Var(X (t)) which depends on the mean in a nonlinear manner.

18 L MFG: background and theory Historical background of noncooperative games MFG: basic theory Generalizations of modeling Comparison of the two approaches in an LQ setting (Huang and Zhou 18): blue route (direct approach); red route (fixed point approach) Problem 3 : player Nash game states: ( 1 ) L strategies: ( L 1 ) costs: ( L 1 ) route 1 A large-scale coupled equation system Example: 1 coupled dynamic programming equations Problem 3 mean field controlled by : Optimal control of a single player state: control: L is fixed and not Performance route 2 route 1 µ W L route 2 MFG equation system: 1 equation of optimal control; 1 equation of mean field dynamics (for ) µ W Example: HJB PDE; FPK PDE (or McKean-Vlasov SDE) Fixed point approach Asymptotic solvability Non-uniqueness (direct approach) dxt i = (A θi Xt i + But i + GX (N) t T { J i (u i, u i ) = E 0 X i t (ΓX (N) t )dt + DdW i t, 1 i N } + η) 2 Q + (ui t) T Rut i dt Definition: Asymptotic solvability The N coupled dynamic programming equations have solutions in addition to mild regularity requirement on the solution behavior. Its necessary and sufficient condition: A non-symmetric Riccati ODE in R n has a solution on [0, T ] where X i t Rn.

19 Historical background of noncooperative games MFG: basic theory Generalizations of modeling Major players with strong influence (example: institutional trader and many smaller traders) Robustness Cooperative decision(team) Partial information

20 (I) Competitive MDP modeling of MFGs Except for LQ cases, general nonlinear MFGs rarely have closed-form solutions We introduce this class of MDP models which have relatively simple solutions (threshold policy)

21 The MF MDP model: (Huang and Ma, 2016) N players (or agents A i, 1 i N) with states x i t, t Z +, as controlled Markov processes (no coupling) State space: S = [0, 1] Action space: A = {a0, a 1 }. a 0 : inaction Interpret state as unfitness or risk Agents are coupled via the costs Examples of binary action space (action or inaction) maintenance of equipment; network security games; vaccination games, etc.

22 Dynamics: The controlled transition kernel for x i t. For t 0 and x S, P(x i t+1 B x i t = x, a i t = a 0 ) = Q 0 (B x), P(x i t+1 = 0 x i t = x, a i t = a 1 ) = 1, Q 0 ( x): stochastic kernel defined for B B(S) (Borel sets). Q 0 ([x, 1] x) = 1. Recall a 0 = inaction. So the state gets worse under inaction. Transition of x i t is not affected by other a j t, j i. The stochastic deterioration is similar to hazard rate modelling in the maintenance literature (Bensoussan and Sehti, 2007; Grall et al. 2002)

23 x i t under inaction. It is getting worse and worse.

24 The cost of A i : J i = E T t=0 ρ t c(x i t, x (N) t, a i t), 1 i N. 0 < T ; ρ (0, 1): discount factor. Population average state: x (N) t = 1 N N i=1 x i t. The one stage cost: c(xt, i x (N) t, at) i = R(xt, i x (N) t ) + γ1 {a i t =a 1 } Motivation: network maintenance game R 0: unfitness-related cost; γ > 0: the effort cost

25 Stationary equation system for MFG [ V (x) = min ρ z = 1 0 xπ(dx) 1 0 V (y)q 0 (dy x) + R(x, z), where π is a probability measure on [0, 1] ] ρv (0) + R(x, z) + γ Remark: The second equation means z = Ex i t in steady state Existence and Uniqueness can be developed under technical assumptions. Solution property: threshold policy

26 Assumptions: (A1) {x0 i, i 1} are independent random variables taking values in S. (A2) R(x, z) is a continuous function on S S. For each fixed z, R(, z) is strictly increasing. (A3) i) Q 0 ( x) satisfies Q 0 ([x, 1] x) = 1 for any x, and is strictly stochastically increasing; ii) Q 0 ( x) has a positive density for all x < 1. (A4) R(x, ) is increasing for each fixed x. (A5) γ > β max z 1 0 [R(y, z) R(0, z)]q 0(dy 0). Remarks: Montonicity in (A2): cost increases when state is poorer. (A3)-i) means dominance of distributions (A5) Effort cost should not be too low; this prevents zero action threshold

27 Theorem (existence) Assume (A1)-(A5). Then [ V (x) = min ρ z = 1 0 xπ(dx) 1 0 V (y)q 0 (dy x) + R(x, z), ] ρv (0) + R(x, z) + γ has a solution (V, z, π, a i ) for which the best response a i is a threshold policy. Uniqueness holds if we further assume R(x, z) = R 1 (x)r 2 (z) and R 2 > 0 is strictly increasing on S. a i : x [0, 1] {a 0, a 1 }, is implicitly specified by the first equation as a threshold policy Show each θ-threshold policy leads to a limiting distribution π θ (verify Doebline s condition for each θ (0, 1)); in fact inf x S P(x 4 = 0 x 0 = x) η > 0.

28 MFG: background and theory III. Mean field team 5 V(x) x 0 iterate Figure : V (x) defined on [0, 1] V(x) Figure : Left: V (x) (threshold 0.49) Right: search of the threshold 0.49

29 (II) Stochastic growth

30 The idea of relative performance in economic literature: Abel (1990), Amer. Econ. Rev. Hori and Shibata (2010), J. Optim. Theory Appl. Turnovsky and Monterio (2007), Euro. Econ. Rev. MFG with relative performance (Espinosa and Touzi, 2013) GBM dynamics for risky assets The performance of agent (manager) i (i = 1, 2,..., N): [ ] EU (1 λ)xt i + λ(x T i X ( i) T ), X ( i) T = 1 N 1 Feature: This is difference-like comparison Our growth model (Huang and Nguyen, AMO 16): Cobb-Douglas production function Relative utility based performance X j T, 0 < λ < 1 j i

31 Dynamics of the N agents: dxt i = [ A(Xt i ) α δxt i Ct i ] dt σx i t dwt i, 1 i N, Xt i : capital stock, X 0 i > 0, EX 0 i <, C t i : consumption rate Ax α, α (0, 1): Cobb-Douglas production function, 0 < α < 1, A > 0 δdt + σdwt i : stochastic depreciation (see e.g. Wälde 11, Feicht and Stummer 10 for stochastic depreciation modeling) {Wt i, 1 i N} are i.i.d. standard Brownian motions. The i.i.d. initial states {X0 i, 1 i N} are also independent of the N Brownian motions.

32 The utility functional of agent i: [ ] T J i (C 1,..., C N ) = E e ρt U(Ct i, C (N,γ) t )dt + e ρt S(X T ), where C (N,γ) t = 1 N Take the utility function 0 N i=1 (C i t ) γ, γ (0, 1). Take S(x) = ηx γ γ, η > 0. [ U(Ct i, C (N,γ) t ) = 1 γ (C t i ) γ(1 λ) (Ct i ) γ ( = [ 1 γ (C i t ) γ ] 1 λ [ C (N,γ) t 1 γ ] λ (C i t )γ C (N,γ) t ] ) λ =: U 1 λ 0 U1 λ, as weighted geometric mean of U 0 (own utility), U 1 (relative utility). Further take γ = 1 α, i.e., equalizing the coefficient of the relative risk aversion to capital share; useful for closed-form solution

33 Results: The individual strategy is a linear feedback [ T Ĉt i = b t e a(t T ) η 1 1 γ + e at t e as b s ds] 1X i t, 1 i N. a: constant; b: determined from fixed point equation (FPE) b t = Γ(b) t, t [0, T ]. Existence of a solution to the FPE by a contraction argument The set of strategies is an ε-nash equilibrium.

34 We solve the fixed equation b = Γ(b) with the following parameters T = 2, A = 1, δ = 0.05, γ = 0.6, η = 0.2, ρ = 0.04, σ = 0.08, λ will take three different values 0.1, 0.3, 0.5 for comparisons. See Feicht and Stummer (2010) for typical parameter values in capital growth models with stochastic depreciation. Time is discretized with step size 0.01.

35 Numerical example λ=0.1 λ=0.3 λ= λ=0.1 λ=0.3 λ= t t Figure : Left: b t solved from b = Γ(b); right: b t Γ 0 (b) t (as control gain) When the agent is more concerned with the relative utility (i.e., larger λ), it consumes with more caution during the late stage.

36 Numerical example MFG: background and theory iterates t Figure : The computation of b t in the first 20 iterates by operator Γ, λ = 0.5.

37 (III) Mean field teams with a major player Related work on social optimization Huang, Caines and Malhamé (TAC, 2012) LQ mean field team with peers Sen, Huang and Malhamé (CDC 16) Nonlinear diffusions with peers Huang and Nguyen (IFAC 2011, CDC 16) LQ model with a major player, no dynamic coupling

38 Motivation for cooperative decision (mean field team) Manage space heaters in large buildings (hotel, apartment building, etc); they can run cooperatively to maintain comfort and good average load (as a mean field) Kizilkale and Malhame (2016) considered related collective target tracking reflecting partial cooperation; linear SDE temperature dynamics

39 Dynamics of the major player A 0, and N minor players A i : dx N 0,t = (A 0 x N 0,t + B 0 u N 0,t + F 0 x (N) t )dt + D 0 dw 0,t, dx N i,t = (Ax N i,t + Bu N i,t + Fx (N) t + Gx N 0,t)dt + DdW i,t, 1 i N. (A1) The initial states x N j,0 = x j(0) for j 0. {x j (0), 0 j N} are independent, and for all 1 i N, Ex i (0) = µ 0. sup i E x i (0) 2 c for a constant c independent of N. Note: The condition of equal initial means can be generalized.

40 The cost for A 0 and A i, 1 i N: J 0 (u N 0, u N 0) = E J i (u N i, u N i) = E T 0 T 0 { x N 0 { x N i where Q 0 0, Q 0 and R 0 > 0, R > 0, Φ(x (N) ) 2 Q 0 + (u N 0 ) T R 0 u N 0 }dt, Ψ(x N 0, x (N) ) 2 Q + (un i ) T Ru N i }dt, u N j = (un 0,..., un j 1, un j+1,..., un N ), x (N) = 1 N N i=1 x N i, Φ(x (N) ) = H 0 x (N) + η 0, Ψ(x N 0, x (N) ) = Hx N 0 + Ĥx (N) + η. The social cost: J (N) soc (u N ) = J 0 + λ N where u N = (u0 N, un 1,..., un N ) and λ > 0. N J k, k=1

41 Notation: u N = (u N 0, un 1,, un N ) u N j = (un 0,..., un j 1, un j+1,..., un N ), j 0 x (N) = 1 N N i=1 x i N, ˆx (N) = 1 N N i=1 ˆx i N, etc û (N) = 1 N N i=1 ûn i ˆx (N) i = 1 N N j i ˆx N j. x (N) i = 1 N N j i x N j. x 0, x i, u i, etc. for the limiting model m, ˆm, m for the mean field

42 Main result 1. The solution of the MFT leads to the FBSDE dˆx 0 = (A 0ˆx 0 + B 0 R 1 0 B T 0 p 0 + F 0 ˆm)dt + D 0 dw 0, d ˆm = [(A + F ) ˆm + G ˆx 0 + B(λR) 1 B T p]dt, dp 0 = { A T 0 p 0 G T p + Q 0 [ˆx 0 (H 0 ˆm + η 0 )] H T λq[(i Ĥ) ˆm Hˆx 0 η]}dt + ξ 0 dw 0, dp = { F T 0 p 0 (A + F ) T p H T 0 Q 0[ˆx 0 (H 0 ˆm + η 0 )] + (I Ĥ) T λq[(i Ĥ) ˆm Hˆx 0 η]}dt + ξdw 0, dˆx i = [Aˆx i + B(λR) 1 B T q i + F ˆm + G ˆx 0 ]dt + DdW i, dq i = { F T 0 p 0 F T p A T q i H T 0 Q 0[ˆx 0 (H 0 ˆm + η 0 )] + λq[ˆx i (Hˆx 0 + Ĥ ˆm + η)] ĤλQ[(I Ĥ) ˆm Hˆx 0 η]}dt + ζ a i dw 0 + ζ b i dw i. where ˆx 0,0 = xn 0,0, ˆm 0 = µ 0, ˆx i,0 = xn i,0, p 0(T ) = p(t ) = p i (T ) = 0. Theorem. This FBSDE has a unique solution. Remark 1: General FBSDEs do not always have a solution.

43 Main result 2. Performance gap Social Optimality Theorem We have J (N) soc (û) inf u J(N) soc (u) = O(1/ N), where each u N j, 0 j N within u is in L 2 F (0, T ; Rn 1 ), and û N 0 = û 0 = R 1 0 BT 0 p 0, û N i = û i = (λr) 1 B T q i. We can further show that p 0 is a linear function of (ˆx 0, ˆm). We may choose F t as the σ-algebra σ(x j (0), W j (τ), 0 j N, τ t). F x (0),W t

44 How is the FBSDE introduced? If (û 0, û 1,, û N ) (as progressively measurable processes) is a social optimum, a perturbation δu k in any single component û k will make the social cost J soc (N) worse off (this property is called person-by-person optimality in team decision theory). This provides variational conditions, one for the major player and the other for a representative minor player. Derive the mean field limit and impose consistency.

45 Basic references and surveys on mean field games Huang, Caines, and Malhame CDC, IEEE TAC 07. Huang, Malhame and Caines, CIS 06. Lasry and Lions. Mean field games, Caines, IEEE Control Systems Society Bode Lecture, 2009; Bensoussan, Frehse, and Yam. Springer Brief Caines. Mean field games, in Encyclopedia of Systems and Control, Springer, Cardaliaguet. Notes on mean field games, Gomes and Saud. Mean field games models a brief survey Caines, Huang and Malhame (2017) Mean field games. (Springer handbook article) Carmona and Delarue (2017) Mean field games, springer. Mean field game with MDP modeling Huang, Malhame, and Caines, CDC 04; Weintraub, Benkard, and Van Roy (2008). Econometrica; Adlakha, Johari, and Weintraub (2015). J. Econ. Theory.

46 Thank you!