Mean Field Control: Selected Topics and Applications

Transcription

1 Mean Field Control: Selected Topics and Applications School of Mathematics and Statistics Carleton University Ottawa, Canada University of Michigan, Ann Arbor, Feb 2015

2 Outline of talk Background and motivation Mean field game (MFG) model: N players (N is large) N interacting particle system modeling; complexity issues Mean field limit ideas Caines, Huang, and Malhamé (03, 06, 07,...); P.E. Caines, IEEE Control Systems Society Bode Lecture, 2009; Lasry and Lions (06, 07,...) An overview by Bensoussan et. al. (2013); Buckdahn et. al. (2011); a survey by Gomes and Saúde (2014) Other modeling issues (major players, common noises, unknown model components,...); applications Remarks and references

3 Example 1: recharging control The aggregate recharging behavior of all Plug-in Electric Vehicles (PEVs) impacts the electricity price p t PEV i s optimization deals with (ut i, p t). ut i : its own recharging rate. For more technical details, see (Ma, Callaway, Hiskens, IEEE Trans. CST 2013).

4 Example 2: flu vaccination game The population vaccination coverage p t. The chance of an outbreak decreases with p t. An individual plays with respect to p t (leading to a mean field model). Trade-off between infection risk and side effects, effort costs.

5 Example 3: relative performance From (Espinosa and Touzi, 2013) The performance of agent (manager) i (i = 1, 2,..., N): EU[(1 λ)x i T + λ(x i T X ( i) T )], 0 < λ < 1 1 non risky asset S t = (S 1 t,..., S d t ): d-dim risky asset (described as a diffusion) X i depends on S and portfolio strategy π i of agent i. Mean field coupling term X ( i) T = 1 N 1 j i X j T occurs in the utility This relative performance is related to human psychology in satisfaction

6 Example 3: production optimization agent i agent j "Market" agent k In a stochastic growth model with many competing producers, let the individual capital stock level be u i (t). The efficiency of production is impacted by u (N) (t) = 1 N N i=1 u i(t). (For instance, a congestion effect due to competitive use of resources) Think of u (N) as a quantity measured by a macroscopic unit.

7 The mean field LQG game Main results Nonlinear models: some more detail We are motivated to develop a general theory for mean field decision problems. To formalize mathematically, we consider stochastic differential games with mean field interactions. For instance, we may consider dynamics and costs: dx i = (1/N) J i (u i, u i ) = E N f ai (x i, u i, x j )dt + σdw i, 1 i N, t 0, j=1 T 0 [ (1/N) N j=1 ] L(x i, u i, x j ) dt, T <.

8 The mean field LQG game Main results Nonlinear models: some more detail The traditional approach: infeasibility Rewrite vector mean field dynamics (controlled diffusion): dx(t) = f (x(t), u 1 (t),..., u N (t))dt + σ N dw (t). Cost of agent i {1,..., N}: J i (u i, u i ) = E T 0 l i(x(t), u i, u i )dt where u i is set of controls of all other agents Dynamic programming (N coupled HJB equations): { [ ] 0 = v i t + min u i f T v i x Tr( 2 v i x σ 2 N σn T ) + l i, v i (T, x) = 0, 1 i N Need too much information since the HJBs give an individual strategy of the form u i (t, x 1,..., x N ). Computation is heavy, or impossible in nonlinear systems. Need a new methodology: mean field stochastic control theory!

9 The mean field LQG game Main results Nonlinear models: some more detail von Neumann and Morgenstern (1944, pp. 12) Their vision on games with a large number of players... When the number of participants becomes really great, some hope emerges that the influence of every particular participant will become negligible, and that the above difficulties may recede and a more conventional theory becomes possible.... In all fairness to the traditional point of view this much ought to be said: It is a well known phenomenon in many branches of the exact and physical sciences that very great numbers are often easier to handle than those of medium size. An almost exact theory of a gas, containing about freely moving particles, is incomparably easier than that of the solar system, made up of 9 major bodies... This is, of course, due to the excellent possibility of applying the laws of statistics and probability in the first case.

10 The mean field LQG game Main results Nonlinear models: some more detail The basic framework of MFGs P 0 Game with N players; Example dx i = f (x i, u i, δ x (N) )dt + σ( )dw i J i (u i, u i ) = E T 0 l(x i, u i, δ x (N) )dt δ x (N) : empirical distribution of (x j ) N j=1 solution HJBs coupled via densities p N i,t, 1 i N +N Fokker-Planck-Kolmogorov equations u i adapted to σ(w i (s), s t) (i.e., restrict to decentralized info for N players); so giving u N i (t, x i ) construct performance? (subseq. convergence) N P Limiting problem, 1 player dx i = f (x i, u i, µ t)dt + σ( )dw i J i (u i ) = E T 0 l(x i, u i, µ t )dt Freeze µ t, as approx. of δ (N) x solution û i (t, x i ) : optimal response HJB (v(t, ) given) : v t = inf ui (f T v xi + l Tr[σσT v xi x i ]) Fokker-Planck-Kolmogorov : p t = div(fp) + (( σσt 2 ) jk p) x j i xk i Coupled via µ t (w. density p t ; p 0 given) The consistency based approach (red) is more popular; related to ideas in statistical physics (McKean-Vlasov eqn); FPK can be replaced by an MV-SDE When a major player or common noise appears, new tools (stochastic mean field dynamics, master equation, etc) are needed

11 The mean field LQG game Main results Nonlinear models: some more detail The main ideas The procedure indicated by the red path is based on (i) freezing mean field, (ii) optimal response, (iii) consistency The procedure indicated by the blue path gives the limiting equation system (See the notes of Cardaliaguet, 2012) Without the decentralized information restriction, the problem is much harder since then the control would be u i (t, x 1,..., x N ). This makes a very difficult problem in deriving the limiting equation system.

12 The mean field LQG game Main results Nonlinear models: some more detail The mean field LQG game Individual dynamics: dz i = (a i z i + bu i )dt + αz (N) dt + σ i dw i, 1 i N. Individual costs: J i = E 0 e ρt [(z i Φ(z (N) )) 2 + ru 2 i ]dt. z i : state of agent i; u i : control; w i : noise a i : dynamic parameter; r > 0; N: population size For simplicity: Take the same control gain b for all agents. z (N) = (1/N) N i=1 z i, Φ: nonlinear function We use this simple scalar model (CDC 03, 04) to illustrate the key idea; generalizations to vector states are obvious

13 The mean field LQG game Main results Nonlinear models: some more detail The methodology of consistent mean field approximation Mass influence z i i u i Play against mass m(t) Consistent mean field approximation In the infinite population limit, individual strategies are optimal responses to the mean field m(t); Closed-loop behaviour of all agents further replicates the same m(t)

14 The mean field LQG game Main results Nonlinear models: some more detail The limiting optimal control problem Recall dz i = (a i z i + bu i )dt + αz (N) dt + σ i dw i J i = E 0 e ρt [(z i Φ(z (N) )) 2 + ru 2 i ]dt Take f, z C b [0, ) (bounded continuous) and construct dẑ i = a i ẑ i dt + bu i dt + αfdt + σ i dw i J i (u i, z ) = E 0 e ρt [(ẑ i z ) 2 + ru 2 i ]dt Riccati Equation : ρπ i = 2a i Π i (b 2 /r)π 2 i + 1, Π i > 0. Optimal Control : û i = b r (Π iz i + s i ) ρs i = ds i dt + a is i b2 r Π is i + απ i f z. How to determine z?

15 The mean field LQG game Main results Nonlinear models: some more detail The mean field solution system Let Π a = Π i ai =a. Assume (i) Ez i (0) = 0, i 1, (ii) The dynamic parameters {a i, i 1} A have limit empirical distribution F (a). Optimal control and consistent mean field approximations (Nash certainty equivalence) = ρs a = ds a dt + as a b2 r Π as a + απ a z z, dz a = (a b2 dt r Π a)z a b2 r s a + α z, z = z a df (a), A z = Φ(z). replicatig step In a system of N agents, agent i uses its own parameter a i to determine u i = b r (Π a i z i + s ai ), 1 i N, decentralized!

16 The mean field LQG game Main results Nonlinear models: some more detail Main results: existence, and ε-nash equilibrium Theorem (Existence and Uniqueness) Under mild assumptions, the mean field solution system has a unique bounded solution ( z a, s a ), a A. Let s i = s ai be pre-computed from the NCE equation system and u 0 i = b r (Π iz i + s i ), 1 i N. Theorem (Nash equilibria, CDC 03, TAC 07) The set of strategies {u 0 i, 1 i N} results in an ε-nash equilibrium w.r.t. costs J i(u i, u i ), 1 i N, i.e. (diminishing value of centralized information), J i (ui 0, u i) 0 ε inf J i (u i, u 0 u i i) J i (ui 0, u i) 0 where 0 < ε 0 as N, and u i depends on (t, z 1,..., z N ).

17 The mean field LQG game Main results Nonlinear models: some more detail The nonlinear case For the nonlinear diffusion model (CIS 06): HJB equation: V t = inf u U { f [x, u, µ t ] V } x + L[x, u, µ t] + σ2 2 V 2 x 2 V (T, x) = 0, (t, x) [0, T ) R. Optimal Control : u t = φ(t, x µ ), (t, x) [0, T ] R. Closed-loop McK-V equation (which can be written as Fokker-Planck equation): dx t = f [x t, φ(t, x µ ), µ t ]dt + σdw t, 0 t T. The NCE methodology amounts to finding a solution (x t, µ t ) in McK-V sense. Extension by V. Kolokoltsov, W. Yang, J. Li. (Preprint 11)

18 Social optimization Major-minor players Robustness Application to Capital accumulation game E1: The model Individual dynamics (N agents): Individual costs: dx i = A(θ i )x i dt + Bu i dt + DdW i, 1 i N. J i =E where Φ(x (N) ) = Γx (N) + η Specification 0 e ρt { x i Φ(x (N) ) 2 Q + u T i Ru i } dt, θ i : dynamic parameter, u i : control, W i : noise x (N) = (1/N) N i=1 x i: mean field coupling term The social cost: J (N) soc = N i=1 J i. The objective: minimize J (N) soc = Pareto optima.

19 Social optimization Major-minor players Robustness Application to Capital accumulation game The SCE equation system The Social Certainty Equivalence (SCE) equation system: ρs θ = ds θ dt + (AT θ Π θ BR 1 B T )s θ [(Γ T Q + QΓ Γ T QΓ) x + (I Γ T )Qη], d x θ = A θ x θ BR 1 B T (Π θ x θ + s θ ), dt x = x θ df (θ), where x θ (0) = m 0 and s θ is sought within C ρ/2 ([0, ), R n ).

20 Social optimization Major-minor players Robustness Application to Capital accumulation game The social optimality theorem Theorem Under some technical conditions, the set of SCE based control laws û i = R 1 B T (Π θi ˆx i + s θi ), 1 i N has asymptotic social optimality, i.e., for û = (û 1,..., û N ), (1/N)J (N) soc (û) inf u U o (1/N)J (N) soc (u) = O(1/ N + ϵ N ), where lim N ϵ N = 0 and U o is defined as a set of centralized information based controls.

21 Social optimization Major-minor players Robustness Application to Capital accumulation game Cost comparison (mean field game v.s. social optimum Social cost per agent NCE based cost Cost difference γ

22 Social optimization Major-minor players Robustness Application to Capital accumulation game E2: Dynamics with a major player The LQG game with mean field coupling: dx 0 (t) = [ A 0 x 0 (t) + B 0 u 0 (t) + F 0 x (N) (t) ] dt + D 0 dw 0 (t), t 0, dx i (t) = [ A(θ i )x i (t) + Bu i (t) + F x (N) (t) + Gx 0 (t) ] dt + DdW i (t), x (N) = 1 N N i=1 x i mean field term (average state of minor players). Major player A 0 with state x 0 (t), minor player A i with state x i (t). W 0, W i are independent standard Brownian motions, 1 i N. We introduce the following assumption: (A1) θ i takes its value from a finite set Θ = {1,..., K} with an empirical distribution F (N), which converges when N.

23 Social optimization Major-minor players Robustness Application to Capital accumulation game Individual costs The cost for A 0 : J 0 (u 0,..., u N ) = E Φ(x (N) ) = H 0 x (N) + η 0 : cost coupling term The cost for A i, 1 i N: J i (u 0,..., u N ) = E 0 0 e ρt{ x0 Φ(x (N) ) 2 Q 0 + u T 0 R 0 u 0 }dt, e ρt{ x i Ψ(x 0, x (N) ) 2 Q + ut i Ru i }dt, Ψ(x 0, x (N) ) = Hx 0 + Ĥx (N) + η: cost coupling term. The presence of x 0 in the dynamics and cost of A i shows the strong influence of the major player A 0.

24 Social optimization Major-minor players Robustness Application to Capital accumulation game A matter of sufficient statistics One might conjecture asymptotic Nash equilibrium strategies of the form: x 0 (t) would be sufficient statistic for A 0 s decision = u 0 (t, x 0 (t)) ; (x 0 (t), x i (t)) would be sufficient statistics for A i s decision = u i (t, x 0 (t), x i (t)). Fact: The above conjecture fails! Theorem (ε-nash equilibrium) Under some technical conditions, a set of decentralized strategies of the form (u 0 [t, x 0 (t), z(t)], u i [t, x 0 (t), z(t), x i (t)]) is an ε-nash equilibrium as N. (see Huang, SICON 10 for detail.) For the case θ i from a continuum, see (Nguyen and Huang, 12): random Gaussian approximation with a kernel function.

25 Social optimization Major-minor players Robustness Application to Capital accumulation game E3: Robustness: local/global unknown disturbance Tembine, Basar, et al. (2012): local disturbance as an adversarial player: embed a saddle point solution of the local players into the MFG. J. Huang and M. Huang (preprint, 2015): a common unknown disturbance, worst case optimization;

26 Social optimization Major-minor players Robustness Application to Capital accumulation game E3: Robustness (ctn) Consider N players, 1 i N: dx i (t) = (Ax i (t) + Bu i (t) + Gx (N) (t) + f (t))dt + DdW i (t), [ T J i (u i, u i, f ) = E ( x i (Γx (N) + η) 2Q + uti Ru i 1γ ) f (t) 2 dt 0 ] + xi T (T )Hx i (T ), where x (N) = 1 N N j=1 x j; f is an unknown L 2 (0, T ; R n ) signal. The worse case cost Ji wo (u i, u i ) = sup J i (u i, u i, f ). f L 2 (0,T ;R n )

27 Social optimization Major-minor players Robustness Application to Capital accumulation game Robustness (ctn) Main result: Under some conditions, we may construct (û 1,..., û N ), where each û i is determined from solving a limiting robust control (minimax control) problem. û i is determined from a forward-backward SDE system driving by W i. The robust ε N -Nash equilibrium for the N players, i.e., Ji wo (û i, û i ) ε N inf u i U Jwo i (u i, û i ) Ji wo (û i, û i ). where ε N 0 as n 0. u i depends on (W 1,..., W N ).

28 Social optimization Major-minor players Robustness Application to Capital accumulation game Mean field capital accumulation game: dynamics X i t : output (or wealth) of agent i, 1 i N u i t [0, X i t ]: capital stock (so no borrowing) c i t = X i t u i t: amount for consumption u (N) t = (1/N) N j=1 uj t: aggregate capital stock level The next stage output, measured by the unit of capital, is Xt+1 i = G(u (N) t, Wt i )ut, i t 0, (3.1) Regard u (N) t as being measured according to a macroscopic unit. See Olson and Roy (2006) for a survey on stochastic growth theory.

29 Social optimization Major-minor players Robustness Application to Capital accumulation game The utility functional The utility functional is J i (u i, u i ) = E T t=0 ρt v(x i t u i t), ρ (0, 1]: the discount factor c i t = X i t u i t: consumption, u i = (, u i 1, u i+1, ) We take the HARA utility v(z) = 1 γ zγ, z 0, γ (0, 1). Main results: (i) The mean field game equation system has a solution (proved by fixed point theorem), (ii) The set of decentralized strategies obtained is an ε-nash equilibrium. (for more detail, see Huang, DGAA 13)

30 Social optimization Major-minor players Robustness Application to Capital accumulation game Mean field dynamics with infinite horizon: nonlinear phenomena p t+1 = Q mf (p t ). The blue curve is Q mf p p p (a) stable equilibrium (b) limit cycle (c) chaos Look for a stationary solution for the infinite horizon mean field capital accumulation game Check stability of the mean field induced from the stationary solution.

31 Social optimization Major-minor players Robustness Application to Capital accumulation game Continuous time modeling: Cobb-Douglas with HARA The dynamics: dx t = A(m t )X α t dt δx t dt C t dt σx t dw t, (3.2) The utility functional: J = 1 [ T ] γ E e ρt Ct γ dt + e ρt ηλ(m T )X γ T. (3.3) 0 F (m, x) = A(m)x α is a mean field version of the Cobb-Douglas production function with capital x and a constant labor size. The function λ > 0 is continuous and decreasing on [0, ). Take the standard choice γ = 1 α (equalizing the coefficient of the relative risk aversion to capital share)

32 Social optimization Major-minor players Robustness Application to Capital accumulation game Continuous time modeling: Cobb-Douglas with HARA The solution equation system of the mean field game reduces to [ ] ṗ(t) = ρ + σ2 γ(1 γ) 2 + δγ p(t) (1 γ)p γ γ 1 (t) ḣ(t) = ρh(t) A(m t )γp(t), { [ dz t = γa(m t ) 1 γ m t = EZt (= EX t ), γδ γφ 1 (t) σ2 γ(1 γ) 2 ] Z t } dt γσz t dw t, where p(t ) = λ(m T )η and h(t ) = 0. φ(t) can be explicitly determined by λ(m T ) and other constant parameters. Existence = fixed point problem. Fix m t ; uniquely solve p, h; further solve Z t (m( )). Then m t = EZt (m( )). 1 γ

33 Mean field game theory via interacting particles has evolved into a major research area with many applications. It adopts ideas from statistical physics.

34 Related literature: peer models (i.e., comparably small; only a partial list) J.M. Lasry and P.L. Lions (2006a,b, JJM 07): Mean field equilibrium; O. Gueant (JMPA 09); GLL 11 (Springer): Human capital optimization G.Y. Weintraub et. el. (NIPS 05, Econometrica 08): Oblivious equilibria for Markov perfect industry dynamics; S. Adlakha, R. Johari, G. Weibtraub, A. Goldsmith (CDC 08): further generalizations with OEs M. Huang, P.E. Caines and R.P. Malhame (CDC 03, 04, CIS 06, TAC 07): Decentralized ε-nash equilibrium in mean field dynamic games; M. Nourian, Caines, et. al. (TAC 12): collective motion and adaptation; A. Kizilkale and P. E. Caines (Preprint 12): adaptive mean field LQG games T. Li and J.-F. Zhang (IEEE TAC 08): Mean field LQG games with long run average cost; M. Bardi (Net. Heter. Media 12) LQG H. Tembine et. al. (GameNets 09): Mean field MDP and team; H. Tembine, Q. Zhu, T. Basar (IFAC 11): Risk sensitive mean field games

35 Related literature (ctn) A. Bensoussan et. al. (2011, 2012, Preprints) Mean field LQG games (and nonlinear diffusion models). H. Yin, P.G. Mehta, S.P. Meyn, U.V. Shanbhag (IEEE TAC 12): Nonlinear oscillator games and phase transition; Yang et. al. (ACC 11); Pequito, Aguiar, Sinopoli, Gomes (NetGCOOP 11): application to filtering/estimation D. Gomes, J. Mohr, Q. Souza (JMPA 10): Finite state space models V. Kolokoltsov, W. Yang, J. Li (preprint 11): Nonlinear markov processes and mean field games Xu and Hajek (2012): mean field supermarket games (cost results from sampling and waiting) Z. Ma, D. Callaway, I. Hiskens (IEEE CST 13): recharging control of large populations of electric vehicles Y. Achdou and I. Capuzzo-Dolcetta (SIAM Numer. 11): Numerical solutions to mean field game equations (coupled PDEs)

36 Related literature (ctn) R. Buckdahn, P. Cardaliaguet, M. Quincampoix (DGA 11): Survey R. Carmona and F. Delarue (Preprint 12): McKean-Vlasov dynamics for players, and probabilistic approach R. E. Lucas Jr and B. Moll (Preprint 11): Economic growth (a trade-off for individuals to allocate time for producing and acquiring new knowldg) M. Balandat and C. J. Tomlin, Efficiency of MFG, ACC 13. Rome University Mean Field Game Workshop, May 2011 A. Bensoussan et. al. (DGAA 13), time consistency strategies in LQG MFGs; B. Djehiche and M. Huang (Preprint 13), time consistency and SMP in nonlinear case. Padova University MFG Workshop, Aug Huang, Caines, Malhamé (2006); Sen and Caines (2014); Kizilkale and Caines (2015): partial information models.

37 Related literature (ctn): major player models Huang (SICON 10): LQG models with minor players parameterized by a finite parameter set; develop state augmentation B.-C. Wang and J.-F. Zhang (Preprint 11): Markovian switching models S. Nguyen and Huang (SICON 12) random Gaussian mean field approximation with continuum parameters; Bensoussan et. al. (2013) M. Nourian and P.E. Caines (SICON 13): Nonlinear diffusion models R. Buckdahn, J. Li and S. Peng (Preprint 13 )

38 Related literature (ctn): Mean field type optimal control: D. Andersson and B. Djehiche (AMO 11): Stochastic maximum principle J. Yong (Preprint 11): control of mean field Volterra integral equations T. Meyer-Brandis, B. Oksendal and X. Y. Zhou (2012): SMP. R. Elliott, X. Li, and Y.-H. Ni (Auotmatica 13): discrete time LQG and Riccati equations. There is only a single decision maker. It affects the mean of the underlying state process.

39 I particular, there are applications of MFGs to economic growth and finance. Guéant, Lasry and Lions (2011): human capital optimization Lucas and Moll (2011): Knowledge growth and allocation of time (JPE in press) Carmona and Lacker (2013): Investment of n brokers Huang (2013): capital accumulation with congestion effect Lachapelle et al. (2013): price formation Espinosa and Touzi (2013): Optimal investment with relative 1 performance concern (depending on N 1 j X j )...

40 Thank you!