LECTURES ON MEAN FIELD GAMES: II. CALCULUS OVER WASSERSTEIN SPACE, CONTROL OF MCKEAN-VLASOV DYNAMICS, AND THE MASTER EQUATION

Transcription

1 LECTURES ON MEAN FIELD GAMES: II. CALCULUS OVER WASSERSTEIN SPACE, CONTROL OF MCKEAN-VLASOV DYNAMICS, AND THE MASTER EQUATION René Carmona Department of Operations Research & Financial Engineering PACM Princeton University Minerva Lectures, Columbia U. October 2016

2 THE ANALYTIC (PDE) APPROACH TO MFGS For fixed µ = (µ t ) t, the value function V µ (t, x) = inf E [ T f (s, X s, µ s, α s)ds + g(x T, µ T ) Xt = x ] (α s) t s T t solves a HJB (backward) equation t V µ (t, x) + inf α [b(t, x, µ t, α) x V µ (t, x) + f (t, x, µ t, α)] with terminal condition V µ (T, x) = g(x, µ T ) 1 2 trace[σ(t, x) σ(t, x) 2 xx V µ (t, x)] = 0 The fixed point step is implemented by requiring that t µ t solves the (forward) Kolmogorov equation t µ t = µ t L t This is also a nonlinear PDE because µ t appears in b... System of strongly coupled nonlinear PDEs! Time goes in both directions

3 HJB EQUATION FROM ITÔ S FORMULA Classical Optimal Control set-up (µ fixed) Dynamic Programming Principle t V µ (t, X t) is a martingale when (X t) 0 t T is optimal Classical Itô formula to compute: d tv µ (t, X t) when (t, x) V µ (t, x) is smooth and dx t = b(t, X t, ˆα t)dt + σ(t, X t, ˆα t)dw t is optimal to set the drift to 0 get HJB

4 MFG COUNTERPART MFG is not an optimization problem per-se Optimal control arguments (for µ fixed) affected by fixed point step What is the effect of last step substitution µ t = P Xt? In equilibrium, do we still have: Dynamic Programming Principle? Martingale property of t V µ (t, X t ) What would be the right Itô formula to compute: d tv µ (t, X t) when dx t = b(t, X t, ˆα t)dt + σ(t, X t, ˆα t)dw t is optimal and µ t = P Xt?

5 MORE REASONS TO DIFFERENTIATE FUNCTIONS OF MEASURES Back to the N-player games (with reduced or distributed controls): dx i t = b ( t, X i t, µn t, φ(t, X i t, µn t ) ) dt + σ ( t, X i t, µn t, φ(t, X i t, µn t ) ) dw i t, t [0, T ], Propagation of Chaos X 1 t,, X k t, become independent in the limit N X i = (X i t ) 0 t T = X = (X t ) 0 t T solution of the McKean Vlasov equation: dx t = b ( t, X t, P Xt, φ(t, X t, P Xt ) ) dt + σ ( t, X t, P Xt, φ(t, X t, P Xt ) ) dw t, t [0, T ], where W = (W t ) 0 t T is a standard Wiener process. Expected Costs: converge to: [ T J i (φ) = E f ( t, X i t, µn t, φ(t, X i t, µn t ) ) dt + g ( X i T, ) ] µn T, 0 [ T J(φ) = E f ( t, X t, P Xt, φ(t, X t, P Xt ) ) dt + g ( ) ] X T, P XT. 0 Optimization after the limit: Control of McKean-Vlasov equations!

6 TAKING STOCK SDE State Dynamics for N players McKean Vlasov Dynamics Fixed Point lim N Optimization Optimization Nash Equilibrium for N players Fixed Point lim N Mean Field Game? Controlled McKean-Vlasov SDE? Is the above diagram commutative?

7 CONTROLLED MCKEAN-VLASOV SDES [ T inf E f ( ) ( ) ] t, X t, P Xt, α t dt + g XT, P XT α=(α t ) 0 t T 0 under dynamical constraint dx t = b ( t, X t, P Xt, α t ) dt + σ ( t, Xt, P Xt, α t ) dwt. State (X t, P Xt ) infinite dimensional State trajectory t (X t, µ t ) is a very thin submanifold due to constraint µ t = P Xt Open loop form: α = (α t ) 0 t T adapted Closed loop form: α t = φ(t, X t, P Xt ) Whether we use Infinite dimensional HJB equation Pontryagin stochastic maximum principle with Hamiltonian H(t, x, µ, y, z, α) = b(t, x, µ, α) y + σ(t, x, µ, α) z + f (t, x, µ, α) and introduce the adjoint equations, WE NEED TO DIFFERENTIATE FUNCTIONS OF MEASURES!

8 DIFFERENTIABILITY OF FUNCTIONS OF MEASURES M(R d ) space of signed (finite) measures on R d Banach space (dual of a space of continuous functions) Classical differential calculus available If M(R d ) m φ(m) R "φ is differentiable" has a meaning For m 0 M(R d ) one can define δφ(m 0 ) δm ( ) as a function on R d in Fréchet or Gâteaux sense Bensoussan-Frehe-Yam alternative is to work only with measures with densities and view φ as a function on L 1 (R d, dx)!

9 TOPOLOGY ON WASSERSTEIN SPACE Measures appearing in MFG theory are probability distributions of random variables!!! Wasserstein space { } P 2 (R d ) = µ P(R d ); x 2 dµ(x) < R d Metric space for the 2-Wasserstein distance [ ] 1/2 W 2 (µ, ν) = inf x y 2 π(dx, dy) π Π(µ,ν) R d R d where Π(µ, ν) is the set of probability measures coupling µ and ν. Topological properties of Wasserstein space well understood as following statements are equivalents µ N µ in Wasserstein space µ N µ weakly and x 2 µ N (dx) x 2 µ(dx)

10 GLIVENKO-CANTELLI IN WASSERSTEIN SPACE X 1, X 2,, i.i.d. random variables in R d with common distribution µ s.t. M q(µ) = x q µ(dx) <. R d If q = 2, [ ] P lim W 2(µ N, µ) = 0 N = 1. where µ N = 1 N N i=1 δ X i is a (random) empirical measure. Standard LLN! Crucial Estimate: Glivenko-Cantelli If q > 4 for each dimension d 1, C = C(d, q, M q(µ)) s.t. for all N 1: E [ W 2 (µ N, µ) 2] N 1/2, if d < 4, C N 1/2 log N, if d = 4, (1) N 2/d, if d > 4.

11 DIFFERENTIAL CALCULUS ON WASSERSTEIN SPACE What does it mean "φ is differentiable" or "φ is convex" for P 2 (R d ) µ φ(µ) R Wasserstein space P 2 (R d ) is a metric space for W 2 Optimal transportation (Monge-Ampere-Kantorovich) Curve length and shortest paths (geodesics) Notion of convex function on P 2 (R d ) Tangent spaces and differential geometry on P 2 (R d ). Differential calculus on Wasserstein space Brenier, Benamou, Ambrosio, Gigli, Otto, Caffarelli, Villani, Carlier,...

12 DIFFERENTIABILITY IN THE SENSE OF P.L.LIONS If P 2 (R d ) µ φ(µ) R is "differentiable" on Wasserstein space what about ( 1 R dn (x 1,, x N ) u(x 1,, x N ) = φ N How does φ(µ) relate to x i u(x 1,, x N )? N ) δ x j? j=1 Lions Solution Lift φ up to L 2 ( Ω, F, P) into φ defined by φ(x) = φ( P X ) Use Fréchet differentials on flat space L 2 Definition of L-differentiability φ is differentiable at µ 0 if φ is Fréchet differentiable at X 0 s.t. P X0 = µ 0 Check definition is intrinsic

13 PROPERTIES OF L-DIFFERENTIALS φ(µ 0 ) = D φ(x 0 ) L 2 ( Ω, F, P) The distribution of the random variable φ(µ 0 ) depends only on µ 0, NOT ON THE RANDOM VARIABLE X 0 used to represent it ξ : R d R d uniquely defined µ 0 a.e. such that φ(µ 0 ) = D φ(x 0 ) = ξ(x 0 ) we use φ(µ 0 )( ) = ξ Examples φ(µ) = φ(µ) = h(x)µ(dx) = φ(µ)( ) = h( ) R d h(x y)µ(dx)µ(dy) = φ(µ)( ) = [2 h( ) µ]( ) R d R d φ(µ) = ϕ(x, µ)µ(dx) = φ(µ)( ) = x ϕ(, µ) + µϕ(x, µ)( )µ(dx ) R d R d

14 TWO MORE EXAMPLES Assume φ : P 2 (R d ) R is L-differentiable and define ( 1 N ) φ N : R d R d (x 1,, x N ) φ N (x 1,, x N ) = φ δ N x i i=1 x i φ N (x 1,, x N ) = 1 N µφ ( 1 N N ) δ x i (x i ) i=1 Assume φ : M 2 (R d ) R has a linear functional derivative (at least in a neighborhood of P 2 (R d ) and that R d x [δφ/δm](m)(x) is differentiable and the derivative [ δφ ] M 2 (R d ) R d (m, x) x (m)(x) R d δm is jointly continuous in (m, x) and is of linear growth in x, then φ is L-differentiable and µφ(µ)( ) = x δφ δm (µ)( ), µ P 2(R d ).

15 CONVEX FUNCTIONS OF MEASURES φ : P 2 (R d ) R is said to be L-convex if whenever P X = µ and P X = µ. µ, µ φ(µ ) φ(µ) E[ µφ(µ)(x) (X X)] 0, Example1 ( ) µ φ(µ) = g ζ(x)dµ(x) R d, for g : R R is non-decreasing convex differentiable and ζ : R d R convex differentiable with derivative of at most of linear growth Example2 µ φ(µ) = g(x, x )dµ(x)dµ(x ) R d R d If g : R d R d R is convex differentiable ( g linear growth) A sobering counter-example. If µ 0 P 2 (E) is fixed, the square distance function may not be convex or even L-differentiable! P 2 (E) µ W 2 (µ 0, µ) 2 R

16 BACK TO THE CONTROL OF MCKEAN-VLASOV EQUATIONS [ T inf E f ( ) ( ) ] t, X t, P Xt, α t dt + g XT, P XT α=(α t ) 0 t T 0 under the dynamical constraint dx t = b ( t, X t, P Xt, α t ) dt + σ ( t, Xt, P Xt, α t ) dwt.

17 EXAMPLE: POTENTIAL MEAN FIELD GAMES Start with Mean Field Game à la Lasry-Lions [ T [ 1 inf E α=(α t ) 0 t T, dx t =α t dt+σdw t 0 2 α t 2 + f (t, X t, µ t ) ] ] dt + g(x T, µ T ) s.t. f and g are differentiable w.r.t. x and there exist differentiable functions F and G x f (t, x, µ) = µf (t, µ)(x) and x g(x, µ) = µg(µ)(x) Solving this MFG is equivalent to solving the central planner optimization problem [ T [ 1 inf E α=(α t ) 0 t T, dx t =α t dt+σdw t 0 2 α t 2 + F (t, P Xt ) ] ] dt + G(P XT ) Special case of McKean-Vlasov optimal control

18 THE ADJOINT EQUATIONS Lifted Hamiltonian H(t, x, X, y, α) = H(t, x, µ, y, α) for any random variable X with distribution µ. Given an admissible control α = (α t ) 0 t T and the corresponding controlled state process X α = (Xt α ) 0 t T, any couple (Y t, Z t ) 0 t T satisfying: dy t = x H(t, Xt α, P X α, Y t t, α t )dt + Z t dw t Ẽ[ µh(t, X t, X, Ỹt, α t )] X=X α dt t Y T = x g(xt α, P X T α ) + Ẽ[ µ g(x, X t )] x=x α T where ( α, X, Ỹ, Z ) is an independent copy of (α, X α, Y, Z ), is called a set of adjoint processes BSDE of Mean Field type according to Buckhdan-Li-Peng!!! Extra terms in red are the ONLY difference between MFG and Control of McKean-Vlasov dynamics!!!

19 PONTRYAGIN MAXIMUM PRINCIPLE (SUFFICIENCY) Assume 1. Coefficients continuously differentiable with bounded derivatives; 2. Terminal cost function g is convex; 3. α = (α t) 0 t T admissible control, X = (X t) 0 t T corresponding dynamics, (Y, Z) = (Y t, Z t) 0 t T adjoint processes and is dt dp a.e. convex, then, if moreover (x, µ, α) H(t, x, µ, Y t, Z t, α) H(t, X t, P Xt, Y t, Z t, α t) = inf α A H(t, Xt, P X t, Y t, α), a.s. Then α is an optimal control, i.e. J(α) = inf β A J(β).

20 PARTICULAR CASE: SCALAR INTERACTIONS b(t, x, µ, α) = b(t, x, ψ, µ, α) σ(t, x, µ, α) = σ(t, x, φ, µ, α) f (t, x, µ, α) = f (t, x, γ, µ, α) g(x, µ) = g(x, ζ, µ ) ψ, φ, γ and ζ differentiable with at most quadratic growth at, b, σ and f differentiable in (x, r) R d R for t, α) fixed g differentiable in (x, r) R d R. Recall that the adjoint process satisfies Y T = xg(x T, P XT ) + Ẽ[ µg( X T, P XT )(X T )]. but since µg(x, µ)(x ) = r g ( x, ζ, µ ) ζ(x ), the terminal condition reads Y T = x g ( X T, E[ζ(X T )] ) + Ẽ [ r g ( XT, E[ζ(X T )] )] ζ(x T ) Convexity in µ follows convexity of g

21 SCALAR INTERACTIONS (CONT.) H(t, x, µ, y, z, α) = b(t, x, ψ, µ, α) y + σ(t, x, φ, µ, α) z + f (t, x, γ, µ, α). µh(t, x, µ, y, z, α) can be identified wih µh(t, x, µ, y, z, α)(x ) = [ r b(t, x, ψ, µ, α) y ] ψ(x ) + [ r σ(t, x, φ, µ, α) z ] φ(x ) + r f (t, x, γ, µ, α) γ(x ) and the adjoint equation rewrites: { dy t = x b(t, Xt, E[ψ(X t )], α t ) Y t + x σ(t, X t, E[φ(X t )], α t ) Z t } + x f (t, Xt, E[γ(X t )], α t ) dt + Z t dw t { Ẽ [ r b(t, Xt, E[ψ( X ] t )], α t ) Ỹt ψ(xt ) + Ẽ [ r σ(t, X t, E[φ( X t )], α t ) Z ] t φ(xt ) Anderson - Djehiche +Ẽ [ r f ((t, Xt, E[γ( X t )], α t ) ] } γ(x t ) dt

22 SOLUTION OF THE MCKV CONTROL PROBLEM Assume b(t, x, µ, α) = b 0 (t) R d xdµ(x) + b 1 (t)x + b 2 (t)α with b 0, b 1 and b 2 is R d d -valued and are bounded. f and g as in MFG problem. Thn there exists a solution (X, Y, Z) = (X t, Y t, Z t ) 0 t T of the McKean-Vlasov FBSDE dx t = b 0 (t)e(x t )dt + b 1 (t)x t dt + b 2 (t)ˆα(t, X t, P Xt, Y t )dt + σdw t, dy t = x H ( ) t, X t, P Xt, Y t, ˆα t dt E [ )] µ H( t, Xt, X t, Ỹt, ˆα t dt + Zt dw t. with Y t = u(t, X t, P Xt ) for a function u : [0, T ] R d P 1 (R d ) (t, x, µ) u(t, x, µ) uniformly of Lip-1 and with linear growth in x.

23 A FINITE PLAYER APPROXIMATE EQUILIBRIUM For N independent Brownian motions (W 1,..., W N ) and for a square integrable exchangeable process β = (β 1,..., β N ), consider the system dx i t = 1 N b 0(t) N j=1 X j t + b 1 (t)x i t + b 2 (t)β i t + σdw i t, X i 0 = ξi 0, and define the common cost [ T J N (β) = E f ( s, X i s, µn s, ) ( βi 1 s ds + g X T, ) ] µn T, with µ N t = 1 N δ 0 N X i. t i=1 Then, there exists a sequence (ɛ N ) N 1, ɛ N 0, s.t. for all β = (β 1,..., β N ), where, α = (α 1,, α N ) with J N (β) J N (α) ɛ N, α i t = ˆα(s, X i i t, u(t, X t ), P X t ) where X and u are from the solution to the controlled McKean Vlasov problem, and ( X 1,..., X N ) is the state of the system controlled by α, i.e. d X i t = 1 N N j=1 b 0 (t) X j t + b 1 (t) X i t + b 2 (t) ˆα(s, X i s, u(s, X i s ), P Xs ) + σdw i t, X i 0 = ξi 0.

24 APPLICATION #2: CHAIN RULE Assume where dx t = b t dt + σ t dw t, X 0 L 2 (Ω, F, P), W = (W t ) t 0 is a F-Brownian motion with values in R d (b t ) t 0 and (σ t ) t 0 are F-progressive processes in R d and R d d Assume T > 0, [ T ( E bt 2 + σ t 4) ] dt < +. 0 Then for any t 0, if µ t = P Xt, and a t = σ t σ t then: t u(µ t ) = u(µ 0 ) + E [ ] 1 t µu(µ s)(x s) b s ds + E [ ( ) ] v µu(µ s) (X s) a s ds

25 CONTROL OF MCKEAN-VLASOV SDES: VERIFICATION THEOREM Problem: if f : P 2 (R d ) R, minimize T J(α) = f ( [ ) T ] 1 P X α dt + E 0 t 0 2 αt 2 dt under the constraint: dx α t = α t dt + dw t, 0 t T, Verification Argument: Assume u : [0, T ] P 2 (R d ) R is C 1,2, and satisfies t u(t, µ) 1 µu(t, µ)(v) 2 dµ(v) + 1 [ ] 2 R d 2 trace v µu(t, µ)(v)dµ(v) + f (µ) = 0, R d then, the McKean-Vlasov SDE d ˆX t = µu ( t, P ˆXt ) ( ˆXt )dt + dw t, 0 t T, has a unique solution ( ˆX t ) 0 t T satisfying E[sup 0 t T ˆX t 2 ] < which is the unique optimal path since ˆα t = µu(t, P ˆXt )( ˆX t ) minimizes the cost: J( ˆα) = inf α A J(α).

26 PROOF (SKETCH OF) For a generic admissible control α = (α t ) 0 t T, set X α t chain rule: du ( ) t, P X α t = [ t u ( ) [ t, P X + E ( )( α αt µu t, P X α X t t [ = f ( ) 1 P X α + t = X 0 + T αsds + Wt and apply the 0 ) αt ] E [trace [ v µu ( t, P X α t )( X α t [ µu 2 E ( )( α) ] t, P X α X 2 t t + E[ ( )( α) ] ] µu t, P X α X t t αt dt [ = f ( ) 1 P X α t 2 E[ α t 2] + 1 [ 2 E α t + ( )( α) ]] µu t, P X α X 2 t t dt )] ]] dt where we used the PDE satisfied by u, and identified a perfect square. Integrate both sides and get: J(α) = u ( [ ) 1 T [ 0, P X0 + 2 E α t + ( )( α) ] ] µu t, P X α X 2 0 t t dt, which shows that α t = µu(t, P X α )(X α t t ) is optimal.

27 JOINT CHAIN RUILE If u is smooth If dξ t = η t dt + γ t dw t If dx t = b t dt + σ t dw t and µ t = P Xt t u(t, ξ t, µ t ) = u(0, ξ 0, µ 0 ) + x u(s, ξ s, µ s) (γ ) sdw s 0 t ( + t u(s, ξ s, µ s) + x u(s, ξ s, µ s) η s trace[ 2 xx u(s, ξs, ] ) µs)γsγ s ds t + Ẽ [ µu(s, ξ s, µ s)( X s) b ] 1 t s ds + Ẽ [ trace ( [ v µu(s, ξ s, µ ] s) ( X s) σ s σ )] s ds where the process ( X t, b t, σ t ) 0 t T is an independent copy of the process (X t, b t, σ t ) 0 t T, on a different probability space ( Ω, F, P)

28 DERIVING THE MASTER EQUATION If (t, x, µ) U(t, x, µ) is the master field ( t U(t, X t, µ t ) f ( s, X s, µ s, ˆα(s, X s, µ s, Y ) ) s) ds 0 0 t T is a martingale whenever (X t, Y t, Z t ) 0 t T is the solution of the forward-backward system characterizing the optimal path under the flow of measures (µ t ) 0 t T. So if we compute its Itô differential, the drift must be 0

29 AN EXAMPLE OF DERIVATION Itô s Formula with µ t = P Xt dx t = b(t, X t, µ t, α t )dt + dw t H(t, x, µ, y, α) = b(t, x, µ, α) y + f (t, x, µ, α) ˆα(t, x, µ, y) = arg inf H(t, x, µ, y, α) α (set ˆα t = ˆα(t, X t, µ t, U(t, X t, µ t )) and b t = b(t, X t, µ t, ˆα t )) du(t, X t, µ t ) = ( t U(t, X t, µ t ) + b t x U(t, X t, µ t ) trace[ 2 xx U(t, X t, µ t )] + f ( t, x, µ, ˆα t ) ) dt [ + E b t µu(t, X t, µ t )(X t ) v µu(t, X t, µ t ) ] ] (X t )] dt + x U(t, X t, µ t )dw t

30 THE ACTUAL MASTER EQUATION t U(t, x, µ) + b ( t, x, µ, ˆα(t, x, µ, U(t, x, µ)) ) x U(t, x, µ) + 1 ] [ 2 trace xx 2 U(t, x, µ) + f ( t, x, µ, ˆα(t, x, µ, U(t, x, µ)) ) [ + b ( t, x, µ, ˆα(t, x, µ, U(t, x, µ)) ) µu(t, x, µ)(x ) R d trace ( v µu(t, x, µ)(x )) ] dµ(x ) = 0, for (t, x, µ) [0, T ] R d P 2 (R d ), with the terminal condition V (T, x, µ) = g(x, µ).