OPTIMAL CONTROL OF MCKEAN-VLASOV DYNAMICS

Transcription

1 OPTIMAL CONTROL OF MCKEAN-VLASOV DYNAMICS René Carmona Deparmen of Operaions Research & Financial Engineering Bendheim Cener for Finance Princeon Universiy CMU, June 3, 2015 Seve Shreve s 65h Birhday

2 CREDITS Join Work wih François Delarue (Nice) (series of papers and forhcoming book)

3 CLASSICAL STOCHASTIC DIFFERENTIAL CONTROL applez T inf E f (, X, )d + g(x T,µ T ) 2A 0 subjec o dx = b(, X, )d + (, X, )dw ; X 0 = x 0. I Analyic Approach (by PDEs) I HJB equaion I Probabilisic Approaches (by FBSDEs) 1. Represen value funcion as soluion of a BSDE 2. Represen he gradien of he value funcion as soluion of a FBSDE (Sochasic Maximum Principle)

4 I. FIRST PROBABILISTIC APPROACH Assumpions I I is unconrolled is inverible Reduced Hamionian For each conrol solve BSDE H(, x, y, )=b(, x, ) y + f (, x, ) dy = H(, X, Z (, X ) 1, )d + Z dw, Y T = g(x T ) Then applez T Y0 = J( ) =E f (, X, )d + g(x T,µ T ) 0 So by comparison heorems for BSDEs, opimal conrol ˆ given by: ˆ =ˆ (, X, Z (, X ) 1 ), wih ˆ (, x, y) 2 argmin 2A H(, x, y, ) and Y 0 = J(ˆ )

5 II. PONTRYAGIN STOCHASTIC MAXIMUM APPROACH Assumpions I Coefficiens b, and f differeniable I f convex in (x, ) and g convex Hamionian H(, x, y, z, )=b(, x, ) y + (, x, ) z + f (, x, ) For each conrol solve BSDE for he adjoin processes Y =(Y ) and Z =(Z ) dy xh(, X, Y, Z, )d + Z dw, Y T xg(x T ) Then, opimal conrol ˆ given by: ˆ =ˆ (, X, Y, Z ), wih ˆ (, x, y, z) 2 argmin 2A H(, x, y, z, ) and Y ˆ 0 = J(ˆ )

6 SUMMARY In boh cases ( unconrolled), need o solve a FBSDE ( dx = B(, X, Y, Z )d + (, X )dw, dy = F(, X, Y, Z )d + Z dw Firs Approach B(, x, y, z) =b, x, ˆ (, x, z (, x) 1 ), F (, x, y, z) = f, x, ˆ (, x, z (, x) 1 z (, x, ) 1 b, x, ˆ (, x, z (, x) 1 ). Second Approach B(, x, y, z) =b(, x, ˆ (, x, y)), F(, x, y, z) xf (, x, ˆ (, x, y)) xb(, x, ˆ (, x, y)).

7 PROPAGATION OF CHAOS &MCKEAN-VLASOV SDES Sysem of N paricles X N,i a ime wih symmeric (Mean Field) ineracions dx N,i = b(, X N,i, µ N X )d + N (, X N,i, µ N X )dw i N, i = 1,, N where µ N X N is he empirical measure µ N x = 1 N P N i=1 x i Large populaion asympoics (N!1) 1. The N processes (X N,i ) 0appleappleT for i = 1,, N become asympoically i.i.d. 2. Each of hem is (asympoically) disribued as he soluion of he McKean-Vlasov SDE dx = b(, X, L(X ))d + (, X, L(X ))dw

8 FORWARD SDES OFMCKEAN-VLASOV TYPE dx = B, X, L(X ) d +, X, L(X ) dw, T 2 [0, T ]. Assumpion. There exiss a consan c 0 such ha (A1) For each (x,µ) 2 R d P 2 (R d ), he processes B(,, x,µ): [0, T ] 3 (!, ) 7! B(!,, x,µ) and (,, x,µ): [0, T ] 3 (!, ) 7! (!,, x,µ) are F-progressively measurable and belong o H 2,d and H 2,d d respecively. (A2) 8 2 [0, T ], 8x, x 0 2 R d, 8µ, µ 0 2P 2 (R d ), wih probabiliy 1 under P, B(, x,µ) B(, x 0,µ 0 ) + (, x,µ) (, x 0,µ 0 ) applec x x 0 +W 2 (µ, µ 0 ), where W 2 denoes he 2-Wassersein disance on he space P 2 (R d ). Resul. if X 0 2 L 2 (, F 0, P; R d ), hen here exiss a unique soluion X =(X ) 0appleappleT in S 2,d s.. for every p 2 [1, 2] h E sup X pi < +1. 0appleappleT Sznimann

9 CONTROLLING LARGE SYMMETRIC POPULATIONS Assume Mean Field Ineracions dx N,i = b(, X N,i, µ N X N, i )d + (, X N,i, µ N X N, )dw i i i = 1,, N Assume disribued sraegies i = (, X N,i ) Assume populaion is large (i.e. N = 1) 1. The N sae processes evolve independenly of each oher 2. Conrolling each of hem reduces o he opimal conrol problem apple Z T inf E f (, X, L(X ), (, X ))d + g(x T, L(X T )) 2 0 s.. dx = b(, X, L(X ), (, X ))d + (, X, L(X ), (, X ))dw X 0 = x 0. Conrol of a McKean-Vlasov SDE (Markovian - closed loop)

10 CONTROL OF MCKEAN-VLASOV DYNAMICS Mahemaical Formulaion 1. Sae dynamics given by an SDE of McKean - Vlasov ype dx = b(, X, L(X ), )d + (, X, L(X ), )dw 2. Objecive funcion o minimize of he McKean-Vlasov ype apple Z T J( ) =E f (, X, L(X ), )d + g(x T, L(X T )) 0 Could use open loop formulaion.

11 CONTROL OF MCKEAN -VLASOV SDES Sae a ime, say (X, L(X )) is infinie dimensional Analyic Approach I Infinie dimensional HJB equaions (Crandall, Lions, Ishii?) Probabilisic Approaches 1. McKean - Vlasov FBSDEs! 2. Ponryagin maximum principle approach I How should we differeniae he Hamilonian w.r.. he measure? More o come

12 N-PLAYER STOCHASTIC DIFFERENTIAL GAMES Assume Mean Field Ineracions (symmeric game) dx N,i = b(, X N,i, µ N X N, i )d + (, X N,i, µ N X N, )dw i i i = 1,, N Assume player i ries o minimize apple Z T J i ( 1,, N )=E 0 Search for Nash equilibria f (, X N,i I Very difficul in general, even if N is small I -Nash equilibria? Sill hard. I How abou in he limi N!1?, µ N X N, )d i + g(x T, µ N X T N ) Mean Field Games Lasry - Lions, Caines-Huang-Malhamé

13 MFG PARADIGM A ypical agen plays agains a coninuum of players whose saes he/she feels hrough heir disribuion µ a ime 1. For each Fixed measure flow (µ ) in P(R), solve he sandard sochasic conrol problem ( Z ) T ˆ = arg inf E f (, X,µ, )d + g(x T,µ T ) 0 subjec o dx = b(, X,µ, )d + (, X,µ, )dw 2. Fixed Poin Problem: deermine (µ ) so ha 8 2 [0, T ], L(X )=µ a.s. Once his is done one expecs ha, if ˆ = (, X ), j = (, X j ), j = 1,, N form an approximae Nash equilibrium for he game wih N players.

14 I. VALUE FUNCTION REPRESENTATION: PREP. Recall and (, x,µ, ) = (, x) uniformly Lip-1 and uniformly ellipic H(, x,µ,y, )=y b(, x,µ, )+f (, x,µ, ) ˆ (, x,µ,y) 2 arg 2 2A H(, x,µ,y, ). (A.1) b is affine in : b(, x,µ, )=b 1 (, x,µ)+b 2 () wih b 1 and b 2 bounded. (A.2) Running cos f srongly convex f (, x 0,µ, 0 ) f (, x,µ, ) h(x 0 x, 0 ),@ (x, ) f (, x,µ, )i 0 2. Then ˆ (, x,µ,y) is unique and [0, T ] R d P 2 (R d ) R d 3 (, x,µ,y)! ˆ (, x,µ,y) is measurable, locally bounded and Lipschiz-coninuous wih respec o (x, y), uniformly in (,µ) 2 [0, T ] P 2 (R d )

15 I. VALUE FUNCTION REPRESENTATION: CONT. If A R k is bounded (no really needed), if X,x =(Xs,x ) applesapplet is he unique srong soluion of dx = (, X )dw over [, T ] s.. X,x = x, and if (Ŷ,x, Ẑ,x ) is a soluion of he BSDE dŷ,x s = H(, Xs,x,µ s, Ẑ s,x (s, X,x s Ẑ,x s dw s, apple s apple T, ) 1, ˆ (s, Xs,x,µ s, Ẑ s,x (s, Xs,x ) 1 ))ds wih ŶT = g(x,x T,µ T ), hen ˆ =ˆ (s, Xs,x,µ s, Ẑ s,x (s, X,x s ) 1 ) is an opimal conrol over he inerval [, T ] and he value of he problem is given by: V (, x) =Ŷ,x. The value funcion appears as he decoupling field of an FBSDE.

16 FIXED POINT STEP =) MCKEAN-VLASOV FBSDE Saring from = 0 and dropping he superscrip,x ( dx = b(, X,µ, ˆ (, X,µ, Z (, X ) 1 ))d + (, X )dw dy = H(, X,µ, Z (, X ) 1, ˆ (, X,µ, Z (, X ) 1 ))d Z dw, for 0 apple apple T, wih ŶT = g(x T,µ T ). Implemening he fixed poin sep µ,! L(X ) gives an FBSDE of McKean-Vlasov ype.

17 II. PONTRYAGIN STOCHASTIC MAXIMUM PRINCIPLE Freeze µ =(µ ) 0appleappleT, Recall (reduced) Hamilonian H(, x,µ,y, )=b(, x,µ, ) y + f (, x,µ, ) Adjoin processes Given an admissible conrol =( ) 0appleappleT and he corresponding conrolled sae process X =(X ) 0appleappleT, any couple (Y, Z ) 0appleappleT saisfying: ( dy xh(, X,µ, Y, )d + Z dw Y T xg(xt,µ T )

18 STOCHASTIC CONTROL STEP Deermine ˆ (, x, µ,y) =arg inf H(, x,µ,y, ) Injec in FORWARD and BACKWARD dynamics and SOLVE ( dx = b(, X,µ, ˆ (, X,µ, Y ))d + (, X )dw dy x H(, X,µ, Y, ˆ (, X,µ, Y ))d + Z dw wih X 0 = x 0 and Y T x g(x T,µ T ) Sandard FBSDE (for each fixed,! µ )

19 FIXED POINT STEP Solve he fixed poin problem (µ ) 0appleappleT! (X ) 0appleappleT! (L(X )) 0appleappleT Noe: if we enforce µ = L(X ) for all 0 apple apple T in FBSDE we have ( dx = b(, X, L(X ), ˆ (, X, L(X ), Y ))d + (, X )dw, dy x H(, X, L(X ), Y, ˆ (, X, L(X ), Y ))d + Z dw wih X 0 = x 0 and Y T x g(x T, L(X T )) FBSDE of McKean-Vlasov ype!!! Very difficul

20 FBSDES OFMCKEAN -VLASOV TYPE In boh probabilisic approaches o he MFG problem he problem reduces o he soluion of an FBSDE ( dx = B(, X, L(X ), Y, Z )d + (, X, L(X ))dw, dy = F(, X, L(X ), Y, Z )d + Z dw wih in he firs approach ( B(, x,µ,y, z) =b(, x,µ,ˆ (, x,µ,z (, x) 1 )), F (, x,µ,y, z) = f (, x,µ,ˆ (, x,µ,z (, x) 1 ) z (, x) 1 b(, x,µ,ˆ (, x,µ,z and in he second: ( B(, x,µ, y, z) =b(, x,µ, ˆ (, x,µ, y)), F(, x,µ,y, z) xf (, x,µ,ˆ (, x,µ,y)) y@ xb(, x,µ,ˆ (, x,µ,y)).

21 ATYPICAL EXISTENCE RESULT ( dx = B, X, Y, Z, P (X,Y ) d +, X, Y, P (X,Y ) dw dy = F, X, Y, Z, P (X,Y ) d + Z dw, 0 apple apple T, wih X 0 = x 0 and Y T = G(X T, L(X T )). Assumpions (A1). B, F, and G are coninuous in µ and uniformly (in µ) Lipschiz in (x, y, z) (A2). and G are bounded and 8 apple R 1/2 >< B(, x, y, z,µ) applel 1 + x + y + z + R d R p (x 0, y 0 ) 2 dµ(x 0, y ) 0, apple R 1/2 >: F (, x, y, z,µ) applel 1 + y + R d R p y 0 2 dµ(x 0, y ) 0. (A3). is uniformly ellipic (, x, y,µ) (, x, y,µ) L 1 I d and [0, T ] 3,! (, 0, 0, (0,0)) is also assumed o be coninuous. Under (A1 3), here exiss a soluion (X, Y, Z) 2 S 2,d S 2,p H 2,p m

22 MEAN FIELD GAMES WITH A COMMON NOISE Saring wih a finie player game, i.e. Simulaneous Minimizaion of ( Z ) T J i ( ) =E f (, X i, µ N, )d i + g(x T, µ N T ), i = 1,, N 0 under consrains (dynamics of players privae saes) dx i = b(, X i, µ N, i )d + (, X i, µ N, i )dw i + 0 (, X i, µ N, )dw i 0 for i.i.d. Wiener processes W k for k = 0, 1,, N.

23 LARGE GAME ASYMPTOTICS (CONT.) Condiional Law of Large Numbers I If we consider exchangeable equilibriums,( 1,, N ), hen I By de Finei LLN lim N!1 µn = P X 1 F 0 I Dynamics of player 1 (or any oher player) becomes dx 1 = b(, X 1,µ, 1 )d + (, X 1,µ, 1 )dw + 0 (, X 1,µ, 1 )dw 0 ; I wih µ = P X 1 F 0. Cos o player 1 (or any oher player) becomes Z T E f (, X,µ, 1 )d + g(x T,µ T ) 0

24 MFG WITH COMMON NOISE PARADIGM 1. For each Fixed measure valued (F 0)-adaped process (µ ) in P(R), solve he sandard sochasic conrol problem ( Z ) T ˆ = arg inf E f (, X,µ, )d + g(x T,µ T ) 0 subjec o dx = b(, X,µ, )d + (, X,µ, )dw + 0 (, X,µ, )dw 0 ; 2. Fixed Poin Problem: deermine (µ ) so ha 8 2 [0, T ], P X F 0 = µ a.s. Once his is done one expecs ha, if ˆ = j = (, X j ), (, X ), for N player game, j = 1,, N form an approximae Nash equilibrium for he game wih N players.

25 EX: PONTRYAGIN STOCHASTIC MAXIMUM PRINCIPLE Freeze µ =(µ ) 0appleappleT, wrie (reduced) Hamilonian H(, x,µ,y, )=b(, x,µ, ) y + f (, x,µ, ) Sandard definiion Given an admissible conrol =( ) 0appleappleT and he corresponding conrolled sae process X =(X ) 0appleappleT, any couple (Y, Z ) 0appleappleT saisfying: ( dy xh(, X,µ, Y, )d + Z dw + Z 0 dw 0 Y T xg(xt,µ T ) is called a se of adjoin processes

26 STOCHASTIC CONTROL STEP SOLUTION Deermine ˆ (, x, µ,y) =arg inf H(, x,µ,y, ) Injec in FORWARD and BACKWARD dynamics and SOLVE ( dx = b(, X,µ, ˆ (, X,µ, Y ))d + (, X )dw + 0 (, X )dw 0, dy x H µ (, X, Y, ˆ (, X,µ, Y ))d + Z dw + Z 0dW 0 wih X 0 = x 0 and Y T x g(x T,µ T ) Sandard FBSDE (for each fixed,! µ )

27 FIXED POINT STEP Solve he fixed poin problem (µ ) 0appleappleT! (X ) 0appleappleT! (P X F ) 0 0appleappleT Noe: if we enforce µ = P X F for all 0 apple apple T in FBSDE we have 0 ( dx = b(, X, P X F, ˆ P 0 X F 0 (, X, Y ))d + (, X )dw + (, X ) dw 0, dy x H P X F 0 (, X, Y, ˆ P X F 0 (, X, Y ))d + Z dw + Z 0dW 0, wih X 0 = x 0 and Y T x g(x T, P XT F 0 T ) Very difficul FBSDE of Condiional McKean-Vlasov ype!!!

28 SEVERAL APPROCHES I Relaxed Conrols (R.C. - Delarue - Lacker) I FBSDEs of Condiional McKean-Vlasov Type (RC - Delarue) I SDEs of Condiional McKean-Vlasov Type (RC - Zhu) I Condiional Propagaion of Chaos (RC - Zhu) I Exisence for a finie common noise (Schauder Theorem) I Weak Soluion by Limiing argumens I Uniqueness via Monooniciy or Srong Convexiy I Srong Soluion via exension of Yamada-Waanabe

29 Back o Conrol of McKean - Vlasov Dynamics Say using Ponryaging Maximum Principle

30 DIFFERENTIABILITY AND CONVEXITY OF µ,! h(µ) I Noions of differeniabiliy for funcions defined on spaces of measures from heory of opimal ransporaion, gradien flows, ec) sudied by Ambrosio, De Giorgi, Oo, Villani, ec I Tailored made noion (Lions Collège de France Lecures, Cardaliague) Lif a funcion µ,! h(µ) o L 2 (, F, P) ino and say X,! h(x) =h( P X ) h is differeniable a µ if h is Fréche differeniable a X whenever P X = µ. A funcion g on R d P 1 (R d ) is said o be convex if for every (x,µ) and (x 0,µ 0 ) in R d P 1 (R d ) we have g(x 0,µ 0 ) xg(x,µ) (x 0 x) Ẽ[@ µg(x, X) ( X 0 X)] 0 whenever P X = µ and P X 0 = µ 0

31 POTENTIAL GAMES Sar wih Mean Field Game à la Lasry-Lions inf =( ) 0appleappleT, dx = d+ apple Z T 1 E dw f (, X,µ ) d + g(x T,µ T ) such ha f and g are differeniable w.r.. x s.. here exis differeniable funcions F and xf (, x,µ)=@ µf (,µ)(x) xg(x,µ)=@ µg(µ)(x) (1) Solving his MFG is equivalen o solving he cenral planner opimizaion problem inf =( ) 0appleappleT, dx = d+ apple Z T 1 E dw F(, L(X )) d + G(L(X T )) Special case of McKean-Vlasov opimal conrol

32 THE ADJOINT EQUATIONS Lifed Hamilonian H(, x, X, y, )=H(, x,µ,y, ) for any random variable X wih disribuion µ. Given an admissible conrol =( ) 0appleappleT and he corresponding conrolled sae process X =(X ) 0appleappleT, any couple (Y, Z ) 0appleappleT saisfying: 8 >< dy xh(, X, P X, Y, )d + Z dw Ẽ[@ µh(, >: X, X, Ỹ, )] X=X d Y T xg(xt, P X T )+Ẽ[@ µg(x, X )] x=x T where (, X, Ỹ, Z ) is an independen copy of (, X, Y, Z ), is called a se of adjoin processes BSDE of Mean Field ype according o Buckhdan-Li-Peng!!! Exra erms in red are he ONLY difference beween MFG and Conrol of McKean-Vlasov dynamics!!!

33 PONTRYAGIN MAXIMUM PRINCIPLE (SUFFICIENCY) Assume 1. Coefficiens coninuously differeniable wih bounded derivaives; 2. Terminal cos funcion g is convex; 3. admissible conrol, X corresponding dynamics, (Y, Z ) adjoin processes and is d dp a.e. convex, hen, if moreover (x,µ, ),! H(, x,µ,y, Z, ) H(, X, P X, Y, Z, )= inf 2A H(, X, P X, Y, ), a.s. Then is an opimal conrol, i.e. J( ) = inf 2A J( )

34 SCALAR INTERACTIONS b(, x,µ, )= b(, x, h, µi, ) f (, x,µ, )= f (, x, h,µi, ) (, x,µ, )= (, x, h, µi, ) g(x,µ)= g(x, h,µi) I,, and differeniable wih a mos quadraic growh a 1, I b, and f differeniable in (x, r) 2 R d R for, ) fixed I g differeniable in (x, r) 2 R d R. Recall ha he adjoin process saisfies Y T x g(x T, P XT )+Ẽ[@ µ g( X T, P XT )(X T )]. bu µ g(x,µ)(x 0 )=@ r g x, (x 0 ), he erminal condiion reads Y T x g X T, E[ (X T )] + r g X T, E[ (X T (X T ) Convexiy in µ follows convexiy of g

35 SCALAR INTERACTIONS (CONT.) H(, x,µ,y, z, )= b(, x, h, µi, ) y+ (, x, h, µi, ) z+ f (, x, h,µi, µ H(, x,µ,y, z, ) can be idenified µh(, x,µ,y, z, )(x 0 r b(, x, h, µi, ) (x 0 ) r (, x, h, µi, ) (x 0 ) r f (, x, h,µi, (x 0 ) and he adjoin equaion rewries: dy x b(, X, E[ (X )], ) Y x (, X, E[ (X )], ) Z x f (, X, E[ (X )], ) d + Z dw r b(, X, E[ ( X )], ) r (, X, E[ ( X )], ) (X) Anderson - Djehiche r f ((, X, E[ ( X )], (X ) d

36 SOLUTION OF THE MCKV CONTROL PROBLEM Assume I b(, x,µ, )=b 0 () R R d xdµ(x)+b 1 ()x + b 2 () wih b 0, b 1 and b 2 is R d d -valued and are bounded. I f and g as in MFG problem. There exiss a soluion (X, Y, Z ) 0 of he McKean-Vlasov FBSDE 8 >< dx = b 0 ()E(X )d + b 1 ()X d + b 2 ()ˆ (, X, P X, Y )d + dw, dy xh, X, P X, Y, ˆ d >: µh, X 0, X, Y 0, ˆ 0 d + ZdW. wih Y = u(, X, P X ) for a funcion u :[0, T ] R d P 1 (R d ) 3 (, x,µ) 7! u(, x,µ) uniformly of Lip-1 and wih linear growh in x.

37 AFINITE PLAYER APPROXIMATE EQUILIBRIUM For N independen Brownian moions (W 1,...,W N ) and for a square inegrable exchangeable process =( 1,..., N ), consider he sysem dx i = 1 N b 0() NX j=1 X j + b 1 ()X i + b 2 () i + dw i, X i 0 = i 0, and define he common cos applez T J N ( )=E f s, Xs i, µn s, i s ds + g XT 1, µn T, wih µ N = 1 NX 0 N i=1 Then, here exiss a sequence ( N ) N 1, N & 0, s.. for all =( 1,..., X i. N ), where, =( 1,, N ) wih J N ( ) J N ( ) N, i =ˆ (s, X i, u(, X i ), P X ) where X and u are from he soluion o he conrolled McKean Vlasov problem, and ( X 1,..., X N ) is he sae of he sysem conrolled by, i.e. d X i = 1 N NX j=1 b 0 () X j + b 1 () X i + b 2 ()ˆ (s, X i s, u(s, X i s ), P X s )+ dw i, X i 0 = i 0.