arxiv: v1 [math.oc] 21 Sep 2018

Transcription

1 Differentia Dynamic Programming for Noninear Dynamic Games Boei Di and Andrew Lampersi arxiv: v mathoc Sep 08 Abstract Dynamic games arise when mutipe agents with differing objectives choose contro inputs to a dynamic system Dynamic games mode a wide variety of appications in economics, defense, and energy systems However, compared to singe-agent contro probems, the computationa methods for dynamic games are reativey imited As in the singe-agent case, ony very speciaized dynamic games can be soved exacty, and so approximation agorithms are required his paper extends the differentia dynamic programming agorithm from singe-agent contro to the case of non-zero sum fu-information dynamic games he method wors by computing quadratic approximations to the dynamic programming equations he approximation resuts in static quadratic games which are soved recursivey Convergence is proved by showing that the agorithm iterates sufficienty cose to iterates of Newton s method to inherit its convergence properties A numerica exampe is provided I INRODUCION Dynamic games arise when mutipe agents with differing objectives act upon a dynamic system In contrast, optima contro can be viewed as the speciaization of dynamic games to the case of a singe agent Dynamic games have many appications incuding pursuit-evasion, active-defense, 3, economics 4 and the smart grid 5 Despite a wide array of appications, the computationa methods for dynamic games are consideraby ess deveoped than the singe-agent case of optima contro his paper shows how the differentia dynamic programming DDP method from optima contro 6 extends to discrete-time non-zero sum dynamic games Cosey reated wors from 7, 8 focus on the case of zero-sum dynamic games Cassica differentia dynamic programming operates by iterativey soving quadratic approximations to the Beman equation from optima contro Our method appies simiar methods to the generaization of the Beman equation for dynamic games 9 Here, at each stage, the agorithm soves a static game formed by taing quadratic approximations to the vaue function of each agent We show that the agorithm converges quadraticay in the neighborhood of a strict Nash equiibrium o prove convergence, we extend arguments from 0, which reate DDP iterates to those of Newton s method, to the case of dynamic games In particuar, we extend the recursive soution for Newton s method to dynamic games, and demonstrate that the soutions produced Newton s method and the DDP method are cose he authors are with the Department of Eectrica and Computer Engineering, University of Minnesota, Minnesota, USA dix047@umnedu, aampers@umnedu A Reated Wor A great dea of wor on agorithmic soutions to dynamic games has been done his subsection reviews reated wor which is a bit more removed from the coser references described above As we wi see, most wors sove somewhat different probems compared to the current paper Methods for finding Nash equiibria via extremum seeing were presented in, 3, 4 In particuar, the controers drive the states of a dynamic system to Nash equiibria of static games A reated method for inear quadratic games was presented in 5 For these wors, each agent ony requires measurements of its own cost However, it is imited to finding equiibria in steady state Our method requires each agent to have expicit mode information, but gives equiibria over finite horizons his is particuary important for games in which trajectories from initia to fina states are desired Severa wors focus on the soution to dynamic potentia games Potentia games are more tractabe than genera dynamic games, as they can be soved using methods from singe-agent optima contro 6, 7, 8, 9, 0, However, potentia games satisfy restrictive symmetry conditions In particuar, the assumption precudes interesting appications with heterogeneous agents B Paper Outine he genera probem is formuated in Section II he agorithm is described in Section III and the convergence proof is setched in Section IV A numerica exampe is described in V Concusions and future directions are discussed in VI whie the proof detais are given in the appendix II DEERMINISIC NONLINEAR DYNAMIC GAME PROBLEM In this section, we introduce deterministic finite-horizon noninear game probem, the notations for the paper, the soution concept and convergence criterion of our proposed method he main probem of interest is a deterministic fuinformation dynamic game of the form beow Probem : Noninear dynamic game Each payer tries to minimize their own cost Subject to constraints J n u 0 x f x,u :, x 0 is fixed c x,u :, a b

2 Here, the state of the system at time is denoted by x R Payer n s input at time is given by u R n un he vector of a payer actions at time is denoted by: u :, u,,u,,,u N, R n u he cost for payer n at time is c x,u :, his encodes the fact that the costs for each payer can depend on the actions of a the payers In ater anaysis, some other notation wi be hepfu he vector payer n s actions over a time is denoted by u n,: u n,0,u n,,,u n, he vector of a actions other than those of payer n is denoted by u n,: u,:,,u n,:,u n,:,,u N,: he vector of a states is denoted by x x 0,x,,x whie the vector of a inputs is given by u u,:,u N,:,,u N,: Note that since the initia state is fixed and the dynamics are deterministic, the costs for each payer can be expressed as functions of the vector of actions, J n u A oca Nash equiibrium for probem is a set of inputs u such that J n u n,:,u n,: J n u, n,,,n 3 for a u n,: in a neighborhood of u n,: In the context of dynamic games, this correponds to an open-oop, oca Nash equiibrium 9 he equiibrium is caed a strict oca Nash equiibrium if the inequaity in 3 is strict for a u n,: u n,: in a neighborhood of u n,: In this paper, we focus on computing Nash equiibria by soving the foowing necessary conditions for oca Nash equiibria: Probem : Necessary conditions J n n,: 0 4 for n,,n For convenient notation, we stac a of the gradient vectors from 4 into a singe vector: J u J,: J,: J N N,: 5 hus, the necessary condition is equivaent to J u 0 Such conditions arise in wors such as, 3 We wi present a method for soving these necessary conditions for a oca Nash equiibrium via differentia dynamic programming DDP In principe, an input vector satisfying the necessary conditions J u 0 coud be found via Newton s method Simiar to the singe-payer case from 0, we anayze the convergence properties of DDP by proving that its soution is cose to that computed by Newton s method o guarantee convergence, we assume that J u satisfies the smoothness and non-degeneracy conditions required by Newton s method 4 For smoothness, we assume that J u is differentiabe with ocay Lipschitz derivatives For non-degeneracy, we assume that J u is invertibe A sufficient condition for the smoothness assumptions is that the functions f and c are twice continuousy differentiabe with Lipschitz second derivatives In our DDP soution, we wi sove a sequence of stage-wise quadratic games As we wi see, a sufficient condition for invertibiity of J u is the unique sovabiity of the stage-wise games near the equiibrium III DIFFERENIAL DYNAMIC PROGRAMMING ALGORIHM his section describes the differentia dynamic programming agorithm for dynamic games of the form in Probem Subsection III-A gives a high-eve description of the agorithm, whie Subsection III-B describes the expicit matrix cacuations used in agorithm A Agorithm Overview he equiibrium soution to the genera dynamic game can be characterized by the Beman recursion: V n, x 0 Q x,u :, c x,u :, V f x,u :, V x min u Q x,u :, 6a 6b 6c In particuar, if a soution to the Beman recursion is found, the corresponding optima strategy for payer n at time woud be the u which minimizes Q x,u :, Note that 6c defines a static game with respect to the u :, variabe at step he idea of the differentiabe dynamic programming DDP is to maintain quadratic approximations of V and Q denoted by Ṽ and Q, respectivey We need some notation for our approximations For a scaar-vaued function, hz, we denote the quadratic approximation near z by: quadhz z δz δz z z h z h z h z z z h z z δz 7a 7b If h : R n R m we form the quadratic approximation by stacing a of the quadratic approximations of the entries: Let quadhz z quadh z z,,quadh m z z 8 x z u :, and et x and ū :, be a trajectory of states and actions satisfying the dynamic equations from he approximate Beman recursion around this trajectory is given by: Ṽ n, x 0 0a Q z quadc z Ṽ f z z 0b Ṽ x min Q x,u :, u 0c Note that 0c is now a quadratic game in the u :, variabes which has unique and ready soution 9 Reca the J u function defined in 5 A sufficient condition for sovabiity of these games is given in terms of J u is given in the foowing emma Its proof is in Appendix I-D 9

3 J ū Lemma : If is invertibe, the game defined by 0c has a unique soution of the form: u :, ū :, K δx s In the notation defined above, we have that δx x x Note that if J u J ū is invertibe, then is invertibe for a ū in a neighborhood of u Here we provide the DDP agorithm for appying DDP game soution in pseudo code Agorithm Differentia Dynamic Programming for Noninear Dynamic Games Generate an initia trajectory x, ū oop Bacward Pass: Perform the approximate Beman recursion from 0 Compute K and s from Forward Pass: Generate a new trajectory using the affine poicy defined by K, s end oop B Impementation Detais A of the operations in the bacwards pass of the DDP agorithm, Agorithm, can be expressed more expicity in terms of matrices o construct the required matrices, we define the foowing approximation terms: A f x,u :, ū, B f x,u :, ū a x :, f f G x x :,,,,, b f :, x f :, δx G δu 0 δx :, δu :, δx G R δx,δu :, δu δx :, δu :, δx G δx M c c x c :, M M x M u δu :, ū c x c x c :, x M x M M ux M u M xu M uu δu :, c :, c x :, c :, ū c d Using the notation from 7, 8 and 9, the second-order approximations of the dynamics and cost are given by: quad f z z f z A δx B δu :, R δz 3a quadc z z M δz 3b δz By construction Ṽ x and Q x,u :, are quadratic, and so there must be matrices and Γ such that Ṽ x x δx x 4a δx Q x,u :, δx Γ δx 4b δu :, δu :, Lemma : he matrices in 4 are defined recursivey by n, 0 and: D Γ M A x B x Γ Γ x Γ u Γ u Γ u N F P Γ u u, Γ u u, x G Γ x Γ Γ u x Γ u x Γ u Nx Γ u Nu N, Γ u x, Γ u x, Γ u Nx N, s F x A A B A D A D ux Γ u Γ xu Γ u u Γ u u Γ u Nu Γ u u, Γ u u, Γ u Nu N,, H H, 0 s 0 I K x B A B Γ u Γ u N Γ xu Γ xu N Γ u u Γ u u N Γ u u Γ u u N Γ u Nu Γ u Nu N, Γ u u N,, Γ u u N, N, Γ u Nu N N, Γ u u Γ u u Γ u Nu Γ u, Γ u, Γ u N N, K F P Γ 0 0 I s K 5a B D xu B D uu 5b 5c 5d 5e 5f, 5g for,,,0 Proof: By construction we must have n, 0 Pugging 3a into 5b and dropping a cubic and higher terms gives 0b Since u :, ū :, δu :, and ū :, is constant, the static game defined in 0c can be soved in the δu :, variabes Differentiating 4b by δu, coecting the derivatives for a payers and setting them to zero eads to

4 the necessary condition for an equiibrium: F δu :, P δx H 0 6 hus, the matrices for the equiibrium strategy are given in 5f Pugging into 4b eads to 5g Remar : he next section wi describe how the agorithm converges to strict Nash equiibria if it begins sufficienty cose o ensure that the agorithm converges regardess of initia condition, a Levenberg-Marquardt stye reguarization can be empoyed Such reguarization has been used in centraized DDP agorithms, 5, 6, to ensure that the required inverses exist and that the soution improves In the current setting, such reguarization woud correspond to using a reguarization of the form F λi where λ 0 is chosen sufficienty arge to ensure that the matrix is positive definite IV CONVERGENCE his section outines the convergence behavior of the DDP agorithm for dynamic games he main resut is heorem which demonstrates quadratic convergence to oca Nash equiibria: heorem : If u is a strict oca equiibrium such that J u is invertibe, then the DDP agorithms converges ocay to u at a quadratic rate he proof depends on severa intermediate resuts Subsection IV-A reformuates Newton s method for the necessary conditions, 4, as the soution to a dynamic game Subsection IV-B demonstrates that the soutions of the dynamic games soved by Newton s method and DDP cose hen, Subsection IV-C finishes the convergence proof by demonstrating that the DDP soution is sufficienty cose to the Newton soution to inherit its convergence property hroughout this section we wi assume that both Newton s method and DDP are starting from the same initia action trajectory, ū Let u N and u D be the updated action trajectories of Newton s method and DDP, respectivey Define update steps, δu N and δu D, by: u N ū δu N u D ū δu D 7 Additionay, we wi assume that u is a strict oca equiibrium with J u invertibe A Dynamic Programming Soution for the Newton Step he proof of heorem proceeds by demonstrating that the soutions from DDP and Newton s method are sufficienty cose that DDP inherits the quadratic convergence of Newton s method o show coseness, we demonstrate that the Newton step can be interpreted as the soution to a dynamic game his dynamic game has a recursive soution that is structuray simiar to the recursions from DDP his subsection derives the corresponding game and soution he Newton step for soving 4 is given by: J ū δu N J ū 8 his rue eads to a quadratic convergence to a root in 4 whenever u J u is ocay Lipschitz and invertibe 4 he next two emmas give game-theoretic interpretations of the Newton step Lemma 3: Soving 8 is equivaent to soving the quadratic game defined by: minj n ū J nū δu n,: δu J n ū δu δu 9 Proof: Under the strict oca equiibrium assumptions, 9 has a unique soution which is found by differentiating with respect to δu n,: and setting the resut to 0 Stacing these equations eads precisey to 8 he next emma shows that 9 can be expressed as a quadratic dynamic game It is proved in Appendix I-A Lemma 4: he quadratic game defined in 9 is equivaent to the dynamic game defined by: min u n,: subject to 0 δx δu :, δx 0 0 x 0 0 M δx M x δu :, x δx A δx B δu :, x A x R δx,δu :, 0a 0b 0c 0d 0e 0,,, 0f Note that the states of the dynamic game are given by δx and x as δx i0 x i0 x δu :,i x,ū δu :,i j0 x δu :, j,,,, :, j x,ū a b It foows that the equiibrium soution of this dynamic game is characterize by the foowing Beman recursion: V n, δx, x 0 Q δx, x,δu :, δx M δx M x δu :, δu :, x a b V A δx B δu :,,A x R δx,δu :, c V δx, x min δu Q δx, x,δu :, d Note that d defines a static quadratic game and V δx, x is found by soving the game and substituting the soution bac to Q δx, x,δu :, he next emma describes an expicit soution to the bacward recursion he ey step in the convergence proof is showing that the matrices used in this recursion are appropriatey cose to the matrices used in DDP

5 Lemma 5: he functions V and Q can be expressed as V δx, x S δx Ω δx x 3a Q δx, x,δu :, δx Γ δx Ω x δu :, δu :, 3b where the matrices S, Γ, and Ω are defined recusrivey by S n, 0, Ω n, 0, and Ω M x Ω A D Γ M n Ω G S A Sx B Sx F Γ Γ x Γ u Γ u Γ u N P Γ u u Γ u u S x A A S A D B S A D ux A S B S 4a 4b S x B B D xu B D uu 4c Γ x Γ u Γ u Γ u N Γ Γ xu Γ xu Γ xu N Γ u x Γ u u Γ u u Γ u u N Γ u x Γ u u Γ u u Γ u u N Γ u Nx Γ u Nu Γ u Nu Γ u Nu N Γ u Nu N Γ u x Γ u x Γ u Nx N s F Γ u u Γ u u Γ u u Γ u u Γ u Nu N Γ u Nu Γ u, H Γ u Γ u N N H, K F S 0 s 0 I K Γ Γ u u N Γ u u N N Γ u Nu N N 4d 4e 4f P 4g 0 0 I 4h s K for,,,0 From this emma, we can see that the matrices used in the recursions for both DDP and Newton s method are very simiar in structure Indeed, the iterations are identica aside from the definitions of the D and D matrices Proof: he proof is very simiar to the proof for Lemma Soving the equiibrium strategy and V δx, x based on Q δx, x,δu :, is the same as how we arrived at 6 and 5g, since the extra terms of x are not couped with δu :, and other terms are of the exact same form he stepping bac in time of Q δx, x,δu :, is acheived by substituting 0d and 0e into 3a, which is sighty different because of the extra terms reated to x Q δx, x,δu :, 5a δx M δx M x δu :, δu :, V A δx B δu,a x R δx,δu :, 5b δx M δx δu :, δu :, S S x A S x B δx A Sx A S A A S B δx δu :, B Sx B S A B S B δu :, M Ω A x δx δx D 5c δu :, δu :, So 4a, 4b and 4c are true B Coseness Lemmas his subsection gives a few emmas which impy that the Newton step, δu N, and the DDP step, δu D, are cose For the rest of the section, we set ū u ε he foowing emma shows that the matrices used in the bacwards recursion are cose It is proved in Appendix I-B Lemma 6: he matrices from the bacwards recursions of DDP and Newton s method are cose in the foowing sense: Γ Γ Γ Γ x Γ u Γ x Γ Γ ux Γ u Γ xu Γ uu Γ Γ x Γ u Γ x Γ Γ ux Γ u Γ xu Γ uu 6a Oε Oε Oε Oε Oε Oε, 6b Oε Oε Oε S x x S Oε Oε Oε Oε S x S x S 6c, 6d sn, K sn, K Oε Oε 6e Furthermore, the foowing matrices are sma: Γ x Γ,u Oε, Γ x Γ,u Oε 6f s Oε, s Oε 6g After showing that the matrices are cose, it can be shown that the states and actions computed in the update steps are cose It is proved in Appendix I-C

6 state owner dog steps a state cum cost cum cost owner dog steps b state distance to equiibrium Fig Owner-dog dynamic game In order to eep the dog around x, the owner has to overshoot and then come bac to x 0 he dog earns to get coser to the owner over iterations, which is what we woud expect given how the probem is formuated As can be seen in Fig b, the cumuative costs for both payers reduces over iterations Lemma 7: he states and actions computed by DDP and Newton s method are cose: δu D :, δun :, Oε δx D δxn Oε Furthermore, the updates are sma: 7a 7b δu D :, Oε, δun :, Oε 7c δx D Oε, δxn Oε 7d C Proof of heorem Lemma 7 impies that δu N δu D Oε Furthermore, the Newton step satisfies: ū δu N u Oε 8 See 4 he proof of quadratic convergence is competed by the foowing steps: ū δu D u ū δu N u δu D δu N ū δu N u δu D δu N 9a 9b Oε 9c V NUMERICAL EXAMPLE We appy the proposed DDP agorithm for deterministic noninear dynamic games to a toy exampes in this section he exampe is impetmented in Python and a derivatives of noninear functions are computed via ensorfow 7 We consider a simpe -D owner-dog probem, with horizon and initia state x :,0, where the dynamics of both the owner and the dog are given respectivey by x 0 x0 tanhu0 x x tanhu 30a 30b he owner cares about going to x 0 and that the dog can stay at x he dog, however, ony tries to catch up with the owner Each payer aso concerns itsef with the energy consumption, therefore has a cost term reated to the iterations Fig his shows the -norm distance between inputs ū and the fina equiibrium u over iterations As can be seen that the error reduces subineary on a og scaed pot, which is evidence that the agorithm converges quadraticay magnitude of its input heir cost functions are formuated as c 0, x,u 0 sigmoidx 0 40x u 0 3a c, x,u tanh x 0 x u 3b Noninear functions are added to the dynamics and costs to create a noninear game rather than for expicit physica meaning We initiaize a trajectory with zero input and initia state, ie ū 0,0,,0 and x,,,,,, We used an identity reguarization matrix with a magnitude of 400 Fig shows the soution via DDP to this probem over iterations, where the more transparent the trajectories, the earier in the iterations they are he starred trajectory is the fina equiibrium soution We simuated 00 iterations after the initia trajectory and piced 0 uniformy spaced ones to show in the figures VI CONCLUSION In this paper we have shown how differentia dynamic programming extends to dynamic games he ey steps were invoved finding expicit forms for both DDP and Newton iterations that enabe cean comparison of their soutions We demonstrated the performance of the agorithm on a simpe noninear dynamic game Many extensions are possibe We wi examine arger exampes and wor on numerica scaing Aso of interest are stochastic dynamic games and probems in which agents have differing, imperfect information sets Additionay, handing scenarios in which agents have imperfect mode information wi of great practica importance REFERENCES I Rusna, he ady, the bandits, and the bodyguards a two team dynamic game, in Proceedings of the 6th word IFAC congress, 005, pp O Proopov and Shima, Linear quadratic optima cooperative strategies for active aircraft protection, Journa of Guidance, Contro, and Dynamics, vo 36, no 3, pp , 03

7 3 E Garcia, D W Casbeer, K Pham, and M Pachter, Cooperative aircraft defense from an attacing missie, in Decision and Contro CDC, 04 IEEE 53rd Annua Conference on IEEE, 04, pp F E Ouardighi, S Jørgensen, and F Pasin, A dynamic game with monopoist manufacturer and price-competing duopoist retaiers, OR spectrum, vo 35, no 4, pp , 03 5 Q Zhu, Z Han, and Başar, A differentia game approach to distributed demand side management in smart grid, in Communications ICC, 0 IEEE Internationa Conference on IEEE, 0, pp D H Jacobson, New second-order and first-order agorithms for determining optima contro: A differentia dynamic programming approach, Journa of Optimization heory and Appications, vo, no 6, pp 4 440, W Sun, E A heodorou, and P siotras, Game theoretic continuous time differentia dynamic programming, in American Contro Conference ACC, 05 IEEE, 05, pp , Stochastic game theoretic trajectory optimization in continuous time, in Decision and Contro CDC, 06 IEEE 55th Conference on IEEE, 06, pp Basar and G J Osder, Dynamic noncooperative game theory Siam, 999, vo 3 0 D Murray and S Yaowitz, Differentia dynamic programming and newton s method for discrete optima contro probems, Journa of Optimization heory and Appications, vo 43, no 3, pp , 984 J C Dunn and D P Bertseas, Efficient dynamic programming impementations of newton s method for unconstrained optima contro probems, Journa of Optimization heory and Appications, vo 63, no, pp 3 38, 989 P Frihauf, M Krstic, and Basar, Nash equiibrium seeing in noncooperative games, IEEE ransactions on Automatic Contro, vo 57, no 5, pp 9 07, 0 3 P Frihauf, M Krstic, and Başar, Finite-horizon q contro for unnown discrete-time inear systems via extremum seeing, European Journa of Contro, vo 9, no 5, pp , 03 4, Nash equiibrium seeing for dynamic systems with nonquadratic payoffs, in Advances in Dynamic Games Springer, 03, pp Y Pan and U Ozgüner, Siding mode extremum seeing contro for inear quadratic dynamic game, A A, vo, p BK, D Gonzáez-Sánchez and O Hernández-Lerma, A survey of static and dynamic potentia games, Science China Mathematics, vo 59, no, pp 075 0, 06 7, Discrete time stochastic contro and dynamic potentia games: the Euer Equation approach Springer Science & Business Media, 03 8, Dynamic potentia games: he discrete-time stochastic case, Dynamic Games and Appications, vo 4, no 3, pp , 04 9 V V Mazaov, A N Rettieva, and K E Avrachenov, Linearquadratic discrete-time dynamic potentia games, Automation and Remote Contro, vo 78, no 8, pp , 07 0 S Zazo, S Vacarce, S Matide, J Zazo, et a, A new framewor for soving dynamic scheduing games, in Acoustics, Speech and Signa Processing ICASSP, 05 IEEE Internationa Conference on IEEE, 05, pp S Zazo, S V Macua, M Sánchez-Fernández, and J Zazo, Dynamic potentia games in communications: Fundamentas and appications, arxiv preprint arxiv:509033, 05 F Facchinei and C Kanzow, Generaized nash equiibrium probems, 4OR, vo 5, no 3, pp 73 0, C Dutang, A survey of gne computation methods: theory and agorithms, 03 4 J Noceda and S J Wright, Numerica optimization, nd ed Springer, L-Z Liao and C A Shoemaer, Convergence in unconstrained discrete-time differentia dynamic programming, IEEE ransactions on Automatic Contro, vo 36, no 6, pp , 99 6 Y assa, heory and Impementation of Biomimetic Motor Controers Hebrew University of Jerusaem, 0 7 M Abadi, A Agarwa, P Barham, E Brevdo, Z Chen, C Citro, G S Corrado, A Davis, J Dean, M Devin, S Ghemawat, I Goodfeow, A Harp, G Irving, M Isard, Y Jia, R Jozefowicz, L Kaiser, M Kudur, J Levenberg, D Mané, R Monga, S Moore, D Murray, C Oah, M Schuster, J Shens, B Steiner, I Sutsever, K awar, P ucer, V Vanhouce, V Vasudevan, F Viégas, O Vinyas, P Warden, M Wattenberg, M Wice, Y Yu, and X Zheng, ensorfow: Large-scae machine earning on heterogeneous systems, 05, software avaiabe from tensorfoworg Onine Avaiabe: APPENDIX I PROOFS OF LEMMAS We first derive a few usefu resuts which we use ater in this chapter First, we bound the spectra radius of F from above ρf ˆF 3 where ˆF is a constant Consider inverting u J u in Newton s method by successivey eiminating δu :, for,,,0 he F matrices are exacty the matrices which woud be inverted when eiminating δu :, Since u J u is Lipschitz continuous, its eigenvaues are bounded away from zero in a neighborhood of u It foows that the eigenvaues of F K must aso be bounded away from zero and F is bounded above Secondy, we assert that Ω x x,ū i c n,i x i,u :,i 33 which means Ω captures the first-order effect of x on the sum of a ater costs for each payer Ω is constructed according to 4a Equation 33 is true for by construction We proof by induction and assume that 33 hods for, ie Ω x i c n,ix i,u :,i, then x,ū Ω M Ω A c x,u :, x x,ū x c x,u :, x x,ū x x,ū x x,ū i c n,i x i,u :,i c n,i x i,u :,i i i c n,i x i,u :,i 34a x x x,ū 34b 34c 34d herefore, 33 hods for And by induction, a 0,,, he third usefu resut is that M u Ω B J nu :, ū Oε 35 J n u ū Oε is true because J :, n u is twice differentiabe hence Lipschitz, ie J nu ū J nu u constant ū u Oε 36 :, :,

8 he first equaity hods because Ju ū :, :, x,ū i0 :, x,ū c x,u :, M u c n,i x i,u :,i :, x,ū x x,ū M u Ω B i :, x,ū i i c n,i x i,u :,i 37a c n,i x i,u :,i 37b c n,i x i,u :,i x :, x,ū 37c 37d hese resuts are used impicity in the ater proofs of emmas For x, row is given by: x i0 i0 j0 δu :,i i0 j0 δu :,i i0 j0 δu f x,u :, ūδu :,i :, j 40a j0 :, j δu f :,i x f x :, j x δu :, j :, j ū x f f x nx x :, j x f x p δu :, j :, j ū p x p δu f :, δu :, δx :, δu f :, δx :, x δx G δx δu :, δu :, p f x δx δx f x p δu :,i i0 j0 p A p xp :, j f x :, δu :, ū δu :, j x p δu :, j :, j 40b 40c 40d A Proof of Lemma 4 First, we prove that the dynamics constraints 0d and 0e are inductive definitions of the foowing approximation terms: o get to each terms in 40c, we used the fact that f :, j 0, for i or j δx i0 x δu :,i o get to 40d, we used the fact i0 x δu :,i 4a 4b δx i0 x i0 x δu :,i x,ū δu :,i j0 x δu :, j,,,, :, j x,ū 38a 38b Note that x 0 is fixed so that 38 hods at 0 Now we hande each of the terms inductivey For δx, we have i0 i0 x δx δu :,i x,ū f x,u :, δu :,i x,ū f x,u :, x x,ū i0 x δu :,i f x,u :, δu :, :, x,ū :, A δx B δu :, 39 f A p x p δu :,i i0 j0 f x f x :, x p ūδu :, j x p :, j f x :, f G x,ū :, 4a 4b 4c Both and p are used to pic out the corresponding eement for a vector or matrix A p means the th row and pth coumn of matrix A Equation 40d actuay describes each eement in 38b, so we ve proven that both are true Next we prove 0a is the quadratic approximation of J n u, ie quadj n uū J n ū J n ū J n ū δu δu δx M δx M x 43 0 δu :, δu :, We used the fact that f x,u :, is zero uness i We need the expicit expressions for the associated

9 derivatives J n u J n u :, j c x,u :, c x,u :, x 44a x c x,u :, :, j 0 x c x,u :, x c x,u :, x x :, j c x,u :, x c x,u :, x x x :, j x :, j 44b We brea down each term in 43 First the second order term δu J n ū δu δu J n u :,i i, j0 ūδu :, j 45a :, j δu c x,u :, :,i i, j,0 ū δu :, j :, j x c x,u :, x i, j,0 δu :,i δu :,i i, j,0 i, j,0 n 0 p 0 0 n δu :, δx 0 p 0 δu :,i δx x ū :, j δu :, j x c x,u :, x ū δu :, j :, j c x,u :, x x ū δu :, j :, j c x,u :, x p ū δu x p :,i δu :, j i, j0 c ūδu :, δx c :, x ū δx c ūδu :, δu c :, x :, :, x ū δx c x,u :, x p x p δu :, c x c :, x c x,u :, x x c x :, c :, δx ū δu :, 45b 45c 45d he first term in 45b to 45c hods because c x,u :, ony depends directy on u :,i and u :, j when i j he others hod because x ony depends on u :,i and u :, j when i, j < he ast term uses the definition of x in 38b he first order term J n ū i,0 0 0 i0 J n u δu :,i 46a c x,u :, c x,u :, x δu :,i 46b x c x,u :, δu :, c x,u :, :, x i0 c x,u :, δu :, c x,u :, δx :, x And constant term J n ū 0 x 46c 46d c x,ū :, 47 From 45, 46, and 47 it foows that 43 is true B Proof of Lemma 6 A more compete version of Lemma 6 is Γ x D D Oε S x x Ω Oε Ω Oε x Sx Oε S Oε Γ u Γ x Γ Γ Oε Γ u Γ u Oε Oε F F Oε P P Oε H H Oε H Oε H Oε s s Oε s Oε s Oε K K Oε, 48a 48b 48c 48d 48e Γ u Oε 48f 48g 48h 48i 48j m 48n 48o 48p 48q 48r of which we give a proof by induction for in this section For, because of the way these variabes are constructed, they are identica, ie Γ n, Γ n, F F P P H H s s K K S n, n, 49a 49b 49c 49d 49e 49f 49g

10 So we have 48d48e48f48g48j484848o48r hod for We aso now that M u n, c n, J nu Oε 50 where the first equaity is by construction, the second is true because u ony appears in J n u in c n, By construction, Γ u n, Γ n, u Mu n, Oε, so 48h48i are true for Simiary, H and H are constructed from Γ u n, and Γ u n,, so 48m48n are true for Because F is bounded above, s F H F Oε Oε Simiary, s Oε Equations 48p48q are true for From 4h and Γ n, M n,, we can get Sn, x Mn, x Mn, u K s Mn, ux Mn, uu K 5a Ω n, OεK OεMn, ux Mn, uu K 5b Ω n, Oε 5c because M n, is bounded Hence 48b is true Further, 48c is aso true he time indices for D n, and D n, go to a maximum of, so to prove things inductivey, we need 48a to hod for he difference between constructions of D n, and D n, is in that the former uses Ω n, and the ater uses n, x But since we ve proven Ω n, n, x Oε, and G is bounded, we can aso concude D n, D n, Oε herefore 48a is true for So far, we ve proved that for the ast step, either or, 48 is true Assuming except for 48a, 48 is true for and 48a is true for If we can prove a equations hod one step bac, our proof by induction woud be done Assume 48a hods for and other equations in 48 hod for Readers be aware that we use these assumptions impicity in the derivations foowing From 4c we can get Γ u Mu, Sx B M, u Ω B OεB Oε 5a 5b 5c Here we used 35 Simiary, we can prove Γ u Oε So 48h and 48i hod for From 4c and 5b we can compute the difference between Γ and Γ as Γ Γ S 0 0 A x 0 0 B x x A 0 0 A A D D 0 0 B A D ux Dux x B 0 0 A B D xu Dxu 53a 0 0 B B D uu Duu Oε A Oε B Oε A Oε A OεA Oε A OεB Oε B Oε B OεA Oε B OεB Oε 53b Oε Oε Oε Oε Oε Oε 53c Oε Oε Oε from which we can see that 48f and 48g are true Once we proved the coseness between Γ and Γ and the specific terms are Oε, ie 48f to 48i, because of they way they are constructed from Γ and Γ, we are safe to say F F Oε P P Oε H H Oε H Oε H Oε 54a 54b 54c 54d 54e herefore, 48j, 48, 48, 48m and 48n are true for Now that we have the resuts with F, F, H, H, P and P, we can move to what are immediatey foowing, ie s, s, K and K s F H F Oε Oε 55 which is true because F is bounded above Simiary, we have s Oε Equations 48p and 48q are true s F H F Oε H Oε 56a F OεH Oε 56b F H F Oε H Oε Oε 56c s Oε 56d K F P F Oε P Oε 56e F OεP Oε 56f F P F P Oε Oε 56g K Oε 56h Equations 48o and 48r are true for

11 Now we are equipped to get coseness/sma resuts for S and S I K Γ 0 0 I I K 0 K s Γ uu s s Γ u s Γuu s s Γu Γ xu Γ uu K s Γ ux Γuu 0 s Γ ux Γ uu K s Γux Γuu K Γ x 0 0 Γ K Γ u Γ Γ x K Γ u Oε Oε Oε 0 Oε Oε Oε Oε Γ 0 K s 0 Γ x Γ u K Γ Γ xu K K Γ uu K Γ x Γu K Γ Γxu K K Γ uu So that 48d and 48e are true for K 0 0 I 0 K 57a 57b 57c 57d S x Sx A Γ u K s Γux Γuu K 58a M x A A Oε K Oε Γ ux Γuu K Oε 58b Ω Oε 58c herefore, 48b hods and then naturay 48c hods We continue to prove that D and D are cose, which is true because So 48a is true D x G 59a Ω OεG 59b Ω G Oε 59c D Oε 59d C Proof of Lemma 7 A more compete version of Lemma 7 is δx D A δx D B δu D Oε δx N A δx N B δu N Oε δu N Oε δu D Oε δx N Oε δx D Oε δu N δud Oε δx N δxd Oε δu N δu D Oε ū δu N u Oε ū δu D u Oε 60a 60b 60c 60d 60e 60f 60g 60h 60i 60j 60 Equation 60a comes directy from the ayor series expansion of and 60b from 0d We prove 60c to 60h by induction For 0, δx N 0 δx D 0 0 and δun 0 s 0, δx D 0 s 0 We now from the proof of emma 6 that s 0 s 0 Oε, s 0 Oε and s 0 Oε, so 60c to 60h hod for 0 Assume 60c to 60h hod for, then δu N K δx N s Oε δu D K δx N s Oε δx N A δx N B δu N Oε Oε δx D A δx D B δu D Oε Oε δu N δud K δx N K δx D s s K δx N K Oεδx N Oε 6a 6b 6c 6d 6e Oε 6f Oεδx N Oε Oε 6g 6h δx N δxd A δx N δxd B δu N δud Oε Oε 6i So 60c to 60h hod for and the proof by induction is done Equation 60i comes directy as a resut Equation 60j is cassic convergence anaysis for Newton s method 4 Equation 60 foows directy from 60i and 60j D Proof of Lemma As discussed in the proof of Lemma, a necessary condition for the soution of 0c is given by 6 hus, a sufficient condition for a unique soution is that F be invertibe At the beginning of the appendix, whe showed that F exists near u and that its spectra radius is bounded Now we show that F exists and ρ F is bounded Lemma 6 impies that F F Oε It foows that F F Oε F F OεF F Oε It foows that F exists and is bounded in a neighborhood of u