arxiv: v1 [math.oc] 5 Oct 2016

Transcription

1 CONTROLLING A SELF-ORGANIZING SYSTEM OF INDIVIDUALS GUIDED BY A FEW EXTERNAL AGENTS PARTICLE DESCRIPTION AND MEAN-FIELD LIMIT MARTIN BURGER, RENÉ PINNAU, ANDREAS ROTH, CLAUDIA TOTZECK, AND OLIVER TSE arxiv: v1 [math.oc] 5 Oct 216 Abstract. Optimal control of large particle systems with collective ynamics by few agents is a subject of high practical importance (e.g. in evacuation ynamics, but still limite mathematical basis. In particular the transition from iscrete optimal control to a continuum setting as the number of particles tens to infinity is by far not fully unerstoo. In this paper we contribute to this issue by stuying a canonical moel of controlling an interacting particle system into a certain spatial region by repulsive forces from few external agents, which might be interprete as shepher ogs leaing sheep to their home. We iscuss the appropriate moelling of such a problem an the associate optimality systems, proviing some connections between the Lagrange multipliers in the iscrete an continuum setting. As control strategies we investigate an Instantaneous Control an a global Optimal Control approach. The solutions of a family of control problems for the particle system with external agents are numerically compare to the mean-fiel controls as the number of particles tens to infinity. In both cases, this leas to a high imensional phase space requiring tailore optimization strategies. All control problems arising are solve using ajoint information to compute the escent irections. The numerical results inicate the convergence of controls for both optimization strategies. 1. Introuction In the last ecaes, the behavior of large particle systems an their mean-fiel limits were extensively investigate with theoretical an computational approaches [1,8,9,21]. Large groups of iniviuals like flocks of birs an schools of fish, an their attractive an repulsive interaction were consiere, which lea to moels of ifferent types of collective behavior such as flocking or milling, an a thorough stuy of their stability [2,16,3,34] (also [9] for etaile a overview. The moels were refine using vision cones, self-propulsion an orientation alignment of neighbors [9,19,37]. As the behavior of large particle groups of same type is well unerstoo, the interest in self-organize systems interacting with few external agents arose. This concept an its kinetic limit was first numerically investigate by Albi an Pareschi in [1]. Using interaction potentials introuce by Cucker-Smale [14] or D Orsogna et al [18], they showe numerically, that the collective behavior of large groups coincies with the behavior of the kinetic moel as the number of iniviuals tens to infinity. While the moels for the interaction of particles became more an more realistic, the complexity was significantly increase. At first, interaction potentials were chosen smooth in orer to have well-efine erivatives [14, 15, 18]. Later it was shown that special classes of singular interaction potentials allow to pass to the mean-fiel limit as well [3, 22]. In the present work, we investigate ifferent control strategies for the interaction of a huge crow of iniviuals with few external agents, with a focus on moelling an computation. This behavior is then compare to the corresponing mean-fiel control problem. As control parameters we use the velocities of the external agents. The cost functionals are esigne in such a way that the external agents aim to lea the crow to a preefine estination using the least amount of energy possible. As an example one may consier a crow of sheep guie by ogs. Analytically, a similar sparse optimization problem was investigate by Fornasier an Solombrino in [2]. They showe the existence of optimal controls on the microscopic level an use the concept of Γ-convergence to perform the mean-fiel limit in a sparse optimal control setting. Numerically, alreay the computation of the state solutions are challenging ue to the high imensional phase space. On the one han, we have the pairwise interactions in the microscopic system an on the other han the mean-fiel equations are of Vlasov type, which yiels in two spatial imensions alreay a four imensional problem. This becomes even worse for the optimization of such problems, since the state systems nee to be solve several times an a huge amount 1

2 2 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE of ata nees to be store. Hence, there is no chance to solve these problems using black-box optimization approaches. Instea, tailore optimization algorithms are require, which are typically base on erivative information [26]. In the literature, ifferent control strategies are available to investigate such problems. For example the Instantaneous Control (IC was applie to traffic flow problems [23] an the Navier- Stokes Equations [24]. The Optimal Control (OC approach was use in moeling of glass cooling [26] or semiconuctors (cf. [32] an the overview in [25]. Due to the large amount of memory neee to store the forwar information, the OC approach is not frequently use for controlling multi-imensional problems as consiere here. The major aim of this manuscript is to construct appropriate numerical approaches an computationally verify that the controls of the microscopic problem converge to the optimal control of the kinetic problem as the number of iniviuals increases for each of the strategies. To this en we implemente numerical escent algorithms, base on first-orer erivative information. These are erive by applying the ajoint calculus to the interacting particle system as well as to the mean-fiel equation. A simple application scenario we have in min is to guie a her of sheep into a given region (stable by few ogs, the interaction between the particles an the external agents thus being of repulsive type. The manuscript is organize as follows. The state systems which constrain the optimization problems are efine for the microscopic an the kinetic case in Section 2. The cost functionals an the corresponing constraine optimization problems are state in Section 3. In Section 4 the associate first orer optimality conitions are erive. Section 5 aresses the numerical schemes an the optimization algorithms. The numerical results for the IC an the OC approach are iscusse in Section 6. Concluing Remarks are given in Section Microscopic an Mean-Fiel Optimal Control Problems We start by escribing the agent-base moel an its corresponing mean-fiel limit, which are the state systems for the control problems consiere later on Microscopic Moel. Let D 1 enote the imension of the spatial an velocity space. The consiere particle system consists of N N particles of the same type an M external agents with N M. Let [, T ] be the time interval of observation, then the particles an agents are represente by state vectors The vectors x i, v i, m, u m : [, T ] R D, for i = 1,... N an m = 1,..., M. x(t = (x i (t i=1,...,n an v(t = (v i (t i=1,...,n, enote the positions an velocities of the particles an (t = ( m (t m=1,...,m an u(t = (u m (t m=1,...,m, the positions an velocities of the external agents, respectively (see also [1]. Since the time epenence is clear, we often write x or x i instea of x(t respectively x i (t to shorten the notation. Note that we write the vectors x an v in bol to make clear when enoting the vectors containing positions an velocities of iniviuals an when we refer to the position an velocity space with variables x an v in the mean-fiel setting later on. The interactions of the iniviuals an external agents are moele using potentials Φ j, j = 1, 2 which satisfy the following assumption (A Φ j : R D R are raially symmetric an continuously ifferentiable, with Φ j locally Lipschitz an globally boune. These rather strict assumptions allow for existence an uniqueness of solutions to the ajoint an state systems. The latter are essential for efining the reuce cost functional neee in the algorithms for numerical investigation. In aition to the interaction terms, our moel inclues (linear friction represente by the friction parameter α >. To simplify the presentation, we enote for any x, y R D the interaction forces K j (x, y = ( Φ j (x y, j = 1, 2. Since Φ j is symmetric, we have that K j (x, y = K j (y, x for x, y R D.

3 (1a ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL 3 Altogether, this leas to the particle system t x i = v i, t v i = 1 N K 1 (x i, x k 1 M k i M K 2 (x i, m αv i, (1b t m = u m. The iniviuals interact pairwise (j = 1 with each other an with the external agents (j = 2. The latter o not interact among themselves nor are influence by the others. Note however that such interactions can easily be consiere by a simple transformation of the optimal control variables analogous to the approach in [6]. For system (1 the velocities of the external agents u are assume to be given. Later, they serve as controls for the optimization problem. We assume that the external agents have a maximal spee u max which inuces a constraint in the space of amissible controls (see (11 below. For notational convenience we further efine the state vector y := (x, v, R D(2N+M an S(y = (S i (y i=1,...,n, m=1 S i (y = 1 K 1 (x i, x j 1 N M Using this notation, the ODE system (1 may be written compactly as (2a j i M K 2 (x i, m αv i. m=1 t y = ( t x, t v, = (v, S(y, u =: F (y, u, t with the mapping F : R D(2N+M R DM R D(2N+M. The microscopic state system is supplemente with the initial conitions (2b x( = x R DN, v( = v R DN, ( = R DM. These initial conitions are enote in short by y( = y Remarks on Well-Poseness. The well-poseness of (2 is rather stanar. However, since it is common in optimal control problems to consier a Hilbert space for the control parameters, we use (3 U := L 2 ((, T, R MD as space of controls. Nevertheless, this is sufficient to prove the well-poseness of (1a. Inee, may be explicitly expresse as (t = + t u(s s, which shows that is absolutely continuous an hence also continuous. Consequently, we obtain from the bouneness of u the existence an uniqueness of a global solution ue to the theorem by Picar an Linelöf. From the expression above we easily euce that H 1 ((, T, R MD. Altogether, we are able to efine the corresponing control-to-state operator G N : U Y which maps any control parameter u U to the unique solution y = G N (u of (2 in the state space Y. In this case, we choose the state space (4 Y := H 1 ((, T, R ND H 1 ((, T, R ND H 1 ((, T, R MD. Note, that the solution y C 1 ((, T, R ND C 1 ((, T, R ND H 1 ((, T, R MD Y is inee in a subspace of Y Mean-Fiel Moel. In orer to efine the limiting problem for an increasing number of iniviuals N explicitly, we consier the empirical measure µ N t (x, v = 1 N δ (x i (t x δ (v i (t v. N i=1 By efinition, µ N t P(R 2D is a Borel probability measure that assigns the probability µ N t (A of fining particles with states within a Borel measurable set A R 2D in the phase space R 2D at time t. If a Borel probability measure µ t P ac (R 2D is absolutely continuous w.r.t. the Lebesgue

4 4 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE measure, we enote its ensity by f t L 1 (R 2D. For later use we introuce the macroscopic ensity ρ t of a Borel probability measure µ t P(R 2D as its first marginal, i.e., ρ t (A := µ t (A R D = f t (x, v x v, A R D for any Borel measurable set A R D. The last equality hols whenever µ t P ac (R 2D. The link between the particle system an the mean-fiel equation is erive formally using ieas from [5, 17, 31]. Let therefore h C c (R 2D be an arbitrary real infinitely ifferentiable function on R 2D with compact support an z i = (x i, v i R 2D. Then µ N t t, h := 1 t N N h(z i = 1 N i=1 which allows for the formal calculation t µ N t, h = 1 N x h(z i v i v h(z i N (5 = i=1 µ N t, x h v v h N x h(z i i=1 ( (K1 ρ N t (xi + 1 M ( (K1 ρ N 1 t (x + M t x i + v h(z i t v i, M K 2 (x i, m + αv i m=1 M K 2 (x, m + αv, where ( K1 ρ N t (x = K 1 (x, x µ N t ( z. R 2D Passing to the limit N an integrating by parts, we arrive at the equation where we efine m=1 = t µ t + v x µ t + v (S(µ t µ t, h, S(µ t (x, v, = ( K 1 ρ t (x 1 M M K 2 (x, m αv. Since h C c (R 2D is arbitrary, we may use the variational lemma to fin that m=1 (6 t µ t + v x µ t + v (S(µ t µ t =, which is the mean-fiel single particle istribution, supplemente with the initial conition µ( = µ. Remark 2.1. Observe that µ N t an µ t satisfy exactly the same equation in the istributional sense. The above limit exists for example in the Wasserstein metric (for a etaile iscussion see [7]. Note that even though µ t has unit mass by construction, we o not nee to explicitly assume that the conition is satisfie, since this is a natural constraint an the control parameter oes not change this property. For this reason the use of the stanar theory of OC base on the L 2 -calculus is justifie. Otherwise we woul enter the regime of probability measures leaing to Wasserstein erivatives. Note that while we pass to the mean-fiel limit N, the number of external agents M remains finite Remarks on Well-Poseness. The existence an uniqueness of solutions for the Vlasov equation (6 may be foun, for example, in the articles [5, 7, 17, 21], where the notion of solution is establishe in the Wasserstein space of Borel probability measures. Definition 2.2. Let P 1 (R 2D enote the space of Borel probability measures on R 2D with finite first moment. We say that µ C([, T ], P 1 (R 2D is a weak measure solution of (6 with initial conition µ P 1 (R 2D if for any test function h Cc ((, T ] R 2D we have ( t h t + v x h t + S(µ t v h t µt t + h µ =. R 2D R 2D (7 On the other han, the equation (6 may be equivalently expresse as a nonlinear flow t Z = ( t x, t v = ( v, S(µ t (Z,, µ t = law(z(t,

5 ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL 5 with the initial conition Z( = Z R 2D, where Z is a ranom variable istribute accoring to µ = law(z. Assuming the solvability of the nonlinear process (7, a weak measure solution of (6 may be represente as the push-forwar of the measure along the flow Z(t, Z, i.e., µ t = Z(t, #µ. If we further assume that the initial measure µ P 1 (R 2D has ensity f L 1 (R 2D, an that the nonlinear flow given by (7 satisfies Z C([, T ], Diff(R 2D, i.e., Z(t, is a iffeomorphism for all t, then µ t P1 ac(r2d has ensity f t L 1 (R 2D an f C([, T ], L 1 (R 2D. In orer to employ the stanar L 2 -calculus we will require more regularity of the states an assume aitionally that (B µ P 1 (R 2D has ensity f L 2 (R 2D with compact support an the flow given by (7 satisfies Z C 1 ([, T ], Diff(R 2D such that f H 1 ((, T, L 2 (R 2D. Remark 2.3. Note that the assumption on the nonlinear flow Z C 1 ([, T ], Diff(R 2D satisfying (7 in (B is not restrictive ue to assumption (A. Inee, since µ C([, T ], P 1 (R 2D, we have that S(µ t is continuous in t an locally Lipschitz in z. Therefore, stanar ODE theory provies the require regularity for the nonlinear flow. Defining the state variable p := (f,, the couple system of (6 an the ODE (1b that moels the movement of the external agents with control w may be written as (8a t p := ( tf t, t = ( v (S(f t f t + v x f t, w =: G(p, w. The initial conitions for this system are (8b f t t= = f L 2 (R 2D, ( = R MD. In the following we shall refer to (8b as p( = p. Since the problem is efine on the whole space R 2D, there are no bounary conitions require. In fact, from (B, we expect f t to have compact support that remains within a boune convex omain Ω R 2D with smooth bounary for all times t [, T ] (cf. [2, Thm 4.4]. For this reason, we may efine the corresponing control-to-state operator G : U Y, u p = G (u, where p satisfies (8 with the state space Y given by (9 Y := H 1 ((, T, L 2 (Ω H 1 ((, T, R MD. Here, Ω R 2D is an a priori given boune convex omain with smooth bounary which contains the support of f t for all times t [, T ]. 3. The Optimal Control Problems The aim of the optimal control problem is to guie the crow to a specifie estination E es R D by controlling the velocity of the external agents. In orer to moel this issue mathematically as an optimal control problem, we will nee a cost functional that reflects the aim of steering the system of particles to a esire estination on the one han, an on the other han it has to converge in the limit N to a suitable cost functional on the mean-fiel level. In the following, we state one possible cost functional base on the center of mass an the variance of the crow, which meets these conitions Optimal Control Problem in the Microscopic Setting. As above x enotes the particle positions. Hence the center of mass E N an variance V N for the particles are efine as E N (x(t = 1 N x i (t, V N (x(t = 1 N x i (t E N (x(t 2, N N i=1 an we efine the cost functional (1 J N (y(u, u = 1 T i=1 σ 1 4 V N(x(t V N 2 + σ 2 2 E N(x(t E es 2 + σ 3 2M u(t 2 R MD t, where V N is the esire variance, e.g. zero. The first part of the functional penalizes the ifference of the sprea of the particles with respect to the esire variance V N. The secon term measures the istance of the center of mass to the estination E es R D. Technically, the thir part ensures the local convexity of the cost functional. Physically, it is an energy term causing the velocities of

6 6 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE the external agents to remain boune. Thus, the energy supplie to the system by the movement of the agents is minimize. Altogether, the above cost functional moels the task of guiing the crow of iniviuals such that the crow is clustere aroun E es R D while utilizing the least amount of kinetic energy possible. The positive parameters σ i allow to ajust the influence of the ifferent parts of the cost functional. Due to the maximal spee constraint on the velocities of the external agents we efine the space of amissible controls U a as (11 U a := { u L 2 ((, T, R MD : u m (t u max, m = 1,..., M }. This yiels a family of constraine optimal control problems on the microscopic level given by Problem 1: Fin (y, u Y U a such that where J N (y, u is given by (1. (y, u = argmin (y,u Y Ua J N (y, u subject to the IVP (2, Remark 3.1. For the IC approach we cut the time interval into slices an solve a stationary control problem on each time slice (see Section 5. The velocities of the external agents are assume to be constant on each time slice. The function u U a is the piecewise constant collection of the solutions on the slices projecte onto the feasible set. Due to the constraint on the maximal velocity (11, we use a projecte graient metho to solve the optimization problems Optimal Control Problem in the Mean-Fiel Setting. As iscusse alreay at the beginning of this section, the control problem on the microscopic level has to match the one of the mean-fiel level as N. Thus, the following cost functional is a irect consequence of (1. The center of mass E an variance V only epen on the macroscopic ensity an are efine as E (f t = xf t x v, V (f t = x E (f t 2 f t x v, R D R D R D R D leaing to the cost functional (12 J (p(w, w = 1 T σ 1 4 V (f t V 2 + σ 2 2 E (f t E es 2 + σ 3 2M w t 2 R MD t. Finally, we efine the control problem in the mean-fiel setting as Problem 2: Fin (p, w Y U a such that (p, w = argmin (p,w Y Ua J (p, w subject to the mean-fiel system (8, where J (p, w is efine in (12. Remark 3.2. An existence result for Problem 1 an 2 may be euce in a straight-forwar way from the results in [2]. Even though the authors work in a sparse control setting, similar arguments may be applie in the present setting as well. 4. First Orer Necessary Conitions As mentione in the introuction we apply ajoint base escent methos to solve the control problems. In this section the ajoints an optimality conitions for the particle an the meanfiel optimal control problem are formally erive with the help of the extene Lagrangians corresponing to Problem 1 an 2. Furthermore, we introuce the reuce cost functional an its graient Ajoint of the Microscopic System. We begin by recalling the spaces U an Y efine in Section 2. Let the control space U an state space Y be the Hilbert spaces U = L 2 ((, T, R MD, Y = [H 1 ((, T, R ND ] 2 H 1 ((, T, R MD, with U a U efine in (11. We further enote X := [L 2 ((, T, R ND ] 2 L 2 ((, T, R MD an Z := X ( [R ND ] 2 R MD, as the space of Lagrange multipliers with Z being its ual.

7 ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL 7 We efine the state operator e N : Y U Z of the microscopic problem as ( e N (y, u = ty F (y, u y( y an the ual pairing e N (y, u, (ξ, η Z,Z = ( t y(t F (y(t, u(t ξ(t t + (y( y η. Let (ξ, η Z enote the Lagrange multiplier which is in fact the ajoint state. Then, the extene Lagrangian corresponing to Problem 1 reas L N (y, u, ξ, η = J N (y, u + e N (y, u, (ξ, η Z,Z. As usual the first-orer optimality conition of Problem 1 is erive by solving L N (y, u, ξ, η =. The erivative w.r.t. the ajoint state results in the state equation while the erivative with respect to the state y yiels the ajoint system an the optimality conition is obtaine by the erivative w.r.t. the control u [26]. For the calculations we enote the three parts of the cost functional by J i, i = 1, 2, 3, as J 1 N(y = σ 1 4T V N (x(t V N 2 t, JN(y 2 = σ 2 2T J 3 N(u = σ 3 2MT u(t 2 R MD t. E N (x(t E es 2 R D t, For any h = (h y = (h x, h v, h, h u Y U, the following Gâteaux erivatives are obtaine x J 1 N(y[h x ] = σ 1 NT x JN(y[h 2 x ] = σ 2 NT u J 3 N(u[h u ] = σ 3 M (V(x(t V N (x(t E(x(t h x (t t, (E(x(t E es h x (t t, u(t h u (t t. For the secon part of the Lagrangian we erive x e N (y, u[h x ], (ξ, η = v e N (y, u[h v ], (ξ, η = e N (y, u[h ], (ξ, η = u e N (y, u[h u ], (ξ, η = t hx (t ξ 1 (t x S(y[h x (t] ξ 2 (t t + h x ( η 1, ( t hv (t + αh v (t ξ 2 (t h v (t ξ 1 (t t + h v ( η 2, t h (t ξ 3 (t S(y[h (t] ξ 2 (t t + h ( η 3, h u (t ξ 3 (t t, where the vanishing erivatives are omitte. Assuming that ξ Y one may formally erive the strong formulation of the ajoint system. Inee, integrating by parts an using the fact that x S(y an S(y are symmetric matrices, we arrive at the following result. Proposition 4.1. The first-orer optimality conition corresponing to Problem 1 reas ( σ3 (13 MT u (t ξ 3 (t (u(t u (t t for all u U a, where ξ = (ξ 1, ξ 2, ξ 3 Y satisfies the ajoint system given by (14a with t ξ 1 = x S(y[ξ 2 ] x J N (t, (14b x J N (t = σ 1 NT t ξ 2 = ξ 1 αξ 2, t ξ 3 = S(y[ξ 2 ], ( (V(x(t V N (x(t E N (x(t + σ 2 (E N (x(t E es, NT

8 8 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE supplemente with the terminal conitions ξ 1 (T =, ξ 2 (T =, ξ 3 (T = Ajoint of the Mean-Fiel System. Here, we assume that p = (f, lies within the state space Y of the PDE optimization problem, where Y = H 1 ((, T, L 2 (Ω H 1 ((, T, R MD. Let X := H 1 ((, T, L 2 (Ω L 2 ((, T, H 1 (Ω L 2 ((, T, R MD an set Z := X ( L 2 (Ω R MD to be the space of ajoint states with ual Z. Note that the control space U is ientical to the one of the particle problem. Now efine the mapping e : Y U Z by ( T e (p, w, (ϕ, η Z,Z = t g t + v x g t + S(f t v g t ft z t + ( Ω t w ϕ t + g(t f(t g(f( z (f( f η f z + (( η, Ω with the ajoint state (ϕ, η X ( L 2 (Ω R MD, ϕ = (g, ϕ an η = (η f, η. Similar to the microscopic case we efine the extene Lagrangian corresponing to Problem 2 as L (p, w, ϕ, η = J (p, w + e (p, w, (ϕ, η Z,Z, Again, we enote the three parts of the cost functional by J i, i = 1, 2, 3, as J 1 (p = σ 1 4T V (f t V 2 t, J (p 2 = σ 2 2T J 3 (w = σ 3 2MT Ω w(t 2 R MD t. E (f t E es 2 t, Analogous to the microscopic case we erive the ajoint system an the optimality conition by calculating the erivatives of L w.r.t. the state variable p in irection h p = (h f, h Y, an the control w in irection h w U. The stanar L 2 -calculus yiels f J 1 (p[h f ] = σ 1 T f J 2 (p[h f ] = σ 2 T w J 3 (w[h w ] = σ 3 MT Ω Ω (V (f t V x E (f t 2 h f t t, x (E (f t E es h f t t, w(t h w (t t. Let ϕ = (g, ϕ be the ajoint state corresponing to p = (f,. Then we obtain for the constraint part of the extene Lagrangian the following Gâteaux erivatives: f e (p, w[h f ( ], (ϕ, η = t g t + v x g t + S(f t v g t h f t z t, Ω f S(f t [h f t ] vg t f t z t Ω + g(t h f (T h f (g( h f (η f z, e (p, w[h ], (ϕ, η = w e (p, w[h w ], (ϕ, η = h w (t ϕ (t t. Ω t h (t ϕ (t t + h ( η, S(f t [h (t] v g t f t z t, Assuming again the ajoint state ϕ to be sufficiently regular, we may integrate by parts to obtain a strong formulation of the ajoint system. For the nontrivial terms we calculate the following Ω

9 ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL 9 representations: f S(f t [h f t ] vg t f t z = K 1 (x, x h f t z vg t (z f t z Ω Ω Ω ( = K 1 ( x, x v g t (z f t z Ω Ω h f t z =: D f (f t [g t ](z h f t z, Ω ( S(f t [h (t] v g t f t z = S(f t [h (t] v g t (z f t z Ω Ω ( = S(f t [ v g t (z] f t z h (t Ω (15 =: D (f t [g t ] h (t. This yiels the following ajoint system an optimality conition. Proposition 4.2. The optimality conition corresponing to Problem 2 reas ( σ3 (16 MT w (t ϕ (t (w(t w (t t for all w U a, where ϕ = (g, ϕ Y satisfies the ajoint system given by (17a (17b where (17c x J (t = σ 1 T t g t + v x g t = S(f t v g t + D f (f t [g t ] x J (t, t ϕ = D (f t [g t ], (V(f t V x E (f t 2 + σ 2 T (E (f t E es x on Ω. supplemente with the terminal conitions g T = an ϕ (T =. Remark 4.1. Note, that the optimality conitions of Problem 1 an 2 coincie. This is ue to the fact that J 3 is ientical in both problems. The mean-fiel limit only affects the ajoint system, but formally they can be ientifie in the following way: along the characteristics of the partial ifferential equation for g, ξ 1 correspons to 1 N xg an ξ 2 to 1 N vg. Finally, ϕ can be ientifie irectly with ξ 3. Remark 4.2. The variational inequalities (13 an (16 may be equivalently expresse as fixe point problems in terms of a projection operator Proj U : U U a which is efine by [26] Proj U (h = argmin u Ua u h U for any h U. Inee, since U is a Hilbert space the following statements are equivalent for any k U an γ > : (1 u U a, k, u u U for all u U a, (2 u = Proj U (u γk. Consequently, the variational inequalities (13 an (16 may be expresse as u = Proj U (u γk U a, where k(u = σ 3 /(MT u ξ 3 for the microscopic case (13 an k(w = σ 3 /(MT w ϕ for the mean-fiel case (16. In our particular case, Proj U has the explicit representation given by (18 Proj U (h(t = { umax h m(t h m(t for h m (t > u max, h m (t otherwise, m = 1,..., M, a.e. in (, T. Due to the high imensionality of the problem, it is not recommene to solve the system consisting of state equation, ajoint equation an optimality conition all at once. Therefore, we employ a graient escent metho. In orer to efine this metho we make use of the graient of the reuce cost functional which is erive in the following for both the microscopic problem an its mean-fiel counterpart.

10 1 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE 4.3. Graient of the Reuce Cost Functional. In this section we introuce the reuce cost functionals ĴN(u an Ĵ (w an formally calculate their graients ĴN(u an Ĵ (w which we nee for the escent algorithms. By means of the control-to-state operators G N : U Y an G : U Y introuce in Section 2, we efine the reuce cost functionals as Ĵ N (u := J N (G N (u, u, Ĵ (w := J (G (w, w. Assuming sufficient regularity for G N an G we further erive the graients of the reuce cost functionals. Making use of the state equation e N (y, u = an e (p, w = we implicitly obtain G N (u an G (w via = u e N (G N (u, u = y e(g N (u, u[g N (u] + u e N (G N (u, u, = w e (G (w, w = p e(g (w, w[g (w] + w e (G (w, w, With the help of the ajoint equations ( y e(y, u [ξ] = y J N (y, u an ( p e(p, w [ϕ] = p J (p, w we may calculate the Gâteaux erivative of Ĵ in the irection h U, which gives ĴN(u[h] = y J N (y, u, G N (u[h] Y,Y + u J N (y, u, h U = σ 3 MT u ξ 3, h U, Ĵ (w[h] = p J (p, w, G (w[h] Y,Y + w J (p, w, h U = σ 3 MT w ϕ, h U. Since U is a Hilbert space, we may use the Riesz representation theorem to ientify the graients (19 ĴN(u = σ 3 MT u ξ 3 an Ĵ (w = σ 3 MT w ϕ. Now, we have all ingreients at han to state the graient escent metho for the numerical simulations. 5. Numerical Schemes an Algorithms Next, we investigate two control strategies, namely the Instantaneous Control (IC an Optimal Control (OC. Both strategies are base on the ajoints an make use of the graients erive in the previous section. As stanar iterative ajoint base optimization methos, the algorithms rely on solvers for the state an ajoint systems an upate the controls using the graient information Numerics for the Forwar an Ajoint IVP. All occurring ODE systems are solve with the explicit fourth orer Runge Kutta solver opri of the Python package python.scipy. The high orer is more relevant for the ajoint problem than for the forwar problem, since the ajoint system is stiffer ue to the aitional term arising from the linearization of the non-linear interaction an the cost functional. In orer to have compatible ata that can be passe from the forwar problem to the ajoint problem an then further to the graient upate, we fix a time iscretization in avance an give the solvers the freeom to make intermeiate steps. To obtain a numerical scheme that is inepenent of the number of particles involve we rescale the ajoint ODE by multiplying with N. This has the effect that the N-epenence of the terms in (14b emerging from the cost functional, is vanishing. In fact, for i = 1,..., N, we set r i (t = Nξi 1(t an s i(t = Nξi 2 (t, an obtain the rescale ajoint ODE system (2a (2b (2c where t r i = 1 N xi K 1 (x i, x j (s i s j 1 xi K 2 (x i, m s i 1 M T x i J N (t, j i t s i = r i αs i, t ϕ i = 1 NM N xi K 2 (x i, m s i. i=1 (2 xi J N (t = σ 1 (V(x(t V N ( x i (t E N (x(t + σ 2 ( EN (x(t E es, with terminal conitions r(t =, s(t = an ϕ(t =. m

11 ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL Numerics for the Mean-fiel Equation an its Ajoint. The forwar an backwar solves for the mean-fiel optimization are realize using the Strang splitting scheme [13]. This scheme applies a semi-lagrangian solver in spatial an a semi-implicit finite-volume scheme in velocity space. Using the following short han notation for (8 we efine the splitting t f = v x f v (S(ff, (21a (21b (21c t f = 1 2 v (S(f f, t f = v x f, t f = 1 2 v (S(ff, f (t = f(t, f (t = f (t + τ, f(t = f (t + τ. A Semi-Lagrangian metho [28, 35] is use to solve (21b. The polynomial reconstruction is one with the help of cubic Bezier curves. As usual, information is trace back along the characteristic curves starting in the respective gri point an interpolate at the en of the characteristic to get the values at the current time. The iscretizations in velocity space, (21a an (21c, use a secon orer finite volume scheme where the avection is approximate by a Lax-Wenroff flux [29, 33]. Oscillations cause by non-smooth solutions are intercepte with the help of a van-leer limiter [36]. For more etails on the scheme see [11]. Basically the same coe is use for the ajoint system, for this sake rewrite S(f v g = v (S(fg + 2αg. Further, we nee to a the term resulting from the linearization of the non-linear interaction an the cost functional. Altogether we obtain for the ajoint system the splitting t g = x J ( v (S(fg + 2αg + D f (f t [g t ], t g = v x g, t g = 1 2 ( v (S(fg + 2αg + D f (f t [g t ], g (t = g(t, g (t = g (t + τ, g(t = g (t + τ. with D f (f t [g t ] as efine in (15 an x J as in (17c The Optimization Algorithms. In the following we iscuss the algorithms for the IC an OC approach employe for the optimization. In general the proceures iffer in the amount of information use to compute the graient to upate the control for the next iteration. The escent irections are base on the graients in the following manner. Let k enote the current iteration, s k the escent irection an c k the control u k or w k, respectively. Then the control is upate ue to (22 c k+1 = c k + ω k s k, where ω k is an amissible step size. Remark 5.1. Note that the interpretation of one iteration iffers for the IC ansatz an the OC approach. For IC the iteration k enotes the current time slice whereas for OC k is the counter of the optimization iterations. For the Instantaneous Control metho we use steepest escent steps to upate the control once on every time slice, therefore s k = Ĵ(u k. For the Optimal Control approach we apply a non-linear conjugate graient metho (NCG, see Algorithm 1 for etails. In the pseuocoe the graient Ĵ(u k is enote by q k to shorten the notation. We choose NCG, since it converges faster than the steepest escent metho an requires less memory than BFGS methos. Nevertheless, the CG factor γ k is chosen such that the escent irection resulting from Algorithm 1 equals the escent irection of a BFGS formula applie one time to the ientity [4, Thm 5.13]. Due to the maximal velocity constraint (11 we nee to make sure that compute the controls are feasible. We therefore project the controls onto the feasible set in every iteration using the operator Proj U efine in (18. The step sizes ω k are obtaine with the help of the projecte Armijo step size rule [26] (see Algorithm 2.

12 12 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE Algorithm 1: Non-Linear Conjugate Graient Direction Data: current graient q k, previous graient q k 1, previous escent irection s k 1 Result: escent irection s k initialization; if k = 1 then s k = q k else γ k 1 = (q k q k 1,q k if en en (q k q k 1,s k 1 an s k = q k γ k 1 s k 1 ; (s k, q k > tol CG then s k = q k Algorithm 2: Projecte Armijo Stepsize Rule Data: Current control c k, graient q k, initial ω, initial γ Result: new control c k+1 initialization; while Ĵ(Proj U[c k ω k q k ] Ĵ + γω k q k 2 o ω k = ω k /2 en The Instantaneous Control is a feeback control strategy. The graient is base on state information of the current time slice. Figuratively, that means that the external agents may react instantaneously on the behavior of the crow an o not take the past an future into account. Thus, the ajoints an graients are base on the forwar ata of one time step only, resulting in avantageous memory consumption (cf. Table 3 below an short computation times. To realize the IC approach we ivie the time interval [, T ] into K equiistant slices an sequentially compute the optimal control for every time slice (cf. [27]. The control problem then reas For k = 1,..., K fin u k R DM such that σ 1 u k = argmin Ĵ N (u k = argmin u k U a u k U a 4 (V N(x k V N (x 2 + σ 2 2 E N(x k E es σ 3 2M u k 2 2 subject to y k = y k 1 + t F (y k 1, u k in the ODE case an For k = 1,..., K fin w k R DM such that σ 1 w k = argmin Ĵ (w k = argmin w k U a w k U a 4 (V (f k V (f 2 + σ 2 2 E (f k E es σ 3 2M w k 2 2 subject to p k = p k 1 + t G(p k 1, w k in the PDE case. The control for time slice k + 1 is initialize with (23 c k+1 =.1c k to make sure that the penalty term of the cost functional is initially non-zero. The final positions an velocities of particles an the positions of the external agents of the time step t k 1 are the initial values for the time slice t k. The controls are assume to be constant in each time slice. The controls for the whole simulation interval [, T ] are obtaine by gluing the solutions on the time slices together leaing to a piecewise constant control function. A pseuocoe for the IC proceure can be foun in Algorithm 3. Remark 5.2. Note that we perform only one graient step on each time slice. Numerical tests showe that more graient steps marginally improve the result but rastically increase the computational time.

13 ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL 13 Algorithm 3: Instantaneous Control Algorithm Data: Initial ata of y or p; parameter values Result: Instantaneous control u or w; the corresponing states y(u or p(w; optimal functional values initialization; t = ; t = T/K; while t < T o. solve state system (2 or (8; 1. solve ajoint problem given in (2 or (17; 2. compute graient corresponing to (19; 3. compute step size using the Armijo rule (2 4. upate controls by steepest escent step (22; 5. project control into the feasible set using (18; 6. initialize controls for the next time slice corresponing to (23; t = t+t en The Optimal Control strategy is an open-loop approach that works on the whole time interval [, T ] at once. The ajoint an graient computations use the information of a full forwar solve on the interval [, T ]. This results in a large amount of memory storage neee (cf. Table 3 below. A vivi escription of this strategy is that the agents plan their whole journey in avance. The optimization algorithm operates as follows. First, the state system is solve for initially given velocities for the external agents. Its solution is then use to compute the ajoint solution backwar in time on the full interval [, T ] as well. Finally, the control for Problem 1 or Problem 2 is upate for the next iteration using the graient corresponing to (19 an (22. The iterations are stoppe when the L 2 norm of two consecutive controls is smaller than the tolerance tol see (27. A pseuocoe for the Optimal Control Algorithms is given in Algorithm 4. Algorithm 4: Optimal Control Algorithm Data: Initial ata of y or p; parameter values Result: Optimal control u or w; the corresponing states y(u or p(w; optimal functional values initialization; solve state system (2 or (8; while eps rel > tol o 1. solve ajoint problem given in (2 or (17; 2. compute graient corresponing to (19; 3. upate controls by NCG step (22 with escent irection of Algorithm 1; 4. project control onto the feasible set using (18; 5. fin appropriate step size by projecte Armijo-rule using Algorithm 2; 6. compute Ĵ, upate eps rel ; en Remark 5.3. Note, that the line search using projecte Armijo Rule might require multiple solves of the forwar system. 6. Numerical Results The numerical simulations for the mean-fiel equation are conucte on the omain Ω R 4 as Ω = [ 1, 1] 2 [ 5, 5] 2, i.e., we set D = 2. For the mean-fiel simulation we use the scale CFL conition τ V T (24 Lh.5

14 14 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE with L = 2, V = 5. The gri parameter h is varie throughout the simulations to investigate the convergence of the scheme. In fact, we use 25, 5 or 1 gri points in each of the two irections leaing to h =.4,.2 or.1. Our particular choice of the interaction potentials are the Morse potentials as propose in [12, 18]. For fixe positive parameters A j, a j, R j, r j we have (25 Φ j (x y = R j exp ( x y r j A j exp ( x y a j, j = 1, 2. The parameters A j, R j enote the attraction an repulsion strengths an a j, r j the raius of interaction. The case j = 1 refers to the interaction of the iniviuals, the interaction of iniviuals with external agents is enote by j = 2. Inspire by [1], we use the values A 1 = 2, R 1 = 5, a 1 = 1, r 1 = 2, A 2 = 5, R 2 = 1, a 2 = 1, r 2 = 5. Remark 6.1. Note that the coefficients of the interaction potentials are scale accoring to Ω. As mentione in the introuction we ha the moel problem of ogs hering sheep to the estination Z in min. The parameters for the sheep-sheep interaction have long range attraction an repulsion on a very short range. The parameters moeling the sheep-og interaction have a larger repulsive influence in orer to reflect the guiing property correctly. Figure 1. Initial configurations. Left: microscopic case with 8 sheep represente by blue ots. Right: initial probability istribution f for the mean-fiel case. In both cases the positions of the ogs are marke by re triangles an the estination E es by a green ot. Further parameters are fixe as follows. The estination E es = ( 2, 2 is epicte in green. The initial support of the sheep is set to Ω = [ 1, 55] [ 2, 55], i.e., f (x, v is the uniform istribution with support Ω an the initial positions an velocities of the sheep are chosen by realizations of i.i.. ranom variables with law(f. In the Figures the sheep an ogs are represente by blue markers an re triangles, respectively. The initial configurations of the microscopic an the mean-fiel case are shown in Figure 1. The parameters for the following results are set to T = 1, Vn =.9V N (x, V =.9V (f an σ 3 =.1. Thus, the esire variance is 1% less than the initial variance given by the initial istribution of the crow. Since the values of the cost functional escent are in the range of 1e 6 on each time slice of the IC Algorithm, we have to choose σ 3 rather small. Otherwise, the Armijo line search oes not succee. The Armijo parameters for IC an OC are set to ω IC = 1, ω OC = 1. Remark 6.2. Note, that in the given time interval [, T ] the task of steering the crow to the estination E es is impossible to realize. In fact, our main focus lies on the comparison of the behaviour of the (sub-optimal controls as N increases. Here sub-optimal refers to the IC simulations an optimal to the OC results.

15 ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL 15 Figure 2. Influence of the cost functional parameters. Left: Setting ICS1 - the ogs positione at the left sie an the bottom of the crow clear the way to the estination. The others push the crow from behin towars the estination. Right: Setting ICS2 - the ogs keep istance an orient towars to corners to reuce the variance of the crow. These observations perfectly agree with the intention we ha when moeling the cost functional Numerical Results Using the IC Algorithm. In this section the results obtaine by the IC algorithm are illustrate. First, we iscuss the influence of the cost functional parameters. We therefore set up three test cases: S1 stresses the variance part of the cost functional, we expect the ogs to move towars the corners of the crow in orer to reuce the sprea of the crow. In S2 the secon part of J is emphasize, thus the ogs shall push the crow towars the estination. For test case S3 we choose the weights of the cost functional such that the focus lies on steering the crow to the estination E es while the variance term has minor influence but cannot be neglecte. In particular, we use the following parameter values (S1 (S2 (S3 σ 1 =.9, σ 2 =.1, σ 1 =.1, σ 2 =.9, σ 1 =.5, σ 2 =.5. These choices assure that J 1 of test case S1 has the same orer of magnitue as J 2 of test case S2 an the other way aroun. The following notation is use in the Figures J 1 = σ ( 1 V N (x t 4 V 2 N an J 2 = σ 2 2 E N(x t E es 2 2 for the microscopic cost functional an analogous for the mean-fiel cost functional. The spatial an velocity gri is iscretize with the same amount of gri points 25, 5 or 1 in each of the two irections for the IC algorithm. Due to the large amount of memory neee for the OC algorithm, we use gris of 25 an 5 points in this case. Further information about the memory consumption of the ifferent simulations can be foun in Table 3. The graphs corresponing to mean-fiel solutions are labelle M# where # enotes the number of gri points. The graphs corresponing to microscopic simulations are enote by their respective number of particles. In Figure 2 the influence of the cost functional parameters is illustrate. We assume that S3 is the most realistic setting for our purpose of leaing the crow to the estination, since the focus lies on J 2 which measures the istance to the estination while the influence of J 1 is not negligible. In Figure 3(left the trajectories of the ogs an the crow at T = 1 are shown. We note that the ogs are not entering the crow as eep as in test case S1. Comparing the values of J, J 1 an J 2 in Figure 3(right an Figure 4 we obviously fin that the part measuring the crow s istance to the estination is superimposing the others. Note that the graphs of the cost functional values are given with respect to time, i.e. it may happen that the values increase. Since the initial positions of the ogs is chosen arbitrarily, they move to appropriate positions first, causing a slight increase of the cost functional. Afterwars the cost functional is ecreasing as expecte. The simulation on the coarse mean-fiel iscretization M 25 overestimates the variance term significantly as can be seen in Figure 4(left. The evolution of

16 16 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE Figure 3. Setting ICS3 - Trajectories of the ogs an evolution of the cost functional. Left: the ogs positione at the left sie an at the bottom of the crow clear the way to the estination. The others push from behin. Note that the ogs on the left to not enter the crow as eep as in setting ICS1. Right: Evolution of the cost functional values. Figure 4. Setting ICS3 - Evolution of the cost functional parts. Left: values of J 1. The simulation M25 overestimates the variance significantly. All other graphs are in goo agreement. Right: values of J 2. All graphs show similar behavior. We see convergence as N an as h. J 1 is in goo resemblance for all other iscretization. Similarly, there is accorance in the graphs showing the evolution of J 2. Furthermore, we see the convergence of the graphs to M1 as N increases an the convergence of the mean-fiel simulations as the gri is refine. In Table 1 we investigate the convergence for N using the following scale norms (26a u u ref = 1 T MV u(t u ref (t R 4 t, (26b J J ref = 1 J(t J ref (t t, T ρ N ρ ref = 1 (26c ρ(t, x, ỹ ρ ref (t, x, ỹ x ỹ t, T where x an ỹ are the spacial measures in x an y irection rescale by 1/L. As reference values we use the results of the simulation M 1. To compute the norms the empirical ensity ρ N of the microscopic simulations is approximate by a histogram which is base on the gri of the corresponing mean-fiel simulation. In the first two columns we see the convergence of the mean-fiel scheme, as expecte the values are ecreasing as the gri is refine. The four columns on the right illustrate the behavior as N increases. All norm values are ecreasing for increasing number of particles from N = 1 to N = 8. The convergence is more transparent for the integral quantities as for the velocities (cf. Remark 6.3.

17 ON THE NUMERICS OF MEAN-FIELD OPTIMAL CONTROL 17 Remark 6.3. Since mean-fiel quantities are average quantities it is very common to compare integral values like J an ρ N in this setting. Note that the velocities an the trajectories of the agents are no such quantities. We therefore expect to have blurre values for u u ref. ICS1 M25 M J J ref 4.882e e e e e e-2 u u ref e e e e e e-2 ρ N ρ ref e e e e-5 ICS2 M25 M J J ref e e e e e e-3 u u ref e e e e e e-2 ρ N ρ ref e e e e-5 ICS3 M25 M J J ref 2.753e e e e-2 5.5e e-3 u u ref 5.222e e e e e e-2 ρ N ρ ref e e e e-5 Table 1. Investigation of convergence as N using the norms in (26 with the respective values of M 1 as reference values. In Figure 5 the evolution of the crow an the ogs is illustrate for the microscopic (N = 8 an the mean-fiel case M1. The re lines show the trajectories of the ogs as above. Aitionally to the information in Figure 3, the re markers inicate the current positions of the ogs at ifferent times. These positions an the form of the crow are in goo accorance Numerical Results Using the Optimal Control Approach. The results for the OC approach are iscusse in this section. As this approach is of open-loop type, we stop the iteration, if w k+1 w L (27 2 ((,T,R MD tol, w L 2 ((,T,R MD with tol =.5 for the computations. Due to the huge amount of memory neee for the simulations we stuy the case S3 for N = 1, 2, 4 an M = 25, 5 only. Remark 6.4. Note, that the reference values of the cost functional graphs in Figure 3(right, Figure 4 an Figure 6 iffer. For the IC simulations the graphs are plotte over time. For the OC simulations the graph is plotte with respect to the iterations, the values have to ecrease in this case. Of course, the values of J N an J are in the same range. The plots of the evolution of the cost functionals over the iterations in Figure 6(left show the typical behavior of Optimal Control computations. In the first optimization iteration the most reuction is obtaine followe by small improvements as the iterations procee. As we have alreay iscusse we cannot expect the ogs to steer the crow to the estination in the short time interval given. Likewise, the cost functionals o reuce only marginal on this short interval. The figure on the right shows the relative norms which are use to evaluate the stopping criterion. Analogous to the behavior of the cost functionals, the graphs of the relative changes in velocity behave very similar for all microscopic an mean-fiel simulations. In Figure 7 (left we see the trajectories resulting from the optimal controls for the ifferent number of particles an the mean-fiel simulations. The trajectories corresponing to the initial controls are epicte in gray. Obviously, in all test cases the eviation from the initial path ue to the optimization is similar. Even though one gets the impression that the og on the right of the mean-fiel simulation stays far away from the crow in Figure 7 (left, we observe in Figure 8 that this istance is cause by the resolution of the gri. We investigate the convergence of the controls an the ensity of the crow as N with the help of the norms efine in (26 in Table 2. For the microscopic simulations the ensity was compute using a histogram base on the gri of the mean-fiel simulation M5. Further, the OC algorithm allows for an investigation of the epenence on the initial ata. To stuy this epenence we escribe ifferent initial controls for the optimization proceure, which

18 18 M. BURGER, R. PINNAU, A. ROTH, C. TOTZECK, AND O. TSE Microscopic an mean-fiel crow together with the ogs at t = 3.3. Microscopic an mean-fiel crow together with the ogs at t = 6.6. Microscopic an mean-fiel crow together with the ogs at t = 1. Figure 5. Setting ICS3 - Comparison of the microscopic (N = 8 an meanfiel (M1 crow an the ogs at ifferent times compute with the IC simulation. u u ref ρ ρ ref e e e e e e-5 M e-2 - Table 2. Setting OCS3 - Differences of the controls measure in the scale norms given in (26.