Characterizing controllability probabilities of stochastic control systems via Zubov s method

Transcription

1 Characterizing controllability probabilities of stochastic control systems via Zubov s method Fabio Camilli Lars Grüne Fabian Wirth, Keywords: Stochastic differential equations, stochastic control system, almost sure controllability, Zubov s method, computational approach. Abstract We consider a controlled stochastic system with an a.s. locally exponentially controllable compact set. Our aim is to characterize the set of points which can be driven by a suitable control to this set with either positive probability or with probability one. This will be obtained by associating to the stochastic system a suitable control problem and the corresponding Bellman equation. We then show that this approach can be used as basis for numerical computations of these sets. Introduction Zubov s method is a general procedure which allows to characterize the domain of attraction of an asymptotically stable fixed point of a deterministic system by the solution of a suitable partial differential equation, the Zubov equation (see f.e. [3 for an account of the various developments of this method). A typical difficulty in the application of this method, i.e. the existence of a regular solution to the Zubov equation, was overcome in [5 by using a suitable notion of weak solution, the Crandall-Lions viscosity solution. The use of weak solutions allows the extension of this method to perturbed and controlled systems, see [9, Chapter VII for an overview. In [6, [3 the Zubov method was applied to Ito stochastic differential equations obtaining in the former a characterization of set of points which are attracted with positive probability to an almost surely exponentially stable fixed point; in the latter a characterization of the points which are attracted with probability (or any fixed probability) to the fixed point. It is worth noting that the Zubov method also yields a Lyapunov function for the deterministic or the stochastic system Sez. di Matematica per l Ingegneria, Dip. di Matematica Pura e Applicata, Università de l Aquila, 674 Monteluco di Roio (AQ), Italy, camilli@ing.univaq.it Mathematisches Institut, Fakultät für Mathematik und Physik, Universität Bayreuth, 9544 Bayreuth, Germany, lars.gruene@uni-bayreuth.de Hamilton Institute, NUI Maynooth, Maynooth, Co. Kildare, Ireland, fabian.wirth@may.ie Supported by Science Foundation Ireland grant /PI./C67. as the unique solution of the Zubov equation. This fact can be used as a basis for numerical computations of the domain of attraction (see [4 in the deterministic case and [3 in the stochastic one). In many applications it is interesting to consider the socalled asymptotic controllability problem, i.e. the possibility of asymptotically driving a nonlinear system to a desired target by a suitable choice of the control law. Whereas in the deterministic case there is huge literature about this problem (see f.e. [6), in the stochastic case it seems to be less considered, also because it request some degeneration of the stochastic part which makes it difficult to handle with classical stochastic techniques. In [ this problem was studied for a deterministic system by means of Zubov s method. Here we use the same approach for a stochastic differential equation. In the stochastic case the Zubov method splits into two parts: In the first step we introduce a suitable control problem, with a fixed positive discount factor δ (chosen equal to for simplicity), associated with the stochastic system. We show that a suitable level set of the corresponding value function v gives the set of initial points for which there exists a control driving the stochastic system to the locally controllable with positive probability. The value function is characterized as the unique viscosity solution of the Zubov equation, which is the Hamilton Jacobi Bellman of the control problem. In the second step we consider as a parameter the discount factor δ and we pass to the limit for δ +. The set of points controllable to the fixed point with probability one is given by the subset of R N where the sequence v δ converges to. The sequence v δ converges to a l.s.c. v which is a ersolution of an Hamilton-Jacobi-Bellman related to an ergodic control problem. In this respect the Zubov equation with positive discount factor can be seen as a regularization of the limit ergodic control problem which gives the appropriate characterization. This paper is organized as follows: In Section we give the setup and study the domain of possible controllability. In Section 3 we analyze the domain of almost sure controllability, and finally, in Section 4 we describe an example where the previous objects are calculated numerically.

2 Domain of possible null-controllability and the Zubov equation We fix a probability space (Ω, F, F t, P), where {F t } t is a right continuous increasing filtration, and consider the controlled stochastic differential equation { dx(t) = b(x(t), α(t)) dt + σ(x(t), α(t)) dw (t) X() = x () where α(t), the control applied to the system, is a progressively measurable process having values in a compact set A R M. We denote by A the set of the admissible control laws α(t). Solutions corresponding to an initial value x and a control law α A will be denoted by X(t, x, α) (or X(t) if there no ambiguity). We assume that the functions b : R N A R N, σ : R N A R N M are continuous and bounded on R N A and Lipschitz in x uniformly with respect to a A and that A. Moreover we assume that there exists a set R N locally a.s. uniformly null-controllable, i.e. there exist r, λ positive and a finite random variable β such that for any x B(, r) = {x R N : d(x, ) r}, there exists α A for which d(x(t, x, α), ) βe λt a.s. for any t >. () In this section we study the domain of possible nullcontrollability C, i.e. the set of points x for which it is possible to design a control law α such that the corresponding trajectory X(t, x, α) is attracted with positive probability to. Hence C = { x R N : there exists α A s.t. P[ lim t + d(x(t, x, α), ) = > }. We introduce a control problem associated to the dynamics in the following way. We consider for x R N and α A the cost functional { + J(x, α) = E g(x(t), α(t))e t } g(x(s),α(s))ds dt = E [e + g(x(t),α(t))dt (3) where g : R N A R is continuous and bounded on R N A and Lipschitz continuous in x uniformly in a A, g(x, a) = for any (x, a) A and inf g(x, a) g >. (R N \B(,r)) A We consider the value function and we can prove v(x) = inf J(x, α) Theorem. C = {x R N : v(x) < }. Proof: Note that by definition v and v(x) > for x. We claim that C is the set of the points x R N for which there exists α A such that E[exp( t(x, α)) >, where t(x, α) = inf{t > : X(t, x, α) B(, r)}. (4) In fact, if x C, then clearly P[{t(x, α) < } > for some α A and therefore E[exp( t(x, α)) >. On the other hand, if E[exp( t(x, α)) > for a control α A, then P[{t(x, α) < } >. By (), we have P[{t(x, α) < + } { lim d(x(t, x, α), ) = } t + = P[{ lim t + d(x(t, x, α), ) = t(x, α) < } = P[{t(x, α) < } = P[{t(x, α) < + }, hence x C. This shows the claim. Now if x C, then for any control α we have E[e t(x,α) =. Hence E [e t(x,α) g(x(t),α(t))dt E[e gt(x,α) =. and therefore v(x) =. If x C, by the previous claim there exists α such that P[t(x, α) < + >. Set τ = t(x, α) and take T and K sufficiently large in such a way P[B := P [{τ T } {β K} η > where β is given as in () For t > T, by () we have E [ E[ X(t, x α) B χ B Then v(x) = E [ E[ X(t τ, X(τ, x, α), α(t τ)) BχB Ke λ(t T ). E [ E[e T g(x(t),α(t)))dt+ + T e (MgT +LgK/λ) < g(x(t),α(t)))dt Bχ B where M g and L g are respectively an upper bound and the Lipschitz constant of g. We have obtained a link between C and v. In the next two propositions we study the properties of these objects in order to get a PDE characterization of v. Proposition. i) B(, r) is a proper subset of C. ii) C is open, connected, weakly positive forward invariant (i.e. there exists α A such that P[X(t, x, α) C for any t >.)

3 iii) E[exp( t(x, α)) if x x C. Proof: The proof of this proposition is similar to the ones of the corresponding results in [6. Hence we only give the details of i) and we refer the interested reader to [6 for the other two statements. Take x B(, r), let α be a control satisfying () and fix b > such that P[B := P[β b ɛ >. From (), there is T > such that P[B {d(x(t, x, α), ) r for t > T } = ɛ (5) Recalling that for any x, y R N and δ > lim P[ X(t, x, α) X(t, y, α) > δ =. x y t [,T select δ such that for any y B(x, δ), defined A = { t [,T X(t, x, α) X(t, y, α) r/}, then P [A c ɛ/ (A c denotes the complement of A in Ω). Set C = A B. From (5), if y B(x, δ) we have that and therefore, from () Moreover P[{d(X(t, y, α), ) r} P[{d(X(t, x, α), ) r/} C P[{ lim d(x(t, y, α), ) = } t + P[{d(X(t, y, α), ) r} P[C P[C = P[A c B c (P[A c + P[B c ) ɛ. It follows that P[{lim t + d(x(t, y, α), ) = } is positive for any y B(x, δ) and therefore B(x, δ) C for any x B(, r). Remark.3 Note that if C does not coincide with all R N, the weakly forward invariance property requires some degeneration of the diffusion part of the stochastic differential equation on the boundary of C, see f.e. [. The typical example we have in mind is a deterministic system driven by a stochastic force, i.e. a coupled system X(t) = (X (t), X (t)) R N R N = R N of the form dx (t) = b (X (t), X (t), α(t))dt dx (t) = b (X (t), α(t)) dt + σ (X (t), α(t)) dw (t), see e.g. [7 for examples of such systems. Note that for systems of this class the diffusion for the overall process X(t) = (X (t), X (t)) is naturally degenerate. Set Σ(x, a) = σ(x, a)σ t (x, a) for any a A and consider the generator of the Markov process associated to the stochastic differential equation L(x, a) = N N Σ i j (x, a) + b i (x, a). (6) x i x j x i i,j= i= Proposition.4 v is continuous on R N and a viscosity solution of Zubov s equation { L(x, a)vδ ( v(x))g(x) } = (7) a A for x R N \. Proof: The only point is to prove that v is continuous on R N. Then from a standard application of the dynamic programming principle it follows immediately that v is a viscosity solution of (7) (see f.e. [7, [8). Note that v in the complement of C. From Prop., if x n C and x n x C we have v(x n ) E[e gt(xn,α) for n + and therefore v is continuous on the boundary of C. To prove that v is continuous on the interior of C, it is sufficient to show that v is continuous in B(, r) since outside g is strictly positive and we can use the argument in [4, part I, Theorem II.. Fix x, y B(, r) and ɛ >. Let b be such that P[B := P [{β b} ɛ/8. Take T in such a way that L g b exp( λt )/λ < ɛ/4, where λ as in (), and let α be a control satisfying () and v(x) E[e + g(x(t,x,α),α(t))dt + ɛ 8 and δ sufficiently small in such a way that E X(t, x, α) X(t, y, α) ɛ/4l g T if x y δ and t T. Hence [ E d(x(t, y, α), )dt χ B T [ E d(x(t + T, y, α( + T )), )dt χ B and be λt /λ. v(y) v(x) E e + g(x(t,y,α),α(t))dt e + g(x(t,x,α),α(t))dt ɛ + 8 ( P(B c ) + E [L T g X(t, y, α) X(t, x, α) dt + T + ɛ 8 ɛ. (d(x(t, x, α), ) + d(x(t, y, α), ))dt )χ B The next theorem gives the characterization of C through the Zubov equation (7). Theorem.5 The value function v is the unique bounded, continuous viscosity solution of (7) which is null on.

4 Proof: We show that if w is a continuous viscosity subsolution of (7) such that w(x) for x, then w v in R N. Using a standard comparison theorem (see f.e. [8), the only problem is the vanishing of g on. Therefore we first prove that w v in B(, r) using (), we then obtain the result in all R N by applying the comparison result in R N \ B(, r). Since w is a continuous viscosity subsolution, it satisfies for any x {δ d(x, ) /δ} w(x) inf E e T τ δ { T τδ g(x(t), α(t))e t g(x(s),α(s))ds dt + } g(x(t),α(t))dt w(x(t τ δ )) for any T > where τ δ = τ δ (α) is the exit time of the process X(t) = X(t, x, α) from {δ d(x, ) /δ}(see [5). Fix ɛ > and let δ > be such that if d(z, ) δ, then w(z), v(z) ɛ. For x B(, r) by the dynamic programming principle we can find α A satisfying () and such that { T τδ v(x) E g(x(t), α(t))e t g(x(s),α(s))ds dt + e T } τ δ g(x(t),α(t))dt v(x(t τ δ )) + ɛ. Therefore we have w(x) v(x) E { e τδ g(x(t),α(t))dt (w(x(τ δ )) v(x(τ δ )) χ {τδ T }} + Me g δ T + ɛ where g δ = inf{g(x, a) : d(x, ) δ, a A} > and M = max{ w, v }. Set B k = {β K} and take T and K sufficiently large in such a way that Me gδt ɛ, MP[Bk c ɛ and, recalling (), P[B k {τ δ T } = P[B k. By (8), we get v(x) w(x) ɛp[b k + MP[B c k + ɛ 4ɛ and for the arbitrariness of ɛ we have w v in B(, r). By a similar argument we can prove that if u is a continuous viscosity ersolution of (7) such that u(x) for x, then u v in R N. Remark.6 The function v is a stochastic control Lyapunov function for the system in the sense that inf E[v(X(t, x, α)) v(x) < for any x C \ and any t >. 3 Domain of almost sure controllability In this section we are interested in a characterization of the set of points which are asymptotically controllable to the set with probability arbitrarily close to one, i.e. in the set D = { x R N : P[ lim d(x(t, x, α), ) = = }. t + (8) We require a slightly stronger stability condition, namely that besides () it is also verified that for any x B(, r), there exists a control α A such that E[d(X(t, x, α), ) q Me µt a.s. for any t > (9) for some q (, and positive constants M, µ. We consider a family of value functions depending in the discount factor on a positive parameter δ v δ (x) = inf E [ + δg(x(t), α(t))e t δg(x(s),α(s))ds dt = inf E[ e + δg(x(t),α(t))dt The main result of this section is Theorem 3. D = {x R N : lim δ v δ (x) = } () Proof: The proof of the result is split in some steps. Claim : For any x B(, r), v δ (x) Cδ for some positive constant C. Since g is Lipschitz continuous in x uniformly in a and g(x, a) = for any (x, a) A, we have g(x, a) min{l g x, M g } C q x q for any q (, and corresponding constant C q. Let α be a control satisfying (9). Then for any δ, by the Lipschitz continuity of g, () and (9) we get v δ (x) E δ [ + + δg(x(t), α(t))e t δg(x(s),α(s))ds dt E [g(x(t), α(t)) dt + δc q E[d(X(t, x, α), ) q dt + δc q Me µt dt hence the claim. Claim : For any x R N, lim E[e δt(x,α) = P[t(x, α) < () δ where t(x, a) is defined as in (4). The proof of the claim is very similar to the one of Lemma 3. in [3, so we just sketch it. Let α A be such that E[e δt(x,α) E[e δt(x,α) + ɛ and T such that exp( δt ) ɛ for T > T. Hence for T > T E[e δt(x,α) E[e δt(x,α) χ {t(x,a)<t } + E[e δt P[t(x, α) < T + ɛ P[t(x, α) < + ɛ from which we get lim δ E[e δt(x,α) P[t(x, α) <.

5 To obtain the other inequality in (), take α A, T sufficiently large and δ small such that and for t < T Hence P[t(x, α) < P[t(x, α) < + ɛ P[t(x, α) < T + ɛ e δt ɛ. E[e δt(x,α) E[e δt(x,α) χ {t(x,α)<t } E[( ɛ)χ {t(x,α)<t } = ( ɛ)p[t(x, α) < T ( ɛ) ( P[t(x, α) < ɛ ). Since ɛ is arbitrary, it follows that lim inf δ Claim 3: For any x R N, E[e δt(x,α) P[t(x, α) <. lim v δ(x) = P[t(x, α) < δ For any α A, we have E[e δg(x(t),α(t))dt E[e δgt(x,α) and therefore by Claim, lim inf δ v δ(x) lim inf δ inf { E[e δgt(x,α) } P[t(x, α) <. Now fix ɛ >, δ > and take T sufficiently large in such a way that exp( δm g T ) ɛ. By the dynamic programming principle, for any α A we have v δ (x) E{ T t(x,α) δg(x(t), α(t))e t δg(x(s),α(s))ds dt+ e T t(x,α) δg(x(t),α(t))dt v(x(t t(x, α))}. () Now using Claim and recalling that v δ we estimate the second term in the right hand side of () by E[e T t(x,α) δg(x(t),α(t))dt v(x(t t(x, α)) E[v(X(t(x, α))χ {t(x,a) T } + E[e T δmgdt χ {t(x,a) T } Cδ + ɛ and the first one by [ T t(x,α) E δg(x(t), α(t))e t [ t(x,α) E δg(x(t), α(t))e t δg(x(s),α(s))ds dt δg(x(s),α(s))ds dt = E[ e t(x,α) δg(x(t),α(t))dt E[ e δmgt(x,α). Substituting the previous inequalities in () we obtain lim δ v δ (x) lim δ inf E[ e δmgt(x,α) + Cδ + ɛ which, recalling Claim, completes the proof of Claim 3. Equality () follows immediately from Claim 3 observing that P[ lim d(x(t, x, α), ) = = P[t(x, a) < t + Remark 3. Note the by the same argument of the previous theorem we can more generally prove that if D p = { x R N : P[lim t + d(x(t, x, α), ) = = p } for p [, then the following characterization holds D p = {x R N : lim δ v δ (x) = p} Remark 3.3 As in Theorem.5, we can prove that for any δ > the value function v δ is the unique viscosity solution of the Zubov equation a A { L(x, a)vδ δ( v δ (x))g(x) } = in R N \ which is null on, where L(x, a) is defined as in (6) 4 A numerical example We illustrate our results by a numerical example. The example is a stochastic version of a creditworthiness model given by with and dx (t) = (α(t) λx (t))dt + σx (t)dw (t) dx (t) = (H(X (t), X (t)) f(x (t), α(t)))dt H(x, x ) = ( α α + x x ) µ θx, x x α θx α, x > x f(x, α) = ax ν α α β x γ. A detailed study of the deterministic model (i.e., with σ = ) can be found in [. In this model k = x is the capital stock of an economic agent, B = x is the debt, j = α is the rate of investment, H is the external finance premium and f is the agent s net income. The goal of the economic agent is to steer the system to the set {x }, i.e., to reduce the debt to. Extending H to negative values of x via H(x, x ) = θx one easily sees that for the deterministic model controllability to {x } becomes equivalent to controllability to = {x /}, furthermore, also for the stochastic model any solution with initial value (x, x ) with x < will converge to, even in finite time, hence satisfies our assumptions.

6 Using the parameters λ =.5, α =, α = (α + ), µ =, θ =.,, a =.9 ν =., β =, γ =.3 and the cost function g(x, x ) = x we have numerically computed the solution v δ for the corresponding Zubov equation with δ = 4 using the scheme described in [3 extended to the controlled case (see [ for more detailed information). For the numerical solution we used the time step h =.5 and an adaptive grid (see [) covering the domain Ω = [, [ /, 3. For the control values we used the set A = [,.5. As boundary conditions for the outflowing trajectories we used v δ = on the upper boundary and v δ = for the lower boundary, on the left boundary no trajectories can exit. On the right boundary we did not impose boundary conditions (since it does not seem reasonable to define this as either inside or outside ). Instead we imposed a state constraint by projecting all trajectories exiting to the right back to Ω. We should remark that both the upper as well as the right boundary condition affect the attraction probabilities, an effect which has to be taken into account in the interpretation of the numerical results. Figure show the numerical results for σ =,. and.5 (top to bottom). In order to improve the visibility, we have excluded the values for x = from the figures (observe that for x = and x > it is impossible to control the system to, hence we obtain v δ in this case) References [ M. Bardi and P. Goatin, Invariant sets for controlled degenerate diffusions: a viscosity solutions approach, in Stochastic analysis, control, optimization and applications, 9 8, Birkhäuser, Boston, MA, 999. [ F. Camilli and M. Falcone, An approximation scheme for the optimal control of diffusion processes, RAIRO, Modélisation Math. Anal. Numér. 9 (995),97. [3 F. Camilli and L. Grüne, Characterizing attraction probabilities via the stochastic Zubov equation, Discrete Contin. Dyn. Syst. Ser. B. 3 (3), [4 F. Camilli, L. Grüne, and F. Wirth. A regularization of Zubov s equation for robust domains of attraction. In Nonlinear Control in the Year, A. Isidori et al., eds., Lecture Notes in Control and Information Sciences, Vol. 58, Springer-Verlag, London,, [5 F. Camilli, L. Grüne and F. Wirth, A generalization of the Zubov s equation to perturbed systems, SIAM J. of Control & Optimization 4 (), [6 F. Camilli and P. Loreti, A Zubov s method for stochastic differential equations, to appear on NoDEA Nonlinear Differential Equations Appl. Figure : Numerically determined controllability probabilities for σ =,.,.5 (top to bottom) [7 F. Colonius, F.J. de la Rubia and W. Kliemann, Stochastic models with multistability and extinction levels, SIAM J. Appl. Math. 56(996), [8 W.H. Fleming and M.H. Soner, Controlled Markov processes and viscosity solutions, Springer Verlag, New York, 993. [9 L. Grüne, Asymptotic Behavior of Dynamical and Control Systems under Perturbation and Discretization, Lecture Notes in Mathematics, Vol. 783, Springer Verlag, Berlin,. [ L. Grüne, Error estimation and adaptive discretization for the discrete stochastic Hamilton Jacobi Bellman equation, Preprint, Universität Bayreuth, 3, submitted. [ L. Grüne, W. Semmler and M. Sieveking, Creditworthiness and Thresholds in a Credit Market Model with Multiple Equilibria, Economic Theory, to appear. [ L. Grüne and F. Wirth, Computing control Lyapunov functions via a Zubov type algorithm. In Proceedings

7 of the 39th IEEE Conference on Decision and Control, Sydney, Australia,, [3 H. K. Khalil, Nonlinear Systems, Prentice-Hall, 996. [4 P.L. Lions, Optimal control of diffusion processes and Hamilton-Jacobi-Bellman equations: I and II, Comm. Partial Differential Equations, 8 (983),-74 and 9-7. [5 P.L. Lions and P. E. Souganidis, Viscosity solutions of second-order equations, stochastic control and stochastic differential games, in Stochastic differential systems, stochastic control theory and applications (Minneapolis, Minn., 986), 93 39, Springer, New York, 988 [6 E.D. Sontag, Stability and stabilization: discontinuities and the effect of disturbances, in Nonlinear analysis, differential equations and control (Montreal, QC, 998), , NATO Sci. Ser. C Math. Phys. Sci., 58, Kluwer Acad. Publ., Dordrecht, 999. [7 J.Yong and X.Y.Zhou, Stochastic Controls: Hamiltonian systems and HJB equations, Springer Verlag, New York, 999. [8 V.I. Zubov, Methods of A.M. Lyapunov and their Application, P. Noordhoff, Groningen, 964.