Optimal Sojourn Time Control within an Interval 1

Transcription

1 Optimal Sojourn Time Control within an Interval Jianghai Hu and Shankar Sastry Department of Electrical Engineering and Computer Sciences University of California at Berkeley Berkeley, CA Abstract In this paper we study the problem of find optimal feedback control that can keep an agent within an interval for at least a certain amount of epected time with the least energy. The dynamics of the agent are subject to the perturbation of random noises, and thus are given by a stochastic differential equation. By formulating the problem as an optimal control problem, we propose a solution by using the Maimum Principle and finding a first integral to the resulting differential equations. Some numerical simulations are presented to illustrate the results. Introduction In many practical applications where safety is the principal concern, for eample, in traffic systems such as air traffic management system [] and automated highway system [5], the evolution of the state of the agents to be controlled can often be modeled as a proper dynamical system subject to the perturbation of random noises, or more precisely, by a stochastic differential equation, with the control to the system of the form of state feedback control. The system is defined to be safe as long as its state evolves inside a certain subset of the state space called the safe region, and whenever this is violated, some emergency procedures have to be evoked in order to reset the state within the safe region. Due to the usually much higher cost of the emergency procedures, it is preferable to design a feedback control law of a reasonable cost that can keep the state of the system within the safe region for as long as possible. Or equivalently, one wishes to find a feedback control with the least cost that can keep the system safe long enough. In this paper we study a simple instance of such a problem. As a concrete eample, consider the following scenario. Suppose that there are three consecutive cars driving in the same direction on a road, numbered,, and 3 from front to end. The body length of each car is This material is based upon work supported by the National Science Foundation under Grant No. EIA d D d Figure : Three cars on a highway. d. Denote by D the distance between car and car 3 and by the distance between car and car 3, both ecluding body length. Hence the distance between car and car is D d (see Fig. ). Suppose that in a period of time, D remains constant, i.e., car and car 3 have identical velocities, while is given by the equation: d(t) = f[(t)] + w(t). dt Here f is the feedback control law for car that depends on the state, and w(t) is the white noise modeling random perturbations such as air resistance and road frictions. Car is safe if its distance from either car or car 3 is at least ɛ >, i.e., if [ɛ, D d ɛ]. Suppose that at time t = car is at its ideal position () = (D d)/. Due to the presence of noises, (t) will eventually venture outside of the safe region. Denote by T the first time (t) escapes from the safe region, which is a positive random variable. A natural question arises: among all the feedback control laws f satisfying the constraint E[T ] > t for some threshold t, find the one with the minimal cost D d ɛ ɛ f () d. The solution to a version of this problem will be discussed in this paper. This paper is organized as following. First of all, the problem under study is defined in its most general form in Section. In Section 3, some properties of its optimal solutions are established, which inspire us to propose a simplified version of the problem. By formulating the simplified problem as an optimal control problem in Section, we use the Maimal Principle to find the equations for the optimal solutions. Due to presence of the two-point boundary conditions, these equations are hard to solve directly. However, by finding

2 a first integral, we can significantly simplify their solution. These results are illustrated through numerical simulations. Finally, conclusions and possible etensions are discussed in Section 5. Problem Formulation g() µ= µ= µ= Given an interval [ a, a] on the real line, consider the solution X t to the following stochastic differential equation (SDE): dx t = f(x t )dt + db t, where B t is the standard Brownian motion, and f is a piecewise Lipschitz continuous function. Denote by T a = inf{t : X t = a} (respectively, T a = inf{t : X t = a}) the first time X t hits a (respectively, a). Then T a T a min{t a, T a } is the first time X t escapes from the interval [ a, a]. For each [ a, a], define g() = E [T a T a ], where E means that the epectation is taken under the initial condition that X =. Thus g() is the epected escaping time from [ a, a] given that X t starts from Figure : Plots of g() for µ =,,. J *(t ) J(f) It is a standard result in stochastic calculus (see, e.g., []) that g satisfies the following ordinary differential equation (ODE): g () + f()g () + =, [ a, a], () with the boundary condition g( a) = g(a) =. Note that g belongs to C, the family of functions on [ a, a] with piecewise Lipschitz continuous second order derivative. Remark For certain special cases, equation () can be solved eplicitly [3]. For eample, if f u on [ a, a], then X t = B t + ut is the BM with drift u, and g() = (a )eua + ( + a)e ua ae u u(e ua e ua. ) In particular, if f on [ a, a], then X t = B t is the standard BM. Let u in the above equation, we have g() = a. Figure plots g() for different values of u. We now formulate our problem. Problem Among all the control functions f for which the corresponding g satisfies g ma ma{g() : [ a, a]} t o a t g() Figure 3: Feasible region of control function f in (g(), J(f)) coordinates. for some fied threshold t >, find the one that minimizes the energy J(f) = a a f () d. In other words, we try to find the minimal energy control f subject to the constraint that the epected escaping time from at least one point in the safe region [ a, a] is no less than t. By Remark, if f has the minimum energy J(f) =, i.e., if f, then g() = a, hence g ma = a. As a result, unless t > a, the solution to Problem is always f. Therefore, we shall in the following assume that t a. For each threshold t, denote by J (t ) the energy of

3 the solution to Problem, i.e., J (t ) = inf{j(f) : f such that g ma t }. Then it is clear that J (t ) is an increasing function of t. J (t ) has the following interpretation. Represent each control function f as a point on a plane such that its coordinate is given by g ma and its y coordinate is given by J(f). In this coordinate system, the feasible region of all possible f is plotted as the shaded region in Fig. 3. The region is bounded from the right by the graph of the function J (t ) for t a, and from the left by another curve of no practical interest. The verte of the region corresponds to the control function f. As a by-product of the above analysis, we have an equivalent (dual) formulation of the problem as follows. Problem Among all the control functions f whose energy is at most J >, find the one for which the corresponding g ma is maimized. The value of g ma for the solution to Problem as a function of the threshold J is eactly the inverse of the function J. We shall focus on Problem only in the rest of the paper. To prove this proposition, we need two technical lemmas on the monotonicity of solutions to the ODE (). Lemma For a fied control function f and R, denote by g(; g, u ) the solution to the ODE () with initial condition g( ) = g and g ( ) = u. Then for any fied and g, g(; g, u ) is an increasing function of u if >, and a decreasing function of u if <. Proof: Suppose that for some u < v, >, we have g () g(; g, u ) > g () g(; g, v ). Then since g () < g () for > in a neighborhood of, there eists an y (, ] where g g achieve its maimum over [, ]. At y we have g (y) < g (y) and g (y) = g (y). Now g () and g () g (y) + g (y) are two solutions to the ODE () with identical values of g(y) and g (y), but nonetheless differ at and, a contradiction to the the uniqueness of solutions to the ODE (). Lemma Suppose that g and g are the solutions to the ODE () with initial condition g( ) = g, g ( ) =, that correspond to the control functions f and f, respectively, and that f f on [, ). Then g g on [, ). 3 Some Properties of Optimal Solutions We first show some useful properties of the optimal solutions to Problem. Proposition For any control function f, the solution g to equation () is strictly concave. Therefore, g, and there is a unique [ a, a] such that g( ) = g ma. Proof: We adopt a probabilistic proof using the definition of g. For any a y < < z a, g() = E [T a T a ] = P (T y < T z )g(y)+ P (T z < T y )g(z) + E [T y T z ], where T y and T z denote the first time X t hits y and z respectively. Note that P (T y < T z )+P (T z < T y ) = and E [T y T z ] >. Therefore, g is strictly concave. Proposition Suppose that f is an optimal solution to Problem, and the corresponding g assumes its maimum at. Then f on [ a, ] and f on [, a]. In other words, the optimal control function f will direct toward from both side. The proof of Lemma follows the same line as that of Lemma, hence is omitted. We now prove Proposition. Suppose otherwise that the optimal solution f is positive on some nontrivial subinterval I = [, ] of [, a]. Define a new control function ˆf such that ˆf = f on I and ˆf = f at everywhere else. Note that ˆf and f have the same energy. We claim that by adopting ˆf as the control function, the epected escaping time, ĝ, is higher at, a contradiction to our assumption that f is optimal. To prove this claim, choose = and g = g( ) in Lemma to conclude that the solution to equation () with initial condition g( ) and g ( ) = corresponding to the new control function ˆf is no larger than g everywhere on [, a]. Piece such a solution on [, a] with g on [ a, ] together to obtain a solution to () on [ a, a] with initial condition g( a) = and g(a) <. Now Lemma shows that a solution to equation () with initial condition g( a) = and g(a) = is smaller than the pieced together solution at any ( a, a]. In other words, the epected escaping time corresponding to the control function ˆf starting from is larger than g( ), a contradiction. The other half of the proposition can be proved similarly. We conjecture that for optimal solution f to Problem, the corresponding g assumes its maimum at =. However, a rigorous proof is yet to be found. If this is

4 indeed the case, then it is clear that there is a version of f that is odd on [ a, a] which generates an escaping time of the same value at. In fact, starting any optimal solution f, if a f () d a f () d, then by choosing ˆf() = f() on [ a, ] and ˆf() = f( ) on [, a], we have a control function ˆf for which the escaping time ĝ satisfies ĝ() = g() on [ a, ] and ĝ() = g( ) on [, a]. Inspired by the above discussions, we propose a restricted version of Problem as follows. Problem 3 Among all the control functions f that satisfy g ma t and that are odd on [ a, a], find the one with minimal energy. Because of the restriction, f is odd and g is even on [ a, a]. Hence we need only to design the functions f and g on [ a, ], and then etend them to the whole interval [ a, a] by their respective symmetries. The equation that g satisfies is now given by g () + f()g () + =, [ a, ], () with initial condition g( a) =, g() = t, g () =. Here we use the obvious fact that the optimal solution f satisfies g() = t. Then the optimal control f satisfies []: and λ = H y =, (5) λ = H y = fλ λ, (6) with the boundary conditions H f = f λ y =, (7) y ( a) =, y () = t, y () =, λ ( a) =. By (5), λ is constant on [ a, ]. From (7) we have f = λ y. (8) Substituting this into () and (6), we obtain y = λ y, (9) λ = λ y λ, () whose boundary conditions are y () = and λ ( a) =. We now try to solve the above dynamic equations. Using equations (9) and (), we have Solution Using Optimal Control Theory Define y () = g(), y () = g (). Then equation () implies and (λ y ) = λ y + λ y = (λ y λ )y λ (λ y + ) = (λ y + λ ), y = y, (3) y = fy, () for [ a, ]. The boundary conditions are y ( a) =, y () = t, and y () =. The above equations specify the dynamics of a control system whose states are (y, y ) and whose input is f. The cost of the system is a f () d = J(f). So Problem is equivalent to Problem Solve the following optimal control problem: Minimize subject to Define the Hamiltonian f () d a { y = y, y = fy, [ a, ], y ( a) =, y () = t, y () =. H(y, y, λ, λ, u) = λ y + λ ( fy ) + f = λ y λ (fy + ) + f. (λ λ y ) = λ λ y = (λ y λ ) + λ (λ y + ) Therefore, = λ y (λ y + λ ). (λ λ y ) = (λ y )(λ y ) = (λ y ). Integrating, we have a first integral λ λ y = λ y + C, [ a, ], () for some constant C. The initial conditions y () = and λ ( a) = imply that C = λ () = λ y ( a). Now λ and y can be solved from () as λ = ± + y (λ y + C) y, y = λ ± λ λ (λ C) λ,

5 Φ[y ( )] 8 6 f opt.5.5 g opt y ( ).5.5 Figure : Plot of Φ[y ( a)] for a =. which, after substitution into (9) and (), yield y = ± + y (λ y + C), λ = ± λ λ (λ C), By Proposition, the sign in the first equation can be easily determined as: y = + y (λ y + C) = + λ y [y y ( a)], () with boundary conditions y () =. Write equation () as dy / + λ y [y y ( a)] = d, and integrating from = a to =, we have Ψ(λ, y ( a)) y( a) dy + λ y [y y ( a)] = a, (3) which defines λ as an implicit function of y ( a) [a, ). In fact, for each fied y ( a) a, since y[y y ( a)] < for < y < y ( a), Ψ(λ, y ( a)) is strictly increasing with λ. Note that Ψ(, y ( a)) = y ( a) a, and lim λ Ψ(λ, y ( a)) = a. So there eists a unique λ, denoted as ϕ[y ( a)], such that Ψ(ϕ[y ( a)], y ( a)) = a. From the above discussion, if y ( a) a, then the solution y to equation () satisfies the boundary condition y () = if and only if we choose λ = ϕ[y ( a)]. To determine y (), note that y satisfies y = y, y ( a) =, y () = t. Therefore, if we define Φ[y ( a)] = a y () d, f opt.5.5 g opt f opt g opt Figure 5: Plots of optimal f and g when a =, with t = (first two rows),t = 3 (middle two rows), and t = 6 (last two rows). where y is the solution to equation () with initial condition y ( a) a and with λ = ϕ[y ( a)], then Φ[y ( a)] = y () y ( a) = t. () From eperiments, Φ[y ( a)] is strictly increasing for y ( a) a. See Figure for a plot of Φ[y ( a)] when a =. Therefore we can find a y ( a) such that Φ[y ( a)] = t if t a. Using this y ( a) and λ = ϕ[y ( a)], the solution to Problem can be found by integrating the following differential equations: y = y, y ( a) =, y = + λ y [y y ( a)]. Note that other boundary conditions y () = t and y () = are automatically satisfied by our choice of y () and λ. Hence the difficulty of solving the two-

6 point boundary problem specified by (9) and () is avoided. Figure 5 plots the optimal f and g computed as above when a = for t =, 3, 6 respectively. The first two rows are the optimal f and g for t =, the middle two rows for t = 3, and the last two rows for t = 6. As epected, as t increases, the optimal feedback control law f becomes more aggressive. An interesting fact is that the optimal f is nearly zero around the center of the interval, while much of the effort is spent near / and 3/ positions of the interval. 5 Conclusion In this paper the problem of optimal sojourn time control for an agent moving in an interval is studied. Using tools from optimal control theory, and by finding a first integral to the resulting equations, we propose a solution to the problem under the symmetric assumption. As future directions, we are currently trying to prove the conjecture that solutions to the symmetric version of the problem are also solutions to the general problem. It is also interesting to see if our approach can be etended to the higher dimensional case, for eample, when the safe region is a disk in R n. References [] A. E. Bryson, Jr. and Y. C. Ho. Applied Optimal Control. Hemisphere Publishing Corp. Washington, D. C., 975. Optimization, Estimation, and Control, Revised printing. [] R. Durrett. Stochastic Calculus: A Practical Introduction. CRC Press, 996. [3] L. C. G. Rogers and D. Williams. Diffusions, Markov Processes, and Martingales, volume I, II. John Wiley & Sons, 99. [] C. Tomlin. Hybrid control of air traffic management system. Ph.D. thesis, EECS Department, UC Berkeley, 997. [5] P. Varaiya. Smart cars on smart roads: problems of control. IEEE Transaction on Automatic Control, AC-38(), 993.