On reachability under uncertainty. A.B.Kurzhanski. Moscow State University. P.Varaiya. University of California at Berkeley

Transcription

1 On reachability under uncertainty A.B.Kurzhanski Moscow State Uniersity P.Varaiya Uniersity of California at Berkeley Abstract. The paper studies the problem of reachability for linear systems in the presence of uncertain (unknown but bounded) input disturbances that may also be interpreted as the action of an adersary in a game-theoretic setting. It denes possible notions of reachability under uncertainty emphasizing the dierences between reachability under open-loop and closed-loop control. Solution schemes for calculating reachability sets are then indicated. The situation when obserations arrie at gien isolated instances of time, leads to problems of anticipatie (maxmin) or nonanticipatie (minmax) piecewise open-loop control with corrections and to the respectie notions of reachability. As the number of corrections tends to innity, one comes in both cases to reachability under nonanticipatie feedback control. It is shown that the closed-loop reach sets under uncertainty may be found through a solution of the forward Hamilton-Jacobi- Bellman-Isaacs equation. The basic relations are deried through the inestigation of superpositions of alue functions for appropriate sequential maxmin or minmax problems of control. Keywords: reachability, reach sets, dierential inclusions, alternated integral, funnel equations, open-loop control, closed-loop control, dynamic programming, uncertainty, dierential games, HJBI equation. AMS subject classications: 34HO5, 34G25, 35F10,49L, 49N70, 91A23. Introduction Recent deelopments in real-time automation hae promoted new interest in the reachability problem the computation of the set of states reachable by a controlled process through aailable controls. Being one of the basic problems of control theory, it was studied from the ery begining of inestigations in this eld (see [18]). The problem was usually studied in the absence of disturbances, under complete information on the system equations and the constraints on the control ariables. It was shown, in particular, that the set of states Research supported by National Science Foundation Grant ECS We thank Oleg Botchkare for the gures. 1

2 reachable at gien time t under bounded controls is one and the same, whether one uses open-loop or closed-loop (feedback) controls. It was also indicated that these \reachability sets" could be calculated as leel sets for the (perhaps generalized) solutions to a \forward" Hamilton- Jacobi-Bellman equation [18], [19], [3], [15], [17]. Howeer, in reality the situation may be more complicated. Namely, if the system is subject to unknown but bounded disturbances, it may become necessary to compute the set of states reachable despite the disturbances or, if exact reachability is impossible, to nd guaranteed errors for reachability. These questions hae implicitly been present in traditional studies on feedback control under uncertainty for continuous-time systems, [10], [28], [4], [9], [12]. They hae also appeared in studies on hybrid and other types of transition systems [1], [29], [21], [5]. This leads us to the topic of the present paper which is the inestigation of reachability under uncertainty for continuous-time linear control systems subjected to unknown input disturbances, with prespecied geometric (hard) bounds on the controls and the unknowns. The paper indicates arious notions of reachability, studies the properties of respectie reach sets and indicates routes for calculating them. The rst question here is to distinguish, whether reachability under open-loop and closedloop controls yield the same reach sets. Indeed, since closed-loop control is based on better information, namely, on the possibility of continuous on-line obserations of the state space ariable (with no knowledge of the disturbance), it must produce, generally speaking, a result which is at least \not worse," for example, than the one by an open-loop control which allows no such obserations, but only the knowledge of the initial state, with no knowledge of the disturbance. An open-loop control of the latter type is further referred to as \nonanticipatie." Howeer, there are many other possibilities of introducing open-loop or piecewise open-loop controls, with or without the aailability of some type of isolated on-line measurements of the state space ariable, as well as with or without an \a priori" knowledge of the disturbance. Thus, in order to study the reachability problem in detail, we introduce a hierarchy of reachability problems formulated under an array of dierent \intermediate" information conditions. These are formulated in terms of some auxiliary extremal problems of the maxmin or minmax type. Starting with open-loop controls, we rst distinguish the case of anticipatie control from nonanticipatie control. The former, for example, is when a reachable set, from a gien initial state x 0, at gien time, is dened as the set X? = X? (; ; x 0 ; ) of such states x, that for any admissible disturbance gien in adance, for the whole interal under consideration, there exists an admissible control that steers the system to a -neighborhood B (x) = fz : (z? x; z? x) 2 g. Here the respectie auxiliary extremal problem is of the maxmin type.(maximum in the disturbance and minimum in the control). On the other hand, for the latter the disturbance is not known in adance. Then the reachability set from a gien initial state is dened as the set X + = X + (; ; x 0 ; ) of such states x whose 2

3 -neighborhoods B (x) may be reached with some admissible control, one and the same for all admissible disturbances, whateer they be. Now the respectie auxiliary problem is of the minmax type. It is shown that always X + X? and that the closed-loop reach set X = X(; ; x 0 ; ) attained under nonanticipatie, but feedback control lies in between, namely, X + X X? : There also are some intermediate situations when the obserations of the state space ariable arrie at gien N isolated instants of time. In that case one has to deal with reachability under possible corrections of the control at these N time instants. Here again we distinguish between corrections implemented through anticipatie control (when the future disturbance is known for each time interal in between the corrections) and nonanticipatie control, when it is unknown. The respectie extremal problems are of sequential maxmin and minmax types accordingly and the controls are piecewise open-loop: at isolated time instants of correction comes information on the state space ariable, while in between these the control is open-loop (either anticipatie or not). Both cases produce respectie sequences X? ;N = X? N (; ; x 0 ; ); X + ;N = X + N (; ; x 0 ; ) of \piecewise open-loop reach sets". The relatie positions of the reach sets in the hierarchical scheme are as follows X + X + ;N X X? ;N X? : Finally, in the limit, as the number of corrections N tends to innity, both sequences of reachability sets conerge to the closed-loop reach set 1. The adopted scheme is based on constructing superpositions of alue functions for openloop control problems. In the limit these relations reect the Priciple of Optimality under set-membership uncertainty. This principle then allows one to describe the closed loop reach set as a leel set for the solution to the forward HJBI (Hamilton-Jacobi-Bellman-Isaacs) equation. The nal results are then presented either in terms of alue functions for this equation or in terms of set-alued relations. Schemes of such type hae been used in synthesizing solution strategies for dierential games and related problems, and were constructed in backward time, [23], [11], [27], [28]. The topics of this paper were motiated by applied problems and also by the need for a theoretical basis for further algorithmic schemes. 1 As indicated in the sequel, this is true when all the sets inoled are nonempty and when the problems satisfy some regularity conditions. 3

4 1 Uncertain dynamics. Reachability under open loop controls In this section we introduce the system under consideration and dene two types of openloop reachability sets. Namely, we discuss reachability under unknown but bounded disturbances for the system _x = A(t)x + B(t)u + C(t)(t); (1) with continuous matrix coecients A(t); B(t); C(t). Here x 2 IR n is the state and u 2 IR p is the control that may be selected either as an open loop control OLC a Lebesgue-measurable function of time t, restricted by the inclusion or as a closed-loop control CLC a set-alued strategy Here 2 IR q is the unknown input disturbance with alues u(t) 2 P(t); a:e:; (2) u = U(t; x) P(t): (3) (t) 2 Q(t); a:e: (4) P(t); Q(t) are set-alued continuous functions with conex compact alues. The class of OLC's u() bounded by inclusion (2) is denoted by U O and the class of input disturbances () bounded by (4) as V O. The strategies U are taken to be in U C the class U C of CLC's that are multialued maps U(t; x) bounded by the inclusion (3), which guarantee the solutions to equation (1), u = U(t; x), (which now turns into a dierential inclusion), for any Lebesgue-measurable function (). 2 We distinguish two types of open loop reach sets the maxmin type and the minmax type. As we will see in the next Section, the names maxmin and minmax assigned to these sets are due to the underlying optimization problems used for their calculation. Denition 1.1 An open loop reach set (OLRS) of the maxmin type (from set X 0 = X( ), at time ) is the set X? (; ; X 0 ) of all ectors x such that for eery disturbance (t) 2 Q(t), there exist an initial state x 0 2 X 0 and an OLC u(t) 2 P(t) which steer the trajectory x(t); t, from state x 0 = x( ) to state x() = x: (5) The set X 0 is assumed conex and compact (X 0 2 conir n ). 2 For example, the class of set-alued functions with alues in compir n, upper semicontinuous in x and continuous in t. 4

5 If X? (; ; X 0 ) turns to be empty, one may introduce the open loop -reachable set X? (; ; X 0 ; ) as in Denition 1.1 except that (5) is replaced by Here x() 2 B (x): B (x) = fx : (x()? x; x()? x) 2 g = x + B (0); 0; is the ball of radius with center x. Thus the OLRS X? (; ; X 0 ) of the maxmin type is the set of points x 2 IR n that can be reached, for any disturbance (t) 2 Q(t) gien in adance, for the whole interal t, from some point x( ) 2 X 0, through some open loop control u() 2 U O. The open loop -reach set X? (; ; X 0 ; ) is the set of points x 2 IR n whose -neighborhood B (x) may be reached, for any disturbance (t) gien in adance, through some x( ) 2 X 0 ; u() 2 U O. By taking 0 large enough, we may assume X? (; ; X 0 ; ) 6= ;. Denote x(t; ; x 0 ju(); ()) to be the unique trajectory corresponding to x( ) = x 0, control u() and disturbance (). Then [fx(t; ; x 0 ju(); ())jx( ) 2 X 0 ; u() 2 U O g = X(t; ; X 0 jp(); ()); is the reach set in the ariable u() 2 U O (at time t from set X 0 ) with xed disturbance input (). Lemma 1.1 X? (; ; X 0 ) = \fx(t; ; X 0 jp(); ())j() 2 V O g: (6) This formula follows from Denition 1.1. (Minkowski) dierence P _?Q of sets P; Q, Recall the denition of the geometrical Then directly from (1) one gets X? (; ; X 0 ) = S( ; )X 0 + P _?Q = fc : c + Q Pg: Here S(s; t) stands for the matrix solution of the adjoint equation In other words the set S(s; )P(s)ds _? S(s; )(?Q(s))ds: t)=@s =?S(s; t)a(t); S(t; t) = I: X? (; ; X 0 ) = X(t; ; X 0 jp(); f0g) _?X(t; ; 0jf0g; Q()) 5

6 is the geometric dierence of two \ordinary" reach sets, namely, the set X(t; ; X 0 jp(); f0g) taken from X( ) = X 0 and calculated in the ariable u, with (t) 0, and the set X(t; ; 0jf0g; Q()) taken from x( ) = 0 and calculated in the ariable, with u() 0. This simple geometrical interpretation is of course due to the linearity of (1). For the -reachable set, we hae the following lemma. Lemma 1.2 The set X? (; ; X 0 ; ) may be expressed as X? (; ; X 0 ; ) = \fx(t; ; X 0 jp(); ()) + B (0)j() 2 V O g (8) also = (X(t; ; X 0 jp(); f0g) + B (0)) _?X(t; ; 0jf0g; Q()); X? (; ; X 0 ; 1 ) X? (; ; X 0 ; 2 ); 1 2 : Remark 1.1. Denition (8) of X? (; ; X 0 ; ) may also be rewritten as X? (; ; X 0 ; ) = \ [ u [ x 0fX(t; ; x 0 ju(); ()) + B (0)jx 0 2 X 0 ; u() 2 U O ; () 2 V O g: We now dene another class of open-loop reach sets under uncertainty the OLRS of the minmax type. Denition 1.2 An open loop -reach set (OLRS) of the minmax type (from set X 0 = X( ), at time ) is the set X + (; ; X 0 ; ) of all x for each of which there exists a control u(t) 2 P(t) that assigns to each (t) 2 Q(t) a ector x 0 2 X 0, such that the respectie trajectory x[t] = x(t; ; x 0 ju(); ()) ends in x[] 2 B (x). Thus the -OLRS of minmax type consists of all x whose -neighborhood B (x) contains the states x[] generated by system (1) under some control u(t) 2 P(t) and all f(t) 2 Q(t); t g with x 0 2 X 0 selected depending on u;. 3 A reasoning similar to the aboe leads to the following lemma. Lemma 1.3 The set X + (; ; X 0 ; ) may be expressed as X + (; ; X 0 ; ) = [f(x(; ; X 0 ju(); f0g) + B (0)) _?X(t; ; 0jf0g; Q())ju() 2 U O g; (9) and X + (; ; X 0 ; 1 ) X + (; ; X 0 ; 2 ); 1 2 : 3 With = 0, and X 0 single-alued, it usually turns out that X + = ;. 6

7 Remark 1.2. Denition (9) of X + (; ; X 0 ; ) may be rewritten as [ u \ [ x 0f(x(t; ; x 0 ju(); f0g)+b (0))?x(t; ; 0jf0g; ())jx 0 2 X 0 ; u() 2 U O ; () 2 V O g: Direct calculation, based on the properties of set-alued operations, allows to conclude the following. Lemma 1.4 When X + (; ; X 0 ; ); X? (; ; X 0 ; ) are both nonempty for some > 0, we hae X + (; ; X 0 ; ) X? (; ; X 0 ; ): We shall now calculate the open-loop reach sets dened aboe, using the techniques of conex analysis ([25], [12], [15]). 2 The calculation of open-loop reach sets Here we shall calculate the two basic types of open-loop reach sets. The relations of this section will also sere as the basic elements for further constructions which will be produced as some superposititions of the relations of this section. The calculations of this section and especially of later sections related to reachability under feedback control require a number of rather cumbersome calculations of geometrical (Minkowski) dierences and their support functions. In order to simplify these calculations we transform system (1) to a simpler form. Taking the transformation z = S(t; )x, one gets _z = B 1 (t)u? C 1 (t); where B 1 (t) = S(t; )B(t); C 1 (t) = S(t; )C(t). Keeping the preious notations x; B; C for z; B 1 ; C 1, we thus come, without loss of generality, to the system _x = B(t)u + C(t); (10) with the same constraints on u; as before. For equation (10) consider the following two problems: Problem (I) Gien a set X 0 and x 2 IR n, nd (; x; ) = max min u under conditions x() 2 B (x); u() 2 U O ; () 2 V O. Problem (II) Gien a set X 0 and x 2 IR n, nd V + (; x; ) = min u max min d(x( ); X 0 ); ; x( ) min d(x( ); X 0 ); ; x( ) 7

8 under conditions x() 2 B (x); u() 2 U O ; () 2 V O. Here and G is a closed set in IR n. Thus d(x; z) 2 = (x? z; x? z); d(x; G) = minfd(x; z)jz 2 G)g; d(x; G) = h + (x; G); where h + (Q; G) is the the Hausdor semidistance between compact sets Q; G; dened as h + (Q; G) = max x minf(x? z; x? z z)1=2 jx 2 Q; z 2 Gg: The Hausdor distance is h(q; M) = maxfh + (Q; G); h + (G; Q)g. In order to calculate the function explicitly, we use the relations x(t) = x 0 + t and (see [10], [15] for the next formula) where (B(s)u(s)? C(s)(s))ds; d(x; G) = maxf(l; x)? (ljg)j(l; l) 1g; (11) (ljg) = supf(l; x)jx 2 Gg is the support function of G [15]. (For compact G, sup may be substituted by max.) We thus need to calculate (; x; ) = max min u minfd(x( ); X 0 )jx() 2 B (x); u() 2 U O ; 2 V O g; x( ) which gies, after an application of (11), and an interchange of min u ; min x( ) and max l (see [7]), (; x; ) (12) = maxf(l; x)? (ljx 0 )? (l; l) 1 2? ((ljb(s)p(s))? (?ljc(s)q(s))dsj(l; l) 1g: Due to (11), the last formula says simply that is gien by where X? (; ; x 0 ; ) = It then follows that (; x; ) = d(x; X? (; ; x 0 ; )); (13) X 0 + B (0) + B(t)P(s)ds _? (?C(s))Q(s)ds; (14) X? (; ; X 0 ; ) = fx : (; x; ) 0g; (15) 8

9 and so (12) implies that x 2 X? (; ; X 0 ; ) i (l; x) (ljx 0 ) + (l; l) ((ljb(t)p(s))? (?ljc(t)q(s)))ds; 8l 2 IR n : This gies, from the denitions of support function and geometrical dierence, l (ljx? (; ; X 0 ; )) = (16) X 0 + B (0) + B(s)P(s)ds _? (?C(s)Q(s))ds ; which, interpreted as integrals of multialued functions, again results in (14). Theorem 2.1 The set X? (; ; X 0 ; ) is gien by formula (14) and its support function (ljx? (; ; X 0 ; )) by (16). It is clear that if the dierence then X? (; ; x 0 ; 0) 6= ;. B(t)P(s)ds _? C(s)(?Q(s))ds 6= ;; Note that function (; x; ) may be also dened as the solution to Problem (I ). Gien X 0, nd (; x; ) = max min u minfd(x(); B (x))jx( ) 2 X 0 ; u() 2 U O ; () 2 V O g: x( ) Direct calculations then produce the formula which gies the same result as Problem (I). Similarly, we may calculate V + (; x; ) = min u fx : (; x; ) 0g = X? (; ; X 0 ; ); (17) max minfd(x( ); X 0 )gjx() 2 B (x); u() 2 U O ; () 2 V O : x( ) Taking into account the minimax theorem of [7] and the fact that we come to V + (; x; ) = max l f(l; x)? max g(l) = max(conc g)(l); (l; l) 1; l l (ljb(s)p(s))ds + (conc(?h))(l)j(l; l) 1g; (18) 9

10 h(l) = (ljx 0 ) + (l; l) 1 2? (?ljc(s)q(s))ds: Here (conc h)(l) is the closed concae hull of h(l). Note that (conc h)(l) =?(con(?h)(l); where (con h)(l) = h (l) is the closed conex hull and also the Fenchel second conjugate h (l) of h(l) (see [25], [12] for the denitions). Therefore where = It then follows that X 0 + B (0) _? V + (; x; ) = d(x; X + (; ; X 0 ; )); (19) X + (; ; X 0 ; ) (20) (?C(s))Q(s)ds + t o B(s)P(s)ds: X + (; ; X 0 ; ) = fx : V + (; x; ) 0g: (21) Similarly, (18) implies that V + (; x; ) 0 i (l; x) (l; x 0 ) + so that the support function = l B(s)P(s)ds + ((ljb(s)p(s))ds? (con h)(l); 8l 2 IR n ; (ljx + (; ; X 0 ; )) (22) l : X 0 + B (0) _? (?C(s))Q(s)ds Theorem 2.2 The set X + (; ; X 0 ; ) is gien by (20) and its support function (ljx + (; ; X 0 ; )) by (22). It can be seen from (22) that X + (; ; x 0 ; 0) may be empty. At the same time, in order that X + (; ; x 0 ; ) 6= ;, it is sucient that B (0) _? which holds for > 0 suciently large. (?C(s))Q(s)ds 6= ;; It is worth mentioning that a minmax OLRS may be also be specied through an alternatie denition. 10

11 Denition 2.1 An open loop -reach set (OLRS) of the minmax type (from set X 0, at time ) is the union where X + (; ; X 0 ; ) = [fx + (; ; x 0 ; )jx 0 2 X 0 g (23) X + (; ; x 0 ; ) = fx : X(; ; x 0 ju(); Q()) B (x)g; for some u() 2 U P with 0 gien and each set X + (; ; x 0 ; ) 6= ;. This leads to Problem (II ). Gien set X 0, and ector x 2 IR n, nd V + (; x; ) = min u max under conditions x( ) 2 X 0 ; u() 2 U O ; () 2 V O. Direct calculations here lead to the formula the same result as Problem II. min d(x(); B (x)); ; x( ) X + (; ; x 0 ; ) = fx : V + (; x; ) 0g; The equialence of Problems II; II means that denitions 1.2 and 2.1 both lead to the same set X + (; ; x 0 ; ). As we shall see, this is not so for the problem of reachability with corrections. A similar obseration holds for problems I; I. Remark 2.1. For the case that X 0 = fx 0 g is a singleton, one should recognize the following. The OLRS of the maxmin type is the set of points reachable at time from a gien point x 0 for any disturbance () 2 V O, proided function (t); t is communicated to the controller in adance, before the selection of control u(t). As mentioned aboe, the control u() is then selected through an anticipatie control procedure. On the other hand, for the construction of the the?reach set of the minmax type there is no information proided in adance on the realization of (), which becomes known only after the selection of u. Indeed, gien point x( ) = x 0, one has to select the control u(t) for the whole time interal t, whateer be the unknown (t) oer the same interal. The control u() is then selected through a nonanticipatie control procedure. Such a denition allows to specify an OLRS as consisting of points x each of which is complemented by a neighborhood B (x) so that X(; ; x 0 ju(); Q()) B (x) for a certain control u() 2 U O. This requires > 0 to be suciently large. As a rst step towards reachability under feedback, we consider piecewise open-loop controls with possibility of corrections at xed instants of time. 11

12 3 Piecewise open-loop controls: reachability with corrections Here we dene and calculate reachability sets under a nite number of corrections. This is done either through the solution of problems sequential maxmin and minmax or through operations on set-alued integrals. Taking a gien instant of time t 2 [ ; t 1 ] = T that diides the interal T in two, namely, T 1 = [ ; + ); T 2 = [ + ; t 1 ]; = t? ; consider the following sequential maxmin problem. Problem (I 1 ) Gien set X 0 ; x 2 IR n and numbers 1 0; 2 0, nd = max min u and then nd = max min u min x( +) 1 ( + ; x; 1 ) fd(x( ); X 0 )jx( + ) 2 B 1 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 1 g; 1 (; x; f 1; 2 g) (24) minf (t ; x( + ); 1 )jx() 2 B 2 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 2 g: x( ) The latter is a problem on nding a sequential maxmin with one \point of correction" t = t. Using the notation [1; 2] = f 1 ; 2 g, denote X? (; t 1 0; [1; 2]) = fx : 1 (; x; [1; 2]) 0g: Let us nd X? (; t 1 0; [1; 2]); 1 (; x; [1; 2]); using the technique of conex analysis. According to section 2, (see (11)), we hae = maxf(l; x( +))? 1 (l; l) 1 2?(ljX 0 )? where? 1 ( + ; ; 1 )) = Substituting this in (24), we hae = max min u min x( ) 1 ( + ; x( + ); 1 ) t0 + = d(x( + );? 1 ( + ; ; 1 )); t0 + X 0 + B 1 (0) + B(s)P(s)ds _? 1 ((ljb(s)p(s))?(?ljc(s)q(s))dsj(l; l) 1g t0 + (?C(s))Q(s))ds: (; x; [1; 2]) = maxf(l; x())? (l; u(s) + (s))ds? (lj? (t ; ; 1 ))j l + 12

13 j(l; l) 1; x() 2 B 2 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 2 g : Continuing the calculation, we come to where 1 (; x; [1; 2]) = (25) = maxf(l; x)? 2 (l; l) 1 2? ((ljb(s)p(s))ds + ((?ljc(s)q(s)))ds? l + + h 1 (l) = (ljx 0 ) + 1 (l; l) 1 2 +?(con h 1 )(l)g; t0 + So (con h 1 )(l) is the support function of the set t0 + X 0 + B 1 (0) + B(s)P(s)ds ((ljb(s)p(s))ds? (?ljc(s)q(s)))ds: _? t0 + (?C(s)Q(s))ds: Together with (25) this allows us as in Section 2, to express 1 (; x; [1; 2]) as where X 0 + B 1 (0) + 1 (; x; [1; 2]) = d(x; X? 1 (; ; x 0 ; [1; 2])); +B 2 (0) + X? 1 (; ; x 0 ; [1; 2]) = (26) t0 + + B(s)P(s)ds B(s)P(s)ds _? _? + t0 + (?C(s))Q(s)ds (?C(s))Q(s))ds: Formula (26) shows that X? (; t 1 0; x 0 ; [1; 2]) (dened as the leel set of 1 ) is also the reach set with one correction. In particular, X? (; t 1 0; x 0 ; 0) consists of all states x that may be reached for any function () 2 V P, whose alues are communicated in two stages, through two consecutie selections of some open-loop control u(t) according to the following scheme. Stage (1): gien at time are the initial state x 0 and function (t) for t 2 T 1, select at time the control u(t) for t 2 T 1. Then at instant of correction t = + comes additional information for stage (2). Stage (2): gien at time t are the state x(t ) and function (t) for t 2 T 2, select at time t = t the control u(t) for t 2 T 2. This proes Theorem

14 Theorem 3.1 The set X? (; t 1 0; x 0 ; [1; 2]) = fx : 1 (; x; [1; 2]) 0g; is the maxmin OLRS with one correction at instant + and is gien by formula (26). We refer to X? 1 (; ; x 0 ; [1; 2]) as the maxmin OLRS with one correction at instant 1 = +. The two-stage scheme may be further propagated to the class of piecewise open-loop controls with k corrections. Taking the interal, T = [ ; ], introduce a partition k = f = 0 ; 1 ; : : :; k ; = k+1 g; i? i?1 = i 0; i = 1; : : :; k + 1; so that the interal T is now diided into k + 1 parts T 1 = [ ; 1 ); T 2 = [ 1 ; 2 ); : : :; T k+1 = [t 1? k ; t 1 ]; where are the points of correction. i = + ix j=1 j ; i = 1; : : :; k; Consider also a nondecreasing continuous function (t) 0; ( ) = 0, denoting 1 = ( 1 )? ( 0 ); i = ( i )? ( i?1 ); i = 1; :::; k + 1; = ( k+1 )? ( 0 ); and also [1; i] = f 1 ; :::; i g; k i > 1: Problem (I k ) Sole the following consecutie optimization problems. Find then nd = max = max min u min u k ( 1; x; 1 ) = minfd(x( ); X 0 )jx( 1 ) 2 B 1 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 1 g; x( 1 ) k ( 2; x; [1; 2]) = minf k ( 1; x( + 1 ); 1 )jx( 2 ) 2 B 2 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 2 g; x( 2 ) then consecutiely, for i = 3; : : :; k, nd k ( i; x; [1; i]) = 14

15 max min minf u k ( i?1; x( i?1 ); [1; i? 1])jx( i ) 2 B i (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T i g; and nally max min u x( i ) k (; x; [1; k + 1]) = minf k ( k; x( k ); [1; : : :; k]jx() 2 B k+1 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T k+1 g: x( ) Direct calculation gies with then with = X k ( 1 ; ; X 0 ; 1 ) = k ( 1; x; 1 ) = d(x; X k ( 1 ; ; X 0 ; 1 )); (27) 1 X 0 + B 1 (0) + B(s)P(s)ds _? 1 (?C(s))Q(s)ds; k ( 2; x; [1; 2]) = d(x; X k ( 2 ; ; X 0 ; [1; 2])) = d(x; X? k ( 2; 1 ; X? k ( 1; ; X 0 ; 1 ); 2 ); 1 X 0 + B 1 (0) + B(s)P(s)ds then consecutiely X k ( 2 ; ; X 0 ; [1; 2]) = _? _? (?C(s)Q(s)ds (?C(s)Q(s)ds; 2 + B 2 (0) + B(s)P(s)ds 1 k ( i; x; [1; i]) = d(x; X? k ( i; ; X 0 ; [1; i]) = d(x; X? k ( i; i?1 ; X? k ( i?1; ; X 0 ; [1; i?1]); i )); with X? k ( 1 i; ; X 0 ; [1; i]) = ::: X 0 + B 1 (0) + B(s)P(s)ds _? i i +B i (0) + B(s)P(s)ds _? and nally where X? k (; ; x 0 ; [1; k+1]) = i?1 i?1 1 (?C(s))Q(s)ds (?C(s))Q(s))ds + ::: k (; x; [1; k + 1]) = d(x; X? k (; ; X 0 ; [1; k + 1])); (28) i +B i (0) + 1 : : : X 0 +B 1 (0)+ B(s)P(s)ds _? i?1 +B k+1 (0) + B(s)P(s)ds k B(s)P(s)ds i _? _? i?1 1 (?C(s))Q(s)ds (?C(s))Q(s))ds +: : : k (?C(s))Q(s)ds + : : : We refer to X? k (; ; [1; k + 1]) as the maxmin OLRS with k corrections at points i ; i = [1 : : :k]. 15 : (29)

16 Theorem 3.2 The set is gien by formula (29). X? k (; ; X 0 ; [1; k + 1]) = fx : k (; ; x; [1; k + 1]) 0g (30) We denote (; x; 0 1) = (; x; 1 ) and also introduce additional notations for the functions i (; x; [1; i + 1]). Denote i (; x; [1; i + 1]) = i (; x; [1; i + 1])j i ( ; ; 0)); emphasizing the dependence of i (; x; [1; i + 1]) on the initial condition i ( ; ; 0) We further assume V ( ; x; 0) = d(x; X 0 ) and also take d(x; X 0 ) = i ( ; x; 0), for all i. Note that the number of nodes j in any partition k is k + 2; as j = 0; :::; k + 1. The partition applied to a function V k is precisely k. Consequently, the increment = ()? ( ) = X k+1 j is presented as a sum of k + 1 increments j 0, once it is applied to a function V k with index k. j=1 A sequence of partitions k is monotone in k if for eery k 1 < k 2 partition k2 the nodes j of partition k1 : contains all Theorem 3.3 Gien are a monotone sequence of partitions k ; k = 1; 2; :::; N; ::: and a continuous nondecreasing function (t) 0; ( ) = 0; that generates for any partition k a sequence of numbers j = ( j )? ( j?1 ); j = 1; :::; k + 1. Gien also are a sequence of alue functions k ( i; ; [1; i]), each of which is formed by the partition k and a sequence j ; j = 1; :::; k + 1; (k is the index of k ). Then the following relations are true. (i) For any xed ; x, one has (; x; 0 1) ::: i (; x; [1; i + 1]) i+1 (; x; [1; i + 2]) ::: (31) ::: k (; x; [1; :::; k + 1]): (ii) For any xed ; x and index i 2 [1; k] one has proided j j ; j = 1; :::; i + 1. i (; x; [1; i + 1]) i (; x; [1; i + 1]); (32) (iii) The following inclusions are true for i 2 [1; k]. where the sets X? i are dened by (30). X? i?1 (; ; X 0 ; [1; i]) X? i (; ; X 0 ; [1; i + 1]); (33) 16

17 The proofs are based on the following properties of the geometrical (Minkowski) sums and dierences of sets P 1 ; P 2 ; P 3, (P 1 + P 2 ) _?P 3 P 1 + (P 2 _?P 3 ); P 1 _?(P 2 + P 3 ) = (P 1 _?P 2 ) _?P 3 ; and the fact that in general a max min does not exceed a min max. Direct calculations indicate that the following superpositions will also be true. Lemma 3.1 The functions k satisfy the following property k ( i; x; [1; i] j k (; ; 0)) = k ( i; x; [j + 1; i] j k ( j; ; [1; j] j k (; ; 0))); (34) proided k + 1 i j 1. This follows from Theorem 3.2 and the denitions of the respectie functions i. Remark 3.1 Formula (34) reects a semigroup property, but only for the selected points of correction i ; i = 1; :::; k: The reasoning aboe indicates, for example, that X? k (; ; x 0 ; 0) is the set of states that may be reached for any function () 2 V O, whose alues are communicated in adance in k stages, through k + 1 consecutie selections of some open-loop control u(t) according to the following scheme. Stage (1): gien at time are initial state x 0 and function (t) for t 2 T 1, { select at time control u(t), for t 2 T 1. Then at instant of correction j comes additional information for stage (j + 1). Stage (j), (j = 2; :::; k): gien at time j are state x( j ) and function (t) for t 2 T j+1, { select at time control u(t) for t 2 T j+1. Remark 3.2. There is a case when all the functions k (; x; 0) taken for all the integers k 0 coincide. This is when system (10) satises the so-called matching conditions: B(t)P(t) (t)c(t)q(t); (t) 2 [0; 1); t 2 [ ; ]: We now pass to the problem of sequential minmax, with one correction at instant + = t, using the notations for Problem (I 1 ). This is Problem (II 1 ). Gien set X 0, ector x 2 IR n, and numbers 1 ; 2 0, nd = min u max min x( +) V + 1 ( + ; x; 1 ) fd(x( ); X 0 )jx( + ) 2 B 1 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 1 g; 17

18 then nd = min u max V + 1 (; x; [1; 2]) = (35) minf (t 1 0 +; x( +); [1; 2])jx() 2 B 2 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 2 g: x( ) The latter is a problem of nding a sequential minmax with one point of correction t = t. Denoting X + (; t 1 0; X 0 ; [1; 2]) = fx : V + 1 (; x; [1; 2]) 0g; let us nd X + (; t 1 0; X 0 ; [1; 2]); V + 1 aboe, with obious changes). (; x; [1; 2]) using the techniques of conex analysis (as This gies where + 1 (t + ; ; 0) = V + 1 ( + ; x; 0) = d(x; + 1 (t + ; ; 1 )); (X 0 + B 1 (0)) _? Continuing the calculations, we hae = min u where max t0 + V + 1 (; x; ) = t0 + (?C(s))Q(s)ds + B(s)P(s)ds: minfd(x(t+); + (t+; t 1 0; 0))jx() 2 B 2 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 2 g = x( ) + (t + ; t 1 0; 0) + B 2 (0) This proes Theorem 3.4 d(x; X + 1 (; ; X 0 ; [1; 2])); X + 1 (; ; X 0 ; [1; 2]) = (36) _? + (?C(s))Q(s)ds + + B(s)P(s)ds: Theorem 3.4 The set X + (; t 1 0; X 0 ; [1; 2]) = fx : V + 1 (; x; [1; 2]) 0g; is the minmax OLRS with one correction at instant t = +, gien by formula (36). Here the problem is again soled in two stages, according to the following scheme. Stage (1): gien at time are set X 0 and x 2 IR n. Select control u(t) (one and the same for all ) and for each (t); t 2 T 1, assign a ector x( ) 2 X 0 that jointly with u; produces x() 2 B 1 (0). 18

19 Then at instant of correction t = + comes additional information for stage (2). Stage (2): gien at time t are x(t ) and ector x 2 IR n. Select control u(t); t 2 T 2 (one and the same for all ) and for each (t); t 2 T 2 assign a ector x(t + ) 2 + ( + ; ; 1 ) that jointly with u; steers the system to state x() 2 B 2 (x). We now propagate this minmax procedure to a sequential minmax problem in the class of piecewise open-loop controls with k corrections, using the notations of Problem (I k ). Problem (II k ). Sole the following consecutie optimization problems. Find = min u max V + k ( 1; x; 1 ) = minfd(x( ); X 0 )jx( 1 ) 2 B 1 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T 1 g; x( 1 ) then consecutiely, for i = 2; :::; k, nd min u max min x( i?1) k ( i; x; [1; i]) = fv + k ( i?1; x( i?1 ); [1; i?1])) jx( i ) 2 B i (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T i g: and nally V + k (; x; [1; k + 1]) = max minfv + u k ( k; x( k ); [1; k + 1]) jx() 2 B k+1 (x); u(t) 2 P(t); (t) 2 Q(t); t 2 T k+1 g: This time direct calculation gies where = ::: (X 0 + B 1 (0)) _? _? i i?1 V + k (; x; ) = d(x; X + k (; ; X 0 ; )); (37) 1 (?C(s))Q(s)ds _? X + k (; ; x 0 ; ) = (38) (?C(s))Q(s)ds + i k (?C(s))Q(s)ds i?1 + 1 B(s)P(s)ds + B 2 (0) B(s)P(s)ds + B i+1 (0) + k B(s)P(s)ds We refer to X + k (; ; x 0 ; )), as the maxmin OLRS with k corrections at points k. : _?::: _?::: Theorem 3.5 The set X k + (; ; x 0 ; [1; k + 1]) = fx : V k + (; x; [1; k + 1]) 0g is then the mimax OLRS with one corection and is gien by formula (38). 19

20 Denote V + (; x; ) = V (; x; ); V i (; x; [1; i + 1]) = V i + (; x; [1; i + 1]jV i + ( ; ; 0)); assuming V + 0 (; x; ) = d(x; X 0 ) and further, taking V + i ( ; x; 0) = d(x; X 0 ); for all i: Under the assumptions and notations of Theorem 3.3, the last results may be summarized in the following proposition. Theorem 3.6 (i) For any xed alues ; x one has V + 0 (; x; 1) ::: V + i (; x; [1; i + 1]) V + i+1 (ii) For any xed ; x and index i 2 [1; k] one has proided 1 2. (; x; [1; i + 2]) ::: V + (iii) The following inclusions are true for i 2 [1; :::; k]; 0: k (; x; [1; k + 1]): (39) V + i (; x; 1 ) V + i (; x; 2 ); (40) X + i?1 (; ; X 0 ; [i? 1]) X + i (; ; X 0 ; [1; i]): (41) (i) The following superpositions will also be true V + k ( i; x; [1; i]jv + k (; ; 0)) = V + k ( i; x; [j + 1; i]jv + k ( j; ; [1; j]jv + k (; ; 0))); proided k + 1 i j 1. In this section we hae considered problems with nite number of possible corrections and additional information coming at xed instants of time, haing presented a hierarchy of piecewise open-loop reach sets of the anticipatie (maxmin) or of the nonanticipatie type. These were presented as leel sets for alue functions which are superpositions of \onestage" alue functions calculated in Section 2. A semigroup-type property (34) for these alue functions was indicated which is true only for the points of correction (Remark 3.1). In the continuous case, howeer, we shall need this property to be true for any points. Then it would be possible to formulate the Principle of Optimality under uncertainty for our class of problems. We shall therefore inestigate some limit transitions with number of corrections tending to innity. This will allow a further possibility of continuous corrections of the control under unknown disturbances. 20

21 4 The alternated integrals and the alue functions We obsered aboe that the open-loop reach sets of both types (maxmin and minmax) are described as the leel sets of some alue functions, namely 4 X? k (; ; X 0 ; ()) = fx : k (; x; ()) 0g; X k + (; ; X 0 ; ()) = fx : V k + (; x; ()) 0g: We now propagate this approach, based on using alue functions, to systems with continuous measurements of the state to allow continuous corrections of the control. First note that inequality i (; x; ()) i (; x; 0) + is always true with equality attained, for example, under the following assumption. Assumption 4.1 There exists a scalar function (t) > 0 such that for all t 2 [ ; ]. B(t)P(t) _?(C(t)Q(t) + (t)b 1 (0)) 6= ;; In order to simplify the further explanations, we shall further deal in this section with the case = 0, omitting the last symbol 0 in the notations for ; V +. 5 Now note that Lemmas 3.1, 3.2 indicate that each of the functions? k (; x; 0 jv k (; ; 0)); V k + may be determined through a sequential procedure, (; x; 0 jv + k (; ; 0)) k (; x j k (; )) =? k (; x jv k (? k; j ::: j k ( + 1 ; j k (; ):::) (42) for + k and a similar one for V k. How could one express this procedure in terms of set-alued representations? For a gien partition k we hae (j i) fx : k ( + ix l=1 = X? k ( + l ; x j k ( + ix l=1 l ; + jx l=1 jx l ; )) 0g = l=1 l ; M? j ); where M? j = fx : k ( + P j -(29)), we may formulate a set-alued analogy of Lemma 3.1. l=1 l; ) 0g Then, in iew of the preious relations (see (27) 4 Here, without abuse of notation for ; X? ; V + ; X +, we shall use symbol () rather than the earlier k k k [1; k + 1], emphasizing the function (t); ()? () = used in the respectie constructions. 5 The case () 6= 0 would add to the length of the expressions, but not to the essence of the scheme. This case could be treated similarly, with obious complements. 21

22 Lemma 4.1 The following relations are true X? k (; ; X 0 ) = (43) X? k (;? k+1; X? k (? k+1;? k+1? k ; :::X? k ( + 2 ; + 1 ; X? k ( + 1 ; ; X 0 ):::): In terms of set-alued integrals (43) is precisely the equialent of (29). Moreoer, = max min u k ( + ix l=1 l ; xj k ( + ::: maxminfd(x(( + u jx l=1 jx l=1 l ; )) = (44) l ); M? j )jx( + ix l=1 l ) = x; u(t) 2 P(t); (t) 2 Q(t); t 2 T j ; :::; u(t) 2 P(t); (t) 2 Q(t); t 2 T i g: Similarly, for the sequential minmax, we hae V + k (; xjv + k (; )) = V + k (; xjv + k (? k+1; j:::jv + k ( + 1 ; jv + k (; ):::): (45) Using notations identical to (42),(43), but with minus changed to plus in the symbols for k ; X? k, we hae Lemma 4.2. Lemma 4.2 The following relations are true X + k (; ; X 0 ) = (46) X + k (;? k+1; X + k (? k+1;? k+1? k ; :::; X + k ( + 2 ; + 1 ; X + k ( + 1 ; ; X 0 ):::): In terms of set-alued integrals, formula (46) is precisely the equialent of (38), proided (t) 0. 6 Moreoer, min u max V + k ( + ::: min u ix l=1 maxfd(x( + l ; xjv + k ( + jx l=1 jx l=1 l ; )) = (47) l ); M + j jx( + ix l=1 l = x; u(t) 2 P(t); (t) 2 Q(t); t 2 T j ; ; :::; u(t) 2 P(t); (t) 2 Q(t); t 2 T i g; where M + j = fx : V + k ( + P j l=1 l; ) 0g. It is important to emphasize that until now all the relations were deried for a xed partition k = f = 0 ; 1 ; :::; k ; = k+1 g; i? i?1 = i ; i = 1; :::; k + 1: 6 Also note that under Assumption 4.1, with X 0 single-alued, one may treat the sets X + (; x; 0) as the k Hausdor limits X + (; x; 0) = k lim!+0 X + (; x; ). k 22

23 What would happen, howeer, if k increases to innity with k+1 X maxf i : i = 1; : : :; k + 1g! 0; k! 1; i =? (48) and would the result depend on the type of partition? Our further discussion will require an important nondegeneracy assumption. i=1 Assumption 4.2 There exist continuous ector functions 1 (t); 2 (t) 2 IR n ; t 2 [ ; t 1 ], and a number > 0 such that (a) 1 ( j ) + B(0) X? j ( j; ; X 0 ) (49) for all the sets X? j ( j; ; X 0 ) = X? j ( j; j?1 ; X? j ( j?1; j?2 ; :::; X? j ( 1; ; X 0 ):::) and for all the sets (b) 2 ( j ) + B(0) X + j ( j; ; X 0 ) (50) X + j ( j; ; X 0 ) = (51) X + j ( j; j?1 ; X + j ( j?1; j?2 ; :::; X + j ( 1; ; X 0 ):::) with j = 1; : : :; k + 1, whateer be the partition k. This last assumption is further taken to be true without further notice. 7 Obsering that (29), (38) hae the form of certain set-alued integral sums, (\the alternated sums"), we introduce the additional notation X? k (; ; X 0 ) = I? (; ; X 0 ; k ); X + k (; ; X 0 ) = I + (; ; X 0 ; k ): Let us now proceed with the limit operation. Take a monotone sequence of partitions k ; k! 1. Due to inclusions (33) and the boundedness of the sequence X? k (; ; X 0 ) from below by any of the sets X i + (; ; X 0 ); i k; the sequence I? (; ; X 0 ; k ) has a set-alued limit. Similarly, the inclusions (40) and the boundedness of the sequence X + k (; ; X 0 ; k ) from aboe ensure that it also has a set-alued limit. A more detailed inestigation of this scheme along the lines of [23] would indicate that under assumption 4.2 (a); (b) these set-alued limits do not depend on the type of partition k. This leads to Theorem If at some stage this assumption is not fullled, it may be applied to sets of type X? j (j; t0; X 0 ; ()); X + j (j; t0; X 0 ; ()) with () suciently large. 23

24 Theorem 4.1 There exist Hausdor limits I? (; ; X 0 ) = X? (; ; X 0 ); I + (; ; X 0 ) = X + (; ; X 0 ): lim h(i? (; ; X 0 ; k ); I? (; ; X 0 )) = 0; with lim h(i + (; ; X 0 ; k ); I + (; ; X 0 )) = 0; k+1 X maxf i : i = 1; : : :; k + 1g! 0; k! 1; These limits do not depend on the type of partition k. i=1 i =? : Moreroer, so that I? (; ; X 0 ) = I + (; ; X 0 ) = I(; ; X 0 ); (52) X? k (; ; X 0 ) = X + k (; ; X 0 ) = X(; ; X 0 ): We refer to I(; ; X 0 ) = X(; ; X 0 ) as the alternated reach set. 8 The proofs of the conergence of the alternated integral sums to their Hausdor limits and of the equalities (52) are not gien here. They follow the lines of those gien in detail in [14] for problems on sequential maxmimin and minmax considered in backward time (see also [23], [22], [13]). Let us now study the behaior of the function i (; xj i ( ; )) under condition (48). According to (38), (31) the sequence i (; x) is increasing in i with i! 1. This sequence is pointwise bounded in x by any solution of Problem (II k ) and therefore has a pointwise limit. Due to (29), Theorem 4.1, and the continuity of the distance function d(x; M) in x; M 2 conir n ; we hae, with k! 1, and therefore we may conclude that lim d(x; I? k ) = d(x; lim I? k ) = d(x; I? ) k (; x)! d(x; I? (; ; X 0 )) = (; x) under condition (48). This yields Theorem 4.2. Theorem 4.2 Under condition (48) there exists a pointwise limit lim k!1 k (; x) = V? (t; x) = d(x; X? (; ; X 0 )); (53) where X? (; ; X 0 ) = I? (; ; X 0 ). This limit does not depend on the type of partititon k. The alternated integral is the leel set of the function (; x), I? (; ; X 0 ) = fx : (; x) 0:g 8 A maxmin construction of the indicated type had been introduced in detail in [23], where it was constructed in backward time, becoming known as the alternated integral of Pontryagin. 24

25 Since (t; x) does not depend on the partition k and due to the properties of minmax we also come to the following conclusion. Theorem 4.3 The function (; x) satises the semigroup property: (; xj ( ; )) = (; xj (t; j ( ; ))); (54) for t 2 [ ; ]. The following inequality is true (t; x) fmax min u (t? ; x(t? ))jx(t) = xg; > 0: (55) Similarly, for the decreasing sequence of functions V k + (; x), we hae Theorem 4.4. Theorem 4.4 (i) Under condition (48) there exists a pointwise limit lim k!1 V + k (; x) = V + (t; x) = d(x; X + (; ; X 0 )); (56) where X + (; ; X 0 ) = I + (; ; X 0 ). This limit does not depend on the type of partititon k. (ii) The alternated integral is the leel set of the function V + (; x), I + (; ; X 0 ) = fx : V + (; x) 0:g (iii) The function V + (; x) satises the semigroup property: for t 2 [ ; ]. (i) The following inequality is true V + (; xjv + ( ; )) = V + (; xjv + (t; jv + ( ; ))); (57) V + (t; x) fmin u max V + (t? ; x(t? ))jx(t) = xg; > 0: (58) A consequence of (52) is the basic assertion, Theorem 4.5. Theorem 4.5 With the initial condition ( ; x) = V + ( ; x) = d(x; X 0 ) = V( ; x), the following equality is true V + (; xjv + ( ; )) = (; xj ( ; )) = V(; xjv( ; )) = d(x; X(; ; X 0 )): (59) The function V(; x) satises the semigroup property V(; xjv( ; )) = V(; xjv(t; jv( ; )): (60) 25

26 The last relation follows from (59), (54), (57). Thus, under the nondegeneracy Assumption 4.2 the two forward alternated integrals I + ; I? coincide and so do the alue functions ; V +. Relations (55), (58), (59) allow us to construct a partial dierential equation for the function V(t; x) the so-called HJBI (Hamilton-Jacobi-Bellman-Isaacs) equation. We now inestigate the existence of the total deriatie dv(t; x)=dt along the trajectories of system (10). Due to (59), (13), we hae V(t; x) = d(x; X(t; ; X 0 )) = maxf(l; x)? (ljx(t; ; X 0 ))j(l; l) 1g: Obsering that for d(x; X(t; ; X 0 )) > 0 the maximizer l 0 (t; x) of (61) is unique and taking l 0 (t; x) = 0 if d(x; X(t; ; X 0 )) = 0, we may apply the rules for dierentiating a \maximum"-type function [6], to get dv(t; x)=dt + (@V=@x; _x) = (l 0 ; 0 jx(t; ; X 0 ))=@t; Direct calculations indicate that the respectie partials exist and are continuous in the domain D[intD 0, where D = fx : d(x; X(t; ; X 0 )) > 0g; D 0 = fx : d(x; X(t; ; X 0 )) = 0g, and intd 0 stands for the interior of the respectie set. To nd the alue of the total deriatie take inequalities (58), (55), which may be rewritten as 0 min maxfv + (t u? ; x(t? ))? V + (t; x)jx(t) = xg (61) and 0 max minf (t u? ; x(t? ))? (t; x)jx(t) = xg: (62) Diiding both relations by > 0 and passing to the limit with! 0, we get max u min dv + (t; x)=dt 0; min max u dv? (t; x)=dt 0: (63) Since in Theorem 4.5 we had V + (t; x) = (t; x) = V(t; x); for the linear system (10) we hae max min dv(t; x)=dt = min max dv(t; x)=dt; u u 2 P(t); 2 Q(t); u which results in the next proposition. Theorem 4.6 In the domain D [ intd 0 the alue function V(t; x) satises the \forward" + max u oer u 2 P(t); 2 Q(t) with boundary condition min (@V=@x; B(t)u + C(t)) = 0 (64) V( ; x) = d(x; X 0 ): (65) 26

27 Equation (63) may be rewritten as + (@V=@xj(?C(t))Q(t)) = 0: (66) The last theorem indicates that the HJBI equation (63) is satised eerywhere in the open domain D [ intd 0. Howeer, the continuity of on the boundary of the domains D; D 0 was not inestigated and in fact may not hold. But it is not dicult to check that with boundary condition (65) the function V(t; x) will be a minmax solution to equation (66) in the sense of [26], which is equialent to the statement that V(t; x) is a iscosity solution ([3], [20]) to (66), (67). This particularly follows from the fact that function V(t; x) is conex, being a pointwise limit of conex functions k (t; x) ([8]). Let us note here that the problem under discussion may be treated not only as aboe but also within the notion of classical solutions to equation (66), (65). Indeed, although all the results aboe were proed for the criterion d(x( ; X 0 )) in the respectie problems, the following assertion is also true. Assertion 4.1 Theorems , , are all true with the criterion d(x( ; X 0 )) in the respectie problems substituted by d 2 (x( ; X 0 )). This assertion follows from direct calculations, as in paper [13], with formula (11) substituted by d 2 (x; G) = maxf(l; x)? (ljg)? (1=4)(l; l) 1=2 g: The respectie alue function similar to V(t; x), denoted further as V 1 (t; x), will now be a solution to (66) with boundary condition V 1 ( ; x) = d 2 (x; X 0 ): (67) Moreoer,V 1 (t; x), together with its rst partials, turns out to be continuous in t; x 2 D[D 0. Thus we come to Theorem 4.7 The function V 1 (t; x) a classical solution to (66), (67) satises the relations fx : V 1 (t; x) 0g = X(t; ; X 0 ) = I(t; ; X 0 ): (68) We hae constructed the set X(t; ; X 0 ) as the limit of OLRS and the leel set of function V(t; x), (or function V 1 (t; x)) the sequential maxmin or minmax of function d(t; X 0 ) (or function d 2 (t; X 0 )) under restriction x(t) = x. It remains to show that X(t; ; X 0 ) is precisely the set of points that may be reached from X 0 with a certain feedback control strategy U(t; x), whateer be the function (t). Prior to the next section, we wish to note the following. Function V(t; x) = V(t; xjv( ; )) may be interpreted as the alue function for the following Problem (IV): nd the alue function V(; x) = min U maxfd(x( ); X 0 )j x() = x); U 2 U C ; x() 2 X U ()g; x() 27

28 where U = U(t; x) 2 U C is a CLC (see Section 1) and X U () is the set of all solutions to the dierential inclusion _x 2 B(t)U(t; x) + C(t)Q(t); x() = x; (69) generated by x() = x; U(t; x); and taken within the interal t 2 [ ; ]. Its leel set X () = fx : V(; x) g is precisely the closed-loop reach set. It is the set of such points x 2 IR n that there exists a strategy U 2 U C which for any solution x(t) of (69), x() = x; t 2 [ ; ], ensures the inequality d(x( ); X 0 ). Due to the structure of (69), (A(t) 0), this is equialent to the following denition of closed-loop reachability sets. Denition 4.1 A closed-loop reachability set X () is the set of such points x 2 IR n for each of which there exists a strategy U 2 U C that for eery () 2 V O assigns a point x 0 2 X 0, such that eery solution x[t] of the dierential inclusion saties the inequality d(x[]; x). _x 2 B(t)U(t; x) + C(t)(t); x( ) = x 0 ; t ; Once the Principle of Optimality (60) is true, it may also be used directly to derie equation (64) - the HJBI equation for the function V(t; x). Therefore, set X () (if nonempty), will be nothing else than the set X(; ; X 0 ) dened earlier as the limit of open-loop reach sets. 5 Closed loop reachability under uncertainty We shall now show that each point of X(t; ; X 0 ) may be reached from X 0 with a certain feedback control strategy U(t; x), whateer be the function (t). In order to do this, we shall need the notion of solability set, (or, in other terms, \the backward reachability set", see [11], [27], [15]) { a set similar to X(t; ; X 0 ), but constructed in backward time. We rst recall from [13] some properties of these sets. Consider Problem (V) : nd the alue function V (t; x) = min U maxfd 2 (x[t 1 ]; M)jU 2 U C ; x() 2 X U g; (70) x() where M is a gien conex compact set (M 2 conir n ) and X U is the ariety of all trajectories x() of the dierential inclusion (69), x() = x; t 2 [ ; t 1 ], generated by a gien strategy U 2 U C. 28

29 The formal HJBI equation for the alue V (t; + min ; B(t)u + C(t) = 0; u 2 P(t); 2 Q(t); (71) with boundary condition Equation (71) may be @V V (t 1 ; x) = d 2 (x; M): jc(t)q(t) = 0: (73) An important feature is that function V (t; x) may be interpreted as a sequential max min similar to the one in section 3. Namely, taking the interal t t 1, introduce a partition k = f = 0 ; 1 ; :::; k ; k+1 = t 1 g, h 1 = k+1? k ; :::; h i+1 = k+i+1? k+i ; :::; h k+1 = 1? 0 ; similar to that of Section 3. For the gien partition, consider the recurrence relations fmax k (t 1? h 1 ; x) = min d 2 (x(t 1 ); u M)j t 1? h 1 t t 1 ; x(t 1? h 1 ) = xg; k (t 1? h 1? h 2 ; x) = = fmax min u k (t 1? h 1 ; x(t 1? h 1 )j t 1? h 1? h 2 t t 1? h 1 ; x(t 1? h 1? h 2 ) = xg k (; x) = fmax min u k ( + h k+1; x( + h k+1 ))j t + h k+1 ; x() = xg; where (t) 2 Q(t); u(t) 2 P(t) almost eerywhere in the respectie interals. Lemma 5.1 ([13]) With there exists a pointwise limit maxfh i : i = 1; : : :; k + 1g! 0; k! 1; k+1 i=1 h i = t 1? ; (74) (; x) = lim k!1 k that does not depend upon the type of partition k. The function (; x) coincides with V (; x): (; x); We shall refer to V? (; x) = V (; x) as the sequential maxmin. This function enjoys properties similar to those of its \forward time" counterpart, the function (; x) of section 3. A similar construction is possible for a \backward" ersion of the sequential minmax. The leel set W(t; t 1 ; M) = fx : V (t; x) 0g 29

30 is referred to as the closed loop solability set CLSS at time = t, from set M. It may be presented as an alternated integral of Pontryagin, { the Hausdor limit of the sequence = : : : M + t 1 t 1? 1 (?B(t)P()d I (t; t 1 ; M; k ) = (75) _? t 1 t 1? 1 (?C(t))Q()d t+ k ; : : :; _? t (?C(t))Q()d ; under conditions (74). Also presumed is a nondegeneracy assumption similar to Assumption 4.2. Assumption 5.1 For a gien set M 2 conir n there exists a continuous function 3 (t) 2 IR n and a number > 0, such that for any i = 1; :::; k + 1, whateer be the partition k. 3 ( i ) + B(0) I ( i ; t 1 ; M; k ); (76) This assumption is presumed in the next lemma. Lemma 5.2 Under condition (76) there exists a Hausdor limit I (t; t 1 ; M): lim h I (t; t 1 ; M; k ) ; I (t; t 1 ; M) = 0: This limit does not depend on the type of partition k and coincides with the CLSS, W(t; t 1 ; M): I (t; t 1 ; M) = W(t; t 1 ; M): (77) From the theory of control under uncertainty and dierential games it is known that if x 2 W(t; t 1 ; M), there exists a feedback strategy U(t; x) 2 U C that steers system (10) from state x(t) = x to set M whateer be the unknown disturbance () ([11], [29], [15]). Therefore, assuming X(t 1 ; t; x ) 6= ; under our assumptions, we just hae to proe the inclusion x 2 W(t; t 1 ; X(t 1 ; t; x )) or, in iew the properties of V(t; x); V (t; x), that x 2 fx : V (t; xjt 1 ; V(t 1 ; x)) 0g: Here V (t; xjt 1 ; V(t 1 ; x)) = V (t; x) is the solution to equation (71) with boundary condition (Recall that V(t 1 ; x) = d 2 (x; X(t 1 ; t; X )). V (t 1 ; x) = V(t 1 ; x): (78) 30