On averaged expected cost control as reliability for 1D ergodic diffusions

Transcription

1 On averaged expected cost control as reliability for 1D ergodic diffsions S.V. Anlova 5 6, H. Mai 7 8, A.Y. Veretennikov 9 10 Mon Nov 27 10:24: Abstract For a Markov model described by a one-dimensional diffsion with ergodic control withot discont on the infinite horizon an ergodic Bellman eqation is proved for the optimal readiness coefficient; convergence of the iteration improvement algorithm is established. 1 Introdction According to textbooks in reliability see, e.g., [7], [19] coefficient of readiness is one of the main characteristics of reliability of the system. In this paper the model nder consideration is presented by an ergodic Markov process described as a one-dimensional diffsion process which is controlled so as to spend more time in a good domain on average on the infinite horizon of time. The crrent readiness of the system is measred by a non-negative fnction f taking vales on the interval [0,1]: one signifies a fll readiness, while zero means that the model is in the break down state. Hence, in particlar, we do not jst split the real line into two parts where f = 1 or f = 0 bt allow a soft transition from fll readiness (f = 1) to a complete failre of the model (f = 0). Both coefficients of the diffsion as well as the fnction f itself may depend on the control. We allow only feedback (Markov) control strategies with vales from some compact set. The main reslt states an ergodic Bellman eqation on the optimal readiness characteristic ρ along with some axiliary fnction; this ρ may be regarded as the most favorable readiness averaged simltaneosly in space and time. Also we state an algorithm of improvement of control which in principle provides a tool to solve the Bellman eqation approximately. Earlier reslts on ergodic control in continos time were obtained in [13], [15], [3], et al. The latest works inclde [1], [2], [18], see also the references therein. In the very first papers and books compact cases with some axiliary bondary conditions so as to simplify ergodicity were stdied; convergence of the improvement control algorithms were stdied only partially. In the 5 Institte for Control Sciences, Moscow, Rssia; ip.r 6 For the first athor this research has been spported by the Rssian Fondation for Basic Research grant no CREST and ENSAE ParisTech, France; gmail.com 8 The second athor thanks the Institt Lois Bachelier for financial spport. 9 University of Leeds, UK, & National Research University Higher School of Economics, & Institte for Information Transmission Problems, Moscow, Rssia; leeds.ac.k 10 The third athor is gratefl to the financial spport by the DFG throgh the CRC 1283 Taming ncertainty and profiting from randomness and low reglarity in analysis, stochastics and their applications at Bielefeld University dring his stay there in Agst 2017; also, for this athor this stdy has been fnded by the Rssian Academic Excellence Project and by the Rssian Fondation for Basic Research grant no _a. All the athors grateflly acknowledge the spport and hospitality of the Oberwolfach Research Institte for Mathematics (MFO) dring the RiP programme in Jne 2014 where this stdy was initiated. 31

2 later investigations noncompact spaces are allowed; however, apparently, ergodic control in the diffsion coefficient σ of the process was not tackled earlier. Abot controlled diffsion processes on a finite horizon, or, on infinite horizon with discont (also known as killing) the reader may conslt in [3], [10]. Discrete time and space theory was developed simltaneosly in the monographs [5], [6, 8], [14], [17] and some others; important jornal references can be fond therein. Technical difficlties related to control in the diffsion coefficients are not an isse in discrete models. Combination of discrete state spaces and continos time can be fond in [18], et al. Reliability was not an isse in most of the cited works; however, it may be introdced in any Markov model. The paper consists of five sections not conting two lines of the Conclsions: 1 Introdction, 2 Setting, 3 Assmptions and Axiliaries, 4 Main reslt and 5 Sketch of the Proof. 2 Setting Given a standard probability space (Ω, F, (F t ), P) and a one-dimensional (F t ) Wiener process B = (B t ) t 0 on it we consider a one-dimensional SDE with coefficients b, σ and a control parameter α described as follows: dx α t = b(α(x α t ), X α t ) dt + σ(α(x α t ), X α t ) dw t, t 0, (1) X α 0 = x R. Its (weak) soltion does exist [11] and nder or conditions 1D, bondedness of all coefficients and niform non-degeneracy (or ellipticity) of σ 2 is weakly niqe. Let a non-empty compact set U R be a range of possible control vales. Withot any frther reminder U being compact is always bonded. Let b: U R R, σ: U R R, α: R U be given Borel fnctions (some more reglarity assmptions will be presented later). Denote the (extended) generator, which corresponds to the eqation (??) with a fixed fnction α( ) by L α : L α (x) = b(α(x), x) + 1 x 2 σ2 (α(x), x) 2 x2, x R. Given a rnning cost fnction f: U R R from a sitable fnction class we aim to choose an optimal (in some relaxed setting, at least, nearly-optimal ) control strategy α: R U (Markov homogeneos, or, in another langage, Markov feedback strategy) sch that the corresponding soltion X α maximizes the averaged cost fnction ρ α (x): = liminf T 1 T T 0 E xf(α(x t α ), X t α ) dt. (2) Recall that the fnction f takes vales 0 f 1, (3) then this rnning cost may be regarded as a measre of crrent readiness of the nderlying device. Namely, any vale between zero and one we can treat as a measre of availability, while the limit ρ α if it exists, can be nderstood as an averaged with respect to time and ensemble availability (=readiness) of the system. This is especially natral for the set of possible vales {0; 1} for sch a fnction; however, the whole interval of vales [0,1] also makes an evident sense in the context of reliability theory. In the seqel we assme that the assmption (3) is satisfied. By K we denote the class of strategies α: R U which are Borel measrable. For convenience for every α K we define the fnction f α : R R, f α (x) = f(α(x), x), x R. Now, instead of(2) we can se the eqivalent form, ρ α 1 (x) = liminf T T T E 0 xf α (X α t ) dt. The maximin cost fnction or, in other terms, the ergodic availability or readiness coefficient of the system is defined by the expression 1 ρ(x): = spliminf α K T T T E 0 xf α (X α t ) dt. (4) Sppose that for every α K the soltion of the eqation (??) X α is an ergodic process, that is, 32

3 there exists a niqe limiting distribtion μ α of X α t, t, the same for all initial conditions X 0 = x R. Then it is tre that for every x R, ρ α (x) ρ α : = f α (x ) μ α (dx ) =: f α, μ α, (5) and ρ(x) ρ: = sp α K f α (x ) μ α (dx ) = sp f α, μ α. (6) α K Note that nder or assmptions ρ does not depend on x. Ergodicity reqires special conditions on the characteristics b, σ, α; they will be later specified in the next section. We also define an axiliary fnction which depends on x and which looks like a cost fnction bt it is not, v α (x): = 0 E x(f α (X t α ) ρ α ) dt, α K. This integral will converge nder the recrrence assmptions below. Soltions of the eqation (??) will be nderstood as weak ones. Correspondlingly, the ergodic Bellman eqation (7) below will be established for weak soltions. The first goal of the paper is to prove that the cost ρ which is a constant in the ergodic setting is the component of the pair (V, ρ), which is a niqe soltion of the ergodic HJB or Bellman s eqation, sp[l V(x) + f (x) ρ] = 0, x R, (7) U where V will be niqe p to an additive constant, while ρ will be niqe in the standard sense. The meaning of the fnction V is that it coincides with v α for the optimal strategy α if the latter exists, and this fnction is the main tool for finding an optimal strategy. Note that de to the nidimensional setting and the non-degeneracy of σ 2 which will be assmed, the eqation (7) is eqivalent to the folowing, spσ 2 (, x) [ 1 b(,x) V (x) + V (x) + f (x) ρ ] = 0, x R. (8) U 2 σ 2 (,x) σ 2 (,x) σ 2 (,x) Frther, de to the non-degeneracy of σ 2 and in particlar becase the right hand sides in (7) and (8) are eqal to zero, we conclde that they are both eqivalent to sp [ 1 b(,x) V (x) + V (x) + f (x) ρ ] = 0, x R. (9) U 2 σ 2 (,x) σ 2 (,x) σ 2 (,x) The second goal is to show that the RIA algorithm ( reward improvement algorithm, or, in some papers, PIA for policy improvement algorithm ) provides a seqence of convergent approximate costs, ρ n ρ, n. Also let s emphasize that nlike in the finite horizon case, here in the average ergodic control setting, the soltion of the HJB eqation is a cople (V, ρ), where ρ is the desired cost while V is some axiliary fnction, which also admits a certain interpretation in terms of control theory. Note that soltions of the eqations (7), (8) and (9) will be stdied in Sobolev classes, hence, (second) derivatives will be defined p to almost everywhere with respect to Lebesge s measre. To keep all strategies Borel, all expressions involving Sobolev derivatives will be derstood as Borel measrable expressions since for any Lebesge s fnction there is a Borel fnction which coincides with the former almost everywhere. Respectively, all HJB or Poisson eqations will be nderstood in the Sobolev sense with Borel versions of any second order Sobolev derivative. First order derivatives are all continos de to Sobolev imbedding theorems. 3 Assmptions and axiliaries To ensre ergodicity of X α nder any feedback control strategy α K, we make the following assmptions on the drift and diffsion coefficients. 1. The fnction b is bonded, C 1 in x, and lim x b(, x) =. (10) sp x U 33

4 2. The fnction σ is bonded, niformly non-degenerate and C 1 in x. 3. The fnction f takes vales in the interval [0,1]. 4. The fnctions σ(, x), b(, x), f(, x) are continos in (, x). 5. The set U R is compact. Lemma 1 Let the assmptions (A1) (A4) be satisfied. Then the fnction v α has the following properties: 1. For any strategy α the fnction v α is continos as well as (v α ), and there exist C, m > 0 sch that sp α ( v α (x) + v α (x) ) C(1 + x m ). 2. v α 2 W p,loc for any p v α C 1,Lip (i.e., (v α ) is locally Lipschitz). 4. v α satisfies a Poisson eqation in the whole space, in the Sobolev sense. L α v α (x) + f α (x) < f α, μ α > = a.e. 0, (11) 5. Soltion of this eqation is niqe p to an additive constant in the class of Sobolev 2 soltions W p,loc with a no more than some (any) polynomial growth. 6. < v α, μ α >= 0. Proof. follows from [21] & [16]; see also [9, Lemma 4.13 and Remark 4.3]. Lemma 2 Let the assmptions (A1) (A3) hold tre. Then, For any C 1, m 1 > 0 there exist C, m > 0 sch that for any strategy α K and for any fnction g growing no faster than C 1 (1 + x m 1), sp E x g(x α t ) C(1 + x m ). (12) t For any strategy α K the fnction ρ α is a constant, and there exists C < sch that sp ρ α C <. (13) α For any α K, the invariant measre μ α integrates any polynomial: x m μ α (dx) <. Proof follows from [21] and [16]. 34

5 4 Main reslt Recall that the state space dimension is D = 1 and that all SDE soltions with any Markov strategy are weak, niqe in distribtion, strong Markov and ergodic. All of these follow from [11] and from the assmptions (A1) and (A2) (see [21] abot ergodicity). The exact RIA reads as follows. Let s start with some homogeneos Markov strategy α 0, which niqely determines ρ 0 = ρ α 0 f α 0, μ α 0 and v 0 = v α 0. Next, for any cople (v, ρ) sch that v C 2 2, or v W p,loc with any p > 0, and for ρ R, define F[v, ρ](x): = sp[l v(x) + f (x) ρ] = max U U [L v(x) + f (x) ρ]. Recall that nless v C 2, we consider a Borel version of the expression in the right hand side. Now, by indction given α n, ρ n and v n, the next improved strategy α n+1 is defined as follows: for any x, L α n+1v n (x) + f α n+1(x) ρ n = F[v n, ρ n ](x). (14) which is eqivalent to L α n+1v n (x) + f α n+1(x) = max [L v n (x) + f (x)] =: G[v n ](x). In the seqel we assme that a Borel measrable version of sch a strategy can be chosen. In or case existence of sch a Borel strategy follows from Stschegolkow s (Shchegolkov s) theorem, see [20], [12, Satz 39], [4, Theorem 1] (the first two references are in German, the last one cites the same reslt in English), which states that if any section of a (nonempty) Borel set E in the direct prodct of two complete separable metric spaces is sigma-compact (i.e., eqals a contable sm of closed sets) then a Borel selection belonging to this set E exists. Now, the vale ρ n+1 is defined as ρ n+1 : = f α n+1, μ α n+1, where, in trn, μ α n+1 is the (niqe) invariant measre, which corresponds to the strategy α n+1. Recall that v n (x) = E 0 x(f α α n(x n t ) ρ n ) dt. Theorem 1 Let the assmptions (A1) (A5) be satisfied. Then the Bellman eqation (7) holds tre for ρ and some axiliary fnction V C 2, soltion of this eqation is niqe for ρ, and for any n, ρ n+1 ρ n, the seqence ρ n is bonded, and there is a limit ρ n ρ, n. 5 Sketch of the Proof Let s show the sketch of the main steps of the proof. noindent 1. From (14) and (11) it may be derived that (L α n+1v n L α n+1v n+1 )(x) a.e. ρ n ρ n+1. Frther, from Dynkin s formla applied to (v n v n+1 )(X t α n+1 ) we obtain, E x v n (X t α n+1 ) E x v n+1 (X t α n+1 ) v n (x) + v n+1 (x) (ρ n ρ n+1 ) t. Since the left hand side here is bonded for a fixed x, after division of all terms by t and at t, we obtain, 0 ρ n ρ n+1, as reqired. Therefore, ρ n ρ n+1, so that ρ n ρ with some ρ. Ths, the RIA does converge, althogh so far we do not know whether ρ = ρ. Clearly, ρ ρ, since ρ is the sp over all Markov strategies, while ρ is the sp over some its contable sbset. Recall that now we want to show that v n v sch that the cople (v, ρ ) satisfies the HJB eqation (7), and that ρ as well as v in some sense here is niqe. 2. What we want to do is to pass to the limit in the eqation 35

6 L α n+1v n+1 (x) + f α n+1(x) ρ n+1 = 0, as n, after having showed compactness of the set (v n ) at least in C 1 (and later on, in C 1,β for any 0 < β < 1). Since ρ n = L α nv n (x) + f α n(x), we obtain after division by σ 2 /2, v n (x) = 2ρ n 2f (x) (x) 2v n σ 2 σ 2 σ2 (x). (15) De to the local bondedness and absolte continity of v n see the Lemma 1 we conclde that the seqence (v n ) is locally (i.e. on any bonded interval) tight in C 1. Hence, there is a sbseqence n sch that v n converges in C 1 on any bonded interval to some fnction v C 1 (in fact, even v C 1,Lip = {g C 1 (R): g Lip } as will be clear in a few lines and follows, e.g., from (15)). Denoting where F 1 [x, v, ρ]: = max [b v + f ρ ](x) max [b v + f ρ a] (x), a (x) = 1 2 (σ (x)) 2, b (x) = b (x)/a (x), f (x) = f (x)/a (x), ρ (x) = ρ/a (x), and by sing the bonds as in [10, Chapter 1], it can be shown the limiting eqation as n v (x) v x (r) + F r 1[s, v (s), ρ ] ds = 0, (16) which implies by differentiation that v (x) + F1 [x, v, ρ ](x) = 0. (17) This eqation is eqivalent to (9) and, hence, to (7), as reqired. In other words, the limiting pair (v, ρ ) satisfies the HJB eqation (7). 3. Uniqeness for ρ. Sppose there are two soltions of the (HJB) eqation, v 1, ρ 1 and v 2, ρ 2 with a polynomial growth for v i. Denote v(x): = v 1 (x) v 2 (x) and consider two Borel strategies α 1 (x) argmax (L v(x)) and α 2 (x) argmin (L i v(x)), and denote by X t a (weak) soltion of the SDE corresponding to each strategy α i. (It exists and is weakly niqe.) Note that h 2 (x): = max (L v(x) ρ 1 + ρ 2 ) = max (L v 1 (x) + f (x) ρ 1 L v 1 (x) f (x) + ρ 2 ) and similarly, max (L v 1 (x) + f (x) ρ 1 ) max (L v 2 (x) + f (x) ρ 2 ) a.e. = 0, h 1 (x): = min(l v(x) ρ 1 + ρ 2 ) = max (L ( v)(x) ρ 2 + ρ 1 ) [max (L v 2 (x) + f (x) ρ 2 ) max (L v 1 (x) + f (x) ρ 1 )] a.e. = 0. We have, L α 2v(x) = h 2 (x) ρ 2 + ρ 1, and L α 1v(x) = h 1 (x) ρ 2 + ρ 1. Frther, Dynkin s formla is applicable. So, E x v(x t 1 ) v(x) = E x t 0 Lα 1v(X s 1 ) ds = E x t h 0 1(X 1 s ) ds + (ρ 1 ρ 2 (h 1 0) ) t (ρ 1 ρ 2 ) t. The last ineqality here is de to the h a.e. 1 0 along with Krylov s bonds [10]. Here the left hand side is bonded (x fixed) de to the Lemma 2, so, we obtain, ρ 1 ρ 2 0. Absoltely similarly we show that also ρ 1 ρ 2 (h 2 0) 0. Ths, eventally, ρ 1 = ρ Proof of the eqality ρ = ρ. We have seen that for any initial (α 0, ρ 0 ), the seqence ρ n 36

7 converges monotonically to ρ, which is a component of soltion of the Bellman eqation (7), as shown earlier in the step 2, and this component ρ is niqe as was jst shown in the step 3. Hence, given some (any) ε > 0, take any initial strategy α 0 sch that ρ 0 = ρ α 0 > ρ ε. Then, clearly, the corresponding limit ρ will satisfy the same ineqality, ρ = limρ n > ρ + ε. n De to niqeness of ρ as a component of soltion of the eqation (7), and since ε > 0 is arbitrary, and becase it is already established that ρ ρ, we now conclde that ρ = ρ. The sketch of the proof of the Theorem 1 is ths completed. 6 Discssion Ths, we have an approach which in principle allows to evalate the ergodic readiness coefficient in certain diffsion Markov models. 7 Addendm: Borel measrability In the presentation of RIA we have assmed existence of a Borel measrable version of sch a strategy to be chosen which maximizes some fnction ofr a fixed x. In or case existence of sch a Borel strategy follows from Stschegolkow s (Shchegolkov s) theorem, see [20], [12, Satz 39], [4, Theorem 1] (the first two references are in German, the last one cites the same reslt in English), which states that if any section of a (nonempty) Borel set E in the direct prodct of two complete separable metric spaces is sigma-compact (i.e., eqals a contable sm of closed sets) then a Borel selection belonging to this set E exists. In or case E = {(, x): F[, x] = φ(x): = max v U F[v, x], x R}. This set is nonempty and closed and, hence, Borel. Indeed, if E ( n, x n ) (, x), n, then F[ n, x n ] F[, x] de to continity of F. Also, de to continity of F, φ(x n ) φ(x). Since each n is a point of argmax (F[, x n ]) where F[ n, x n ] = φ(x n ) we have, F[, x] = lim n F[ n, x n ] = lim n φ(x n ) = φ(x), we find that (, x) E, i.e., E is closed. Frther, any section E x of E is also closed itself again de to continity of F, as if ( n, x) E and n, then F[ n, x] F[, x], i.e., actally, F[ n, x] = F[, x]. Ths, Stschegolkow s theorem is applicable. References [1] A. Arapostathis. On the policy iteration algorithm for nondegenerate controlled diffsions nder the ergodic criterion in book. In: Optimization, control, and applications of stochastic systems, Systems Control Fond. Appl., Birkhäser/Springer, New York, [2] A. Arapostathis, V. S. Borkar, M. K. Ghosh. Ergodic control of diffsion processes. Encyclopedia of Mathematics and its Applications 143. Cambridge: Cambridge University Press, [3] V. S. Borkar. Optimal control of diffsion processes. Harlow: Longman Scientific & Technical; New York: John Wiley & Sons, [4] L.D. Brown, R. Prves. Measrable selections of extrema. Ann. Stat., 1: , [5] E.B. Dynkin and A.A. Yshkevich. Upravlyaemye markovskie protsessy i ikh prilozheniya, Moskva: Naka, 1975 (in Rssian). [6] R. A. Howard. Dynamic programming and Markov processes. New York-London: John Wiley &; Sons, Inc. and the Technology Press of the Massachsetts Institte of Technology, [7] B. V. Gnedenko, Y. K. Belyaev, A. D. Solovyev. Mathematical Methods in Reliability 37

8 Theory. Academic Press, New York, [8] R. A. Howard. Dynamic probabilistic systems. Vol. II: Semi-Markov and decision processes. Reprint of the 1971 original ed. Mineola, NY: Dover Pblications, , [9] R. Khasminskii. Stochastic stability of differential eqations. With contribtions by G.N. Milstein and M.B. Nevelson. 2nd completely revised and enlarged ed. Berlin: Springer, [10] N.V. Krylov. Controlled diffsion processes, 2nd ed. Berlin, et al., Springer, [11] N. V. Krylov, On the selection of a Markov process from a system of processes and the constrction of qasi-diffsion processes. Math. USSR Izv. 7: , [12] A.A. Ljapnow, E.A. Stschegolkow, and W.J. Arsenin. Arbeiten zr deskriptiven Mengenlehre. Mathematische Forschngsberichte. 1. Berlin: VEB Detscher Verlag der Wissenschaften, [13] P. Mandl. Analytical treatment of one-dimensional Markov processes. Berlin-Heidelberg- New York: Springer, [14] H. Mine and S. Osaki. Markovian decision processes. New York: American Elsevier Pblishing Company, Inc., [15] R. Morton. On the optimal control of stationary diffsion processes with inaccessible bondaries and no disconting. J. Appl. Probab., 8: , [16] È. Pardox and A.Y. Veretennikov. On Poisson eqation and diffsion approximation. II. Ann. Probab., 31(3): , [17] M. L. Pterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., Hoboken, [18] V. V. Rykov, Controllable Qeeing Systems: From the Very Beginning p to Nowadays, RT&A, 2(45), vol. 12, 39-61, [19] A. D. Solovyev. Basics of mathematical reliability theory, vol. 1. Moscow: Znanie, 1975 (in Rssian). [20] E.A. Shchegol kov. Über die Uniformisierng gewisser B-Mengen. Dokl. Akad. Nak SSSR, n. Ser., 59: , [21] A.Y. Veretennikov. On polynomial mixing for SDEs with a gradient-type drift. Theory Probab. Appl., 45(1): ,