INVARIANT PROBABILITIES FOR

Transcription

1 Applied Mathematics and Stochastic Analysis 8, Number 4, 1995, INVARIANT PROBABILITIES FOR FELLER-MARKOV CHAINS 1 ONtSIMO HERNNDEZ-LERMA CINVESTA V-IPN Departamento de Matemticas A. Postal 1-70, Mxico D.F., Mxico ohernand@math.cinvestav.mx JEAN B. LASSERRE LAAS-CNRS 7 A v. du Colonel Roche Toulouse Cedex, France lasserre@laas.fr (Received April, 1995; Revised August, 1995) ABSTRACT We give necessary and sufficient conditions for the existence of invariant probability measures for Markov chains that satisfy the Feller property. Key words: Markov Chains, Feller Property, Invariant Measures. AMS (MOS) subject classifications: 60J05, 93E Introduction The existence of invariant probabilities for Markov chains is an important issue for studying the long-term behavior of the chains and also for analyzing Markov control processes under the long-run expected average cost criterion. Inspired by the latter control problems, we present in this paper, two necessary and sufficient conditions for the existence of invariant probabilities for Markov chains that satisfy the Feller property. Our study extends previous results using stronger assumptions, such as the strong Feller property in Bene [1], nondegeneracy assumptions (see condition (2) in Bene [2]), and a uniform countable-additivity hypothesis in Liu and Susko [8]. As can be seen in the references, it is also worth noting that there are many reported results providing (only) sufficient conditions for the existence of invariant measures; in contrast however, our conditions are also necessary. The setting for this paper is specified in Section 2, and our main result is presented in Section 1This work is part of a research project jointly sponsored by CONACYT (Mxico) and CNRS (France). The work of the first author was also partially supported by CONACYT grant E9206. Printed in the U.S.A. ()1995 by North Atlantic Science Publishing Company 341

2 342 ON]SIMO HERNNDEZ-LERMA and JEAN B. LASSERRE 2. Notation and Definitions Let X be a a-compact metric space, and let {xt, t 0, 1,...} be an X-valued Markov chain with time-homogeneous kernel P, i.e., P(BIx) Prob(x + 1 ( B Ixt x)w 0,1,..., x x, B e %(X), where (X) denotes the Borel r-algebra of X. to be invariant for P if #(B) J P(B x)#(dx VB %(X). x A probability measure (p.m.) # on (X)is said Here, we give necessary and sufficient conditions for the existence of invariant p.m. s when P satisfies the Feller property: J x--, u(y)p(dy x) is in C(X) whenever u C(X), (1) where C(X) denotes the space of all bounded and continuous functions on X. a moment function, defined as follows. _ sup M Our conditions use Definition: A nonnegative Borel-measurable function v on X is said to be a moment if, as noc, inf {v(x) x gn}oo for some sequence of compact sets KnTX. Moment functions have been used by several authors to study the existence of invariant measures for Markov processes (e.g., see Bene [1, 2], Hernndez-Lerma [6], Liu and Susko [8], and Meyn and Tweedie [9]). The key feature used in these studies is the following (easily proved) fact. Lemma: Let M be a family of p.m. s on X. If there exists a moment v on X such that f vd# < cx), then M is tight, i.e., for every positive e there exists a compact set K such that #(K) > 1- e for all p M. Therefore by Prohorov s Theorem [3, 9], the family M in the lemma is relatively com m_p_.t, i.e., every sequence in M contains a weakly convergent subsequence. Our theorem below (see Section 3) extends a result by Bene [2] where our conditions (a) and (b) are new and, most importantly, we do not require Bene "nondegeneracy" condition, according to which xlkrnpt(kix) 0 for t 1,2,...,K compact, with pt(. ix being the t-step transition probability given the initial state x 0 x. This condition excludes important classes of ergodic Markov chains, such as those that have a "m inorant" see e.g., mynkin and Yushkevich [5], or condition RI in Hernndez-Lerma et al. [7]. See also Remarks 2 and 3 (Section 3) for additional comments on related results. 3. The Theorem If t, is a p.m. on X, E(. stands for the expectation given the "initial distribution" Theorem: If P satisfies the Feller property, then the following conditions (a), (b), and (c) are equivalent:

3 Invariant Probabilities for Feller-Markov Chains 343 (a) There exists a p.m. v and a moment v such that lim supjn(u < cx, wherejn(u) -n-le, E t-- There exists a p.m. u and a moment v such that limsupva() < c, where Va(u)"-(1-a)E u [kt=0ttv(xt)]; (c) There exists an invariant probability for P. Proof: We will show that (a)=(b)(c)=v(a). (a) implies (b): This follows from a well-known Abelian theorem (e.g., see Sznajder and Filar [11], Theorem 2.2), which states that lim supva() <_ lim supjn( ). n-,oo al"l (Since a direct proof that (a) implies (c) is surprisingly simple, it will also be included here; see Remark 1 below.) (b) implies (c): Suppose that (b) holds and for each c E (0, 1), let Pa be the probability measure on X defined as Then we may write Va(u)as Vc,(u and, by (b), p(b) (1 -a) a pt(b z)(dz), B e (X). t=0 X f limal,lsupv,(u) =nli,rnv(n(u) =n_..,oolim Let {c%} be a sequence in (0,1) such that c%t1 f vd#a n < oo. By the lemma in Section 2, {#a } is tight and therefore, by Prohorov s Theorem, {#a } contains a weakly convergent subsequence, which we denote by {p } again; that is, there existsa probability measure # on X such that We claim that # is invariant for P. lim n / udiza n ud# Vu C(X). To see this, first note that by the Markov property, we may write Hence, for any u G C(X), f udp- #(B) (1 ()u(b) + ( / P(B (1- c)j u(z)u(dz)+c// as Vc (0, 1), B (X). u(y)p(dy x)#a(dx),

4 344 ONISIMO HERN/NDEZ-LERMA and JEAN B. LASSERRE and furthermore, note that by the Feller property (1), f u(y)p(dy )is in C(X). Thus, replacing c by c% and letting ncx, we obtain / udlz //u(y)p(dy,x)iz(dx). (2) Finally, since u E C(X) was arbitrary, we conclude from (2) that/ is invariant for P. (c) implies (a): Let v be an invariant probability for P, and let {Kn} be an increasing sequence of compact sets such that KnTX and (K n + 1- Kn) < 1/n3, n- 1,2, (Here we have used the fact that every p.m. on a a-compact metric space is tight; see [3], p. 9.) Define a functionv(.) -0onKlandv(x):=nforxKn+l-Kn,nl. Then v is a moment and limsupjn(u v(x)u(dx) n-2 <. n=l Remark 1: We will prove directly that (a) implies (c). Suppose that (a) holds and for every 1, 2,..., let #n be the probability measure on X defined as [ t=o J so that we may rewrite the condition in (a) as Hence, by the lemma in Section 2, probability measure u. We will show that (cf. (2)) sup / lim vd #n < cx:) J {#,} has a subsequence {/,.} which converges weakly to some 0 / c c(x), (3) where Lu(x):= f u(y)p(dylx u(x), thus showing that # is invariant for P. Indeed, for any bounded measurable function u on X, the sequence with Mo(u := U(Xo) n--1 Mn(u) -u(xn)-elu(xt), n 1,2,..., t--o is a martingale, which implies Eu[Mn(u)]-- Ev[Mo(u)]Vn Finally, let u be in C(X); replace n by ni; multiply by 1/ni; and then let i--,c to get (3). Remark 2: In [8], it is shown that sup f f g(y)pt(dylz)u(dz)< (4) t>l for some moment g and initial p.m. v, is also a necessary and sufficient condition for existence of invariant probabilities provided that the Markov chain satisfies the uniform countable-additivity property

5 Invariant Probabilities for Feller-Markov Chains 345 for every compact set K in x. lim sup P(A x)- 0 (5) AIO x E K Note that (4) is stronger than our condition (a) and that (5) implies: For every compact set g C X, the family of p.m. s (P(. x)}x e K is tight. Remark 3." It is worth noting that the theorem still holds if we replace "lim sup" by "lim inf" in both conditions (a) and (b). Now, (b):=(a) by a well-known Abelian theorem [11]. With similar arguments as in Remark 1, (a):=v(c). We finally prove (c):=v(b) by exhibiting the same moment function v and show that In conclusion, lim,nfvc(u -lim].nf(1- c)e ve tv(xt v()u(d) <_ n- < c. t=o t=o we mention that the theorem can be extended in the obvious way to contin- Conditions for uniqueness and ergodicity of invariant mea- uous-time Markov processes, as in [2]. sures can be found, for instance, in [4, 7, 10] and references therein. References [1] [4] [5] [7] [9] [10] [11] Bench, V.E., Existence of finite invariant measures for Markov processes, Proc. A mer. Math. Soc. 18 (1967), Bene, V.E., Finite regular invariant measures for Feller processes, J. Appl. Prob. 5 (1968), Billingsley, P., Convergence of Probability Measures, Wiley, New York Borovkov, A.A., Conditions for ergodicity of Markov chains which are not associated with Harris irreducibility, Siberian Math. J. 32 (1992), Dynkin, E.B. and Yushkevich, A.A., Controlled Markov Processes, Springer-Verlag, New York tiernndez-lerma,,o., Lyapunov criteria for stability of differential equations with Markov parameters, Boletin Soc. Mat. Mexicana 24 (1979), Hernafidez-Lerma, O., Montes de Oca, R. and Cavazos-Cadena, R., Recurrence conditions for Markov decision processes with Borel state space: a survey, Ann. Op. Res. 28 (1991), Liu, J. and Susko, E., On strict stationarity nd ergodicity of a nonlinear ARMA model, J. Appl. Probab. 29 (1992), Meyn, S.P. and Tweedie, R.L., Markov Chains and Stochastic Stability, Springer-Verlag, London Skorokhod, A.V., Topologically recurrent Markov chains: ergodic properties, Theory Prob. Appl. 31 (1986), Sznajder, R. and Filar, J.A., Some comments on a theorem of Hardy and Littlewood, J. Optim. Theory Appl. 75 (1992),

6 Advances in Operations Research Advances in Decision Sciences Applied Mathematics Algebra Probability and Statistics The Scientific World Journal International Differential Equations Submit your manuscripts at International Advances in Combinatorics Mathematical Physics Complex Analysis International Mathematics and Mathematical Sciences Mathematical Problems in Engineering Mathematics Discrete Mathematics Discrete Dynamics in Nature and Society Function Spaces Abstract and Applied Analysis International Stochastic Analysis Optimization