Journal of Mathematical Analysis and Applications

Transcription

1 J. Mah. Anal. Appl Conens liss available a ScienceDirec Jornal of Mahemaical Analysis and Applicaions On solions of Kolmogorov s eqaions for nonhomogeneos jmp Markov processes Egene A. Feinberg a,,1, Manasa Mandava a,1,albern.shiryaev b a Deparmen of Applied Mahemaics and Saisics, Sony rook Universiy, Sony rook, NY, USA b Seklov Mahemaical Insie, 8, Gbkina Sr., Moscow, Rssia aricle info absrac Aricle hisory: Received 23 Febrary 2013 Available online 27 Sepember 2013 Sbmied by U. Sadmeller Keywords: Jmp Markov processes ackward Kolmogorov eqaion Forward Kolmogorov eqaion Minimal non-negaive solion Transiion fncion Compensaor This paper sdies hree ways o consrc a nonhomogeneos jmp Markov process: i via a compensaor of he random measre of a mlivariae poin process, ii as a minimal solion of he backward Kolmogorov eqaion, and iii as a minimal solion of he forward Kolmogorov eqaion. The main conclsion of his paper is ha, for a given measrable ransiion inensiy, commonly called a Q -fncion, all hese consrcions define he same ransiion fncion. If his ransiion fncion is reglar, ha is, he probabiliy of accmlaion of jmps is zero, hen his ransiion fncion is he niqe solion of he backward and forward Kolmogorov eqaions. For coninos Q -fncions, Kolmogorov eqaions were sdied in Feller s seminal paper. In pariclar, his paper exends Feller s resls for coninos Q -fncions o measrable Q -fncions and provides addiional resls Elsevier Inc. All righs reserved. 1. Inrodcion Le, be a sandard orel space, ha is,, is a measrable space for which here exiss a measrable injecion ono a orel sbse of he real line endowed wih is orel σ -field. For a orel sbse E of he exended real line, we denoe by E is orel σ -field. A fncion P, x;,, where, R + := ]0, [, <, x, and, is called a ransiion fncion if i akes vales in [0, 1] and saisfies he following properies: i for all, x, he fncion P, x;, is a measre on, ; ii for all he fncion P, x;, is orel measrable in, x, ; iii P, x;, saisfies he Chapman Kolmogorov eqaion P, x;, = Ps, y;, P, x; s, dy, < s <. 1 If P, x;, = 1forall, x,, hen he ransiion fncion P is called a reglar ransiion fncion. A sochasic process { : 0} wih vales in, defined on a probabiliy space Ω, F, P wih a filraion {F } 0, is called Markov if P F = P, P-a.s. for all, R + wih < and for all. And each Markov process has a ransiion fncion P sch ha P = P, ;,, P-a.s.; see Kznesov [19], where he * Corresponding ahor. addresses: egene.feinberg@sonybrook.ed E.A. Feinberg, mmandava@ams.snsyb.ed M. Mandava, albersh@mi.ras.r A.N. Shiryaev. 1 This research was parially sppored by NSF grans CMMI and CMMI /$ see fron maer 2013 Elsevier Inc. All righs reserved. hp://dx.doi.org/ /j.jmaa

2 262 E.A. Feinberg e al. / J. Mah. Anal. Appl eqivalence of he wo approaches o define a Markov process from Kolmogorov [18] is esablished. If each sample pah of he Markov process is a righ-coninos piecewise-consan fncion ha has a finie nmber of disconiniy poins on each inerval [0, ] for <, where is he lifeime of he process, hen he Markov process is called a jmp Markov process; Gikhman and Skorokhod [9, p. 187]. In his paper, measrabiliy and orel measrabiliy are sed synonymosly and all condiional probabiliies are defined almos sre, even when his is no explicily saed. We now inrodce he following fncion q ha can be inerpreed as he ransiion inensiies of a nonhomogeneos jmp Markov process. A fncion qx,,, where x, R +, and, is called a Q -fncion if i saisfies he following properies: i for all x, he fncion qx,, is a signed measre on, sch ha qx,, 0 and 0 qx,, \{x}< for all ; ii for all he fncion qx,, is measrable in x,. In addiion o properies i and ii, if qx,, = 0 for all x,, hen he Q -fncion q is called conservaive. Noe ha any Q -fncion can be ransformed ino a conservaive Q -fncion by adding a sae x o wih qx,, { x} := qx,,, q x,, := 0, and q x,, { x} := 0, where x and R +. To simplify he presenaion, in his paper we always assme ha q is conservaive. If here is no assmpion ha q is conservaive, Remark 4.1 explains how he main formlaions change. A Q -fncion q is called coninos if i is coninos in R +. Le qx, := qx,, {x}. Ase is called q-bonded if sp x, R+ qx, < and he Q -fncion q is called sable if he se {x} is q-bonded for all x. The following assmpion inrodced by Feller [8] holds hrogho he paper. Assmpion 1.1. The Q -fncion q is sable. Le n := {x : sp R+ qx, <n + 1} for all n 0. If he Q -fncion q is sable, hen n as n. Ths, Assmpion 1.1 is eqivalen o he exisence of a seqence of q-bonded ses { n } n 0 sch ha n as n.this way a sable Q -fncion was defined in Feller [8]. In his paper, a non-negaive solion f in a cerain class of solions of a fncional eqaion is called he minimal non-negaive solion if for any non-negaive solion f of his eqaion from ha class f x f x for all vales of he argmen x. Feller [8] sdied he backward and forward Kolmogorov eqaions for coninos Q -fncions. For a sable coninos Q -fncion, Feller [8] provided explici formlae for a ransiion fncion ha saisfies boh he backward and forward Kolmogorov eqaions. If he consrced ransiion fncion is reglar, Feller [8, Theorem 3] showed ha his ransiion fncion is he niqe non-negaive solion o he backward Kolmogorov eqaion. Thogh Feller [8] focsed on reglar ransiion fncions, i follows from he proof of Theorem 3 in Feller [8] ha he ransiion fncion consrced here is he minimal non-negaive solion o he backward Kolmogorov eqaion. For homogeneos Markov processes, ha is Q -fncions do no depend on he ime parameer, Doob [4], [5, Chap. 6] provided an explici consrcion for mliple ransiion fncions saisfying he backward Kolmogorov eqaion. For conable sae homogeneos Markov processes, Kendall [14], Kendall and Reer [15], and Reer [20] gave examples wih non-niqe solions o Kolmogorov eqaions, and Reer [20] provided necessary and sfficien condiions for heir niqeness; see also Anderson [1] and Chen e al. [3]. Ye, Go, and Hernández-Lerma [21] proved he exisence of a ransiion fncion ha is he minimal non-negaive solion o boh he backward and forward Kolmogorov eqaions for a conable sae problem wih measrable Q -fncions. A conservaive Q -fncion can be sed o consrc a predicable random measre. According o Jacod [13, Theorem 3.6], an iniial sae disribion and a predicable random measre define niqely a mlivariae poin process. This paper sdies he backward and forward Kolmogorov eqaions for measrable Q -fncions and sandard orel sae spaces. I exends Feller s [8] resls for coninos Q -fncions o measrable Q -fncions and esablishes addiional resls. Theorem 2.2 below saes ha he sochasic process associaed wih he mlivariae poin process defined by a sable conservaive Q -fncion and an iniial sae disribion is a jmp Markov process wih he ransiion fncion defined in Feller [8]. Theorems 3.1 and 4.1 sae ha his ransiion fncion saisfies he backward and forward Kolmogorov eqaions. In addiion, his ransiion fncion is he minimal non-negaive solion of boh he backward and forward Kolmogorov eqaions and, if his ransiion fncion is reglar, hen i is he niqe non-negaive solion of he backward and forward Kolmogorov eqaions; Theorems 3.2, 4.3. Ths, he minimal non-negaive solion of boh he backward and forward Kolmogorov eqaions is he ransiion fncion of a jmp Markov process associaed wih a mlivariae poin process whose compensaor is defined by he conservaive Q -fncion. In addiion o answering he fndamenal qesion on how o consrc a jmp Markov process wih a given Q -fncion, or ineres in his sdy is moivaed by is applicaions o conrol of coninos-ime jmp Markov processes. Here we menion wo of hem: i For a conable sae space, each Markov policy along wih a given iniial sae disribion defines a jmp Markov process wih he ransiion fncion being he minimal non-negaive solion of he forward Kolmogorov eqaion; Go and Hernández-Lerma [10, Secion 2.2]. An arbirary policy defines a mlivariae poin process via he compensaor of is random measre; Kiaev [16], Kiaev and Rykov [17, Secion 4.6], Feinberg [6,7], Go and Pinovskiy [11]. The resls of his paper imply ha for Markov policies hese wo definiions are eqivalen for problems wih orel sae spaces.

3 E.A. Feinberg e al. / J. Mah. Anal. Appl ii Feller s [8] resls are broadly sed in he lierare on coninos-ime Markov decision processes o define Markov processes corresponding o Markov policies, and his leads o he nnecessary assmpion ha decisions depend coninosly on ime; see, e.g., Go and Rieder [12, Definiion 2.2]. For conable sae problems, he resls of Ye, Go, and Hernández-Lerma [21] removed he necessiy o assme his coniniy. The resls of he crren paper imply ha his coniniy assmpion is nnecessary for Markov decision processes wih orel sae spaces. 2. Relaion beween jmp Markov processes and Q -fncions The main goals of his secion are o show ha an iniial sae disribion and a sable Q -fncion q define a jmp Markov process and o consrc is ransiion fncion. Le x / be an isolaed poin adjoined o he space. Denoe = {x } and R + =]0, ]. Consider he orel σ -field = σ, {x } on, which is he minimal σ -field conaining and {x }.Le R + be he se of all seqences x 0, 1, x 1, 2, x 2,... wih x n and n+1 R + for all n 0. This se is endowed wih he σ -field generaed by he prodcs of he orel σ -fields and R +. Denoe by Ω he sbse of all seqences ω = x 0, 1, x 1, 2, x 2,... from R + sch ha: i x 0 ; ii if n <, hen n < n+1 and x n, and, if n =, hen n+1 = n and x n = x, for all n 1. Observe ha Ω is a measrable sbse of R +. Consider he measrable space Ω, F, where F is he σ -field of he measrable sbses of Ω. Then, x n ω = x n and n+1 ω = n+1, n 0, are random variables defined on he measrable space Ω, F. Le 0 := 0, ω := lim n n ω, ω Ω, and for all 0leF := σ, G, where G := σ I{x n }I{ n s}: n 1, 0 s,. Throgho his paper, we omi ω whenever possible and also follow he sandard convenion ha 0 =0. For a given Q -fncion q, consider he random measre ν on R +, R + defined by ν ω;]0, ], = 0 I{ n < s n+1 }q x n, s, \{x n } ds, R +,. 2 n 0 Noe ha ν{}, = ν[, [, = 0 and 2 can be rearranged as ν ]0, ], = n 1 m+1 m I{ n < n+1 } q x m, m + s, \{x m } n ds + n q x n, n + s, \{x n } ds. 3 As he expression in he parenheses on he righ-hand side of 3 is an F n -measrable process for each, i follows from Jacod [13, Lemma 3.3] ha he process {ν]0, ], : R + } is predicable. Therefore, he measre ν is a predicable random measre. According o Jacod [13, Theorem 3.6], he predicable random measre ν defined in 2 and a probabiliy measre μ on define a niqe probabiliy measre P on Ω, F sch ha Px 0 = μ,, and ν is he compensaor of he random measre of he mlivariae poin process n, x n n 1 defined by he riple Ω, F, P. Consider he process { : 0}, ω := I{ n < n+1 }x n + I{ }x, 4 n 0 defined on Ω, F, P. Weabbreviaeheprocess{ : 0} as. The main resl of his secion, Theorem 2.2, shows ha he process is a jmp Markov process and provides is ransiion fncion. For an F -measrable sopping ime τ,lenτ := max{n = 0, 1,...: τ n }.SinceNτ = and τ ={x } when τ, we follow he convenion ha +1 = and x +1 = x. Denoe by G τ ω;, and H τ ω; respecively he reglar condiional laws of Nτ +1, x Nτ +1 and Nτ +1 wih respec o F τ ; H τ ω; = G τ ω;,. In pariclar, Gn ω;, and H n ω;, where n = 0, 1,..., denoe he condiional laws of n+1, x n+1 and n+1 wih respec o F n. We remark ha he noaions G n and H n correspond o he noaions G n and H n in Jacod [13, p. 241]. Lemma 2.1. For all, R +,<, H [, ] = e q,s ds, N<, 5 G d, = e q,s ds q,, \{ } d,, N<. 6 Proof. According o Jacod [13, Proposiion 3.1], forall R +,, and n = 0, 1,... νd, = G n d, H n [, ], n < n+1. 7 In pariclar, for =, from7 and from he propery ha x n+1 when n+1 <,

4 264 E.A. Feinberg e al. / J. Mah. Anal. Appl νd, = G n d, H n [, ] = H n d H n [, ], n < n+1. This eqaliy implies ha νd, is he hazard rae fncion corresponding o he disribion H n when n < n+1. Therefore, H n [, ] = e ν] n,],i{ n < n+1 }, R +, > n. 8 From 2 and 8, forall R +, H n [, ] = e n qx n,s ds, > n, 9 and from 2, 7, and 9, forall R +,, G n d, = e n qx n,s ds q xn,, \{x n } d, > n. 10 To compe G, observe ha for all, R +, <, and, G d, = P N+1 [, + d[, x N+1 F = n 0 P N+1 [, + d[, x N+1 F I { N = n } = n 0 P n+1 [, + d[, x n+1 F, N = n I { N = n }, 11 where he firs eqaliy follows from he definiion of G, he second eqaliy holds becase {N = } {N = n} n=0,1,... is an F -measrable pariion of Ω and x N+1 = x / when N =, and he hird eqaliy follows from N = n and from {N = n} F. Observe ha for any random variable Z on Ω, F P Z F, N = n I { N = n } = P Z Fn, N = n I { N = n } = PZ F n, n, n+1 > I { N = n } = PZ F n, n+1 > I { N = n } = PZ, n+1 > F n { I N = } n, 12 P n+1 > F n where he firs eqaliy follows from rémad [2, Theorem T32, p. 308], he second eqaliy holds becase { n, n+1 > } ={N = n}, he hird eqaliy holds becase { n } F n, and he las one follows from he definiion of condiional probabiliies. Le Z ={ n+1 [, + d[, x n+1 }, where R +,. Then11 and 12 imply G d, = n 0 = n 0 P n+1 [, + d[, x n+1 F n { I N = } n P n+1 > F n e n qx n,s ds qxn,, \{x n } d { I N = } n e n qx n,s ds = e q,s ds q,, \{ } d, 13 where he firs eqaliy holds becase { n+1 [, + d[, n+1 > } ={ n+1 [, + d[} when >, he second eqaliy follows from 9 and 10, and he las eqaliy holds since x n = when N = n. Forall, R +, <, i follows from he propery ha x N+1 when N+1 < and from 13 ha H [, ] saisfies 5. Following Feller [8, p. 501], forx,, R +, <, and, define P 0, x;, = I{x }e qx,s ds, 14 and for n 1define P n, x;, = \{x} e s qx,θ dθ qx, s, dy P n 1 s, y;, ds. 15

5 E.A. Feinberg e al. / J. Mah. Anal. Appl Se P, x;, := P n, x;,. n=0 16 Observe ha P is a ransiion fncion. For sable coninos Q -fncions, Feller [8, Theorems 2, 5] proved ha a for fixed, x, he fncion P, x;, is a measre on, sch ha 0 P, x;, 1, and b for all, x,, he fncion P, x;, saisfies he Chapman Kolmogorov eqaion 1. The proofs remain correc for measrable Q -fncions q. The measrabiliy of P, x;, in, x, for all is sraighforward from he definiions 14, 15, and 16. Therefore, he fncion P saisfies properies i iii from he definiion of a ransiion fncion. Theorem 2.2. For a given iniial sae disribion and for a sable Q -fncion q, he process defined in 4 is a jmp Markov process wih he ransiion fncion P. Proof. Observe ha he sample pahs of he process are righ-coninos piecewise-consan fncions ha have finie nmber of disconiniies on each inerval [0, ] for <.Thsif,forall, R +, <, and, P F = P = P, ;,, <, 17 hen he process is a jmp Markov process wih he ransiion fncion P.Toprove17, we firs esablish by indcion ha for all n = 0, 1,...,, R +, <, and P, N ],] = n F = P n, ;,, <, 18 where N ],] := N N when < and N ],] := when.eq.18 holds for n = 0becasefor < P, N ],] = 0 F = P, N+1 > F = I{ }H ], ] = I{ }e q,s ds = P 0, ;,, 19 where he firs eqaliy holds becase he corresponding evens coincide, he second eqaliy holds becase { } F and from he definiion of H, he hird eqaliy is correc becase of 5, and he las eqaliy is 14. For some n 0, assme ha 18 holds. Then for < P, N ],] = n + 1 F = = = \{ } \{ } P, N ]N+1,] = n F, N+1, x N+1 G d N+1, dx N+1 P, N ]N+1,] = n F N+1 G d N+1, dx N+1 q, s, dye s q,θ dθ P n s, y;, ds = P n+1, ;,, 20 \{ } where he firs eqaliy holds since N ],] = 1 + N ]N+1,] for N ],] 1 and since EEZ D = EZ for any random variable Z and any σ -field D, he second eqaliy holds since σ F, N+1, x N+1 = F N+1, he hird eqaliy follows from 6 and 18, and he las eqaliy is 15. Eqaliy18 is proved. Observe ha for, R +, <,, P F = P F I{ < }+P F I{ } = n 0 P, N ],] = n F I{ < }= n 0 P n, ;, I{ < } = P, ;, I{ < }= P, ;, I{ }, 21 where he firs eqaliy holds since {{ < }, { }} is a pariion of Ω and { < }, { } F, he second eqaliy holds since implies <, he hird eqaliy follows from 18, he forh eqaliy follows from 16, and he las one holds since { < }={ }. As follows from 21, he fncion P F is σ -measrable. Ths,

6 266 E.A. Feinberg e al. / J. Mah. Anal. Appl P F = P P F = P, 22 where he second eqaliy holds becase σ F ; see e.g. rémad [2, p. 280]. Ths, 17 follows from 21 and ackward Kolmogorov eqaion In his secion, we show ha he ransiion fncion P defined in 16 is he minimal non-negaive solion o he backward Kolmogorov eqaion. For a coninos Q -fncion q, relevan resls were esablished by Feller [8, Theorems 2, 3]. Theorem 3.1. The fncion P, x;, saisfies he following properies: i P, x;, is for fixed x,, an absolely coninos fncion in and saisfies niformly in he bondary condiion lim P, x;, = I{x }. 23 ii For all x,,, he fncion P, x;, saisfies for almos every < he backward Kolmogorov eqaion P, x;, = qx, P, x;, \{x} qx,, dyp, y;,. 24 Proof. i For all x,, R +, <, and, P, x;, = P n, x;, n=0 = I{x }e qx,s ds + n=1 = I{x }e qx,s ds + = I{x }e qx,s ds + e s qx,θ dθ e s qx,θ dθ e s qx,θ dθ \{x} \{x} \{x} qx, s, dy P n 1 s, y;, ds qx, s, dy P n 1 s, y;, ds n=1 qx, s, dy Ps, y;, ds, 25 where he firs eqaliy is 16, he second eqaliy follows from 14 and 15, he hird eqaliy is obained by inerchanging he inegral and sm, and he las one follows from 16. Forfixedx,,, Eq.25 implies ha P, x;, is he sm of wo absolely coninos fncions in. Ths, P, x;, is for fixed x,, an absolely coninos fncion in. Observe ha P n, x;, P, x;, 1foralln 0, x,, R+, <, and. Thenfrom15, P n, x;, e s qx,σ dσ qx, s ds, n This ineqaliy and 14 imply ha, for any sable Q -fncion q, lim P n, x;, = 0 for all n 1 and lim P 0, x;, = I{x } 27 niformly wih respec o. Ths,16 and 27 imply 23. ii Since an absolely coninos real-valed fncion is differeniable almos everywhere on is domain, for all x,, he fncion P, x;, is differeniable in almos everywhere on ]0, [. y differeniaing 25, foralmosevery <, P, x;, = I{x }e qx,s ds qx, + s e qx,θ dθ \{x} \{x} qx,, dy P, y;, qx, s, dy Ps, y;, ds

7 = I{x }e qx,s ds qx, + E.A. Feinberg e al. / J. Mah. Anal. Appl e s qx,θ dθ qx, \{x} \{x} qx,, dy P, y;, qx, s, dy Ps, y;, ds. 28 In view of 25, he sm of he firs and he las erms in he las expression of 28 is eqal o he firs erm on he righ-hand side of 24. As shown in Feller [8, Theorem 2], for a sable coninos Q -fncion q, he ransiion fncion P saisfies he backward Kolmogorov eqaion for all, while Theorem 3.1ii saes ha his eqaion holds for almos every. This difference in formlaions akes place becase he coniniy of he Q -fncion q and he finieness of each inegrand in he las expression of 25 garanee he exisence of he derivaive P, x;, for all. Definiion 3.1. AfncionP wih he same domain as P is a solion of he backward Kolmogorov eqaion 24 if he fncion P saisfies he properies saed in Theorem 3.1. The nex heorem describes he minimal and niqeness properies of he solion P of he backward Kolmogorov eqaion 24. Theorem 3.2. The fncion P is he minimal non-negaive solion of he backward Kolmogorov eqaion 24. Also,if P isareglar ransiion fncion ha is, P, x;, = 1 for all, x, inhedomainof P, hen P is he niqe non-negaive solion of he backward Kolmogoroveqaion 24 hais a measreon, for fixed, x, wih < and akes vales in [0, 1]. Proof. The proof of minimaliy is similar o he proof of Theorem 3 in Feller [8]. We provide i here for compleeness. Le P wih he same domain as P be a non-negaive solion of he backward Kolmogorov eqaion 24. Inegraing 24 from o and by sing he bondary condiion 23, P, x;, = I{x }e qx,s ds + Since he las erm of 29 is non-negaive, \{x} e s qx,θ dθ qx, s, dyp s, y;, ds. 29 P, x;, I{x }e qx,s ds = P 0, x;,, 30 where he las eqaliy is 14. Forall, x,, wih <, assme P, x;, n P m, x;, for some n 0. Then from 29 P, x;, I{x }e qx,s ds + = P 0, x;, + \{x} e s qx,θ dθ qx, s, dy n P m+1, x;, = n+1 n P m s, y;, ds P m, x;,, where he firs eqaliy follows from he assmpion ha P, x;, n P m, x;, for all, x,, wih <, he second eqaliy follows from 14 and 15, and he hird eqaliy is sraighforward. Ths, by indcion, P, x;, n P m, x;, for all n 0, x,, R +, <, and, which implies ha P, x;, P, x;, for all, x,,. To prove he second par of he heorem, le he solion P be a measre on, for fixed, x, and wih vales in [0, 1]. Assme ha P, x;, P, x;, for a leas one ple, x,,. Then, P, x;, = P, x;, + P, x;, c > P, x;, + P, x;, c = P, x;, = 1, where he ineqaliy holds becase P, x,, P, x,, for all, x,. SinceP akes vales in [0, 1], he assmpion ha P, x;, P, x;, for aleas one ple, x,, leads o a conradicion.

8 268 E.A. Feinberg e al. / J. Mah. Anal. Appl Forward Kolmogorov eqaion For he forward Kolmogorov eqaion, his secion provides he resls similar o he resls on backward Kolmogorov eqaion in Secion 3. Theorem 4.1. The fncion P, x;, saisfies he following properies: i P, x;, is for fixed, x, an absolely coninos fncion in and saisfies niformly in he bondary condiion lim P, x;, = I{x } ii For all, x, and q-bonded ses, he fncion P, x;, saisfies for almos every > he forward Kolmogorov eqaion P, x;, = qy, P, x;, dy + q y,, \{y} P, x;, dy. 32 Proof. i For all x,, R +, <, and, Eq.25 implies ha he fncion P, x;, is absolely coninos in for fixed, x,. Also, Eqs. 14 and 26 imply ha, for any sable Q -fncion q, lim P n, x;, = 0 for all n 1 and lim P 0, x;, = I{x } niformly wih respec o. Ths,16 and 33 imply 31. ii Consider he following non-negaive fncion defined on he domain of P by Π, x;, = qx,, dye qy,s ds. 34 \{x} According o Feller [8, Theorem 4], he fncion P n, x;,, n 1, saisfies he recrsion P n, x;, = Πs, y;, P n 1, x; s, dy ds. 35 Thogh he fncion P n, x;, is defined for coninos Q -fncions in Feller [8], he proof given here is correc for orel Q -fncions. From 14, 16, and 35, P, x;, = P n, x;, n=0 = I{x }e qx,s ds + = I{x }e qx,s ds + = I{x }e qx,s ds + n=1 Πs, y;, P n 1, x; s, dy ds Πs, y;, P n 1, x; s, dy ds n=1 Πs, y;, P, x; s, dy ds. 36 Since P, x;, is an absolely coninos fncion in for fixed, x,, hederivaive P, x;, exiss for almos every ], [. y differeniaing 36, foralmosevery >, P, x;, = I{x }e qx,s ds qx, + Π, y;, P, x;, dy + Πs, y;, P, x; s, dy ds. 37

9 E.A. Feinberg e al. / J. Mah. Anal. Appl y differeniaing 34 wih respec o, forallq-bonded ses, Π, x;, = qx,, dy e qy,s ds = qy, Π, x;, dy. 38 \{x} Combining 37 and 38 and observing ha Π, y;, = qy,, \{y}, forallq-bonded ses, P, x;, = I{x }e qx,s ds qx, + q y,, \{y} P, x;, dy qz, Πs, y;, dz P, x; s, dy ds, 39 for almos every >. y sbsiing P, x;, dz in he lef-hand side of he following eqaliy wih he final expression in 36, qz, P, x;, dz = I{x }e qx,s ds qx, + Formlae 39 and 40 imply saemen ii of he heorem. qz, Πs, y;, dz P, x; s, dy ds. 40 Corollary 4.2. See Feller [8, Eq. 37]. For all x,, R +,<, and q-bonded ses, he fncion P, x;, defined in 16 saisfies P, x;, = I{x }+ ds q y, s, \{y} P, x; s, dy ds qy, sp, x; s, dy. 41 Proof. Inegraing 32 from o and by sing he bondary condiion 31, we ge41. Ths, by Theorem 4.1, for all x,, R +, <, and q-bonded ses, he fncion P, x;, saisfies 41. Definiion 4.1. A fncion P wih he same domain as P is a solion of he forward Kolmogorov eqaion 32 if he fncion P saisfies he properies saed in Theorem 4.1. Following he proof of Theorem 3.2, we esablish he minimal and niqeness properies of he solion P of he forward Kolmogorov eqaion 32 in Theorem 4.3. We remark ha he fncion P is he minimal non-negaive solion of 32 on aresriceddomain, R +, <, x, and q-bonded ses. Theorem 4.3. The fncion P, x;,, being resriced o q-bonded ses, is he minimal non-negaive solion of he forward Kolmogorov eqaion 32. Also,if P is a reglar ransiion fncion ha is, P, x;, = 1 for all, x, inhedomainof P, hen P is he niqe non-negaive solion of he forward Kolmogorov eqaion 32 ha is a measre on, for fixed, x, wih < and akes vales in [0, 1]. Proof. Le P defined on he same domain as P be a non-negaive solion of he forward Kolmogorov eqaion 32. Inegraing 32 from o and by sing he bondary condiion 31, forallx,, R + wih <, and q-bonded ses, P, x;, = I{x }e qx,s ds + ds Πs, y;, P, x; s, dy, 42 for all q-bonded ses. Since he las erm of42 is non-negaive, P, x;, I{x }e qx,s ds = P 0, x;,, 43 where he las eqaliy is 14. Forallx,, wih < and q-bonded ses, assme P, x;, n P m, x;, for some n 0. Then from 42

10 270 E.A. Feinberg e al. / J. Mah. Anal. Appl P, x;, I{x }e qx,s ds + = P 0, x;, + ds Πs, y;, n P m+1, x;, = n P m, x; s, dy n+1 P m, x;,. Ths, by indcion, P, x;, n P m, x;, for all n 0, x,, R + wih <, and q-bonded ses, which implies ha P, x;, P, x;, for all, x, wih <, and for all q-bonded ses. To prove he niqeness propery of P,lehesolionP be a measre on, for fixed, x, wih < and wih vales in [0, 1]. I follows from saemen i of he heorem ha for all P, x;, = lim n P, x;, n lim P, x;, n = P, x;,, 44 n where { n } n 0 is an increasing seqence of q-bonded ses sch ha n as n, whose exisence is garaneed by Assmpion 1.1. If P, x;, = 1 for all, x,, hen he niqeness of P wihin he se of solions o he forward Kolmogorov eqaion ha ake vales in [0, 1] and ha are measres on, for fixed, x, wih < follows from he minimaliy of P 44 and from he same argmens as in he proof of niqeness in Theorem 3.2. Remark 4.1. The resls of his paper can be exended o non-conservaive Q -fncions. As menioned in Secion 1, any non-conservaive Q -fncion q can be ransformed ino a conservaive Q -fncion by adding a sae x o wih qx,, { x} := qx,,, q x,, := 0, and q x,, { x} := 0, where x and R +. According o Theorem 2.2, hereisa ransiion fncion P of a jmp Markov process wih he sae space = { x}, and his process is deermined by he iniial sae disribion and by he compensaor defined by he modified Q -fncion. The proofs of he resls of Secions 3 and 4 do no se he assmpion ha he Q -fncion q is conservaive. Therefore, hese resls remain valid for non-conservaive Q -fncions. However, he validiy of he condiion P, x;, = 1forallx,, wih < in Theorems 3.2 and 4.3 is possible only if qx,, = 0 almos everywhere in for each x. Ths,infac,q is conservaive, if P, x;, = 1 for all x,, wih <. I is also easy o see ha he minimal non-negaive solions of boh he backward and forward Kolmogorov eqaions are eqal o P, x;,, when x and, where he ransiion fncion P is described in he previos paragraph for a broader domain. Remark 4.2. In his paper, ransiion fncions P, x;, and P, x;, are defined for > 0. All he resls of Secions 3 and 4 hold for 0 wih he same proofs. When x is he iniial sae of he process, he resls of Secion 2 also hold for 0. References [1] W.J. Anderson, Coninos-Time Markov Chains: An Applicaions-Oriened Approach, Springer Ser. Sais., Springer-Verlag, New York, [2] P. rémad, Poin Processes and Qees: Maringale Dynamics, Springer Ser. Sais., Springer-Verlag, New York, [3] A. Chen, P. Polle, H. Zhang,. Cairns, Uniqeness crieria for coninos-ime Markov chains wih general ransiion srcres, Adv. in Appl. Probab [4] J.L. Doob, Markoff chains denmerable case, Trans. Amer. Mah. Soc [5] J.L. Doob, Sochasic Process, John Wiley, New York, 1990, reprin of he 1953 original. [6] E.A. Feinberg, Coninos ime disconed jmp Markov decision processes: A discree-even approach, Mah. Oper. Res [7] E.A. Feinberg, Redcion of disconed coninos-ime MDPs wih nbonded jmp and reward raes o discree-ime oal-reward MDPs, in: D. Hernández, J.A. Minjárez-Sosa Eds., Opimizaion, Conrol, and Applicaions of Sochasic Sysems, irkhäser, oson, 2012, pp [8] W. Feller, On he inegro-differenial eqaions of prely-disconinos Markoff processes, Trans. Amer. Mah. Soc , Trans. Amer. Mah. Soc Erram. [9] I.I. Gikhman, A.V. Skorokhod, The Theory of Sochasic Processes II, Springer-Verlag, erlin, [10]. Go, O. Hernández-Lerma, Coninos-Time Markov Decision Processes: Theory and Applicaions, Springer-Verlag, erlin, [11]. Go, A.. Pinovskiy, Disconed coninos-ime Markov decision processes wih consrains: nbonded ransiion and loss raes, Mah. Oper. Res [12]. Go, U. Rieder, Average opimaliy for coninos-ime Markov decision processes in Polish spaces, Ann. Appl. Probab [13] J. Jacod, Mlivariae poin processes: predicable projecion, Radon Nikodym derivaives, represenaion of maringales, Probab. Theory Relaed Fields [14] D.G. Kendall, Some frher pahological examples in he heory of denmerable Markov processes, Q. J. Mah [15] D.G. Kendall, G.E.H. Reer, Some pahological Markov processes wih a denmerable infiniy of saes and he associaed semigrops of operaors on l, in: Proceedings of he Inernaional Congress of Mahemaicians, vol. III, Amserdam, 1954, pp [16] M.Y. Kiaev, Semi-Markov and jmp Markov conrolled models: average cos crierion, Theory Probab. Appl [17] M.Y. Kiaev, V.V. Rykov, Conrolled Qeeing Sysems, CRC Press, oca Raon, [18] A.N. Kolmogorov, On analyic mehods in probabiliy heory, in: A.N. Shiryaev Ed., Seleced Works of A.N. Kolmogorov, vol. II, in: Probab. Theory Mah. Sais., Springer-Verlag, 1992 in Rssian [19] S.E. Kznesov, Any Markov process in a orel space has a ransiion fncion, Theory Probab. Appl [20] G.E.H. Reer, Denmerable Markov processes and he associaed conracion semigrops on l, Aca Mah [21] L. Ye,. Go, O. Hernández-Lerma, Exisence and reglariy of a nonhomogeneos ransiion marix nder measrabiliy condiions, J. Theore. Probab