Chapter 6. Markov processes. 6.1 Introduction

Size: px

Start display at page:

Download "Chapter 6. Markov processes. 6.1 Introduction"

Leslie George
6 years ago
Views:

1 Chapter 6 Markov processes 6.1 Introduction It is not uncommon for a Markov process to be defined as a sextuple (, F, F t,x t, t, P x ), and for additional notation (e.g.,,, S,P t,r,etc.) tobe introduced rather rapidly. This can be intimidating for the beginner. We will explain this notation in as gentle a manner as possible. We will consider amarkovprocesstobeapair(x t, P x )(ratherthanasextuple),wherex t is asinglestochasticprocessand{p x } is a family of probability measures, one probability measure P x corresponding to each element x of the state space. The idea that a Markov process consists of one process and many probabilities is one that takes some getting used to. To explain this, let us first look at an example. Suppose X 1,X 2,... is a Markov chain with stationary transition probabilities with 5 states: 1, 2,...,5. Everything we want to know about X can be determined if we know p(i, j) =P(X 1 = j X 0 = i) for each i and j and µ(i) =P(X 0 = i) foreachi. We sometimes think of having a di erent Markov chain for every choice of starting distribution µ =(µ(1),...,µ(5)). But instead let us define a new probability space by taking 0 to be the collection of all sequences! =(! 0,! 1,...)suchthateach! n takes one of the values 1,...,5. Define X n (!) =! n.definef n to be the -field generated by X 0,...,X n ; this is the same as the -field generated by sets of the form {! :! 0 = a 0,...,! n = a n },wherea 0,...,a n 2{1, 2,...,5}. 85

2 86 CHAPTER 6. MARKOV PROCESSES For each x =1, 2,...,5, define a probability measure P x on 0 by P x (X 0 = x 0,X 1 = x 1,...X n = x n ) (6.1) =1 {x} (x 0 )p(x 0,x 1 ) p(x n 1,x n ). We have 5 di erent probability measures, one for each of x =1, 2,...,5, and we can start with an arbitrary probability distribution µ if we define P µ (A) = P 5 i=1 Pi (A)µ(i). We have lost no information by this redefinition, and it turns out this works much better when doing technical details. The value of X 0 (!) =! 0 can be any of 1, 2,...,5; the notion of starting at x is captured by P x,notbyx 0.TheprobabilitymeasureP x is concentrated on those! s for which! 0 = x and P x gives no mass to any other!. Let us now look at a Lévy process, and see how this framework plays out there. Let P be a probability measure and let Z t be a Lévy process with respect to P started at 0. Then Z x t = x + Z t is a Lévy process started at x. Let 0 be the set of right continuous left limit functions from [0, 1) tor, so that each element! in 0 is a right continuous left limit function. (We do not require that!(0) = 0 or that!(0) take any particular value of x.) Define X t (!) =!(t). (6.2) This will be our process. Let F be the -field on 0,therightcontinuous left limit functions, generated by the cylindrical subsets. Now define P x to be the law of Z x.thismeansthatp x is the probability measure on ( 0, F) defined by P x (X 2 A) =P(Z x 2 A), x 2 R,A2F. (6.3) The probability measure P x is determined by the fact that if n applet n,andb 1,...,B n are Borel subsets of R, then P(X t1 2 B 1,...,X tn 2 B n )=P(Z x t 1 2 B 1,...,Z x t n 2 B n ). 1, t 1 apple 6.2 Definition of a Markov process We want to allow our Markov processes to take values in spaces other than the Euclidean ones. For now, we take our state space S to be a separable metric space, furnished with the Borel -field. For the first time around, just think of R in place of S.

3 6.2. DEFINITION OF A MARKOV PROCESS 87 To define a Markov process, we start with a measurable space (, F) and we suppose we have a filtration {F t } (not necessarily satisfying the usual conditions). Definition 6.1 A Markov process (X t, P x ) is a stochastic process X :[0, 1)!S and a family of probability measures {P x : x 2S}on (, F) satisfying the following. (1) For each t, X t is F t measurable. (2) For each t and each Borel subset A of S, the map x! P x (X t 2 A) is Borel measurable. (3) For each s, t 0, each Borel subset A of S, and each x 2S, we have P x (X s+t 2 A F s )=P Xs (X t 2 A), P x a.s. (6.4) Some explanation is definitely in order. Let '(x) =P x (X t 2 A), (6.5) so that ' is a function mapping S to R. Part of the definition of filtration is that each F t F. Since we are requiring X t to be F t measurable, that means that (X t 2 A) isinf and it makes sense to talk about P x (X t 2 A). Definition 6.1(2) says that the function ' is Borel measurable. This is a very mild assumption, and will be satisfied in the examples we look at. The expression P Xs (X t 2 A) on the right hand side of (6.4) is a random variable and its value at! 2 isdefinedtobe'(x s (!)), with ' given by (6.5). Note that the randomness in P Xs (X t 2 A) isthusallduetothe X s term and not the X t term. Definition 6.1(3) can be rephrased as saying that for each s, t, eacha, andeachx, thereisasetn s,t,x,a thatisa null set with respect to P x and for!/2 N s,t,x,a,theconditionalexpectation P x (X s+t 2 A F s )isequalto'(x s ). We have now explained all the terms in the sextuple (, F, F t,x t, t, P x ) except for t. These are called shift operators and are maps from! such that X s t = X s+t. We defer the precise meaning of the t and the rationale for them until Section 6.4, where they will appear in a natural way.

4 88 CHAPTER 6. MARKOV PROCESSES In the remainder of the section and in Section 6.3 we define some of the additional notation commonly used for Markov processes. The first one is almost self-explanatory. We use E x for expectation with respect to P x. As with P Xs (X t 2 A), the notation E Xs f(x t ), where f is bounded and Borel measurable, is to be taken to mean (X s )with (y) =E y f(x t ). If we want to talk about our Markov process started with distribution µ, we define Z P µ (B) = P x (B) µ(dx), and similarly for E µ ;hereµ is a probability on S. 6.3 Transition probabilities If B is the Borel -field on a metric space S, akernel Q(x, A) ons is a map from S B!R satisfying the following. (1) For each x 2S, Q(x, ) isameasureon(s, B). (2) For each A 2B,thefunctionx! Q(x, A) isborelmeasurable. The definition of Markov transition probabilities or simply transition probabilities is the following. 0} are Markov transi- Definition 6.2 A collection of kernels {P t (x, A); t tion probabilities for a Markov process (X t, P x ) if (1) P t (x, S) =1for each t 0 and each x 2S. (2) For each x 2S, each Borel subset A of S, and each s, t 0, Z P t+s (x, A) = P t (y, A)P s (x, dy). (6.6) (3) For each x 2S, each Borel subset A of S, and each t 0, S P t (x, A) =P x (X t 2 A). (6.7) Definition 6.2(3) can be rephrased as saying that for each x, themeasures P t (x, dy) andp x (X t 2 dy) arethesame.wedefine Z P t f(x) = f(y)p t (x, dy) (6.8)

5 6.3. TRANSITION PROBABILITIES 89 when f : S!R is Borel measurable and either bounded or non-negative. The equations (6.6) are known as the Chapman-Kolmogorov equations. They can be rephrased in terms of equality of measures: for each x Z P s+t (x, dz) = P t (y, dz)p s (x, dy). (6.9) y2s Multiplying (6.9) by a bounded Borel measurable function f(z) andintegrating gives Z P s+t f(x) = P t f(y)p s (x, dy). (6.10) The right hand side is the same as P s (P t f)(x), so we have i.e., the functions P s+t f and P s P t f are the same. known as the semigroup property. P s+t f(x) =P s P t f(x), (6.11) The equation (6.11) is P t is a linear operator on the space of bounded Borel measurable functions on S. Wecanthenrephrase(6.11)simplyas P s+t = P s P t. (6.12) Operators satisfying (6.12) are called a semigroup, and are much studied in functional analysis. One more observation about semigroups: if we take expectations in (6.4), we obtain i P x (X s+t 2 A) =E hp x Xs (X t 2 A). The left hand side is P s+t 1 A (x) andtherighthandsideis E x [P t 1 A (X s )] = P s P t 1 A (x), and so (6.4) encodes the semigroup property. The resolvent or -potential of a semigroup P t is defined by R f(x) = Z 1 0 e t P t f(x) dt, 0, x 2S.

6 90 CHAPTER 6. MARKOV PROCESSES This can be recognized as the Laplace transform of P t.bythefubinitheorem, we see that Z 1 R f(x) =E x e t f(x t ) dt. 0 Resolvents are useful because they are typically easier to work with than semigroups. When practitioners of stochastic calculus tire of a martingale, they stop it. Markov process theorists are a harsher lot and they kill their processes. To be precise, attach an isolated point to S. ThusonelooksatS b = S[, and the topology on S b is the one generated by the open sets of S and { }. is called the cemetery point. All functions on S are extended to S b by defining them to be 0 at. At some random time the Markov process is killed, which means that X t = forallt. Thetime is called the lifetime of the Markov process. 6.4 The canonical process and shift operators Suppose we have a Markov process (X t, P x )wheref t = (X s ; s apple t). Suppose that X t has right continuous left limit paths. For this to even make sense, we need the set {t! X t is not right continuous left limit} to be in F, and then we require this event to be P x -null for each x. Define tobethe e set of right continuous left limit functions on [0, 1). If e! 2, e set X e t = e!(t). Define F e t = ( X e s ; s apple t) andf e 1 = _ t 0Ft e. Finally define P ex on (, e F e 1 )by ep x ( X e 2 )=P x (X 2 ). Thus P ex is specified uniquely by ep x ( e X t1 2 A 1,..., e X tn 2 A n )=P x (X t1 2 A 1,...,X tn 2 A n ) for n 1, A 1,...,A n Borel subsets of S, andt 1 < <t n.clearlythereis so far no loss (nor gain) by looking at the Markov process ( X e t, P ex ), which is called the canonical process. Let us now suppose we are working with the canonical process, and we drop the tildes everywhere. We define the shift operators t :! as follows. t (!) will be an element of and therefore is a continuous function from [0, 1) tos. Define t (!)(s) =!(t + s).

7 6.5. ENLARGING THE FILTRATION 91 Then X s t (!) =X s ( t (!)) = t (!)(s) =!(t + s) =X t+s (!). The shift operator t takes the path of X and chops o and discards the part of the path before time t. We will use expressions like f(x s ) or f(x s ) t = f(x s+t ). t.ifweapplythisto! 2, then (f(x s ) t )(!) =f(x s ( t (!))) = f(x s+t (!)), Even if we are not in this canonical setup, from now on we will suppose there exist shift operators mapping into itself so that X s t = X s+t. 6.5 Enlarging the filtration Throughout the remainder of this chapter we assume that X has paths that are right continuous with left limits. To be more precise, if N = {! : the function t! X t (!) isnotrightcontinuouswithleftlimits}, then we assume N 2F and N is P x -null for every x 2S. Let us first introduce some notation. Define F 00 t = (X s ; s apple t), t 0. (6.13) This is the smallest -field with respect to which each X s is measurable for s apple t. WeletFt 0 be the completion of Ft 00,butweneedtobecarefulwhat we mean by completion here, because we have more than one probability measure present. Let N be the collection of sets that are P x -null for every x 2S.ThusN 2N if (P x ) (N) =0foreachx 2S,where(P x ) is the outer probability corresponding to P x.theouterprobability(p x ) is defined by (P x ) (S) =inf{p x (B) :A B,B 2F}. Let F 0 t = (F 00 t [N). (6.14)

8 92 CHAPTER 6. MARKOV PROCESSES Finally, let F t = F 0 t+ = \ ">0 F 0 t+". (6.15) We call {F t } the minimal augmented filtration generated by X. The reason for worrying about which filtrations to use is that {Ft 00 } is too small to include many interesting sets (such as those arising in the law of the iterated logarithm, for example), while if the filtration is too large, the Markov property will not hold for that filtration. The filtration matters when defining a Markov process; see Definition 6.1(3). We will make the following assumption. Assumption 6.3 Suppose P t f is continuous on S whenever f is bounded and continuous on S. Markov processes satisfying Assumption 6.3 are called Feller processes or weak Feller processes. IfP t f is continuous whenever f is bounded and Borel measurable, then the Markov process is said to be a strong Feller process. One can show that under Assumption 6.3 we have P x (X s+t 2 A F s )=P Xs (X t 2 A), P x a.s. 6.6 The Markov property We start with the Markov property: E x [f(x s+t ) F s ]=E Xs [f(x t )], P x a.s. (6.16) Since f(x s+t )=f(x t ) s,ifwewritey for the random variable f(x t ), we have E x [Y s F s ]=E Xs Y, P x a.s. (6.17) We wish to generalize this to other random variables Y. Proposition 6.4 Let (X t, P x ) be a Markov process and suppose (6.16) holds. Suppose Y = Q n i=1 f i(x ti s), where the f i are bounded, Borel measurable, and s apple t 1 apple...apple t n. Then (6.17) holds.

9 6.7. STRONG MARKOV PROPERTY 93 Proof. We will prove this by induction on n. The case n =1is(6.16),so we suppose the equality holds for n and prove it for n +1. Let V = Q n+1 j=2 f j(x tj t 1 )andh(y) =E y V.Bytheinductionhypothesis, h n+1 Y i E x f j (X tj ) F s j=1 = E x h E x [V t1 F t1 ]f 1 (X t1 ) F s i = E x h (E Xt 1 V )f1 (X t1 ) F s i = E x [(hf 1 )(X t1 ) F s ]. By (6.16) this is E Xs [(hf 1 )(X t1 s)]. For any y, E y [(hf 1 )(X t1 s)] = E y [(E X t 1 s V )f 1 (X t1 s)] i = E he y y [V t1 s F t1 s]f 1 (X t1 s) = E y [(V t1 s)f 1 (X t1 s)]. If we replace V by its definition, replace y by X s,andusethedefinitionof t1 s,wegetthedesiredequalityforn +1andhencetheinductionstep. We now come to the general version of the Markov property. As usual, F 1 = _ t 0 F t. The expression Y t for general Y may seem puzzling at first. Theorem 6.5 Let (X t, P x ) be a Markov process and suppose (6.16) holds. Suppose Y is bounded and measurable with respect to F 1. Then E x [Y s F s ]=E Xs Y, P x a.s. (6.18) The proof follows from the previous proposition by a monotone class argument. 6.7 Strong Markov property Given a stopping time T, recall that the -field of events known up to time T is defined to be F T = A 2F 1 : A \ (T apple t) 2F t for all t>0

10 94 CHAPTER 6. MARKOV PROCESSES We define T by T (!)(t) =!(T (!) +t). Thus, for example, X t T (!) = X T (!)+t (!) andx T (!) =X T (!) (!). Now we can state the strong Markov property. Suppose (X t, P x )isamarkovprocesswithrespectto{f t }. The strong Markov property is said to hold if whenever T is a finite stopping time and Y is bounded and measurable with respect to F 1,then E x [Y T F T ]=E X T Y, P x a.s. Recall that we are restricting our attention to Markov processes whose paths are right continuous with left limits. If we have a Markov process (X t, P x ) whose paths are right continuous with left limits, which has shift operators { t },andwhichsatisfiestheconclusionoftheorem6.6,whether or not Assumption 6.3 holds, then we say that (X t, P x )isastrong Markov process. AstrongMarkovprocessissaidtobequasi-left continuous if X Tn! X T,a.s.,on{T <1} whenever T n are stopping times increasing up to T. Unlike in the definition of predictable stopping times, we are not requiring the T n to be strictly less than T.AHunt process is a strong Markov process that is quasi-left continuous. Quasi-left continuity does not imply left continuity; consider the Poisson process.

Let (Ω, F) be a measureable space. A filtration in discrete time is a sequence of. F s F t

2.2 Filtrations Let (Ω, F) be a measureable space. A filtration in discrete time is a sequence of σ algebras {F t } such that F t F and F t F t+1 for all t = 0, 1,.... In continuous time, the second condition