Some classical results on stationary distributions of continuous time Markov processes

Transcription

1 Some classical results on stationary distributions of continuous time Markov processes Chris Janjigian March 25, 24 These presentation notes are for my talk in the graduate probability seminar at UW Madison in Spring 24. The topics covered are some classical results on the existence, uniqueness, and regularity of stationary distributions for Markov processes on nice state spaces. These notes are primarily based on lecture notes by Martin Hairer [5] [6]. The literature on each of these three subjects is massive, so I make no attempt at completeness. The big picture If we ignore regularity of the Markov process, it isn t too hard to guess what the relationship between topological properties and measure theoretic properties of a Markov process should be. Compactness properties give existence of the stationary distribution and irreducibility properties along with an analogue of aperiodicity give uniqueness. Working on a general state space makes the technical conditions more difficult to satisfy and provide some interesting counterexamples. Finite dimensional discrete time Markov Chains As is often the case, we can get some good intuition about the behavior of nicely behaved Markov processes by considering the finite dimensional, discrete time case. We start by recalling the topological properties which give results on existence and uniqueness of stationary distributions.

2 Existence Call the Markov transition matrix P and fix any initial distribution µ. If the state space is finite we equip it with the discrete topology, so that the sequence of measures µ P n is tight. We would like to say that µ P n µ for some µ, but of course this is not true in general. As is often the case in analysis a little averaging helps here, so instead consider µ N = N N n= µ P n, which is also clearly tight. Take a subsequence N k along which µ Nk µ and consider a test function f µ P µ, f = lim N k = lim N k = lim N k ( N k N k n= µ P n ) P, f µ (P Nk+ P ), f N k 2 f N k ( N k N k n= µ P n ) A more standard proof of this result would be to explicitly write down a stationary distribution. If x is a recurrent state for the Markov Chain, then we can define a stationary measure by fixing x and defining the mass of y to be the expected number of visits to y before returning to x, µ x (y) = E x [ T x i= {Xi =y}]. There is a generalization of this for recurrent diffusions. See Lemma 2. and Theorem 2. in [7]. Remark. It is actually not hard to show that this Cesaró average N N n= P n converges without taking subsequences and the limit of N N n= p(n) (x, y) is the stationary distribution that we would have written down explicitly., f Uniqueness I will not go into the standard uniqueness proof for Markov chains, but the basic idea is that if a stationary measure exists then uniqueness follows from irreducibility. The additional assumption of aperiodicity implies that the transition probabilities actually converge to the invariant distribution. When we move away from the discrete setting the analogue of aperiodicity (mixing) becomes relevant again. Ergodicity corresponds to the chain being irreducible and if the chain is reducible, then we can just restrict our attention to the irreducible components. The difference is that on a discrete space, we can throw away states that have probability 2

3 zero until we get to a chain where all states have positive probability. In a continuous setting, that no longer works, so we need to introduce stronger assumptions. Preliminaries The study of Markov processes is a little overrun with notation, so to spare time, I am going to introduce as little as possible. There are some technical issues in the notes that I based this on. I will assume away as many of them as possible. The gist here is that we want an abstract way to handle the operator f( ) E [f(x(t))] where X is a Markov process. Part of the issue here is that there really isn t much hope of getting a sharp characterization of when this map will make sense without looking at a particular model. A Markov operator over a Polish E space is a bounded linear operator P : B b (E) B b (E) so that. P = 2. If ϕ then P ϕ 3. P ϕ ϕ 4. If ϕ n B b (E) is a uniformly bounded sequence with ϕ n ϕ B b (E) pointwise, then P ϕ n P ϕ pointwise. Remark 2. This definition deviates from [5] in two ways. First, I include the contraction property as I could not see how to prove it without some extra regularity. Second, property 4.) has the added assumption of uniform boundedness, since I do not think that the usual definition of Markov transition probabilities satisfies the corresponding definition in [5]. As a comment, this is equivalent to the construction of a Markov transition kernel via the identification P ( A )(x) = P (x, A). We may define the action of a Markov operator P on a measure µ via P µ(a) = E P ( A)(x)µ(dx). By density this extends to P µ(ϕ) = E P ϕ(x)µ(dx). A family of Markov operators indexed by time, (P t ) t is a Markov semi-group if P t P s = P t+s and for all ϕ B b (E), the map (t, x) P t ϕ(x) is Borel measurable in the prodcut sigma algebra. Remark 3. [5] does not include the measurability requirement, but I cannot see how to prove that the integrals in the first proof of the next section make sense without it. 3

4 We can verify that the usual definition of Markov transition probabilities agrees with this one via the identifiction E x [f(x(t))] = P t (f)(x) And to see that it is a semigroup, observe that by the Markov property P t+s (f)(x) = E x [f(x(t + s))] = E x [E[f(X(t + s)) F s ]] = E x [E Xs f(x(t))] = E x [P t (f)(x(s))] = P s P t (f)(x) We will say that a Markov semigroup (P t ) t is Feller if t and ϕ C b (E) we have P t ϕ C b (E). Finally, a probability measure µ is stationary for the semi-group (P t ) t if P t µ = µ for all t. Existence Necessary and sufficient condition in a Polish space On sufficiently nice cases there is an exact characterization of when a stationary distribution exists. As in the finite dimensional discrete time case, the important fact is compactness. We will essentially repeat the same proof as before in this case. Theorem 4. (Kyrlov-Bogolioubov)[5] Let (P t ) t be a Feller-Markov semigroup over a Polish space E. Then there exists an invariant distribution µ M (E) if and only if there exists µ M (E) such that {P (t)µ } t is tight. Proof. Necessity follows from the definition of stationarity and the fact that in a Polish space, all probability measures are tight. For sufficiency, suppose that µ M (E) such that {P (t)µ } t is tight. Then µ t (A) = t t (P s µ )( A )ds 4

5 is well defined by our measurability assumptions and tight by taking the same compact set. Take any weak limit µ with µ tn ϕ C b (E) (P t µ ) (ϕ) µ (ϕ) = µ (P t ϕ) µ (ϕ) µ and t n. We claim that µ is invariant. Fix = lim µ t n (P t ϕ) µ tn (ϕ) t n tn = lim P s P t ϕ(x)µ (dx)ds t n t n tn = lim P t+s ϕ(x)µ (dx)ds t n t n t+tn = lim P s ϕ(x)µ (dx)ds t n t n t t+tn = lim P s ϕ(x)µ (dx)ds t n t n lim n 2t ϕ t n = We needed Feller continuity in the second line. t tn tn tn t P s ϕ(x)µ(dx)ds P s ϕ(x)µ(dx)ds P s ϕ(x)µ(dx)ds P s ϕ(x)µ(dx)ds Corollary 5. If E is a compact Polish space and (P t ) t is a Feller-Markov semi-group, then there exists a stationary distribution for (P t ) t. If E is not compact, it can be non-trivial to check the tightness condition. In such cases, one can either proceed directly and produce a measure satisfying the tightness conditions or work with some additional structure of the model. Example: reflected Brownian motion with drift It is sometimes possible to prove existence directly from the model without recourse to the abstract construction above. The easiest way is to have the exact distribution of P (t)µ for some measure µ. An example where this is possible is the reflected Brownian Motion with drift. Let µ < and B t be standard Brownian motion. Define dβ = µ sign(β)dt + db t and let Z = β. I am not going to prove that this is genuinely a reflected Brownian motion with drift or that such a process exists (though the SDE solution clearly does). One can use a Girsanov change of measure and the fact that this process satisfies a version of Levy s 5

6 distributional laws for the absolute value of Brownian motion[4] that the distribution of this process started from zero is P (P (t)δ z) = P (Z(t) z) ( ) ( ) z µt z µt = Φ e 2µz Φ t t e 2µz It follows from this that the exponential distribution with parameter 2µ is invariant. Diffusions One can also use the generator of the diffusion directly to show existence of an invariant measure µ with P (t)ϕ(x)µ(dx) = ϕ(x)µ(dx). Defining the generator to be the pair P (t) (L, D(L)), where L = lim t and D(L) is the set of functions in our domain for t which this limit makes sense. One can show the previous equation holds if and only if Lϕ(x)µ(dx) = for all ϕ for which the limit makes sense. It can be shown that if the multidimensional diffusion X satisfies dx = b(x)dt + σ(x)db t for Lipschitz b and σ, the generator of X is given by L = i,j d a ij (x) ij + i d b i (x) i where ( 2 σσt ) ij = a ij. Gibbs measures for gradient systems The next result generalizes quite a bit. It is an exact formula for the stationary distribution of diffusions of the form dx = Φ(X)dt + 2 β db t Theorem 6. Suppose that L is the generator of a Markov process and is of the form L = β Φ where Φ C (R d ) and b C(R d )) and suppose that e βφ(x) is integrable. Then µ = Z β e βφ(x) dx is stationary. 6

7 Proof. (mostly) It is necessary and sufficient to show that µ(dx) = Z β e βφ(x) dx satisfies Lϕ(x)µ(dx) = for all ϕ D(L). It is true, but not obvious that it suffices to check this R d condition on Cc (R d )[2]. The key observation to this proof is that Lϕ = β eβφ div ( e βφ ϕ ) Since we assumed that e β L we know that Z β < and since ϕ Cc that β eβφ div ( e βφ ϕ ) is integrable with respect to µ. Then Lϕdµ = div ( e βφ(x) ϕ(x) ) dx R βz d β R d = (R d ), we know where the last equality follows from the divergence theorem. Lyapunov functions for diffusions Theorem 7. (Khasminskii, 96)[8] Suppose that L is of the form above, then a sufficient condition for the existence of an invariant probability measure is that there exists a C 2 (R d ) (Lyapunov) function V (x) with the proprties lim V (x) = x lim LV (x) = x Remark 8. In this setting there are some domain issues to deal with, which I ignore. Proof. (sketch based on [2]) For simplicity, we will assume that transition densities exist. The idea of this proof is to show that P (t)v (x) V (x) T T (without loss of generality) that V (x) then this implies that T P (t)lv (x)dt. If we assume V (x) T T T P (t)lv (x)dt. Recalling that P (t)lv (x) = p(t, x, y)lv (y)dy R d we can observe that LV M uniformly for some M. Moreover, for each ɛ > there is a compact set K ɛ so that LV (x). Then recall that ɛ T p(t, x, y)lv (y)dydt = T p(t, x, y)lv (y)dydt T R T d K ɛ + T p(t, x, y)lv (y)dydt T Kɛ c M T p(t, x, y)dydt ɛ T 7 K c ɛ

8 It then follows that T T K c ɛ which implies that the measures A T ( p(t, x, y)dydt ɛ M + V (x) ) T T A p(t, x, y)dydt are tight. Remark 9. Using Lyapunov functions to control recurrence properties (which essentially gives you a stationary measure) goes back to at least the 95s, when Foster [3] showed that the existence of a discrete Lyapunov function implied recurrence properties of a Markov chain. Examples Non-compact with no stationary distribution The classical examples of a continuous time process with no stationary distribution is standard Brownian motion. This is an explicit computation using the known semigroup. Compact state space, but no stationary distribution Such a process has to look odd, since it cannot be Feller. This example comes from Hairer [5]. Let our state space be [, ] with transition probabilities P (x, ) = δ x 2 {x>} + δ {x=}. We can see that µ({}) = for any invariant measure, from which it follows that µ({ 2, ]}) = and then inductively that µ([, ]) =. Lyapunov function for a diffusion Consider the Ornstein-Uhlenbeck process given by then the generator of X is dx t = X t dt + 2dB t L = x d dx + d2 dx 2 Take a smooth function V (x) which is equal to e x near x =. Then clearly lim x V (x) = and lim x LV (x) =. Therefore a stationary distribution exists. Notice that this (being one dimensional with a continuous drift) is also a gradient diffusion and therefore it has a stationary measure proportional to e xdx = e 2 x2. 8

9 Uniquness Ergodic theorems Proofs of uniqueness for invariant measures of Markov chains and processes tend to rely quite heavily on ergodic theorems. The key fact is the observation that the set of invariant probability measures for a dynamical system forms a convex set and that the extreme points are the ergodic measures. The proof of this result is not terribly hard, but it is tedious and I will omit it. I will call this result the structure theorem. We need to fit Markov chains into the framework of dynamical systems to use the ergodic theory (and define what it means to be ergodic). Definition. Let T be N, Z, R or R +. A dynamical system on a Polish space E is a collection of maps {θ t } t T with θ t : E E and the properties. (t, ω) θ t (ω) is jointly measurable 2. θ t θ s = θ t+s A measure µ is said to be invariant for θ t if the pushforward θ t µ = µ for all t. The invariant sigma algebra of a dynamical system is I(θ) = {A B E : θ t (A) = A t T }. We recall Birkhoff s ergodic theorem Theorem. (Birkhoff) Let θ t be a dynamical system and let µ be invariant for θ t, then for each f L (dµ) and µ almost every x E lim N N N f(θ n (x)) = E µ [f I(θ)](x) n= This theorem immediately implies the main result that we can use to prove the uniqueness of invariant measures. Notice that there is a unique stationary distribution if and only if the set of invariant probability measures is a singleton. In this case, the measure must be ergodic, by the structure theorem. We call a measure ergodic for θ t if µ(a) {, } for all A I(θ). If µ is ergodic for θ, then for each f and µ almost every x we have E µ [f I(θ)](x) = E µ [f] Corollary 2. If µ and µ 2 are both ergodic for θ t, then they are mutually singular. 9

10 Proof. If µ µ 2 there is a bounded measurable function f with E µ [f] E µ2 [f]. But for each i, we have µ i there is a set Ω i with µ i (Ω i ) = and for all x Ω i lim N N N f(θ n (x)) = E µi [f] from which it follows Ω Ω 2 =, so that µ and µ 2 are singular. n= Our goal in proving uniqueness is to take this measure theoretic result and turn it into a pointwise result. In paricular, we want to show that the sets of measure one in the previous results actually have a point in common. This is difficult, since one cannot generally infer pointwise information from measure theoretic information. The main idea for the general uniqueness result will be to show that a little extra regularity on our Markov processes implies that this measure theoretic condition extends to a topological condition. Markov processes as dynamical systems To use the theory in the previous section, we need to construct a dynamical system with the property that being invariant for this dynamical system is the same as being stationary for the Markov process. Before doing this in generality, let s start with an example Lemma 3. (SLLN) Suppose that X i is an iid sequence of real valued random variables and E[ X i ] <. Then n n i= X i E[X ]. Proof. Then since R is Polish we may apply Kolmogorov s extension theorem to realize this sequence on a single probability space ( R N, B N R, P N) where P law of X. If f(x) = x is the projection to the first coordinate, then E P N[ f ] < by hypothesis. Let θ i be the shift map: if x = (x, x 2,... ) then θ i (x) = (x i+, x i+2,... ). Then for each x R N n n x i = n f(θ i x) n i= i= E[f I(θ)](x) but the invariant σ- algebra of θ is a subset of the tail σ-algebra, which is trivial by the Kolmogorov law, so for P RN almost every x R N we have E P R N [f I(θ)](x) = E P RN [f] = E[X ]. The key idea behind this proof is to realize the entire stochastic process on a single product probability and then have our dynamical system be the shift map.

11 Formally, given a Markov semigroup P t over a Polish space E and an invariant measure for P t, we can construct the probability space (E T, B T, P µ ) where P µ is the measure obtained from the extension theorem applied to the finite dimensional distributions on t < t 2 <... t n given on cylinder sets by the following expression, where P µ (A t A tn ) really denotes the measure of the set where all but the t i projections are all of E P µ (A t A tn ) =... P tn tn (x n, dx n )... P t2 t (x, dx 2 )µ(dx ) A t A tn These specifications are consistent by the Markov property, so generate a measure and it is not hard to see that since µ is invariant P µ is invariant for the shift map θ t x(s) = x(t + s). This now lets us define ergodicity for Markov processes: µ is ergodic for P t if and only if P µ is ergodic for θ t. There is a bit of measure theory that needs to be done to explain how this translates to the original space. It is handled in [5]. Return to reflected Brownian motion There is some general theory for showing uniqueness of stationary distributions, but first I want to go over an example where we can show it directly. This example also serves to highlight the connection between regularity and uniqueness. The goal here will be to show that any invariant measure is equivalent to a single reference measure (for example, the Lebesgue measure). This will imply that any two invariant measures must also be equivalent and therefore they are equal. Recall that we have the exact formula ( ) ( ) z µt z µt P (Z(t) z) = Φ e 2µz Φ t t A similar formula exists for general x and it is not hard to see from this that P x (Z(t) A) = iff m(a) =, where m is Lebesgue. In particular then, if π is an invariant measure, then π(a) = P x (Z() A)π(dx) R + Since for every x R +, P x (Z() A) = if and only if m(a) =, the above integrand is zero if and only if m(a) =. It follows that every invariant measure is equivalent to the Lebesgue measure in the sense of absolute continuity and therefore invariant measures are unique.

12 The strong Feller property Typically, if we want to exchange a measure theoretic property which holds on almost every point of a set for one which holds on the entire set, some kind of continuity is necessary. A strong, but not completely unreasonable, condition is the strong Markov property: given a Feller Markov semigroup {P } t, we say that {P } t is strong Feller if P t (B b (E)) C b (E) for all t. Lemma 4. If P t is a strong Feller semigroup over a Polish space E and µ and µ 2 are mutually singular invariant measures for P t then µ and µ 2 have disjoint topological supports. Proof. Let A be a set with µ (A) = and µ 2 (A) =. Then since we have (by invariance) P E t( A )(x)µ i (dx) = µ i (A), we must have P ( A )(x) = for µ almost every x and P ( A )(x) = for µ 2 almost every x. But then by continuity since P t ( A ) is constant of µ i almost everywhere, it is constant with the same value on the topological support of µ i. Thus the topological supports of the µ i must then be disjoint. With this result in hand, we would like nice conditions that allow us to check the strong Feller property and that every x belongs to the topological support of each invariant measure. The most general way I found to check the strong Feller property is actually to check regularity, so I will postpone that. Ideally, we would like to show the first condition by showing that for each x E, there is a t so that for any initial condition we can reach x by time t with positive probability. For discrete models, this argument can often be carried out directly. For models with a continuous state space, it is more subtle. Stroock Varadhan support theorem for diffusions Suppose we are given a stochastic differential equation in Stratonovich form with smooth coefficients dx t = f (X)dt + m f i (X) dω i (t) i= then we have a theorem of Stroock and Varadhan Theorem 5. (Stroock and Varadhan)[5] Given a diffusion of the form above, then the support of P t (x, ) is the closure of the set of points in R d that can be reached in time t by 2

13 solutions to the control problem ẋ(t) = f (x(t)) + x() = x m f i (x(t))u i (t) i= where the u i are arbitrary smooth functions. Proof. (extremely rough sketch; see [] for an actual proof) This result is hard. The idea behind the proof is to use the Cameron Martin theorem to find the support of Brownian motion in an appropriate space of continuous functions, then to push this result forward to the differential equations. The essential difficulty in this scheme lies in the fact that the Ito solution map is extremely discontinuous. The appropriate theory for this version of the proof is called rough paths. Recall that if h lies in the Cameron Martin space W,2 = {h L 2 (, T ) : k L 2 (, T ) with h(t) = t k(s)ds} then the Weiner measure P is absolutely continuous with respect to the measure P h given by P h (A) = P (A h ) where A h = {x h : x A}. Call the support of B in the space of functions we are working on F. From the version of Cameron Martin for this space, we find that if x F, then x h F for each h W,2. If we take a smooth sequence x n converging to x, then we see that F. From this, it follows that C is also in F. The reverse inclusion uses properties of the topology, which I am not introducing. For sufficiently regular h, consider the ordinary differential equation dx h t = f (x h t ) + m f i (x h t )dh i (t) i= Recall that Brownian motion is the uniform limit of piecewise linear functions. If we call these approximating functions B n, then in an appropriate sense solutions to the above equations for x Bn will converge to the solution for x B. Since these functions are all in W,2, we can see that the support of X lies in the support of {x h : h W,2 }. The reverse inclusion is nontrivial and again uses the properties of the topology. The rough paths interpretation of these differential equations agrees with the Stratonovich formulation of the SDE, which is where that requirement comes from. We come to the connection between regularity and uniqueness. A common way to check that the strong Feller property holds is to show that not only is P t a map from B b C b, but 3

14 it is actually a map from B b Cb. This result is due to Hörmander, though probabilistic proofs of it were the main reason for the development of the Malliavin calculus. Regularity Unfortunately, regularity is complicated and I will not be able to go into any detail about the proof of this result. It is something that can be checked on a case-by-case basis though. Hörmander s condition Again, we will work with the Stratonovich formulation of the model, since that is the framework one uses in the proof of this result using the Malliavin calculus. Theorem 6. Suppose that dx t = f (X t )dt + m f i (X t ) db t i= and define vector fields V k by V = {f i : i m} and V k+ = V k {[U, f j ] : U V k, j m} where [, ] is the Lie bracket [V, U](x) = (DV )(x)u(x) (DU)(x)V (x). If the V k are bounded and k span{v k (x)} = R d for each x R d, the semigroup admits smooth transition densities and maps measurable functions into smooth functions. Remark 7. We can translate between Ito and Stratonovich equations in one dimension via dx = a(t, X)dt + b(t, X)dB dx = a(t, X) 2 b(t, X) xb(t, X)dt + b(t, X) db and there is a similar result in higher dimensions. Example 8. Ornstein-Uhlenbeck process (note: this is overkill, since the equation is uniformly elliptic) Let Y (t) be the solution to the Ito form SDE dy = Y dt + db t Y () = x 4

15 Then Y t also solves the Stratonovich equation dy = Y dt + db t In the language above, f (x) = x and f (x) =. Then V = {} and V = {} {[, g] : g {, x}}. [, x] = d x d x = and [, ] =. But since the span of is R for all x, dx dx this satisfies Hörmander s condition. References [] F. Baudoin, Rough paths Unpublished lecture notes [2] S. Fornaro, Regularity properties for second order partial differential operators with unbounded coefficients 24 [3] F.G. Foster, On the stochastic matrices associated with certain queuing processes. Ann. Math. Statist. Volume 24, Number 3 (953), [4] S.E. Graversen and A.N. Shiryaev, An Extension of P. Lévy s Distributional Properties to the Case of a Brownian Motion with Drift. Bernoulli, Vol. 6, No. 4 (Aug., 2), pp [5] M. Hairer, Ergodic theory for Stochastic PDEs. Unpublished lecture notes, 28. [6] M. Hairer, On Malliavin s proof of Hörmander s theorem. Unpublished lecture notes, 2. [7] R.Z. Khasminskii, Ergodic Properties of Recurrent Diffusion Processes and Stabilization of the Solution to the Cauchy Problem for Parabolic Equations. Theory Probab. Appl. Volume V, Number 2 (96), [8] R.Z. Khasminskii, Stochastic Stability of Differential Equations Springer-Verlag, Germany, 2nd Edition, 2. 5