Lecture 6 Classification of states We have shown that all states of an irreducible countable state Markov chain must of the same tye. This gives rise to the following classification. Definition. [Classification of irreducible countable state Markov chains] An irreducible Markov chain is called transient, res. null recurrent or ositive recurrent, if all its states are transient, res. null recurrent or ositive recurrent. Recall that τ y := inf{n : X n = y} and ρ xy = P x (τ y < ). The restriction to irreducible Markov chains is artly justified by the following result. Theorem.2 [Decomosition theorem] Let R = {x S : ρ xx = } be the set of recurrent states of a countable state Markov chain. Then R can be decomosed as the disjoint union of R i, each of which is irreducible and closed in the sense that if x R i and ρ xy > 0, then y R i. For each i, the states in R i are either all null recurrent or all ositive recurrent. Proof. If x R and ρ xy > 0, then we have shown that y R and ρ yx =. If ρ xy > 0 and ρ yz > 0, then also ρ xz ρ xy ρ yz > 0. Therefore the relation x y for x and y with ρ xy > 0 defines an equivalence relation on R. The equivalence classes {R i } are what we seek. Restricted to each R i, we have an irreducible Markov chain whose states are all of the same tye. Examle.3 The symmetric simle random walk on Z is recurrent, while the asymmetric simle random walk on Z is transient. 2 Recurrence/transience, harmonic functions and martingales We have shown that the Green s function, G(x, y) = n=0 P x(x n = y), of an irreducible countable state Markov chain (X n ) n 0 is finite if and only if the chain is transient. This rovides one way of determining the transience or recurrence of an irreducible Markov chain. As an examle, we consider the simle symmetric random walk on Z d defined by X 0 = x Z d and X n = X 0 + n i= ξ i, where ξ i are i.i.d. with P(ξ = e) = 2d for each of the 2d unit vectors e Z d. Theorem 2. [Simle symmetric random walk on Z d ] The simle symmetric random walk on Z d is recurrent in dimensions d =, 2, and transient in dimensions d 3. Proof. We will comute the Green s function for the simle symmetric random walk (X n ) n 0 on Z d using Fourier transform. Clearly X is an irreducible Markov chain on Z d. Since all states are of the same tye, we may assume X 0 = 0, the origin. By translation invariance,
we can let n (x, y) := n (y x) denote the n-ste transition robability of the random walk. Then for x Z d, n (x) = (x), x,,x n Z d (x ) (x 2 x ) (x x n ) = n where n denotes the n-fold convolution of. For any k [ π, π] d, n has Fourier transform ˆ n (k) = x Z d e i<k,x> n (x) = φ n (k), where φ(k) = d d i= cos k i is the Fourier transform of. By Fourier inversion, n (0) = (2π) d φ n (k)dk, [ π,π] d and hence G(0, 0) = n (0) = n=0 n=0 (2π) d φ n (k)dk = [ π,π] d (2π) d [ π,π] d φ(k) dk. Note that φ(k) = if and only if k = 0, and φ(k) = k 2 2d + o( k 2 ) where k denotes the Euclidean norm. For k > ɛ and k [ π, π] d, φ(k) is uniformly bounded. Therefore to determine whether G(0, 0) is finite, we can relace φ(k) by its Taylor exansion at 0, and consider 2d (2π) d k 2 dk = 2d ɛ (2π) d r d 3 dr, k ɛ which is finite if and only if d 3. Therefore G(0, 0) < and the simle symmetric random walk is transient if and only if d 3. 0 Exercise 2.2 Prove that if the random walk increments ξ i = X i X i, i N, are i.i.d. with E[ξ i ] = 0 and E[ ξ i 2 ] <, then the walk on Z d is also recurrent in dimensions d = and 2, and transient in dimensions d 3. On the other hand, if E[ξ i ] 0, then the walk is always transient regardless of the dimension. In general, it is not ossible to comute exlicitly the Green s function of a Markov chain. An alternative aroach is by finding suitable non-negative (suer) martingales. Let us first illustrate this technique by a concrete examle. Examle 2.3 [Recurrence of a simle symmetric random walk on Z] Let X n = X 0 + n i= ξ i be a simle symmetric random walk on Z where ξ i are i.i.d. with P(ξ i = ±) = 2. To rove recurrence, it suffices to show that P (τ 0 < ) =, where P denotes robability law for X with X 0 =, and τ 0 = inf{n : X n = 0}. Note that X n is a martingale, and X n τ0 τ L is a bounded martingale for any L N. Since τ 0 τ L < almost surely, by the otional stoing theorem, X 0 = = E [X τ0 τ L ] = LP (τ L < τ 0 ), and hence P (τ 0 < τ L ) = L. Let L, we deduce that P (τ 0 < ) =. The crucial fact we use here is that f(x n ), where f(x) = x, is a martingale, and f is non-negative on [0, ) and increases to at. Note that the roof still goes through if f(x n ) is a suer-martingale instead of a martingale. 2
Examle 2.4 [Transience of an asymmetric simle random walk on Z] Let X n = X 0 + n i= ξ i where ξ i are i.i.d. with P(ξ i = ) = P(ξ i = ) = for some > 2. To rove transience, it suffices to show that P (τ 0 < ) <. Note that f(x n ) := ( )Xn is a martingale. For X 0 =, τ 0 τ L < almost surely for any L N, therefore, [( ) Xτ0 ] ( τ L ) LP = E = P (τ 0 < τ L ) + (τ L < τ 0 ) P (τ 0 < τ L ). Let L, we deduce that P (τ 0 < ) <, thus roving transience. The crucial fact we use here is that f(x n ), where f(x) = ( )x, is a non-negative suer-martingale, and f(0) > f(). We now generalize the above roof aroach to general irreducible Markov chains. Definition 2.5 [Harmonic functions for a Markov chain] Let X be a time-homogeneous irreducible Markov chain with countable state sace S and one-ste transition robability matrix Π(x, y). A function f : S R is said to be harmonic for X at x S if (Πf)(x) = y S Π(x, y)f(y) = f(x), (2.) and sub-harmonic, res. suer-harmonic, at x if f(x) (Πf)(x), res. f(x) (Πf)(x). If f is (sub/suer)-harmonic at all x S, then f is said to be (sub/suer)-harmonic for X. Lemma 2.6 If f is a harmonic function for an irreducible Markov chain X, then f(x n ) is a martingale. If f is sub/suer-harmonic, then f(x n ) is a sub/suer-martingale. Proof. Let (F n ) n 0 be the canonical filtration generated by (X n ) n 0. By the Markov roerty, E[f(X n+ ) F n ] = E[f(X n+ ) X n ] = y S Π(X n, y)f(y) = f(x n ). Therefore f(x n ) is a martingale. The case when f is sub/suer-harmonic is similar. Theorem 2.7 [Sufficient condition for recurrence] Let X be an irreducible Markov chain with countable state sace S. If there exists a finite set F S and a function φ : S [0, ) such that φ is suer-harmonic for X at all x S\F, and φ(x) as x in the sense that the level set {x S : φ(x) M} is finite for all M > 0, then X is recurrent. Proof. Let the Markov chain start from x 0 S\F. Let τ F = inf{n : X n F }, and let τ M = inf{n : φ(x n ) > M} for any M > φ(x 0 ). By our assumtion on φ, φ(x n τf τ M ) is a suer-martingale. Therefore φ(x 0 ) E[φ(X n τf τ M )] MP(τ M < n τ F ). (2.2) Note that because {x : φ(x) M} is a finite set, by the irreducibility of the Markov chain, τ M < almost surely. Sending n in (2.2) then yields P(τ M < τ F ) φ(x 0 )/M, and hence P(τ F < τ M < ) φ(x 0) M. Sending M then gives P(τ F < ) =. Therefore the set F is recurrent in the sense that X will return to F infinitely often almost surely. In articular, y F G(x 0, y) =. Since F is a finite set, G(x 0, y) = for some y F, which imlies that X is recurrent. 3
Theorem 2.8 [Necessary and sufficient condition for transience] Let X be an irreducible Markov chain with countable state sace S. A necessary and sufficient condition for X to be transient is the existence of a non-constant, non-negative suer-harmonic function φ. Proof. Suose that X is recurrent and φ is a non-negative non-constant suer-harmonic function. We will derive a contradiction. Let x, y S with φ(x) < φ(y). Let the Markov chain start at x, whose law we denote by P x. Let τ y = inf{n : X n = y}. Then τ y is a stoing time, and φ(x n τy ) is a suer-martingale. In articular, φ(x) E x [φ(x n τy )] φ(y)p x (τ y n). By recurrence, τ y < almost surely. Therefore sending n leads to φ(x) φ(y), which is a contradiction. For the converse, we claim that if X is transient, then there exists a non-negative nonconstant suer-harmonic function. Fix an x 0 S. We claim that φ(x) = P x ( τ x0 < ) is a suer-harmonic function, where τ x0 = inf{n 0 : X n = x 0 }. Note that φ [0, ], φ(x 0 ) =, and by transience, there exists y S with Π(x 0, y) > 0 and φ(y) <. By examining the Markov chain starting at x after one ste, we note that, if x = x 0, P x ( τ x0 < ) = Π(x, y)p y ( τ x0 < ), if x x 0. y S Therefore φ is harmonic at x x 0 and suer-harmonic at x 0, and hence φ is a non-negative non-constant suer-harmonic function. We now illustrate Theorems 2.7 and 2.8 by an examle. Examle 2.9 [Birth-death chains] Let X be a time-homogeneous Markov chain on {0,, 2, } with one-ste transition robabilities (i, i+) = i, (i, i ) = q i and (i, i) = r i with q 0 = 0. This is called the birth-death chain. We assume that i > 0 for i 0 and q i > 0 for i so that the chain is irreducible. Let us find a non-negative φ which is harmonic for the birth-death chain at all x N. W.l.o.g., we may assume φ(0) = 0 and φ() =. For φ to be harmonic at each x N, we must have which sets u the recursion relation Therefore and φ(x) = x φ(x + ) + r x φ(x) + q x φ(x ), φ(x + ) φ(x) = q x x (φ(x) φ(x )). φ(x) φ(x ) = φ(x) = + x i x i= i=2 j= By construction, φ is harmonic for X at all x N and is non-negative and monotonically increasing. We claim that the chain is recurrent if and only if φ( ) := lim x φ(x) = + 4 q i i, q j j. i i=2 j= q j j =.
Indeed, if φ( ) =, then Theorem 2.7 alies with F = {0}. If φ( ) <, then φ(x) = φ( ) φ(x) defines a non-negative non-constant suer-harmonic function for X, so that Theorem 2.8 alies. More directly, we note that φ(x n τ0 τ N ) is a martingale for any N N, which by the otional stoing theorem imlies that if the chain starts at X 0 =, then and hence φ() = P (τ 0 < τ N )φ(0) + P (τ N < τ 0 )φ(n), P (τ N < τ 0 ) = φ(n). Sending N, we conclude that 0 is a recurrent state if and only if φ( ) =. A secial case of the birth-death chain is an asymmetric simle random walk where i = q i = for all i. Then x ( ) i. φ(x) = i= Therefore φ( ) = if and only if 2. If > 2, then P (τ 0 = ) = φ( ) = 2. 3 Dirichlet roblem and Poisson equation We now elaborate more on the connection between Markov chains and otential theory. The first link concerns the so-called the Dirichlet roblem. Assume that X is an irreducible recurrent Markov chain with transition matrix Π, and let A be a given set. We are interested in finding bounded solutions of the Dirichlet boundary value roblem (Π I)f(x) = 0 for x A, f(x) = g(x) for x / A, (3.3) where g is a given bounded function on A c. Observe that such an f is harmonic for X at all x A. Therefore f(x n τa c ) is a bounded martingale, where τ A c = inf{n 0 : X n / A} is finite a.s. by the recurrence of X. By the otional stoing theorem, f(x) = E x [f(x τa c )]. (3.4) Therefore (3.3) has a unique bounded solution, which is given by (3.4). If we let g(x) = {x B} for some B A c, then the solution to (3.3) is f(x) = P x (X τa c B), which is the robability that the Markov chain starting from x enters the set A c by entering the subset B. The collection P x (X τa c = y) for y A c defines a robability measure on A c, called the harmonic measure. Thus comuting the hitting robabilities for a Markov chain is equivalent to solving a Dirichlet roblem with suitable boundary conditions. When X is transient, bounded solutions to (3.3) are not unique in general, as can be seen from the fact that when g = on A c, taken to be a finite set, both f and f(x) = P x (τ A c < ) are bounded solutions to (3.3). However, the second solution can be easily seen to be the minimal non-negative solution, using the fact that any solution induces a martingale. 5
Another general way of generating martingales for Markov chains is to take any bounded function f on the state sace S, and then extract a martingale from f(x n ) by subtracting a suitable redictable sequence. More recisely, let Then n = E[f(X n ) f(x n ) X n ] = (Π I)f(X n ). M n = f(x n ) f(x 0 ) n n = f(x n ) f(x 0 ) i= n (Π I)f(X i ) (3.5) is a martingale. If we want to comute the exected hitting time of a set A, i.e., E x [τ A ], then it suffices to solve the equation (Π I)f(x) = for x A, f(x) = 0 for x / A. i= (3.6) This is sometimes called Dynkin s equation. Indeed, if f is a bounded solution of (3.6), then by (3.5), f(x n τa c ) f(x 0 ) + n τ A c is a martingale. If the Markov chain is recurrent, then it is easy to see that E x [τ A c] = f(x). More generally, comuting E x [ τa c i=0 g(x i ) ] reduces to solving (3.6) with (Π I)f = g on A, which is called the Poisson equation. Of course to establish this corresondence using martingales and the otional stoing theorem, one has to be careful with (uniform) integrability issues. Analogues of (3.5) and (3.6) also exist for continuous time Markov rocesses, with (Π I) relaced by the generator of the Markov rocess, the most well known being the Lalacian, which is the generator of a Brownian motion. We can even find robabilistic reresentations for solutions of equations of the following general form: (Π I + V )f(x) = g(x) for x A, (3.7) f(x) = h(x) for x / A, where to avoid integrability issues, let us assume that A is a bounded set, V : A R satisfies max x A V (x) <, h : A c R is a bounded function, and g = 0 on A c. Indeed, the solution of the equation (Π I + V )f(x) = 0 for x A, f(x) = h(x) for x / A (3.8) admits the reresentation f(x) = h(x) if x A c [ and f(x) = E x h(xτa c ) τ A c i=0 V (X i )], using the observation that f(x n ) n i=0 V (X i ) is a martingale if f solves (3.8). This is called the Feynman-Kac formula. On the other hand, the solution of the equation (Π I + V )f(x) = g(x) for x A, f(x) = 0 for x / A [ admits the reresentation f(x) = 0 when x A c τa c, and f(x) = E x i=0 g(x i ) i j=0 when x A, observing that if f solves (3.9) then n f(x n ) i=0 n V (X i ) + i=0 i g(x i ) j=0 V (X j ) (3.9) ] V (X j ) is a martingale. Finally note that by linearity, any solution to (3.7) can be written as the sum of a solution to (3.8) and another solution to (3.9). 6