MA3H2 Markov Processes and Percolation theory. Stefan Adams

MA3H2 Markov Processes and Percolation theory Stefan Adams 2011, update 13.03.2015

Contents 1 Simple random walk 1 1.1 Nearest neighbour random walk on Z.............. 1 1.2 How often random walkers return to the origin?........ 6 1.3 Transition function........................ 9 1.4 Boundary value problems..................... 15 1.5 Summary............................. 20 2 Markov processes 21 2.1 Poisson process.......................... 23 2.2 Compound Poisson process on Z d................ 25 2.3 Markov processes with bounded rates.............. 28 2.4 The Q-matrix and Kolmogorov s backward equations..... 30 2.5 Jump chain and holding times.................. 41 2.6 Summary - Markov processes.................. 44 3 Examples, hitting times, and long-time behaviour 46 3.1 Birth-and-death process..................... 46 3.2 Hitting times and probabilities. Recurrence and transience.. 48 4 Percolation theory 61 4.1 Introduction............................ 61 4.2 Some basic techniques...................... 69 4.3 Bond percolation in Z 2...................... 69

Preface These notes are from 2011-2015. Stefan Adams 1 Simple random walk 1.1 Nearest neighbour random walk on Z Pick p (0, 1), and suppose that (X n : n N) is a sequence (family) of { 1, +1}-valued, identically distributed Bernoulli random variables with P(X i = 1) = p and P(X i = 1) = 1 p = q for all i N. That is, for any n N and sequence E = (e 1,..., e n ) { 1, 1} n, P(X 1 = e 1,..., X n = e n ) = p N(E) q n N(E), where N(E) = {m: e m = 1} = n+ n m=1 em is the number of 1 s in the 2 sequence E. Imagine a walker moving randomly on the integers Z. The walker starts at a Z and at every integer time n N the walker flips a coin and moves one step to the right if it comes up heads (P({head}) = P(X n = 1) = p) and moves one step to the left if it comes up tails. Denote the position of the walker at time n by S n. The position S n is a random variable, it depends on the outcome of the n flips of the coin. We set S 0 = a and S n = S 0 + n X i. (1.1) Then S = (S n ) n N is often called a nearest neighbour random walk on Z. The random walk is called symmetric if p = q = 1. We may record the motion 2 of the walker as the set {(n, S n ): n N 0 } of Cartesian coordinates of points in the plane (x-axis is the time and y-axis is the position S n ). We write P a for the conditional probability P( S 0 = a) when we set S 0 = a implying P(S 0 = a) = 1. It will be clear from the context which deterministic starting point we consider. Lemma 1.1 (a) The random walk is spatially homogeneous, i.e., P a (S n = j) = P a+b (S n = j + b), j, b, a Z. (b) The random walk is temporally homogeneous, i.e., P(S n = j S 0 = a) = P(S n+m = j S m = a). 1 i=1

(c) Markov property Proof. (b) P(S m+n = j S 0, S 1,..., S m ) = P(S m+n = j S m ), n 0. (a) P a (S n = j) = P a ( n i=1 X i = j a) = P a+b ( n i=1 X i = j a). LHS = P( n X i = j a) = P( i=1 m+n i=m+1 X i = j a) = RHS. (c) If one knows the value of S m, then the distribution of S m+n depends only on the jumps X m+1,..., X m+n, and cannot depend on further information concerning the values of S 0, S 1,..., S m 1. Having that, we get the following stochastic process oriented description of (S n ) n N0 replacing (1.1). Let (S n ) n N0 be a family of random variables S n taking values in Z such that P (S 0 = a) = 1 and { p, if e = 1 P(S n S n 1 = e S 0,..., S n 1 ) = q, if e = 1. (1.2) Markov property: conditional upon the present, the future does not depend on the past. The set of realisations of the walk is the set of sequences S = (s 0, s 1,...) with s 0 = a and s i+1 s i = ±1 for all i N 0, and such a sequence may be thought of as a sample path of the walk, drawn as in figure 1. Let us assume that S 0 = 0 and p = 1. Natural questions to ask are: 2 On the average, how far is the walker from the starting point? What is the probability that at a particular time the walker is at the origin? More generally, what is the probability distribution for the position of the walker? Does the random walker keep returning to the origin or does the walker eventually leave forever? We easily get that E(S n ) = 0 when p = 1. For the average distance from 2 the origin we compute the squared position at time n, i.e., E(S 2 n) = E ( (X 1 + + X n ) 2) = n j=1 E(X 2 j ) + i j E(X i X j ). 2

Now X 2 j = 1 and the independence of the X i s gives E(X i X j ) = E(X i )E(X j ) = 0 whenever i j. Hence, E(S 2 n) = n, and the expected distance from the origin is c n for some constant c > 0. In order to get more detailed information of the random walk at a given time n we consider the set of possible sample paths. The probability that the first n steps of the walk follow a given path S = (s 0, s 1,..., s n ) is p r q l, where r = of steps of S to the right = {i: s i+1 s i = 1} l = of steps of S to the left = {i: s i+1 s i = 1}. Hence, any event for the random walk may be expressed in terms of an appropriate set of paths. P(S n = b) = r M r n(a, b)p r q n r, P(S 0 = a) = 1, where M r n(a, b) is the number of paths (s 0, s 1,..., s n ) with s 0 = a and s n = b having exactly r rightward steps. Note that r + l = n and r l = b a. Hence, If 1 2 r = 1 2 (n + b a) and l = 1 (n b + a). 2 (n + b a) {0, 1,..., n}, then ( ) n P(S n = b) = p 1 2 (n+b a) q 1 2 (n b+a), (1.3) (n + b a) 1 2 and P(S n = b) = 0 otherwise, since there are exactly ( n r) paths with length n having r rightward steps and n r leftward steps. Thus to compute probabilities of certain random walk events we shall count the corresponding set of paths. The following result is an important tool for this counting. Notation: N n (a, b) = of possible paths from (0, a) to (n, b). We denote by N 0 n(a, b) the number of possible paths from (0, a) to (n, b) which touch the origin, i.e., which contain some point (k, 0), 1 k < n. Theorem 1.2 (The reflection principle) If a, b > 0 then N 0 n(a, b) = N n ( a, b). Proof. Each path from (0, a) to (n, b) intersects the x-axis at some earliest point (k, 0). Reflect the segment of the path with times 0 m k in the x-axis to obtain a path joining (0, a) and (n, b) which intersects/touches the x-axis, see figure 2. This operation gives a one-one correspondence between the collections of such paths, and the theorem is proved. 3

Lemma 1.3 ( N n (a, b) = 1 2 ) n. (n + b a) Proof. Choose a path from (0, a) to (n, b) and let α and β be the numbers of positive and negative steps, respectively, in this path. Then α + β = n and α β = b a, so that α = 1 (n+b a). Now the number of such paths is 2 exactly the number of ways picking α positive steps out of n available steps. Hence, ( ) n N n (a, b) =. α Corollary 1.4 (Ballot theorem) If b > 0 then the number of paths from (0, 0) to (n, b) which do not revisit the x-axis (origin) equals b n N n(0, b). Proof. The first step of all such paths is to (1, 1), and so the number of such paths is N n 1 (1, b) Nn 1(1, 0 b) = N n 1 (1, b) N n 1 ( 1, b) ( ) ( n 1 = (n 1 + b 1) Elementary computations give the result. 1 2 1 2 ) n 1. (n 1 + b + 1) What can be deduced from the reflection principle? We first consider the probability that the walk does not revisit its starting point in the first n steps. Theorem 1.5 Let S 0 = 0, b Z, and p (0, 1). Then P(S 1 S 2 S n 0, S n = b) = b n P(S n = b), implying P(S 1 S 2 S n 0) = 1 n E( S n ). Proof. Pick b > 0. The admissible paths for the event in question do not visit the x-axis in the time interval [1, n], and the number of such paths is by the Ballot theorem exactly b N n n(0, b), and each path has 1 (n + b) rightward 2 and 1 (n b) leftward steps. 2 P(S 1 S 2 S n 0, S n = b) = b n N n(0, b)p 1 2 (n+b) q 1 2 (n b) = b n P(S n = b). 4

The case for b < 0 follows similar, and b = 0 is obvious. Surprisingly, the last expression can be used to get the probability that the walk reaches a new maximum at a particular time. Denote by M n = max{s i : 0 i n} the maximum value up to time n (S 0 = 0). Theorem 1.6 (Maximum and hitting time theorem) Let S 0 = 0 and p (0, 1). (a) For r 1 it follows that { P(M n r, S n = b) = P(S n = b) if b r ( q p )r b P(S n = 2r b) if b < r. (b) The probability f b (n) that the walk hits b for the first time at the n-th step is f b (n) = b n P(S n = b). Proof. (a) The case b r is clear. Assume r 1 and b < r. Let N r n(0, b) denote the number of paths from (0, 0) to (n, b) which include some point having height r (i.e., some point (i, r) with time 0 < i < n). Call such a path π and (i π, r) the earliest such hitting point of the height r. Now reflect the segment with times larger than i π in the horizontal height axis (x-axis shifted in vertical direction by r), see figure 3. The reflected path π (with his segment up to time i π equal to the one of π) is a path joining (0, 0) and (n, 2r b). Here, 2r b is the result of b + 2(r b) which is the terminal point of π. There is again a one-one correspondence between paths π π, and hence N r n(0, b) = N n (0, 2r b). Thus, P(M n 1 r, S n = b) = N r n(0, b)p 1 2 (n+b) q 1 2 (n b) = ( q p )r b N n (0, 2r b)p 1 2 (n+2r b) q 1 2 (n 2r+b) = ( q p )r b P(S n = b). (b) Pick b > 0 (the case for b < 0 follows similar). Then, using (a) and the fact that {M n 1 = b} = {M n 1 b 1} \ {M n 1 b}, we get f b (n) = P(M n 1 = S n 1 = b 1, S n = b) = p ( P(M n 1 b 1, S n 1 = b 1) P(M n 1 b, S n 1 = b 1) ) = p ( P(S n 1 = b 1) ( q p )P(S n 1 = b + 1) ) = b n P(S n = b). 5

1.2 How often random walkers return to the origin? We are going to discuss in an heuristic way the question how often the random walker returns to the origin. The walker always moves from an even integer to an odd integer or from an odd integer to an even integer, so we know for sure the position S n of the walker is at an even integer if n is even or an at an odd integer if n is odd. Example. Symmetric Bernoulli random walk, p = 1 2, S 0 = 0: P(S 2n = 2j) = and in particular ( ) 2n 2 2n = 2 2n (2n)! n + j (n + j)!(n j)!, j Z, 2n (2n)! P(S 2n = 0) = 2 n!n!, and with Stirling s formula we finally get n! n n e n 2πn, 2n (2n)! P(S 2n = 0) = 2 n!n! 2 2n 2 2n = 1. π n πn This fact is consistent with what we already know. We know that the walker tends to go a distance about a constant times n, and there are about c n such integer points that are in distance within n from the origin. Henceforth, it is very reasonable that a particular one is chosen with probability a constant times n 1/2. We consider the number of times that the random walker returns to the origin. Define J n = 1l{S n = 0}, n N 0. Here the indicator function 1l is defined as follows. Let E be an event, then 1l{E} is the random variable that takes the value 1 if the event occurs and 0 if it does not occur. The total number of visits of the random walker to the origin is then We compute the expectation E(V ) = V = J 2n. n=0 E(J 2n ) = n=0 6 P(S 2n = 0). n=0

We know that P(S 2n = 0) c/ n as n for some c > 0. Therefore, E(V ) =. Note however that it is possible for a random variable to be finite yet have an infinite expectation, so we shall do more work to prove that V is actually infinite. For that, let q be the probability that the random walker ever returns to the origin after time 0. Claim: q = 1. We shall prove this claim by contradiction. Suppose that q < 1. Then we can give the distribution of V. P(V = 1) = (1 q) since V is one if and only if the walker never returns after time zero. P(V = k) = q k 1 (1 q), k = 1, 2, 3,.... Thus E(V ) = kp(v = k) = k=1 k=1 kq k 1 (1 q) = 1 1 q <. The latter equation is easily seen by observing that q 0 (1 q) + 2q(1 q) + 3q 2 (1 q) + = 1 q + 2q 2q 2 + 3q 2 3q 3 +. We know that E(V ) = and henceforth q = 1. With the claim we have shown the following result. Theorem 1.7 The probability that a one-dimensional simple random walker returns to the origin infinitely often is one. Remark 1.8 (i) E(V ) = 1 + qe(v ) leads to E(V ) = (1 q) 1. The 1 represents the first visit; q is the probability of returning to the origin; and the key observation is that the expected number of visits after the first visit, given there is a second visit, is exactly the expected number of visits starting at the origin. (ii) If the random walker starts at x 0, then the probability that he will get to the origin is one. What happens in higher dimensions? Let s consider Z d, d 1, and x Z d, x = (x 1,..., x d ). We study the simple random walk on Z d. The walker starts at the origin (i.e. S 0 = 0) and at each integer time n he moves to one of the nearest neighbours with equal probability. Nearest neighbour refers here to the Euclidean distance, 7

x = ( d i=1 (xi ) 2) 1/2, and any lattice site in Z d has exactly 2d nearest neighbours. Hence, the walkers jumps with probability 1 to one of its nearest 2d neighbours. Denote the position of the walker after n N time steps by S n = (S n (1),..., S n (d) ) and write S n = X 1 + +X n, where X i = (X (1) i,..., X (d) i ) are independent random vectors with P(X i = y) = 1 2d for all y Z d with y = 1, i.e., for all y Z d that are in distance one from the origin. We compute similarly as above and E( S n 2 ) = E((S (1) n ) 2 + + (S (d) n ) 2 ) = de((s (1) n ) 2 ), E((S (1) n ) 2 ) = n j=1 E((X (1) j ) 2 ) + E(X (1) i X (1) j ). i j The probability that the walker moves within the first coordinate (either +1 or 1) is 1 (1), thus E((X d j ) 2 ) = 1 and E( S d n 2 ) = n. Consider again an even time 2n and take n sufficiently large, then (law of large number, local central limit theorem) approximately 2n expected steps will be done by the walker d in each of the d component directions. To be at the origin after 2n steps, the walker will have had to have an even number of steps in each of the d component directions. Now for n large the probability for this happening is about ( 1 2 )d 1. Whether or not an even number of steps have been taken in each of the first d 1 component directions are almost independent events; however, we know that if an even number of steps have been taken in the first d 1 component directions then an even number of steps have been taken in the last component as well since the total number of steps taken is even. 2n d steps in each component direction gives P(S (i) d 2n = 0) 1 π 2n, i = 1,..., d. Hence, ( P(S 2n = 0) 2 1 d d 1 ) d ( d d/2 ) = n d/2. π 2n 2 d 1 2 d/2 π d/2 This is again consistent with what we already know. We know that the mean distance is n from the origin, and there are about n d/2 points in Z d that are within distance n from the origin. Hence, we expect that the probability of choosing a particular one would be of order n d/2 ). As in d = 1 the expected number of visits to the origin up to time n is { E(V ) = P(S 2n = 0) = 1 + const n d/2 <, d 3, = =, d = 1, 2. n=0 8 j=1

We summarise these observations in the following theorem. Theorem 1.9 Let (S n ) n N0 be the simple random walk in Z d with S 0 = 0. If d = 1, 2, the random walk is recurrent, i.e., with probability one it returns to the origin infinitely often. If d 3, the random walk is transient, i.e., with probability one it returns to the origin only finitely often. In addition we have that P(S n 0 for all n > 0) > 0 if d 3. The statement for d 3 is an application of the Borel-Cantelli Lemma which we recap here. Lemma 1.10 (Borel-Cantelli Lemma) Suppose E 1, E 2,... is a collection of events such that P(E n ) <. n=1 Then with probability one at most finitely many of the events occur. For a proof we refer to standard books in probability or the first support class. We apply this Lemma to the simple random walk in dimension d 3. We let E n be the event that S n = 0. From our previous consideration we get that n=1 P(E n) <, and hence with probability one, only finitely many of the events E n occur. Henceforth with probability one the random walk revisits the origin only finitely often. 1.3 Transition function We study random walks on Z d and connect them to a particular function, the so-called transition function or transition matrix. For each pair x and y in Z d we define a real number P (x, y), and this function will be called transition function or transition matrix. Definition 1.11 (Transition function/matrix) Let P : Z d Z d R be given such that (i) 0 P (x, y) = P (0, y x) for all x, y Z d, (ii) x Z d P (0, x) = 1. The function P is called transition function or transition matrix on Z d. 9

It will turn out that this function actually determines completely a random walk on Z d. That is, we are now finished - not in the sense that there is no need for further definitions, for there is, but in the sense that all further definitions will be given in terms of P. How is a random walk S = (S n ) n N0 connected with a transition function (matrix)? We consider random walks which are homogeneous in time, that is This motivates to define P(S n+1 = j S n = i) = P(S 1 = j S 0 = i). P (x, y) = P(S n+1 = y S n = x), for all x, y Z d. (1.4) Hence, P (0, x) corresponds to our intuitive notion of the probability of a one-step transition from 0 to x. Then it is useful to define P n (x, y) as the n-step transition probability, i.e., the probability that a random walker (particle) starting at the origin 0 finds itself at x after n transitions (time steps) governed by P. Example. Bernoulli random walk: The n-step transition probability is given as ( ) P n (0, x) = p (n+x)/2 q (n x)/2 n (n + x)/2 when n is even, x n, and P n (0, x) = 0 otherwise. Example. Simple random walk in Z d : Any lattice site in Z d has exactly 2d nearest neighbours. Hence, the transition function (matrix) reads as P (0, x) = { 1 2d, if x = 1, 0, otherwise. Notation 1.12 The n-step transition function (matrix) of the a random walk S = (S n ) n N0 is defined by P n (x, y) = P(S m+n = y S m = x), m N, x, y Z d, and we write P 1 (x, y) = P (x, y) and P 0 (x, y) = δ x,y. The n-step transition function can be written as P n (x, y) = P (x, x 1 )P (x 1, x 2 ) P (x n 1, y), n 2. (1.5) x i Z d,i=1,...,n 1 This is proved in the following statement. 10

Theorem 1.13 For any pair r, s N 0 satisfying r + s = n N 0 we have P n (x, y) = z Z d P r (x, z)p s (z, y), x, y Z d. Proof. The proof for n = 0, 1 is clear. We give a proof for n = 2. Induction will give the proof for the other cases as well. The event of going from x to y in two transitions (time steps) can be realised in the mutually exclusive ways of going to some intermediate lattice site z Z d in the first transition and then going from site z Z d to y in the second transition. The Markov property implies that the probability of the second transition is P (z, y), and that of the first transition is clearly P (x, z). Using the Markov property and the relation P(A B C) = P(A B C)P(B C), we get for any m N, P 2 (x, y) = P(S m+2 = y S m = x) = z Z d P(S m+2 = y, S m+1 = z S m = x) = z Z d P(S m+2 = y S m+1 = z, S m = x)p(s m+1 = z S m = x) = z Z d P(S m+2 = y S m+1 = z)p(s m+1 = z S m = x) = z Z d P (x, z)p (z, y). The probability interpretation of P n (x, y) is evident, it represents the probability that a particle, executing a random walk and starting at the lattice site x at time 0, will be at the lattice site y Z d at time n. We now define a function of a similar type, namely, we are asking for the probability (starting at x at time 0), that the first visit to the lattice site y should occur at time n. Definition 1.14 For all x, y Z d and n 2 define F 0 (x, y) := 0, F 1 (x, y) := P (x, y), F n (x, y) := P (x, x 1 )P (x 1, x 2 ) P (x n 1, y). x i Z d \{y} i=1,...,n 1 11

Important properties of the function F n, n 2, are summarised as follows. Proposition 1.15 For all x, y Z d : (a) F n (x, y) = F n (0, y x). (b) n k=1 F k(x, y) 1. (c) P n (x, y) = n k=1 F k(x, y)p n k (y, y). Proof. (a) is clear from the definition and the known properties of P. (b) The claim is somehow obvious. However, we shall give a proof. For n N put Ω n = {ω = (x 0, x 1,..., x n ): x 0 = x, x i Z d, i = 1,..., n}. Clearly, Ω n is countable, and we define a probability for any elementary event ω Ω n by p(ω) := P (x, x 1 )P (x 1, x 2 ) P (x n 1, x n ), ω = (x 0, x 1,..., x n 1 ) Ω n. Clearly, ω Ω: x n=y p(ω) = P n (x, y), and ω Ω n p(ω) = y Z d P n (x, y) = 1. The sets A k, A k = {ω Ω n : x 1 y, x 2 y,..., x k 1 y, x k = y}, 1 k n, are disjoint subsets of Ω n and F k (x, y) = ω A k p(ω) implies that n F k (x, y) p(ω) = 1. ω Ω n k=1 (c) This can be proved in a very similar fashion, or by induction. We skip the details. We come up now with a third (and last) function of the type above. This time we are after the expected number of visits of a random walk to a given point within a given time. More precisely, we denote by G n (x, y) the expected number of visits of the random walk, starting at x, to the point y up to time n. Notation 1.16 G n (x, y) = n P k (x, y), n N 0, x, y Z d. k=0 12

One can easily convince oneself that G n (x, y) G n (0, 0) for all n N 0, x, y Z d : it suffices to consider x 0, using Proposition 1.15(c) we get that G n (x, 0) = = n n k P k (x, 0) = F k j (x, 0)P j (0, 0) k=1 k=1 j=0 n j n P j (0, 0) F i (x, 0) j=0 i=0 j=0 n P j (0, 0) = G n (0, 0). We are now in the position to classify the random walks according to whether they are recurrent or transient (non-recurrent). The idea is that n k=1 F k(0, 0) represents the probability of a return to the origin before or at time n. The sequence of sums n k=1 F k(0, 0) is non-decreasing as n increases, and by Proposition 1.15 bounded by one. Call the limit by F 1. Further, call G the limit of the monotone sequence (G n (0, 0)) n N0. Notation 1.17 (a) G(x, y) = n=0 P n(x, y) for all x, y Z d, G n (0, 0) := G n and G := G(0, 0). (b) F (x, y) = n=1 F n(x, y) 1 or all x, y Z d, F n (0, 0) := F n and F := F (0, 0). Definition 1.18 The random walk (on Z d ) defined by the transition function P is said to be recurrent if F = 1 and transient if F < 1. Proposition 1.19 G = 1 1 F with G = + when F = 1 and F = 1 when G = +. Proof. (The most convenient way is to prove it is using generating functions). We sketch a direct method. P n (0, 0) = n F k P n k (0, 0), n N. (1.6) k=0 Summing (1.6) over n = 1,..., m, and adding P 0 (0, 0) = 1 gives G m (0, 0) = m F k G m k (0, 0) + 1, m N. (1.7) k=0 13

Letting m we get G = 1 + lim m m F k G m k 1 + G k=0 and thus G 1 + GF. Now (1.7) gives 1 = G m m G k F m k G m G m k=0 and henceforth 1 G(1 F ). N F k, for all N N, k=0 m k=1 F m k G m (1 F ), Example. Bernoulli random walk: P (0, 0) = p, and P (0, 1) = q = 1 p, p [0, 1]. ( ) ( ) 2n P(S 2n = 0) = P 2n (0, 0) = (pq) n = ( 1) n (4pq) n 1 2, n n where we used that ( ) ( ) 2n = ( 1) n 4 n 1 2 n n (1.8) holds for any n N. The binomial coefficients for general numbers r are defined as ( ) r r(r 1) (r k + 1) =, k N 0. k k! We prove the claim (1.8) by induction: For n = 1 the LHS= 2! and RHS= 1!1! ( 1)4 1/2. Suppose the claim is vlid for n N. We shall show the claim for 1! n + 1 as well. We get ( ) 2(n + 1) (2n)!(2n + 1)(2(n + 1)) = n + 1 (n + 1)n!(n + 1)n! ( ) 1/2 = ( 1) n 4 n 2 n (2n + 1) n + 1 = ( 4)n+1 ( n 1/2)( 1/2)( 1/2 1) ( 1/2 n + 1) (n + 1)! ( ) 1/2 = ( 1) n+1 4 n+1. n + 1 Further, using Newton s generalised Binomial theorem, that is, ( ) r (x + y) r = x r k y k, (1.9) k k=0 14

we - noting that 0 p = 1 q implies that 4pq 1 - get that t n P 2n (0, 0) = (1 4pqt) 1/2, t < 1. n=0 Thus lim t 1,t<1 t n P 2n (0, 0) = n=0 P 2n (0, 0) = n=0 P n (0, 0) = G, n=0 henceforth G = { (1 4pq) 1/2 <, if p q, +, if p = q. The Bernoulli random walk (on Z) is recurrent if and only if p = q = 1 2. Example. Simple random walk in Z d : The simple random walk is for d = 1 recurrent, d = 2 recurrent, d 3 transient. 1.4 Boundary value problems We give a brief introduction to boundary value problems. We start off with the so-called gambler s ruin. Suppose N is a positive integer and a random walker (p = 1 2 ) starts at x {0, 1,..., N}. Let S n denote the position of the walker at time n. Suppose that the random walker stops when he reaches 0 or N. Define the random time T = min{n N: S n = 0 or S n = N}. Then the position of the walker at time n is Ŝn = S n T, where n T denotes the minimum of n and T. We can easily see that with probability one T <, that is, the walker eventually reaches 0 or N and then stops. Define the function F : {0, 1,..., N} [0, 1] by F (x) = P(S T = N S 0 = x). The gambling interpretations is as follows. Suppose S n is the amount of pounds currently held by the gambler (after n games). The gambler is playing a fair game (starting with exactly x pounds) where at each time duration the gambler wins or loses one pound. The gambler plays until he or she 15

has N pounds or has gone bankrupt. The chance the gambler does not go bankrupt before attaining N is F (x). Clearly, F (0) = 0 and F (N) = 1. Let now 0 < x < N. After the first game, the gambler has either x 1 or x + 1 pounds, and each of theses outcomes is equally likely. Therefore, F (x) = 1 2 F (x 1) + 1 F (x + 1), x = 1,..., N 1. (1.10) 2 A function F that satisfies (1.10) with the boundary conditions F (0) = 0 and F (N) = 1 is the linear function F (x) = x. The next theorem shows N that this is the only solution to (1.10) with the given boundary conditions. Theorem 1.20 Let a, b R and N N. Then the only function F : {0, 1,..., N} R satisfying (1.10) with F (0) = a and F (N) = b is the linear function F 0 (x) = a + x(b a), x {0, 1,..., N}. N Proof. The proof is relatively easy and there are many different ways to prove it. We immediately see that F 0 solves the equation with the given boundary conditions. We will give two different proofs. Version 1: Suppose F is a solution to (1.10). Then for each 0 < x < N, F (x) max{f (x 1), F (x + 1)}. With this we can easily see that the maximum of F is attained at 0 or N. Similarly, we can show that F attains its minimum at {0, N}. Suppose that F (0) = F (N) = 0. Henceforth F is identical zero. Suppose now F (0) = a and F (N) = b and with at least one of a and b not equal to zero. Let F 0 be the linear function with these same boundary conditions. Then F F 0 satisfies (1.10) with zero boundary condition and henceforth F F 0 0. This implies F = F 0. Version 2: Suppose F is a solution of (1.10). Let the random walker start at x. We shall prove that E[F (S n T )] = F (x), n N 0, x {0, 1,..., N}. We prove this via induction. If n = 0 we have F (S 0 ) = F (x), and hence E[F (S 0 )] = F (x). We assume the claim holds for n and show the inductive step n n + 1. We compute the expectation and get N E[F (S (n+1) T )] = P(S n T = y)e[f (S n T ) S n T = y]. y=0 16

If y = 0 or y = N and S n T = y, then S (n+1) T = y and hence If 0 < x < N and S n T = y, then E[F (S (n+1) T ) S n T = y] = F (y). E[F (S (n+1) T ) S n T = y] = 1 2 F (y + 1) + 1 F (y 1) = F (y). 2 Therefore we obtain E[F (S (n+1) T )] = N P(S n T = y)f (y) = E[F (S n T )] = F (x), y=0 where the last equality holding by inductive hypothesis and the next to the least equality is the computation of the expectation value. As the right hand side of our claim does not depend on the time n we can pass to the limit and obtain F (x) = lim n E[F (S n T )] = lim n N P(S n T = y)f (y) y=0 = P (S T = 0)F (0) + P(S T = N)F (N) = (1 P(S T = N))F (0) + P(S T = N)F (N) = P(S T = N), where the last equality is obtained using the boundary conditions F (0) = 0 and F (N) = 1. Now, P(S T = N) = x finishing the proof showing that N F (x) = F (0) + x (F (N) F (0)). N Why we have P(S T = N) = x? We can show this using the optimal sampling N theorem. Let M n := S n T. Then M n is a bounded martingale and the optimal sampling theorem states that for 0 x N we have E x [M 0 ] = E x [M T ] where the index x at the expectation means starting at x. Martingales are families of random variables, e.g., a fair game is a typically martingale as the expected win does not depend when the game is stopped. We have x = E x [M 0 ] and we compute E x [M T ] = 0P(S T = 0) + NP(S T = N) to get that P(S T = N) = x/n. We give briefly the definition for martingales. Definition 1.21 A sequence (Y n ) n N0 of random variables is a martingale with respect to the sequence (X n ) n N0 of random variables if, for all n N 0, 17

(i) E[ Y n ] <, (ii) E[Y n+1 X 0,..., X n ] = Y n. As an example we consider the random walk in Z with probability p going to the right. S n = X 1 + + X n and E[S n+1 X 0,..., X n ] = Y n. Therefore Y n + S n n(p q) is a martingale with respect to the random walk (S n ) n N0. This follows immediately via E[S n+1 (n + 1)(p q) X 0,..., X n ] = S n n(p q) = Y n. We shall generalise our previous theorem to higher dimensions. We replace {1,..., N 1} with an arbitrary finite subset A Z d. The (outer) boundary of A is A = {z Z d \ A: dist(z, A) = 1}, and we let A = A A be the closure of A. Definition 1.22 The linear operators Q and L acting on functions F : Z d R are defined by QF (x) = 1 F (y) 2d and LF (x) = (Q 1l)F (x) = 1 2d y Z d : x y =1 y Z d : x y =1 The operator L is called the discrete Laplacian. We easily see that LF (x) = E[F (S 1 ) F (S 0 ) S 0 = x]. F (y) F (x). The name Laplacian is motivated by the continuous space variant. That is, for any function f : R d R, the Laplacian is f(x) = d i=1 2 f (x). x 2 i 18

For any i = 1,..., d we denote by e i the basis vector in Z d with 1 in the ith-component and all other components equal to zero. The finite difference operators are defined as Then i F (x) = F (x + e i ) F (x); i F (x) = F (x e i ) F (x); 2 i F (x) = 1 2 F (x + e i) + 1 2 F (x e i) F (x), x Z d. LF (x) = 1 d d 2 i F (x) = i=1 d i i F (x), which justifies the name Laplacian for the linear operator L. Definition 1.23 (i) The function F : Z d R is (discrete) harmonic at x if LF (x) = 0, and F is harmonic in A, A Z d, if LF (x) = 0 for all x A. (ii) Dirichlet problem for harmonic functions: Given a set A Z d and a function F : A R find an extension of F to A such that F is harmonic in A, i.e., LF (x) = 0 for all x A. For the case d = 1 and A = {1,..., N 1}, it is easy to guess the solution. In higher dimensions this is no longer the case. However, we will show that our last proof of Theorem 1.20 has a straightforward extension to higher dimensions. We let T A = min{n N 0 : S n / A}. Theorem 1.24 If A Z d finite, then for every F : A R, there is a unique extension of F to A that satisfies LF (x) = 0 for all x A. The extension is given by F 0 (x) = E[F (S TA ) S 0 = x] = y A P(S TA = y S 0 = x)f (y). i=1 Proof. We only give a brief sketch of the proof. We can easily show that F 0 is indeed a solution. This is left as an exercise to the reader. We are left to show uniqueness of the solution. Suppose F is harmonic on A, S 0 = x A, and let M n = F (S n TA ). Then LF (x) = 0 for all x A can be rewritten as E[M n+1 S 0,..., S n ] = F (S n TA ) = M n. Recall that a process (M n ) n N0 that satisfies E[M n+1 S 0,..., S n ] = M n is called a martingale with respect to the random walk. One can show that F 19

being harmonic in A is essentially equivalent to F (S n TA ) being a martingale. The optimal sampling theorem gives that E[M n ] = E[M 0 ], S 0 = x, and therefore P(S n TA )F (y) = E[M n ] = E[M 0 ] = F (x). y A As a next step one needs to show that with probability one T A < (exercise for the reader). We can therefore take limits and obtain F (x) = lim P(S n TA = y)f (y) = P(S TA = y)f (y). n y A y A 1.5 Summary The simple random walks on Z d (discrete time) are examples of Markov chains on Z d. Definition 1.25 Let I be a countable set, λ M 1 (I) be a probability measure (vector) on I, and P = (P (i, j)) i,j I be a transition function (stochastic matrix). A sequence X = (X n ) n N0 of random variables X n taking values in I is called a Markov chain with state space I and transition matrix P and initial distribution λ, if P(X n+1 = i n+1 X 0 = i 0,..., X n = i n ) = P (i n, i n+1 ) and P(X 0 = i) = λ(i), i I, for every n N 0 and every i 0,..., i n+1 I with P(X 0 = i 0,..., X n = i n ) > 0. We call the family X = (X n ) n N0 a (λ, P )-Markov chain. Note that for every n N 0, i 0,..., i n I, the probabilities are computed as P(X 0 = i 0,..., X n = i n ) = λ(i 0 )P (i 0, i 1 )P (i 1, i 2 ) P (i n 1, i n ). A vector λ = (λ(i)) i I is called a stationary distribution of the Markov chain if the following holds: (a) λ(i) 0 for all i I, and i I λ(i) = 1. (b) λ = λp, that is, λ(j) = i I λ(i)p (i, j) for all j I. Without proof we state the following result which will we prove later in the continuous time setting. Theorem 1.26 Let I be a finite set and P : I I R + be a transition function (matrix). Suppose for some i I that P n (i, j) λ(j) as n for all j I. Then λ = (λ(j)) j I is an invariant distribution. 20

2 Markov processes In this chapter we introduce continuous-time Markov processes with a countable state space I. Throughout the chapter we assume that X = (X t ) t 0 is a family of I-valued random variables. The family X = (X t ) t 0 is called a continuous-time random process. We shall specify the probabilistic behaviour (or law) of X = (X t ) t 0. However, there are subtleties in this problem not present in the discrete-time case. They arise because the probability of a countable disjoint union is the sum of the single probabilities, whereas for a noncountable union there is no such rule. To avoid these subtleties we shall consider only continous-time processes which are right continuous. This means that with probability one, for all t 0, lim h 0 X t+h = X t. By a standard result of measure theory the probability of any event depending on a right-continuous process can be determined from its finite-dimensional distributions, that is, from the probabilities P(X t0 = i 0,..., X tn = t n ) for n N 0, 0 t 0 t n and i 0,..., i n I. Throughout we are using both writings, X t and X(t) respectively. Definition 2.1 The process X = (X t ) t 0 is said to satisfy the Markov property if P(X(t n ) = j X(t 0 ) = i 0,... X(t n 1 ) = i n 1 ) = P(X(t n ) = j X(t n 1 ) = i n 1 ) for all j, i 0,..., i n 1 I and any sequence t 0 < t 1 < < t n of times. We studied in the first chapter the simplest discrete time Markov process (Markov chain) having independent, identically distributed increments (Bernoulli random variables). The simplest continuous time Markov processes are those whose increments are mutually independent and homogeneous in the sense that the distribution of an increment depends only on the length of the time interval over which the increment is taken. More precisely, we are dealing with stochastic processes X = (X t ) t 0 having the property that P(X 0 = x 0 ) = 1 for some X 0 I and P(X(t 1 ) X(t 0 ) = i 1,..., X(t n ) X(t n 1 ) = i n ) = n P(X(t m ) X(t m 1 ) = i m ) for n N, i 1,..., i n I and all times t 0 < t 1 < < t n. We introduce in the first subsection the Poisson process on N. Before that we shall collect some basic facts from probability theory. Definition 2.2 (Exponential distribution) A random variable T having values in [0, ) has exponential distribution of parameter λ [0, ) if P(T > 21 m=1

t) = e λt for all t 0. The exponential distribution is the probability measure on [0, ) having the (Lebesgue-) density function f T (t) = λe λt 1l{t 0}. We write T E(λ) for short. The mean (expectation) of T is given by E(T ) = 0 P(T > t) dt = λ 1. The other important distribution is the so-called Gamma distribution. We consider random time points in the interval (0, ) (e.g. incoming claims in an insurance company or phone calls arriving at a telephone switchboard). The heuristic reasoning is that, for every t > 0, the number of points in (0, t] is Poisson distributed with parameter λt, where λ > 0 represents the average number of points per time. We look for a model of the r-th random point. What is the probability measure P describing the distribution of the r-th random point? P ((0, t]) = probability that the r-th point arrives no later than t (i.e. at least r points/arrivals in (0, t]). Denote by P λt the Poisson distribution with parameter λt. We get the probability in question using the complementary event as P ((0, t]) = 1 P λt ({0,..., r 1}) r 1 = 1 e λt k=0 (λt) k k! = t 0 λ r (r 1)! xr 1 e λx dx. The last equality can be checked when differentiating with respect to t. Recall the definition of Euler s Gamma function, Γ(r) = y r 1 e y dy, r > 0, and 0 Γ(r) = (r 1)! for all r N. Definition 2.3 (Gamma distribution) For every λ, r > 0, the probability measure Γ λ,r on [0, ) with (Lebesgue-) density function γ λ,r (x) = λr Γ(r) xr 1 e λx, x 0, is called the Gamma distribution with scale parameter λ and shape parameter r. Note that Γ λ,1 is the exponential distribution with parameter λ. Lemma 2.4 (Sum of exponential random variables) If X i E(λ), i = 1,..., n, independently, and Z = X 1 + + X n then Z is Γ λ,n distributed. Proof. Exercise of example sheet 2. 22

2.1 Poisson process In this Subsection we will introduce a basic intuitive construction of the Poisson process. The Poisson process is the backbone of the theory of Markov processes in continuous time having values in a countable state space. We will study later more general settings. Pick a parameter λ > 0 and let (E i ) i N be a sequence of i.i.d. (independent identically distributed) random variables (having values in R + ) that are exponentially distributed with parameter λ (existence of such a sequence is guaranteed - see measure theory). Now, E i is the time gap (waiting or holding time) between the (i 1)-th (time) point and the i-th point. Then the sum J k = is the k-th random point in time (see figure). Furthermore, let k i=1 E i N t = k N 1l (0,t] (J k ) be the number of points in the interval (0, t]. Thus, for s < t, N t N s is the number of points in (s, t]. Clearly, for t [J k, J k+1 ) one has N t = k. Theorem 2.5 (Construction of the Poisson process) The N t, t 0, are random variables having values in N 0, and, for 0 = t 0 < t 1 < < t n, the increments N ti N ti 1 are independent and Poisson distributed with parameter λ(t i t i 1 ), 1 i n. Definition 2.6 A family (N t ) t 0 of N 0 -valued random variables satisfying the properties of Theorem 2.5 with N 0 = N(0) = 0 is called a Poisson process with intensity λ > 0. We can also write J k = inf{t > 0: N t k}, k 1, in other words, J k is the k-th time point at which the sample path t N t of the Poisson process performs a jump of size 1. These times are therefore called jump times of the Poisson process, and (N t ) t 0 and (J k ) k N are two manifestations of the same mathematical object. Proof of Theorem 2.5. First note that {N t = k} = {J k t < J k+1 }. (2.11) We consider here n = 2 to keep the notation simple. The general case follows analogously. Pick 0 < s < t and k, l N. It suffices to show that P(N s = k, N t s = l) = ( λs (λs)k )( e e λ(t s) (λ(t s))l ). (2.12) k! l! 23

Having (2.12), summing over l and k, respectively, we conclude that N s and N t s are Poisson distributed (and are independent, see right hand side of (2.12)). The joint distribution of the (holding) times (E j ) 1 j k+l+1 has the product density f(x 1,..., x k+l+1 ) = λ k+l+1 e λτ k+l+1(x), where for convenience we write τ k+l+1 (x) = x 1 + + x k+l+1. Using the equality of the events in (2.11), the left hand side of (2.12) reads as P(N s = k, N t s N s = l) = P(J k s < J k+1 J k+l t < J k+l+1 ) = 0 0 dx 1 dx k+l+1 λ k+l+1 e λτ k+l+1(x) 1l{τ k (x) s < τ k+1 (x) t < τ k+l+1 (x)}. We integrate step by step starting from the innermost integral and moving outwards. Fix x 1,..., x k+l and set z = τ k+l+1 (x), 0 dx k+l+1 λe λτ k+l+1(x) 1l{τ k+l+1 (x) > t} = t dzλe λz = e λt. Fix x 1,..., x k and make the substitution y 1 = τ k+1 (x) s, y 2 = x k+2,..., y l = x k+l to obtain 0 = 0 0 dx k+1 dx k+l 1l{s < τ k+1 (x) τ k+l (x) t} 0 dy 1 dy l 1l{y 1 + + y l t s} = which can be proved via induction on l. In a similar way one gets 0 0 dx 1 dx k 1l{τ k (x) s} = sk k!. Combing all our steps above, we obtain finally P(N s = k, N t s = l) = e λt k+l sk λ k! (t s) l. l! (t s)l, l! The following statement shows that the Poisson process satisfies the Markov property. 24

Theorem 2.7 Let N = (N t ) t 0 be a Poisson process with intensity λ > 0, then P(N s+t N s = k N τ, τ [0, s]) = P(N t = k), k N. That is for all s > 0, the past (N τ ) τ [0s) is independent of the future (N s+t N s ) t 0. In other words, for all s > 0 the process after time s and counted from the level N s remains a Poisson process with intensity λ independent of its past (N τ ) τ [0s). 2.2 Compound Poisson process on Z d We can easily construct a rich class of processes which are the continuous time analogs of the random walks on Z d in Section 1. Pick a probability vector (probability measure) µ = (µ k ) k Z d M 1 (Z d ) such that µ 0 = 0, i.e., µ k [0, 1] k Z d and k Z µ d k = 1. The compound Poisson process on Z d with jump distribution µ and rate λ (0, ) is the stochastic process (X t ) t 0 which starts at the origin, sits there for an exponential holding time having mean value λ 1, at which time it jumps by the amount k Z d with probability µ k, sits where it lands for another, independent holding time with mean λ 1, jumps again and so on. If d = 1, λ = 1, and µ 1 = 1 we say (X t ) t 0 is the simple Poisson process, which, once restricted to the state space N 0, is the Poisson process from the previous section. The jump distribution allows only jumps by +1, i.e., only jumps to the right, because µ k = 0 for all k 0. Construction of the compound Poisson process: Choose a family (B n ) n N of mutually independent Z d -valued random variables with distribution µ M 1 (Z d ) with µ 0 = 0. This determines a random walk (Y n ) n N0 (in discrete time) on Z d. Y 0 = 0 and Y n = n B m, n 1. For any rate λ (0, ), a family (X t ) t 0 of Z d -valued random variables is defined by X t := Y N(λt), t 0, where (N t ) t 0 is a simple Poisson process which is independent of the random variables B m (simple means intensity one). The following facts are easily seen from the construction. X 0 = 0 and [0, ) t X t is a piecewise constant, right continuous Z d - valued path. The number of jumps during a time interval (s, t] is precisely 25 m=1

N(λt) N(λs) and B n = Y n Y n 1 is the amount of the n-th jump. We let J 0 = 0 and J n, n 1, denote the time of the n-th jump. Then N(λt) = n J n λt < J n+1 and X Jn X Jn 1 = B n. If (E i ) i N is the family of unit exponential holding times (i.e. E i E(1)) of the simple Poisson process (N t ) t 0, then the holding times of the process (X t ) t 0 are given via the jump times as J n J n 1 = E n λ ; X(t) X(t ) = 0 for t (J n 1, J n ). We call (X t ) t 0 the compound Poisson process with jump distribution µ and rate λ. In the next lemma we shall show that a compound process moves along in homogeneous, mutually independent increments. Lemma 2.8 P(X(s + t) X(s) = k X(τ), τ [0, s]) = P(X(t) = k), k Z d. (2.13) Proof. Given A σ({x(τ): τ [0, s]}) it suffices to show that P({X(s + t) X(s) = k} A) = P({X(s + t) X(s) = k})p(a). W.l.o.g. we assume that, for some m N, N(λs) = m on A. Then the event A is independent of σ({y m+n Y m : n 0} {N(λ(s + t)) N(λs)}), and so P({X(s + t) X(s) = k} A) = P({X(s + t) X(s) = k; N(λ(s + t) N(λs) = n} A) = = n=0 P({Y m+n Y m = k; N(λ(s + t) N(λs) = n} A) n=0 P(Y n = k; N(λt) = n)p(a) = P(X(t) = k)p(a). n=0 Finally, we shall compute the distribution of the compound Poisson process (X t ) t 0. Recall that the distribution of the sum of k independent, identically distributed random variables is the n-fold convolution of their distribution. 26

Definition 2.9 (Convolution) If µ, ν M 1 (Z d ) are two probability vectors, the convolution µ ν M 1 (Z d ) of µ and ν is defined by µ ν(m) = k Z d µ k ν m k, m Z d. The convolution of µ M 1 (Z d ) with itself is defined as (µ ( 0) ) k = δ 0,k, (µ ( n) ) k = j Z d (µ ( (n 1)) ) k j µ j, n 1. Clearly, we have Henceforth we obtain the following, P(X(t) = k) = P(Y n = k) = µ ( n) k. P(Y n = k, N(λt) = n) = e λt n=0 n=0 (λt) n (µ ( n) ) k. n! Now with Lemma 2.8 we compute for an event A σ(n(τ): τ [0, s]). P({X(s + t) = k} A) = j Z d P({X(s + t) = k} A {X(s) = j}) = j Z d P({X(s + t) X(s) = k j} A {X(s) = j}) =: j Z d P t (j, k)p(a {X(s) = j}) = E ( P t (X(s), k)1l{a} ), where P t (k, l) = e λt m=0 (λt) m (µ ( m) ) k j, l, k Z d. m! We have thus proved that (X t ) t 0 is a continuous time Markov process with transition probability P t in the sense that P(X(s + t) = k X(τ), τ [0, s]) = P t (X(s), k). As a consequence of the last equation, we find that (P t ) t 0 is a semigroup. That is, it satisfies the Chapman-Kolmogorov equation P s+t = P s P t, s, t [0, ). (2.14) 27

This can be seen as follows, P s+t (0, k) = j Z d P(X(s + t) = k; X(s) = j) = j Z d P t (j, k)p s (0, j) = j Z d P s (0, j)p t (j, k) = (P s P t ) 0,k, where we used that P t (k, l) = P t (0, l k). 2.3 Markov processes with bounded rates There are two possible directions in which one can generalise the previous construction of the Poisson respectively the compound Poisson process: jump distribution depends on where the process is at the time of the jump. holding time depends on the particular state the process occupies. Assumptions: I countable state space and Π = (π(x, y)) x,y I a transition probability matrix such that π(x, x) = 0 for all x I. Λ = {λ(i) λ i : i I} (0, ) a family of bounded rates such that sup i I {λ i } <. Proposition 2.10 With the above assumptions, a continuous time Markov process on I with rates Λ and transition probability matrix Π is an I-valued family (X t ) t 0 of random variables having the properties that (a) t X(t) is piecewise constant and right continuous, (b) If J 0 = 0 and, for n 1, J n is the time of the n-th jump, then P(J n > J n 1 +t; X(J n ) = j X(τ), τ [0, J n )) = e tλ(x(j n 1)) π(x(j n 1 ), j) (2.15) on {J n 1 < }. Proof. We have to show two things. First (see step 1 below) we have to show that Proposition 2.10 together with an initial distribution uniquely determines the distribution of the family (X t ) t 0. Secondly (see step 2 below), we have to show that the family (X t ) t 0 possesses the Markov property. 28

Step 1: The assumption on the rates ensures that P(J n < ) = 1 for all n N 0, henceforth we assume that J n < for all n N 0. We now set Y n = X(J n ) and E n = Jn J n 1 λ(y n 1 for n N. Then (2.15) shows that ) P(E n > t, Y n = j {E 1,..., E n 1 } {Y 0,..., Y n 1 }) = e t π(y n 1, j). Hence, (Y n ) n N0 is a Markov chain with transition probability matrix Π and the same initial distribution as (X t ) t 0. Furthermore, (E n ) n N is a family of mutually independent, unit exponential random variables, and σ({y n : n N 0 }) is independent of σ({e n : n N}). Thus, the joint distribution of {Y n : n N 0 } and {E n : n N} is uniquely determined. We can recover the process (X t ) t 0 from the Markov chain (Y n ) n N0 and the family (E n ) n N in the following way. Given (e 1, e 2,...) (0, ) N and (j 0, j 1,...) I N, define Φ (Λ,Π) (t; (e 1, e 2,...), (j 0, j 1,...)) = j n for ξ n t < ξ n+1, where we put ξ 0 = 0 and ξ n = n m=1 λ j m 1 e m. Then X(t) = Φ (Λ,Π) (t; (E 1,...), (Y 0, Y 1,...)) for 0 t m=1 λ 1 j m 1 E m. Now, the distribution of (X t ) t 0 is uniquely determined once we check that m=1 λ 1 (j m 1 )E m = with probability one. At this stage our assumptions on the rates come into play. Namely, by the Strong Law of Large Numbers we know that m=1 E m = with probability one. Step 2: We show that the family (X t ) t 0 possesses the Markov property: P(X(s + t) = j X(τ), τ [0, s]) = P (t) X(s),j, (2.16) where P t (i, j) := P(X(t) = j X(0) = i). We use the Markov chain (Y n ) n N0 again. For that we consider the Φ (Λ,Π) defined above. Recall from the definition that ξ n+m t + s < ξ n+m+1 corresponds to the state j n+m of the process at time t. If we are ahead of m time steps (for the Markov chain), that is ξ n = n l=1 λ(j m+l 1)e m+l s+ξ m = s+ξ n+m, we observe that x n t < ξ n+1 corresponds to the state j n+m as well because of s+ξ n+m+1 t s+ξ n+m. This leads us to the following observation for ξ m t < ξ m+1, Φ (Λ,Π) (s + t; (e 1,...), (j 0,...)) = Φ (Λ,Π) (t; (e m+1 λ jm (s ξ m ), e m+2,...), (j m,...)). (2.17) Pick an event A σ({x(τ): τ [0, s]}) and assume that X(s) = j on A. To prove (2.16) it suffices to show that P({X(s + t) = j} A) = P t (i, j)p(a). 29

To see this, put A m = A {X(s) = m} = {E m+1 > λ(i)(s J m )} B m where B m is an event depending on {E 1,..., E m } {Y 0,..., Y m }. Clearly, B m {J m s}. We get P({X(s + t) = j} A) = = P({X(s + t) = j} A m ) m=0 P({X(s + t) = j; E m+1 > λ i (s J m )} B m ). m=0 By the memoryless property, (2.15), and our observation (2.17) we get P({X(s + t) = j; E m+1 > λ i (s J m )} B m ) m=0 = P ( {Φ (Λ,Π) (t; (E m+1 λ i (s J m ), E m+2,..., E m+n,...), (i, Y m+1,..., Y m+n,...)) = j} {E m+1 > λ i (s J m )} B m ) = P(X(t) = j X(0) = i)e(e λ i(s J m), B m ) = P t (i, j)p(a m ), from which we finally get the Markov property. The Markov property immediately shows that the family (P t ) t 0 is a semigroup because of P t+s (i, j) = k I = k I P(X(s + t) = j; X(s) = k X(0) = i) P t (k, j)p(x(s) = k) = k I P t (k, j)p s (i, k) = (P s P t ) i,j. 2.4 The Q-matrix and Kolmogorov s backward equations In this Subsection we learn how to construct a process X = (X t ) t 0 taking values in some countable state space I satisfying the Markov property in Definition 2.1. We will do this in several steps. First, we proceed like for Markov chains (discrete time, countable state space). In continuous time we do not have a unit length of time and hence no exact analogue of the transition function P : I I [0, 1]. Notation 2.11 (Transition probability) Let X = (X t ) t 0 be a Markov process on a countable state space I. 30

(a) The transition probability P s,t (i, j) of the Markov process X is defined as P s,t (i, j) = P(X t = j X s = i) for s t; i, j I. (b) The Markov process X is called (time-) homogeneous if P s,t (i, j) = P 0,t s (i, j) for all i, j I; t s 0. We consider homogeneous Markov processes only in the following, hence we write P t for P 0,t. We write P t for the I I -matrix. The family P = (P t ) t 0 is called transition semigroup of the Markov process. For continuous time processes it can happen that rows of the transition matrix P t do not sum up to one. This motivates the following definition for families of matrices on the state space I. Definition 2.12 ((Sub-) stochastic semigroup) A family P = (P t ) t 0 of matrices on the countable set I is called (Sub-) stochastic semigroup on I if the following conditions hold. (a) P t (i, j) 0 for all i, j I. (b) j I P t(i, j) = 1 (respectively j I P t(i, j) 1). (c) Chapman-Kolmogorov equations P t+s (i, j) = k I P t (i, k)p s (k, j), t, s 0. We call the family P = (P t ) t 0 standard if in addition to (a)-(c) lim t 0 P t (i, j) = δ i,j for all i, j I holds. As we have seen in the previous section, apart from its initial distribution, the distribution of a Markov process is completely determined by the semigroup (P t ) t 0. Thus, it is important to develop methods for calculating the transition probabilities P t (j, k), j, k I, directly from the data contained in the rates Λ and the transition probability matrix Π. The Chapman-Kolmogorov equations (semigroup property) leads one to suspect that P t can be written as e tq for some Q. In fact, Q should be derived by differentiation of the semigroup at t = 0. To prove these speculations, we shall first show that 31