Homework set 2 - Solutions Math 495 Renato Feres Simulating a Markov chain in R Generating sample sequences of a finite state Markov chain. The following is a simple program for generating sample sequences of a Markov chain. It consists of a function that takes in three arguments: N (number of steps), π 0 (the initial probability distribution), P (the transition probabilities matrix), and produces a vector of length N : (X 1,..., X N ) where each X i lies in the state space {1,2,..., s. Note that π 0 must be a vector of length s and P must be an s-by-s matrix. Here is the program: #It is assumed that the states are {1,2,...,s #where s is the length of pi0, and P is s-by-s. #pi0 is the probability distribution of the initial step #and P is the transition probabilities matrix. Markov=function(N,pi0,P) { #Number of states s=length(pi0) X=matrix(0,1,N) a=sample(c(1:s),1,replace=true,pi0) X[1]=a for (i in 2:N) { a=sample(c(1:s),1,replace=true,p[a,]) X[i]=a X To test that it works as expected, let us do the following experiment. Let π 0 = (1,0) and P = ( 0.8 0.2 0.3 0.7 We generate N = 10 5 steps of the sequence: X=Markov(100000,pi0,P) and compute the relative frequency of occurrences of state 1 (the set of states being {1, 2.) It is not difficult to compute the exact stationary distribution: π = (0.6,0.4). Let us compute the approximate stationary probability of 1 using the Markov chain simulation: ). #Define the probability distribution of X0 pi0=c(1,0)
#Define the transition probabilities matrix P=matrix(0,2,2) P[1,]=c(0.8,0.2) P[2,]=c(0.3,0.7) #Number of steps N=10^5 > X=Markov(N,pi0,P) > sum(x==1)/n [1] 0.60168 The result is sufficiently close to 0.6 to make me believe that the program has no bugs. Here is a variant of the above program. We wish to find the number of steps, beginning at state i, till the chain with transition matrix P reaches a different state j. We count step 0 but not the step at which the chain is at j. #This program gives the number of steps in one #run of the Markov chain with transition probabilities #Matrix P, starting at i, till it reaches j. We count #the 0th step but not the step when the chain is at j. #Be careful: if the probability of eventually reaching j from #i is 0, the program will not stop! Number_of_steps=function(i,j,P) { T=0 u=dim(p) s=u[1] X=i while (X!=j){ X=sample(c(1:s),1,replace=TRUE,P[X,]) T=T+1 return(t) Suppose we wish to find the expected number of steps it takes to get to state 2 starting at 1 for the two-state chain with transition matrix P given above. We can do the following: P=matrix(0,2,2) P[1,]=c(0.8,0.2) P[2,]=c(0.3,0.7) N=1000 T=0*c(1:N) for (n in 1:N){ T[n]=Number_of_steps(1,2,P) sum(t)/n 2
Note that we are approximating the expected time by the mean number of steps of N sample values. One run of this program, for N = 1000, gave me the value 5.07. It is not difficult (using the material of section 1.4 of the textbook) to show that the exact number is 5. Let us do one more experiment based on the chain of the first example above. Here it will be convenient to label the states as 0 or 1, so the state set is S = {0,1. We are interested in the frequency over the long run of patterns of 4 consecutive symbols: 0010, 1001, etc. Each pattern corresponds to an integer expressed in base 2. So, for example, 0110 6, 1101 13, and so forth. In general, the correspondence between patterns and integers is given by a 0 a 1 a 2 a 3 a 0 2 0 + a 1 2 1 + a 2 w 2 + a 3 2 3. This naturally defines another Markov chain with 16 states and state transitions a i a i+1 a i+2 a i+3 a i+1 a i+2 a i+3 a i+4. #Define the probability distribution of X0 pi0=c(1,0) #Define the transition probabilities matrix P=matrix(0,2,2) P[1,]=c(0.8,0.2) P[2,]=c(0.3,0.7) #Number of steps N=10^5 X=Markov(N,pi0,P) sum(x==1)/n #Subtract 1 from each entry of X so that states are 0 and 1 X=X-1 #Initialize an array Y. It will have integer entries from 0 to 15 #Make it all zeros initially: Y=matrix(0,1,N-3) #Now for each index i associate the integer from 0 to 15 corresponding #to the binary string of 4 symbols given by X[i]... X[i+3] for (i in 1:(N-3)) { Y[i]=X[i]*1+X[i+1]*2+X[i+2]*4+X[i+3]*8 #We can get an idea of the relative frequencies of the various binary p #patterns by plotting a histogram: hist(y,freq=false,breaks=seq(from=-0.5,to=15.5,by=1)) #We can also get the relative frequencies of the 16 patterns directly: rel_freq = matrix(0,1,16) for (k in 0:15) { rel_freq[k+1]=sum(y==k)/(n-3) #The same vector of relative frequencies could have been obtained #from the histogram as follows: h=hist(y,freq=false,breaks=seq(from=-0.5,to=15.5,by=1)) rel_freq_from_hist = h$density 3
The histogram is shown next. Figure 1: Histogram for the relative frequencies of patterns of 4 symbols for a two-state Markov chain. Problems 1. (Similar to Exercise 1.5, page 36 in text.) Consider the Markov chain with state space S = {0,...,5 and transition matrix 0.1 0 0 0 0.9 0 0 0.7 0.3 0 0 0 0 0.5 0.5 0 0 0 P =. 0 0.25 0.25 0 0.25 0.25 0.7 0 0 0 0.3 0 0 0.2 0 0.2 0.2 0.4 (a) Draw a states and transitions graph. (b) What are the communication classes? (c) Which communication classes are recurrent and which are transient? (d) Suppose the system starts in state 2. What is the probability that it will be in state 2 at some large time? (e) Suppose the system starts in state 5. What is the probability that it will be in state 5 at some large time? (f) Numerically obtain P 500. Are your answers to d and e supported by the values of the entries of this matrix? Solution. (a) The transitions diagram is shown below in Figure 2 (b) It is clear from the diagram that the communication classes are {1,2, {3,5, and {0,4. 4
Figure 2: Graph of Problem 2. (c) The classes {1,2 and {0,4 are recurrent while {3,5 is transient. (d) The subset of states {1, 2 is invariant. This means that if the Markov chain has initial probability distribution supported ( on this ) set, it will never leave the set. In this case, the chain behavior is determined by the matrix 0.7 0.3 block. The stationary distribution is given by the left-eigenvector associated to the eigenvalues 1. 0.5 0.5 This this the row vector π that satisfies π(i P) = 0. In detail, (π 1,π 2 ) ( 0.3 0.3 0.5 0.5 which gives 0.3π 1 = 0.5π 2. Since π i 0 and π 1 + π 2 = 1 we obtain ( 5 π = 8, 3 ). 8 ) = (0,0) Thus in the long run the probability that the chain will be back at 2 is 3/8. (e) Because 5 is a transient state, the probability that the state will be 5 at some large time approaches 0. (f) I will do this in R. > P=matrix(0,nrow=6,ncol=6) > P[1,]=c(0.1,0,0,0,0.9,0) > P[2,]=c(0,0.7,0.3,0,0,0) > P[3,]=c(0,0.5,0.5,0,0,0) > P[4,]=c(0,0.25,0.25,0,0.25,0.25) > P[5,]=c(0.7,0,0,0,0.3,0) > P[6,]=c(0,0.2,0,0.2,0.2,0.4) > P%^%500 [,1] [,2] [,3] [,4] [,5] [,6] [1,] 0.4375000 0.0000000 0.0000000 0.000000e+00 0.5625000 0.00000e+00 [2,] 0.0000000 0.6250000 0.3750000 0.000000e+00 0.0000000 0.00000e+00 [3,] 0.0000000 0.6250000 0.3750000 0.000000e+00 0.0000000 0.00000e+00 [4,] 0.1590909 0.3977273 0.2386364 5.091561e-152 0.2045455 1.27289e-151 [5,] 0.4375000 0.0000000 0.0000000 0.000000e+00 0.5625000 0.00000e+00 [6,] 0.1988636 0.3409091 0.2045455 1.018312e-151 0.2556818 2.54578e-151 > 3/8 5
[1] 0.375 The matrix P 500 confirms our conclusions: the probability of 2, having started at 2, seems to stabilize at 0.375, which is indeed 3/8. Regardless of where the chain begins, the probability of being at state 5 (this corresponds to the 6th column) at step 500 is negligibly small. 2. (Text, Exercise 1.19, page 40.) Suppose we flip a fair coin repeatedly until we have flipped four consecutive heads. (a) Give a Markov chain (by drawing its transition diagram with the transition probabilities indicated next to the arrows) that represents this random experiment. (Hint: consider a Markov chain with state space S = {0,1,2,3,4. The state at any given time is the number of consecutive heads since the last time tail came up. The state is reset to 0 at each occurrence of tail and 4 is an absorbing state.) (b) Write down the transitions matrix P. (c) Describe the communication classes and indicate which are transient and which are recurrent. (d) What is the expected number of flips that are needed? (e) Do a computer simulation to verify that your answer is reasonable. I suggest the following: modify the Markov chain program given in the preliminaries section of this assignment so that it gives the number of steps till a given state is reached (in this case, the state 4) for each run of the chain. Then run the chain, starting at 0, a large number of times so as to get a large number of sample values of the random variable T 4. Then compute the sample mean of T 4. This mean should, by the law of large numbers, approximate the expected value we want. Solution. (a) The transitions diagram is given in Figure 3. Figure 3: Diagram for Problem 2. (b) The transitions matrix is P = 1/2 1/2 0 0 0 1/2 0 1/2 0 0 1/2 0 0 1/2 0 1/2 0 0 0 1/2 0 0 0 0 1 6
(c) It is clear from the diagram that {0,1,2,3 is a transient communication class and {4 is a recurrent communication class. (d) My solution to this problem will seem longer than necessary because it will derive the relevant part of the theory given in section 1.4 of the textbook rather than use the results given there. Define the substochastic matrix 1/2 1/2 0 0 Q = 1/2 0 1/2 0 1/2 0 0 1/2. 1/2 0 0 0 Note that Q gives the transition probabilities among the transient states. Let N (k) be the number of times the chain passes through transient state k before reaching the recurrent state 4 and let 1 k be the indicator function of the state k. Thus 1 k (X n ) = 1 if at time step n the state of the chain is k, and 0 if not. Thus N (k) = 1 k (X 0 ) +1 k (X 1 ) + and the expected number of visits to k before arriving at 4 is E[N (k) X 0 = 0] = E[1 k (X n ) X 0 = 0] = Q n 0k. n=0 n=0 Also note that the number of steps before getting to state 4 (counting time 0 as one step) is T 4 = N (0) + N (1) + N (2) + N (3) and the expected value of T 4 is the sum of the expected values of the N (k), k 4. A key remark is I +Q +Q 2 + = (I Q) 1 =: M. Therefore, E[T 4 X 0 = 0] = M 00 + M 01 + M 02 + M 03. A matrix inversion exercise (you may use R or other software if you like) gives 16 8 4 2 M = 14 8 4 2 12 6 4 2. 8 4 2 2 The value we want is then E[T 4 X 0 = 0] = 16 + 8 + 4 + 2 = 30. (e) See the example in the preliminaries to this homework set. The expected value we want can be computed as follows: P=matrix(0,nrow=5,ncol=5) P[1:4,1]=1/2 P[1,2]=1/2 P[2,3]=1/2 7
P[3,4]=1/2 P[4,5]=1/2 P[5,5]=1 N=10000 T=0*c(1:N) for (n in 1:N){ T[n]=Number_of_steps(1,5,P) sum(t)/n With N = 10000 sample values of T 4, here are a few values for the sample mean I obtained: 30.22,29.51,29.90. This seems to confirm our solution 30. 3. (Text, Exercise 1.8, page 36.) Consider simple random walk on the graph below. (Recall that a simple random walk on a graph is the Markov chain which at each time moves to an adjacent vertex, each adjacent vertex having the same probability.) (a) Draw the transitions diagram with the values of the transition probabilities next to each arrow. (b) Write the transitions matrix P. (States A,B, etc. correspond to row and column indices 1,2, etc.) (c) In the long run, about what fraction of time is spent in vertex A? (d) Suppose a walker starts in vertex A. What is the expected number of steps until the walker returns to A? (e) Suppose a walker starts in vertex C. What is the expected number of visits to B before reaching A? (f) Suppose a walker starts in vertex D. What is the probability that the walker reaches A before reaching B? (g) Again assume the walker starts in C. What is the expected number of steps until the walker reaches A? A B C D E Figure 4: Graph of Problem 3. Solution. (a) The transitions diagram is shown in Figure 7. (b) The matrix P is immediately read from the diagram: P = 0 1/3 1/3 1/3 0 1/3 0 1/3 0 1/3 1/2 1/2 0 0 0 1/2 0 0 0 1/2 0 1/2 0 1/2 0. 8
A B C D E Figure 5: Transition probabilities diagram for the random walk of Problem 3. (c) This chain is clearly a recurrent since it is both irreducible and finite. We know that, in the long run, the fraction of time spent in a given vertex equals the stationary probability of that state. Thus we need to compute the stationary probability vector π. Recall that, by definition, πp = π. This equation amounts to a linear system, which can be solved in several ways. In R you may use the command eigen(t(p))$vec[,1] to obtain the (unnormalized) right-eigenvector of the transpose of P for the eigenvalue 1. (You will want to also look at the eigenvalues, eigen(t(p))$val to be sure that the first eigenvalues is indeed 1.) It is also not hard to solve the system by the method of row-reduction. In any event, you should get π = (1/4,1/4,1/6,1/6,1/6). From this we immediately read the fraction of time 1/4 that the chain spends in A in the long run. (d) We know that the expected number of steps back to A is 1/π(A), which is 4. (e) The approach here is similar to that of problem 2 (d). We first modify the matrix P so that state A becomes absorbing: 1 0 0 0 0 1/3 0 1/3 0 1/3 P = 1/2 1/2 0 0 0. 1/2 0 0 0 1/2 0 1/2 0 1/2 0 The problem is now to find to total number of visits to B for a chain that starts at C having transition matrix P. Recall that the key is to study the (southeast) block submatrix 0 1/3 0 1/3 Q = 1/2 0 0 0 0 0 0 1/2. 1/2 0 1/2 0 Let M = (I Q) 1. Then by the same argument used above in Problem 2 (d), the expected number of times the walker visits B starting at C is given by the entry M 21. I will calculate M using R: P=matrix(0,5,5) P[1,2:4]=1/3 9
P[2,c(1,3,5)]=1/3 P[3,1:2]=1/2 P[4,c(1,5)]=1/2 P[5,c(2,4)]=1/2 Q=P[2:5,2:5] solve(diag(4)-q) [,1] [,2] [,3] [,4] [1,] 1.6363636 0.5454545 0.3636364 0.7272727 [2,] 0.8181818 1.2727273 0.1818182 0.3636364 [3,] 0.5454545 0.1818182 1.4545455 0.9090909 [4,] 1.0909091 0.3636364 0.9090909 1.8181818 This gives the values M 21 = 0.8181818. With some guesswork based on this matrix we can find the exact inverse, and the value M 21 = 9/11. (f) The basic idea is explained in page 29 of the textbook. Let us change the matrix P so that states A and B are now absorbing. I will call the new transition matrix P. So 1 0 0 0 0 0 1 0 0 0 P = 1/2 1/2 0 0 0. 1/2 0 0 0 1/2 0 1/2 0 1/2 0 ( I 0 ) Thus P = S Q where S = 1/2 1/2 1/2 0 0 1/2, Q = 0 0 0 0 0 1/2 0 1/2 0. Define the matrix α as the solution to the equation α = S +Qα. Therefore, α = (I Q) 1 S. Then the probability that the chain starting at D will reach A before B is α 2,1. It is now an easy computation to obtain (I Q) 1 = 1 0 0 0 4/3 2/3 0 2/3 4/3 and α = (I Q) 1 S = 1 0 0 0 4/3 2/3 0 2/3 4/3 1/2 1/2 1/2 0 0 1/2 = 1/2 1/2 2/3 1/3 1/3 2/3. We conclude that the probability of reaching A before B, starting at D is 2/3. (g) As we saw in problem 2, this is given by M 21 + M 22 + M 23 + M 24 = 2.63636. 10
4. (From somewhere on the internet.) I assume for simplicity that you are the only player of this board game. Start at square 1 and move at each step to one, two, or three squares ahead according to the outcome of spinning a roulette. If you land at the bottom of a ladder, immediate climb it. If you land on the head of a snake, slide down to the tip of the snake s tail. The game ends at reaching square 12. The only way to reach 12 from squares 10 and 11 is by drawing 2 or 1, respectively; if the roulette gives a higher number than necessary, stay where you are. We wish to run a stochastic simulation of this game by using the Markov chain program. Figure 6: The snakes-and-ladders game. (a) What is the expected number of steps to go from 1 to 12? What is its standard deviation? (b) Draw a histogram of the distribution of the number of steps. (Be careful with the breaks!) Before writing your program, convince yourself that my transition probabilities diagram given below correctly interprets the rules of the game. The simulation consists of running the game till finish a large number of times and obtaining the statistics asked in the problem. Figure 7: Markov transitions diagram for the snakes-and-ladders game. Remarks about this problem: First note that our original program Markov needs some changes. In that program we have a deterministic stopping time N, but now this time is random and we cannot tell in advance what it is. The following modifications should take care of this problem. The input variable TerminalStates is an array 11
giving the states at which the chain should stop. In this example we have the single element array c(8) since 8 is the index of state 12. (The mathematical chain simply goes on forever inside this terminal set; for the snakes and ladders example, a sample path of the chain could be 1 3 5 8 10 10 11 12 12 12 but in the program we wish to interrupt it at the first occurrence of 12.) ################################ #We modify the first Markov program to allow #for a set of terminal states, at which the #chain must stop. Let TerminalStates be the array of #such states. Markov_stopped=function(pi0,P,TerminalStates) { #Number of states s=length(pi0) X=matrix(0,1,1000) a=sample(c(1:s),1,replace=true,pi0) X[1]=a i=1 while (X[i] %in% TerminalStates == FALSE) { a=sample(c(1:s),1,replace=true,p[a,]) i=i+1 X[i]=a U=X[1:i] return(u) ############################### Because the states and their indices do not coincide, it could be helpful to translate the output of the previous program (a string of indices) into a string of actual states. This is a minor point, but the following script does the translation: ############################### #This function associates to each of #1, 2, 3, 4, 5, 6, 7, 8 the corresponding states: #1, 2, 3, 5, 8, 10, 11, 12 Substitute_state = function(x) { a1 = which(x==4) a2 = which(x==5) a3 = which(x==6) a4 = which(x==7) a5 = which(x==8) u = x u[a1] = 5 12
u[a2] = 8 u[a3] = 10 u[a4] = 11 u[a5] = 12 return(u) ################################ To test the program, here s what one trial run would look like (you ll need to define pi0=c(1, 0, 0, 0, 0, 0, 0, 0) and P; the latter is the matrix of transition probabilities.) > Substitute_state(Markov_stopped(pi0,P,c(8))) [1] 1 3 5 8 10 10 11 12 Having a program to simulated the Markov chain with a stopping time, we can now answer the questions posed. For example, for part (a), run the chain from state 1 till it stops a large number times (say, n = 10000). Collect the number of steps in each run into a vector U then find mean(u) and sd(u). Solution. (a) Before running the new Markov chain program, we need the transition probabilities matrix and the initial probability vector. They are: P = matrix(0,8,8,byrow=true) a = 1/3 b = 2/3 P[1,]=c(0, a, a, a, 0, 0, 0, 0) P[2,]=c(0, 0, a, b, 0, 0, 0, 0) P[3,]=c(0, 0, 0, b, 0, 0, a, 0) P[4,]=c(0, a, 0, 0, a, 0, a, 0) P[5,]=c(0, 0, 0, 0, a, a, a, 0) P[6,]=c(0, 0, 0, 0, 0, a, a, a) P[7,]=c(0, 0, 0, 0, 0, 0, b, a) P[8,]=c(0, 0, 0, 0, 0, 0, 0, 1) pi0=c(1, 0, 0, 0, 0, 0, 0, 0) We then run the experiment: #Part (a): number of steps to reach terminal state 12 #The number of trial runs (sample paths) of the Markov chain is N = 10000 # #For each trial we keep a record of the number os steps in a vector U. #I ll initialize U as a vector of zeros: U = 0*c(1:N) 13
#We can now run the experiment: for (i in 1:N) { X=Markov_stopped(pi0, P, c(8)) U[i]=length(X) #The values we want are > mean(u) [1] 8.0648 > sd(u) [1] 3.227111 Therefore, if U is the random variable giving the number of steps till reaching the terminal state, then Mean value of U = 8.0648; Standard deviation of U = 3.2271. (b) The histogram is obtained with the R-command hist(u,breaks=seq(from=0.5, to=35.5, by=1),freq=false) Figure 8: Histogram for the number of steps till reaching the terminal state; part (a) of problem 4. 14