EE266 Homework 3 Solutions

Transcription

1 EE266, Spring Professor S. Lall EE266 Homework 3 Solutions 1. Second passage time. In this problem we will consider the following Markov chain. Note that self-loops are omitted from this figure. The transition matrix for this chain is P = This matrix is given in passage_time_data.m so you don t have to type it into MATLAB. We will compute several things about this Markov chain. Let the destination set be the single state E = {1}, and let the initial state be i = 2. Recall that the first passage time to the set E is τ E = min{t > 0 x t E} The kth passage time for a Markov chain is defined recursively as where τ (1) E = τ E. τ (k) E (k 1) = min{t > τ x t E}, (a) For the Markov chain above, compute and plot the distribution of the first passage time conditioned on the initial state, i.e., compute as a function of t. E f(t) = Prob(τ E = t x 0 = i) 1

2 f (b) One may calculate the distribution of the second passage time by constructing a new Markov chain, whose first passage time is equal to the second passage time of the original chain. What is the transition graph of this chain? (c) For t = 1, 2,..., use your construction to compute the distribution of the second passage time, which is s(t) = Prob(τ (2) E = t x 0 = i). Use your method to compute and plot this distribution also. (d) Plot the sum of s(t) and f(t), and also plot Prob(x t E). Explain what you see. (e) For a given state j, the recurrence time is the first passage time to the state j given the initial condition x 0 = j. This has distribution r(t) = Prob(τ {j} = t x 0 = j). Let j = 1. Explain how to compute this, compute it, and plot it. (f) Show that s = f r, where denotes convolution. Verify this numerically, and interpret this result. (g) Give a method to compute the distribution of the kth passage time, for any k. How does your method scale with k? Solution: (a) To calculate the first passage time, we will make the nodes in E absorbing, with the corresponding transition probability matrix denoted by P a. Using distribution propagation, i.e., π t+1 = π t P a, we can calculate p(t) = Prob(x t E x 0 = i) for t = 0, 1,..., where π 0 (i) = 1 for i = x 0 and zero otherwise. Then, we can calculate the first passage time of the original Markov chain as follows: { p(t) t = 0 f(t) = Prob(τ E = t x 0 = i) = p(t) p(t 1) t > t 2

3 You can find the matlab code to calculate the first passage time below. function fp = first_passage(p,e,x0,t) n = size(p,1); % make E absorbing P_a = P; P_a(E,:) = 0; P_a(E,E) = 1; % first passage time using distribution propagation pi = zeros(t,n); prob = zeros(t,1); pi(1,x0) = 1; for t = 1 : T prob(t) = pi(t,e); pi(t+1,:) = pi(t,:)*p_a; fp = prob - [0; prob(1:-1)]; (b) We will construct a Markov chain whose first passage time gives the second passage time of the original Markov chain as follows. We replicate the Markov chain, so now we have 2n states. Then, link each state i E in the first copy to the corresponding nodes j in the second copy, as dictated by P ij. We will denote the second copy of E by E 2, i.e., E 2 = {i + n i E}. The first passage time through E 2 for the newly constructed Markov chain will give the second passage time of the original Markov chain. The transition probability matrix for this Markov chain is P i,j if 1 i, j n P (2) P ij = i,j n if i E, n + 1 j 2n P i n,j n if n + 1 i, j 2n 0 otherwise To calculate the first passage time of this Markov chain, we will need to use the method of part (a). (c) The distribution is plotted below. 3

4 s t (d) Below see the plots of f(t) + s(t), as well as Prob(x t E) for t = 0,..., 100. We notice that f(t) + s(t) = Prob(x t E) for the first few time steps, after which it seems to have converged to a value around fpluss probofxine (e) Let s denote the first passage time starting from x 0 = i by f i (t). Then, r(t) = { Prob(τ E = t x 0 E) 0 t = 0 = n i=1 P (E, i)f i(t 1) t > 0 The distribution of recurrence time, r(t), is shown below. 4

5 r t (f) We will show that s(t) = f(t) r(t) as follows: f(t) r(t) f(m)r(t m) f(m)r(t m) = tm=0 Prob(τ E = m x 0 = i) Prob(τ E = t m x 0 E) = tm=0 Prob(τ E = m x 0 = i) Prob(τ E = t x m E) E = t x 0 = i) = m= = tm=0 = t m=0 Prob(τ E = m, τ (2) = Prob(τ (2) E = t x 0 = i) = s(t). Below you can see a plot of f(t) r(t) and s(t) s(t) f(t)*r(t) (g) To compute the k th passage time, we would need to make k copies of the Markov chain, so the number of states would be nk. We would connect the nodes in the 5

6 E for the ith copy to the corresponding nodes in the (i + 1)th copy. Calculating the first passage time involves evaluating π t P for t = 0, 1,..., which scales with O(k 2 ) as k grows. You can find the matlab implementation of parts (a)-(f) below. clear all; close all; passage_time_data; E = 1; % destination node x0 = 2; % initial state do_print = 1; %% part a T = 200; fp = first_passage(p,e,x0,t); plot([0:t-1], fp, b.-, LineWidth, 1, MarkerSize,16) set(gca, xlim, [0 50]) xlabel( t ) ylabel( f ) if do_print saveas(gcf,../figures/passage_1.eps, psc2 ); %% part c % second passage time n = 6; P2 = zeros(2*n,2*n); P2(1:n,1:n) = P; P2(n+1:2*n,n+1:2*n) = P; P2(E,:) = [zeros(1,n) P(E,:)]; % make links from E to the second copy E2 = E+n; sp = first_passage(p2,e2,x0,t); figure; plot([0:t-1], sp,.-, LineWidth, 1, MarkerSize,16); set(gca, xlim, [0 100]) xlabel( t ) ylabel( s ) if do_print saveas(gcf,../figures/passage_2.eps, psc2 ); %% part d 6

7 % Prob(x_t\in E x_0) pi = zeros(t,n); pi(1,x0) = 1; for t = 1:T-1 pi(t+1,:) = pi(t,:)*p; figure; plot([0:t-1], fp+sp,.-, LineWidth, 1, MarkerSize,16) hold on plot([0:t-1], pi(:,e), ro-, LineWidth, 1); set(gca, xlim, [0 100]); leg = leg( fpluss, probofxine ); set(leg, FontSize, 20); if do_print saveas(gcf,../figures/passage_fs_vs_prob.eps, psc2 ); %% part e: recurrence time % loop over next states r = zeros(t,1); for i = 1 : n if (P(E,i) ~= 0) fp_i = first_passage(p, E, i, T); r = r + P(E,i)*[0 ;fp_i(1:-1)]; figure; plot([0:t-1], r,.-, LineWidth, 1, MarkerSize,16); hold on set(gca, xlim, [0 100]); xlabel( t ) ylabel( r ) if do_print saveas(gcf,../figures/passage_recurrence.eps, psc2 ); %% part f figure; plot([0:t-1], sp, b.-, LineWidth, 1, MarkerSize,16); hold on c= conv(fp,r); 7

8 plot([0:t-1], c(1:t), ro-, LineWidth, 1); set(gca, xlim, [0 100]); leg=leg( s(t), f(t)*r(t) ); set(leg, FontSize, 24); if do_print saveas(gcf,../figures/passage_conv.eps, psc2 ); 2. Class decomposition of a Markov chain. In lecture we claimed that the states of a Markov chain can be ordered so that the probability-transition matrix has the form [ ] P11 P P = 12, 0 P 22 where P 11 is block upper triangular with irreducible blocks on the diagonal, and P 22 is block diagonal with irreducible blocks. Each of the blocks on the diagonal of P 11 represents a transient class, while each of the blocks on the diagonal of P 22 represents a recurrent class. In this problem you will write code to find an ordering of the states that puts P in this form. We will use standard graph-theory terminology throughout the problem. Feel free to consult Wikipedia or other external sources if you encounter any unfamiliar concepts. The file class_decomposition_data.m defines a specific probability-transition matrix that you should use throughout this problem. However, your code must work for any probability-transition matrix. (a) Let G be a graph with adjacency matrix A R n n. We write i j if there is a path from node i to node j in G. We define the reachability matrix R R n n such that { 1 i j, R ij = 0 otherwise. An algorithm for constructing R is given in algorithm 1. R = A /* A is the adjacency matrix */ repeat for i = 1 to n do /* iterate over all states k we know are reachable from i */ for k such that R ik = 1 do /* iterate over all states j we know are reachable from k */ for j such that R kj = 1 do /* j is reachable from i through k */ R ij = 1 until no changes are made to R ij Algorithm 1: Computing the reachability matrix Write a function that implements algorithm 1; use the following function header. 8

9 function R = reachable_states(a) For reference, our implementation of reachable_states is less than twenty lines. Hint. The function find if you are using MATLAB may be useful. (b) We write i j if i j and j i, and we define the communication matrix C R n n such that { 1 i = j or i j, C ij = 0 otherwise. Explain how to construct the communication matrix from the reachability matrix. (c) We define the transience vector t R n such that { 1 i is a transient state, t i = 0 otherwise. Explain how to construct the transience vector from the communication and reachability matrices. (d) Write a function that implements algorithm 2. L Empty List /* L will contain sorted nodes */ S Set of all nodes with no incoming edges while S is non empty do Remove a node n from S Add n to tail of L for each node m with edge e from n to m do Remove edge e from graph if m has no other incoming edges then Insert m into S if Graph has edges then return error: graph has at least one cycle else return L /* L a topologically sorted order */ Algorithm 2: Computing topological sort Use the following function header. function L = topological_sort(a) For reference, our implementation of topological_sort is less than twenty lines. (e) Suppose that duplicate rows of C have been removed, and the rows have been sorted so that the first n t rows represent the transient classes. (You need to write code to do this; the commands unique and sort may be useful.) Note that there is now a unique row of C corresponding to each class. For two classes 9

10 C i, C j {1,..., n}, we write C i C j if x y for some x C i and y C j. The adjacency matrix A t R nt nt for the set of transient classes is defined such that { 1 C i C j, (A t ) ij = 0 otherwise. Explain how to use the reachability matrix R of the Markov chain to find the adjacency matrix A t for the set of transient classes. (f) You should now have all the tools you need to construct an ordering of the states that puts the probability transition matrix in the desired form. Apply your method to the matrix in class_decomposition.data.m. Attach a plot of calling spy on your reordered matrix (the data file generates a plot of the original matrix). For reference, our solution has thirty-five lines (not including reachable_states and topological_sort). Solution: (a) An example implementation of reachable_states is given below. function R = reachable_states(a) n = size(a,1); R = A; no_changes = false; while ~no_changes no_changes = true; for i = 1:n for k = find(r(i,:)) for j = find(r(k,:)) if ~R(i,j) no_changes = false; R(i,j) = 1; (b) Note that (R T ) ij = 1 if j i and (R T ) ij = 0 otherwise. Form the matrix R R T, where denotes entrywise and. Then, we have that (R R T ) ij = 1 if i j, and (R R T ) ij = 0 otherwise. This is very close to the definition of C; the only difference is that the diagonal entries of R R T may not be equal to one. We can correct this deficiency using the formula C = (R R T ) I, where denotes entrwise or, and I is the identity matrix. 10

11 (c) Observe that i is a transient state if and only if n R ij > j=1 n C ij. j=1 This condition says that we can reach more states from i than there are states that communicate with i; this is what it means for i to be transient. We can use this condition to construct t from R and C. (d) An example implementation of topological_sort is given below. function L = topological_sort(a) L = []; S = find(sum(a) == 0); while ~isempty(s) n = S(1); S = S(2:); L = [L n]; for m = find(a(n,:)) A(n,m) = 0; if sum(a(:,m)) == 0 S = [S m]; if any(a(:)) L = []; (e) Let x and y be fixed representatives of C i and C j, respectively. We claim that (A t ) ij = 1 if and only if R xy = 1. The definition of A t says that (A t ) ij = 1 if and only if there exist x C i and ỹ C j such that x ỹ. Then, we have that x x ỹ y, so the properties of the communication relation imply that x y if and only if x ỹ. This proves the claim. (f) The following script uses the tools and observations developed above to compute the class decomposition of a probability-transition matrix. clear all; close all; clc; class_decomposition_data; R = reachable_states(p > 0); C = (R & R ) eye(n); 11

12 t = any(r & ~C,2); [C, idx] = unique(c, rows ); t = t(idx); [t, idx] = sort(t, desc ); C = C(idx,:); nt = sum(t); At = zeros(nt); for i = 1:nt x = find(c(i,:),1); for j = 1:nt y = find(c(j,:),1); if i ~= j && R(x,y) At(i,j) = 1; L = topological_sort(at); C(1:nt,:) = C(L,:); idx = []; for i = 1:size(C,1) idx = [idx, find(c(i,:))]; P = P(idx,idx); figure(); spy(p); print -depsc class_decomposition.eps The reordered probability-transition matrix is as follows. 12

13 nz =

14 3. Markov web surfing model. A set of n web pages labeled 1,..., n contain (directed) links to other pages. We define the link matrix L R n n as { 1 if page i links to page j L ij = 0 otherwise. We define o R n as o = L1, which gives the number of outgoing links from each page. A very crude model of a web surfer is a Markov chain on the pages, with transitions described as follows. The model includes a parameter θ (0, 1), which (roughly) gives the probability that the surfer follows a link from the current page. For a page with o i > 0 (i.e., with at least one outgoing link) the surfer moves to each of the linkedto pages with probability θ/o i, and jumps to a page not linked to i with probability (1 θ)/(n o i 1). For a page with no outgoing links (i.e., o i = 0) the surfer jumps to a random page, chosen from a uniform distribution on (the other) pages. We will assume that web surfer starts at a random page, uniformly distributed. We earn a payment when the surfer follows (i.e., clicks on) a link, given by R ij 0. This payment matrix satisfies R ij = 0 when L ij = 0 (i.e., we are not paid for random jumps; only following links). The following questions concern the specific instance of the problem with data given in link_matrix_data.m. (a) What is the most likely page the surfer is on, at time t = 10? at t = 100? (b) Let J denote the expected total payment over t = 0,..., 50. Compute J three ways: Monte Carlo simulation (which gives an estimate of J, not the exact value). Distribution propagation. Value iteration. Be sure to check that the values are consistent. Remark. The Markov model described in this problem leads to Google s famous PageRank, which corresponds to the fraction of time spent at each site, when T. (The current version of PageRank is based on far more than just the link topology, but the first versions really did make heavy use of the Markov surfing model.) Solution: (a) To find the most likely page the surfer is on, we will calculate the state distributions at times t = 10 and t = 100 by distribution propagation. The most likely page at t = 10 is page 96, and the most likely page at t = 100 is page 83. Below you can see a plot of the most likely page for all time t = 0,..., 100. Note that for t = 1, all pages are equally likely to be visited, so the choice of page 1 in the plot is arbitrary. 14

15 pmax t (b) We check the expected total payment using the three methods. We get J = using value iteration, J = using distribution propagation, and J = using Monte Carlo simulation. The code is shown below. clear all; close all; link_matrix_data; % create the probability transition matrix P = zeros(n,n); for i = 1 : n n_o = sum(l(i,:)); if n_o > 0 P(i,:) = (1-theta)/(n-n_o-1); P(i,L(i,:)>0) = theta/n_o; else P(i,:) = 1/(n-1); P(i,i) = 0; % distribution propagation T = 101; pi = zeros(t,n); pi(1,:) = 1/n; exp_g = zeros(t,1); for t = 1 : T 15

16 pi(t+1,:) = pi(t,:)*p; exp_g(t) = sum(pi(t,:)*(r.*p)); % most likely page at t max_prob = zeros(t,1); for t = 1:T [xx max_prob(t)] = max(pi(t,:)); stairs(0:t-1,max_prob, LineWidth, 2); xlabel( t ); ylabel( pmax ); set(gca, Ylim, [1 n]); saveas(gcf,../figures/markovian_pagerank_ml_page.eps, psc2 ); % cost over 50 time steps using distribution propagation T = 51; J_distprop = sum(exp_g(1:t)); % value iteration V = zeros(n,t+1); V(:,T+1) = 0; for t = T:-1:1 V(:,t) = sum(p.*r,2) + P*V(:,t+1); J_valiter = mean(v(:,1)); % monte carlo N = 1000; cost = zeros(n,1); for iter = 1 : N x = randsample(n,1,true,(1/n)*ones(n,1)); cost(iter) = 0; for t = 1 : T y = randsample(n,1,true,p(x,:)); cost(iter) = cost(iter) + R(x,y); x = y; J_mc = mean(cost); 16