Markovské řetězce se spojitým parametrem

Markovské řetězce se spojitým parametrem Mgr. Rudolf B. Blažek, Ph.D. prof. RNDr. Roman Kotecký, DrSc. Katedra počítačových systémů Katedra teoretické informatiky Fakulta informačních technologií České vysoké učení technické v Praze Rudolf Blažek & Roman Kotecký, 2011 Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 20 Evropský sociální fond Praha & EU: Investujeme do vaší budoucnos@

Mgr. Rudolf B. Blažek, Ph.D. prof. RNDr. Roman Kotecký, DrSc. Department of Computer Systems Department of Theoretical Informatics Faculty of Information Technologies Czech Technical University in Prague Rudolf Blažek & Roman Kotecký, 2011 Statistics for Informatics MI-SPI, ZS 2011/12, Lecture 20 The European Social Fund Prague & EU: We Invest in Your Future

Recall Discrete-time Markov Chains The Markov Property for a discrete-time Markov chains P(X n+1 = j X n = i, X n 1 = i n 1,..., X 0 = i 0 )= = P(X n+1 = j X n = i) = P i,j = p i,j = p(i, j) In continuous time difficult to define conditional probability given Xr for all r < s we instead work with all choices of 0 s0 < s1,..., sn < s (for all n 1) 3

Definition!!! Markovský řetezec se spojitým parametrem Xt, t 0 is a continuous-time Markov chain if for any times 0 s0 < s1,..., sn < s and any states i0, i1,..., in, i, j (for all n 1) we have P(X t+s = j X s = i, X = i sn n,..., X = i s0 0 )= = P(X t = j X 0 = i) = p t (i, j) 4

Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example Let N(t), t 0 be a Poisson Process with rate λ. Let Yn be a discrete-time Markov chain with transition probabilities u(i,j). Assume N(t) is independent of Yn. Then Xt = YN(t) is a continuous-time Markov chain. Xt jumps according to u(i,j) at each arrival of N(t). Most continuous-time Markov chains can be constructed is a similar way. We will show how later today. 5

Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example N(t), t 0 is a Poisson Process with rate λ, therefore the number of arrivals in (0,t) is a Poisson random variable * * * * * * * N(t) ~ Poisson(λt) t P(N(t) =n) =e ( t)n n! Therefore p t (i, j) =P(X t = j X 0 = i) 1X = P(N(t) =n, X t = j X 0 = i) n=0 6

Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example 1X p t (i, j) = P(N(t) =n, X t = j X 0 = i) n=0 1X Xt = YN(t) N(t) is independent of the MC Yn = = n=0 1X n=0 P(N(t) =n, Y N(t) = j Y N(0) = i) P(N(t) =n) P(Y n = j Y 0 = i) 7

Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example 1X p t (i, j) = P(N(t) =n) P(Y n = j Y 0 = i) n=0 = 1X n=0 e t ( t)n n! u n (i, j) N(t) ~ Poisson(λt) n-step transition probability for discretetime Markov chain Yn 8

Chapman-Kolmogorov Equation Chapman-Kolmogorov Equation Theorem X k p s (i, k) p t (k, j) =p s+t (i, j) Idea of the Proof We want to go from i to j in time s + t First we must go to some state k in time s Then we must finish by going from k to j in time t We must consider all states k, thus we sum over k 9

Chapman-Kolmogorov Equation Chapman-Kolmogorov Equation Theorem X k p s (i, k) p t (k, j) =p s+t (i, j) Assume we know pt(i,j) for all t t0 (and for all i,j) The theorem allows us to calculate pt(i,j) for all t > 0!!! Let t0 0, consider the derivative at 0: lim t 0!0 +(p t (i, j) 0)/t 0 0 The derivative at 0 should be be enough to determine pt(i,j) for all t > 0. Well, it is!! (We will show this later today.) 10

Jump Rates Jump Rates Definition!!!!!!!!!!! Intenzita přechodu Denote the derivative of pt(i,j) with respect to time at 0 as If the derivative exists, then we call q(i,j) the jump rate from i to j. q(i, j) = lim h!0 + p h (i, j) h for j 6= i Note that we do not calculate q(i,i) Why is it called a jump rate? Let s look at our construction of the Markov chain... 11

Jump Rates Why is q(i,j) called a jump rate? Previous Example Xt = YN(t); Yn = discrete-time MC with transition prob. u(i,j). N(t), t 0 is a Poisson Process with rate λ, indep. of Yn. If Xt is at i, it makes jumps with rate λ (a Poisson process) It goes to j with probability u(i,j) This is a Poisson process thinning: Xt jumps from i to j as a Poisson Process with rate λ u(i,j) Next we will show that in this example q(i,j) = λ u(i,j) That is why we call q(i,j) the jump rate from i to j 12

Jump Rates Why is q(i,j) called a jump rate? Previous Example Xt = YN(t); Yn = discrete-time MC with transition prob. u(i,j). N(t), t 0 is a Poisson Process with rate λ, indep. of Yn. We got before p t (i, j) = 1X n=0 e t ( t)n n! u n (i, j) Therefore p h (i, j) h = 1 h 1X n=0 e h ( h)n n! u n (i, j) 13

Jump Rates Why is q(i,j) called a jump rate? Previous Example p t (i, j) = 1 h h 1X n=0 h e ( h)n u n (i, j) n! Let h 0 1 λ u(i,j) p h (i, j) h = 1 e h ( h)0 u 0 (i, j) +e h 0! + e h 1 1X n=2 n h n 1 h ( )1 1! u 1 (i, j) = 0 j i (transition in 0 steps) 0 u n (i, j) 0 n! 14

Jump Rates Why is q(i,j) called a jump rate? Summary of Previous Example Xt = YN(t); Yn = discrete-time MC with transition prob. u(i,j). N(t), t 0 is a Poisson Process with rate λ, indep. of Yn. Xt jumps from i to j as a Poisson Process w/ rate λ u(i,j) The jump rates of the process Xt are q(i, j) = lim h!0 p h (i, j) h = lim h!0 e h u(i, j) = u(i, j) Most Markov chains can be constructed in a similar way!! That is why for any continuous-time Markov chain we call q(i,j) the jump rates from i to j. 15

Jump Rates Poisson Process as a Markov Chain Simple Example!!!!!!!!! Poisson Process Consider a Poisson process with rate λ. Let X(t) be number of arrivals of the process in (0,t). X(t) increases from n to n + 1 at rate λ: q(n, n+1) = λ for all n 0 This simple example will allow us to create other more complicated examples... 16

Jump Rates Queueing System M / M / s Infinite FCFS Queue m s Parallel Servers w/ Exp(!) service time Infinite Customer Population Arriving Customers Poisson Process w/ rate " Departing Customers 17

Jump Rates Queueing System M / M / s Example!!!!!!!!!!!! M / M / s Queue Consider load-balancing s replicated database servers. A request is routed to the next available server. Requests line-up in a single queue if all servers are busy. Requests arrive at times of a Poisson Process w/ rate λ: q(n, n+1) = λ for all n 0 Service times are random independent ~ Exp(μ): q(n, n 1) = nμ q(n, n 1) = sμ if all 0 n s if all n s 18

Jump Rates Queueing System M / M / s with Balking Example!!!!!!!! M / M / s Queue with Balking The same setup; requests arrive as Poisson Process (λ). The load balancer forwards requests randomly to a different data center with probability (1 an) if there are n requests. I.e., the new request stays with probability an: q(n, n+1) = λ an for all n 0 Service times are as before independent ~ Exp(μ): q(n, n 1) = nμ q(n, n 1) = sμ if all 0 n s if all n s 19

Jump Rates Jump Rates vs. Transition Probabilities Example In our example the jump rates are q(i, j) = u(i, j) We leave i with rate λ (a Poisson process)... * * * * * * * *... and go to j with probability u(i,j) Note this can be reversed u(i, j) =q(i, j)/ This is how we can construct the MC from its jump rates... * * * * * * * *... but what is λ? 20

Constructing Markov Chains from Jump Rates From Jump Rates To a Markov Chain Constructing the MC from its jump rates Let λi = j i q(i,j). This is the rate at which Xt leaves i. λi =... Xt leaves i immediately... so we assume λi <. For λi > 0 we define (a routing matrix) r(i, j) =q(i, j)/ i If Xt is in state i with λi = 0, then Xt stays in i forever. If λi > 0, then Xt waits in i for a random time ~ Exp(λi). Then Xt jumps to j with probability r(i,j). We have many Poisson processes with λi for each state i 21

Constructing Markov Chains from Jump Rates A Single Poisson Process for Markov Chains with Bounded Rates Constructing the MC from bounded jump rates Let λi = j i q(i,j) and Λ = maxi λi. Assume that Λ < (bounded rates) and define u(i, j) =q(i, j)/ u(i, i) =1 X j6= i for j 6= i u(i, j) Xt tries to make a transitions with rate Λ (Poisson Process). Xt leaves i for j with probability u(i,j), but stays in i with u(i,i). Any MC with bounded rates can be constructed as our initial example! 22

Kolmogorov s Backward and Forward Equations Computing Transition Probabilities Our goal is to use the jump rates q(i,j) to find the transition probabilities p t (i, j) =P(X t = j X 0 = i) Recall the Chapman-Kolmogorov equation: X p s (i, k) p t (k, j) =p s+t (i, j) k It can be differentiated to show that p 0 X t (i, j) = q(i, k)p t (k, j) ip t (i, j) k6= i 23

Kolmogorov s Backward and Forward Equations Computing Transition Probabilities Kolmogorov s Backward Equation Define a matrix The equation Q(i, j) = q(i, j) if j 6= i i if j = i. p 0 X t (i, j) = q(i, k)p t (k, j) ip t (i, j) k6= i can be rewritten in matrix notation as p 0 = Qp t t. This is the Kolmogorov s Backward Equation. 24

Kolmogorov s Backward and Forward Equations Computing Transition Probabilities Kolmogorov s Backward Equation The Kolmogorov s Backward Equation is p 0 t = Qp t Kolmogorov s Forward Equation Using similar techniques, one can obtain Kolmogorov s Forward Equation p 0 = p t t Q 25

Kolmogorov s Backward and Forward Equations Solving Kolmogorov Backward Equation Theorem The solution of the Kolmogorov s backward equation can be obtained as e Qt p 0 t = Qp t (similarly as if Q were a number). The exponential function for matrix Q is defined as Proof: By direct differentiation. 26