Markovské řetězce se spojitým parametrem

Transcription

1 Markovské řetězce se spojitým parametrem Mgr. Rudolf B. Blažek, Ph.D. prof. RNDr. Roman Kotecký, DrSc. Katedra počítačových systémů Katedra teoretické informatiky Fakulta informačních technologií České vysoké učení technické v Praze Rudolf Blažek & Roman Kotecký, 2011 Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 20 Evropský sociální fond Praha & EU: Investujeme do vaší budoucnos@

2 Mgr. Rudolf B. Blažek, Ph.D. prof. RNDr. Roman Kotecký, DrSc. Department of Computer Systems Department of Theoretical Informatics Faculty of Information Technologies Czech Technical University in Prague Rudolf Blažek & Roman Kotecký, 2011 Statistics for Informatics MI-SPI, ZS 2011/12, Lecture 20 The European Social Fund Prague & EU: We Invest in Your Future

3 Recall Discrete-time Markov Chains The Markov Property for a discrete-time Markov chains P(X n+1 = j X n = i, X n 1 = i n 1,..., X 0 = i 0 )= = P(X n+1 = j X n = i) = P i,j = p i,j = p(i, j) In continuous time difficult to define conditional probability given Xr for all r < s we instead work with all choices of 0 s0 < s1,..., sn < s (for all n 1) 3

4 Definition!!! Markovský řetezec se spojitým parametrem Xt, t 0 is a continuous-time Markov chain if for any times 0 s0 < s1,..., sn < s and any states i0, i1,..., in, i, j (for all n 1) we have P(X t+s = j X s = i, X = i sn n,..., X = i s0 0 )= = P(X t = j X 0 = i) = p t (i, j) 4

5 Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example Let N(t), t 0 be a Poisson Process with rate λ. Let Yn be a discrete-time Markov chain with transition probabilities u(i,j). Assume N(t) is independent of Yn. Then Xt = YN(t) is a continuous-time Markov chain. Xt jumps according to u(i,j) at each arrival of N(t). Most continuous-time Markov chains can be constructed is a similar way. We will show how later today. 5

6 Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example N(t), t 0 is a Poisson Process with rate λ, therefore the number of arrivals in (0,t) is a Poisson random variable * * * * * * * N(t) ~ Poisson(λt) t P(N(t) =n) =e ( t)n n! Therefore p t (i, j) =P(X t = j X 0 = i) 1X = P(N(t) =n, X t = j X 0 = i) n=0 6

7 Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example 1X p t (i, j) = P(N(t) =n, X t = j X 0 = i) n=0 1X Xt = YN(t) N(t) is independent of the MC Yn = = n=0 1X n=0 P(N(t) =n, Y N(t) = j Y N(0) = i) P(N(t) =n) P(Y n = j Y 0 = i) 7

8 Construction Construction: Discrete-time Markov Chain with Poisson Process Timing Example 1X p t (i, j) = P(N(t) =n) P(Y n = j Y 0 = i) n=0 = 1X n=0 e t ( t)n n! u n (i, j) N(t) ~ Poisson(λt) n-step transition probability for discretetime Markov chain Yn 8

9 Chapman-Kolmogorov Equation Chapman-Kolmogorov Equation Theorem X k p s (i, k) p t (k, j) =p s+t (i, j) Idea of the Proof We want to go from i to j in time s + t First we must go to some state k in time s Then we must finish by going from k to j in time t We must consider all states k, thus we sum over k 9

10 Chapman-Kolmogorov Equation Chapman-Kolmogorov Equation Theorem X k p s (i, k) p t (k, j) =p s+t (i, j) Assume we know pt(i,j) for all t t0 (and for all i,j) The theorem allows us to calculate pt(i,j) for all t > 0!!! Let t0 0, consider the derivative at 0: lim t 0!0 +(p t (i, j) 0)/t 0 0 The derivative at 0 should be be enough to determine pt(i,j) for all t > 0. Well, it is!! (We will show this later today.) 10

11 Jump Rates Jump Rates Definition!!!!!!!!!!! Intenzita přechodu Denote the derivative of pt(i,j) with respect to time at 0 as If the derivative exists, then we call q(i,j) the jump rate from i to j. q(i, j) = lim h!0 + p h (i, j) h for j 6= i Note that we do not calculate q(i,i) Why is it called a jump rate? Let s look at our construction of the Markov chain... 11

12 Jump Rates Why is q(i,j) called a jump rate? Previous Example Xt = YN(t); Yn = discrete-time MC with transition prob. u(i,j). N(t), t 0 is a Poisson Process with rate λ, indep. of Yn. If Xt is at i, it makes jumps with rate λ (a Poisson process) It goes to j with probability u(i,j) This is a Poisson process thinning: Xt jumps from i to j as a Poisson Process with rate λ u(i,j) Next we will show that in this example q(i,j) = λ u(i,j) That is why we call q(i,j) the jump rate from i to j 12

13 Jump Rates Why is q(i,j) called a jump rate? Previous Example Xt = YN(t); Yn = discrete-time MC with transition prob. u(i,j). N(t), t 0 is a Poisson Process with rate λ, indep. of Yn. We got before p t (i, j) = 1X n=0 e t ( t)n n! u n (i, j) Therefore p h (i, j) h = 1 h 1X n=0 e h ( h)n n! u n (i, j) 13

14 Jump Rates Why is q(i,j) called a jump rate? Previous Example p t (i, j) = 1 h h 1X n=0 h e ( h)n u n (i, j) n! Let h 0 1 λ u(i,j) p h (i, j) h = 1 e h ( h)0 u 0 (i, j) +e h 0! + e h 1 1X n=2 n h n 1 h ( )1 1! u 1 (i, j) = 0 j i (transition in 0 steps) 0 u n (i, j) 0 n! 14

15 Jump Rates Why is q(i,j) called a jump rate? Summary of Previous Example Xt = YN(t); Yn = discrete-time MC with transition prob. u(i,j). N(t), t 0 is a Poisson Process with rate λ, indep. of Yn. Xt jumps from i to j as a Poisson Process w/ rate λ u(i,j) The jump rates of the process Xt are q(i, j) = lim h!0 p h (i, j) h = lim h!0 e h u(i, j) = u(i, j) Most Markov chains can be constructed in a similar way!! That is why for any continuous-time Markov chain we call q(i,j) the jump rates from i to j. 15

16 Jump Rates Poisson Process as a Markov Chain Simple Example!!!!!!!!! Poisson Process Consider a Poisson process with rate λ. Let X(t) be number of arrivals of the process in (0,t). X(t) increases from n to n + 1 at rate λ: q(n, n+1) = λ for all n 0 This simple example will allow us to create other more complicated examples... 16

17 Jump Rates Queueing System M / M / s Infinite FCFS Queue m s Parallel Servers w/ Exp(!) service time Infinite Customer Population Arriving Customers Poisson Process w/ rate " Departing Customers 17

18 Jump Rates Queueing System M / M / s Example!!!!!!!!!!!! M / M / s Queue Consider load-balancing s replicated database servers. A request is routed to the next available server. Requests line-up in a single queue if all servers are busy. Requests arrive at times of a Poisson Process w/ rate λ: q(n, n+1) = λ for all n 0 Service times are random independent ~ Exp(μ): q(n, n 1) = nμ q(n, n 1) = sμ if all 0 n s if all n s 18

19 Jump Rates Queueing System M / M / s with Balking Example!!!!!!!! M / M / s Queue with Balking The same setup; requests arrive as Poisson Process (λ). The load balancer forwards requests randomly to a different data center with probability (1 an) if there are n requests. I.e., the new request stays with probability an: q(n, n+1) = λ an for all n 0 Service times are as before independent ~ Exp(μ): q(n, n 1) = nμ q(n, n 1) = sμ if all 0 n s if all n s 19

20 Jump Rates Jump Rates vs. Transition Probabilities Example In our example the jump rates are q(i, j) = u(i, j) We leave i with rate λ (a Poisson process)... * * * * * * * *... and go to j with probability u(i,j) Note this can be reversed u(i, j) =q(i, j)/ This is how we can construct the MC from its jump rates... * * * * * * * *... but what is λ? 20

21 Constructing Markov Chains from Jump Rates From Jump Rates To a Markov Chain Constructing the MC from its jump rates Let λi = j i q(i,j). This is the rate at which Xt leaves i. λi =... Xt leaves i immediately... so we assume λi <. For λi > 0 we define (a routing matrix) r(i, j) =q(i, j)/ i If Xt is in state i with λi = 0, then Xt stays in i forever. If λi > 0, then Xt waits in i for a random time ~ Exp(λi). Then Xt jumps to j with probability r(i,j). We have many Poisson processes with λi for each state i 21

22 Constructing Markov Chains from Jump Rates A Single Poisson Process for Markov Chains with Bounded Rates Constructing the MC from bounded jump rates Let λi = j i q(i,j) and Λ = maxi λi. Assume that Λ < (bounded rates) and define u(i, j) =q(i, j)/ u(i, i) =1 X j6= i for j 6= i u(i, j) Xt tries to make a transitions with rate Λ (Poisson Process). Xt leaves i for j with probability u(i,j), but stays in i with u(i,i). Any MC with bounded rates can be constructed as our initial example! 22

23 Kolmogorov s Backward and Forward Equations Computing Transition Probabilities Our goal is to use the jump rates q(i,j) to find the transition probabilities p t (i, j) =P(X t = j X 0 = i) Recall the Chapman-Kolmogorov equation: X p s (i, k) p t (k, j) =p s+t (i, j) k It can be differentiated to show that p 0 X t (i, j) = q(i, k)p t (k, j) ip t (i, j) k6= i 23

24 Kolmogorov s Backward and Forward Equations Computing Transition Probabilities Kolmogorov s Backward Equation Define a matrix The equation Q(i, j) = q(i, j) if j 6= i i if j = i. p 0 X t (i, j) = q(i, k)p t (k, j) ip t (i, j) k6= i can be rewritten in matrix notation as p 0 = Qp t t. This is the Kolmogorov s Backward Equation. 24

25 Kolmogorov s Backward and Forward Equations Computing Transition Probabilities Kolmogorov s Backward Equation The Kolmogorov s Backward Equation is p 0 t = Qp t Kolmogorov s Forward Equation Using similar techniques, one can obtain Kolmogorov s Forward Equation p 0 = p t t Q 25

26 Kolmogorov s Backward and Forward Equations Solving Kolmogorov Backward Equation Theorem The solution of the Kolmogorov s backward equation can be obtained as e Qt p 0 t = Qp t (similarly as if Q were a number). The exponential function for matrix Q is defined as Proof: By direct differentiation. 26