Markov processes and queueing networks

Size: px
Start display at page:

Download "Markov processes and queueing networks"

Transcription

1 Inria September 22, 2015

2 Outline Poisson processes Markov jump processes Some queueing networks

3 The Poisson distribution (Siméon-Denis Poisson, ) { } e λ λ n n! As prevalent as Gaussian distribution n N

4 The Poisson distribution (Siméon-Denis Poisson, ) { } e λ λ n n! As prevalent as Gaussian distribution n N Law of rare events (a.k.a. law of small numbers) p n,i 0 such that lim n sup i p n,i = 0, lim n i p n,i = λ > 0

5 The Poisson distribution (Siméon-Denis Poisson, ) { } e λ λ n n! As prevalent as Gaussian distribution n N Law of rare events (a.k.a. law of small numbers) p n,i 0 such that lim n sup i p n,i = 0, lim n i p n,i = λ > 0 Then X n = i Z n,i with Z n,i : independent Bernoulli(p n,i ) verifies X n D Poisson(λ)

6 Point process on R + Definition Point process on R + : Collection of random times {T n } n>0 with 0 < T 1 < T 2... Alternative description Collection {N t } t R+ with N t := n>0 I T n [0,t] Yet another description Collection {N(C)} for all measurable C R + where N(C) := n>0 I Tn C

7 Poisson process on R + Definition Point process such that for all s 0 = 0 < s 1 < s 2 <... < s n, 1 Increments {N si N si 1 } 1 i n independent 2 Law of N t+s N s only depends on t 3 for some λ > 0, N t Poisson(λt)

8 Poisson process on R + Definition Point process such that for all s 0 = 0 < s 1 < s 2 <... < s n, 1 Increments {N si N si 1 } 1 i n independent 2 Law of N t+s N s only depends on t 3 for some λ > 0, N t Poisson(λt) In fact, (3) follows from (1) (2)

9 Poisson process on R + Definition Point process such that for all s 0 = 0 < s 1 < s 2 <... < s n, 1 Increments {N si N si 1 } 1 i n independent 2 Law of N t+s N s only depends on t 3 for some λ > 0, N t Poisson(λt) In fact, (3) follows from (1) (2) λ is called the intensity of the process

10 A first construction Given λn i.i.d. numbers U n,i, uniform on [0, n], let N (n) t := n i=1 I U n,i t Then for any k N, s 0 = 0 < s 1 < s 2 <... < s n, {N (n) s i N (n) s i 1 } 1 i k D 1 i k Poisson(λ(s i s i 1 ))

11 A first construction Given λn i.i.d. numbers U n,i, uniform on [0, n], let N (n) t := n i=1 I U n,i t Then for any k N, s 0 = 0 < s 1 < s 2 <... < s n, {N (n) s i N (n) s i 1 } 1 i k D 1 i k Poisson(λ(s i s i 1 )) Proof: Multinomial distribution of {N (n) s i N (n) s i 1 } 1 i k Convergence of Laplace transform E exp( k i=1 α i(n s (n) i N s (n) i 1 )) for all α1 k Rk +

12 A first construction Given λn i.i.d. numbers U n,i, uniform on [0, n], let N (n) t := n i=1 I U n,i t Then for any k N, s 0 = 0 < s 1 < s 2 <... < s n, {N (n) s i N (n) s i 1 } 1 i k D 1 i k Poisson(λ(s i s i 1 )) Proof: Multinomial distribution of {N (n) s i N (n) s i 1 } 1 i k Convergence of Laplace transform E exp( k i=1 α i(n s (n) i N s (n) i 1 )) for all α1 k Rk + Suggests Poisson processes exist and are limits of this construction

13 A second characterization Proposition For Poisson process {T n } n>0 of intensity λ, its interarrival times τ i = T i+1 T i, where T 0 = 0, verify {τ n } n 0 i.i.d. with common distribution Exp(λ)

14 A second characterization Proposition For Poisson process {T n } n>0 of intensity λ, its interarrival times τ i = T i+1 T i, where T 0 = 0, verify {τ n } n 0 i.i.d. with common distribution Exp(λ) Density of Exp(λ): λe λx I x>0

15 A second characterization Proposition For Poisson process {T n } n>0 of intensity λ, its interarrival times τ i = T i+1 T i, where T 0 = 0, verify {τ n } n 0 i.i.d. with common distribution Exp(λ) Density of Exp(λ): λe λx I x>0 Key property: Exponential random variable τ is memoryless, i.e. t > 0, P(τ t τ > t) = P(τ )

16 A third characterization Proposition Process with i.i.d., Exp(λ) interarrival times {τ i } i 0 can be constructed on [0, t] by 1) Drawing N t Poisson(λt) 2) Putting N t points U 1,..., U Nt on [0, t] where U i : i.i.d. uniform on [0, t]

17 A third characterization Proposition Process with i.i.d., Exp(λ) interarrival times {τ i } i 0 can be constructed on [0, t] by 1) Drawing N t Poisson(λt) 2) Putting N t points U 1,..., U Nt on [0, t] where U i : i.i.d. uniform on [0, t] Proof. Establish identity for all n N, φ : R n + R: where S n 1 E[φ(τ 0, τ 0 + τ 1,..., τ τ n 1 )I Nt=n] = e λt (λt) n n! n! (0,t] n φ(s 1, s 2,..., s n )I s1 <s 2 <...<s n n i=1 ds i = P(Poisson(λt) = n) E[φ(S 1,..., S n )] : sorted version of i.i.d. variables uniform on [0, t]

18 Laplace transform of Poisson processes Definition The Laplace transform of point process N {T n } n>0 is the functional whose evaluation at f : R + R + is L N (f ) := E exp( N(f )) = E(exp( n>0 f (T n ))).

19 Laplace transform of Poisson processes Definition The Laplace transform of point process N {T n } n>0 is the functional whose evaluation at f : R + R + is L N (f ) := E exp( N(f )) = E(exp( n>0 f (T n ))). Proposition: Knowledge of L N (f ) on sufficiently rich class of functions f : R + R + (e.g. piecewise continuous with compact support) characterizes law of point process N.

20 Laplace transform of Poisson processes Definition The Laplace transform of point process N {T n } n>0 is the functional whose evaluation at f : R + R + is L N (f ) := E exp( N(f )) = E(exp( n>0 f (T n ))). Proposition: Knowledge of L N (f ) on sufficiently rich class of functions f : R + R + (e.g. piecewise continuous with compact support) characterizes law of point process N. Laplace transform of Poisson process with intensity λ: L N (f ) = exp( λ(1 e f (x) )dx) R +

21 Laplace transform of Poisson processes Definition The Laplace transform of point process N {T n } n>0 is the functional whose evaluation at f : R + R + is L N (f ) := E exp( N(f )) = E(exp( n>0 f (T n ))). Proposition: Knowledge of L N (f ) on sufficiently rich class of functions f : R + R + (e.g. piecewise continuous with compact support) characterizes law of point process N. Laplace transform of Poisson process with intensity λ: L N (f ) = exp( λ(1 e f (x) )dx) R + (i) Previous construction yields expression for L N (f ) (ii) For f = i α i I Ci N(C i ) Poisson(λ C i dx), with independence for disjoint C i. Hence existence of Poisson process...

22 Poisson process with general space and intensity Definition For λ : R d R + locally integrable function, N {T n } n>0 point process on R d is Poisson with intensity function λ if and only if for measurable, disjoint C i R d, i = 1,..., n, N(C i ) independent, Poisson( C i λ(x)dx)

23 Poisson process with general space and intensity Definition For λ : R d R + locally integrable function, N {T n } n>0 point process on R d is Poisson with intensity function λ if and only if for measurable, disjoint C i R d, i = 1,..., n, N(C i ) independent, Poisson( C i λ(x)dx) Proposition Such a process exists and admits Laplace transform ( ) L N (f ) = exp λ(x)(1 e f (x) )dx R d

24 Further properties Strong Markov property: Poisson process N with intensity λ, stopping time T (i.e. t 0, {T t} σ(n s, s t)) then on {T < + }, {N T +t N T } t 0 : Poisson with intensity λ and independent of {N s } s T

25 Further properties Strong Markov property: Poisson process N with intensity λ, stopping time T (i.e. t 0, {T t} σ(n s, s t)) then on {T < + }, {N T +t N T } t 0 : Poisson with intensity λ and independent of {N s } s T Superposition: For independent Poisson processes N i with intensities λ i, i = 1,..., n then N = i N i: Poisson with intensity i λ i

26 Further properties Strong Markov property: Poisson process N with intensity λ, stopping time T (i.e. t 0, {T t} σ(n s, s t)) then on {T < + }, {N T +t N T } t 0 : Poisson with intensity λ and independent of {N s } s T Superposition: For independent Poisson processes N i with intensities λ i, i = 1,..., n then N = i N i: Poisson with intensity i λ i Thinning: For Poisson process N {T n } n>0 with intensity λ, {Z n } n>0 independent of N, i.i.d., valued in [k], processes N i : N i (C) = n>0 I T n C I Zn=i are independent, Poisson with intensities λ i = λp(z n = i)

27 Markov jump processes Process {X t } t R+ with values in E, countable or finite, is Markov if P(X tn = x n X tn 1 = x n 1,... X t1 = x 1 ) = P(X tn = x n X tn 1 = x n 1 ), t n 1 Rn +, t 1 < < t n, x n 1 E n Homogeneous if P(X t+s = y X s = x) =: p xy (t) independent of s, s, t R +, x, y E

28 Markov jump processes Process {X t } t R+ with values in E, countable or finite, is Markov if P(X tn = x n X tn 1 = x n 1,... X t1 = x 1 ) = P(X tn = x n X tn 1 = x n 1 ), t n 1 Rn +, t 1 < < t n, x n 1 E n Homogeneous if P(X t+s = y X s = x) =: p xy (t) independent of s, s, t R +, x, y E Semi-group property p xy (t + s) = z E p xz(t)p zy (s), or P(t + s) = P(t)P(s) with P(t) = {p xy (t)} x,y E

29 Markov jump processes Process {X t } t R+ with values in E, countable or finite, is Markov if P(X tn = x n X tn 1 = x n 1,... X t1 = x 1 ) = P(X tn = x n X tn 1 = x n 1 ), t n 1 Rn +, t 1 < < t n, x n 1 E n Homogeneous if P(X t+s = y X s = x) =: p xy (t) independent of s, s, t R +, x, y E Semi-group property p xy (t + s) = z E p xz(t)p zy (s), or P(t + s) = P(t)P(s) with P(t) = {p xy (t)} x,y E Definition {X t } t R+ is a pure jump Markov process if in addition (i) It spends with probability 1 a strictly positive time in each state (ii) Trajectories t X t are right-continuous

30 Markov jump processes: examples Poisson process {N t } t R+ : then Markov jump process with p xy (t) = P(Poisson(λt) = y x)

31 Markov jump processes: examples Poisson process {N t } t R+ : then Markov jump process with p xy (t) = P(Poisson(λt) = y x) Single-server queue, FIFO ( First-in-first-out ) discipline, arrival times: N Poisson (λ), service times: i.i.d. Exp(µ) independent of N X t = number of customers present at time t: Markov jump process by Memoryless property of Exponential distribution + Markov property of Poisson process (the M/M/1/ queue)

32 Markov jump processes: examples Poisson process {N t } t R+ : then Markov jump process with p xy (t) = P(Poisson(λt) = y x) Single-server queue, FIFO ( First-in-first-out ) discipline, arrival times: N Poisson (λ), service times: i.i.d. Exp(µ) independent of N X t = number of customers present at time t: Markov jump process by Memoryless property of Exponential distribution + Markov property of Poisson process (the M/M/1/ queue) Infinite server queue with Poisson arrivals and Exponential service times: customer arrived at T n stays in system till T n + σ n, where σ n : service time X t = number of customers present at time t: Markov jump process (the M/M/ / queue)

33 Structure of Markov jump processes Infinitesimal Generator 1 p x, y, y x E, limits q x := lim xx (t) p t 0 t, q xy = lim xy (t) t 0 t exist in R + and satisfy y x q xy = q x q xy : Jump rate from x to y Q := {q xy } x,y E where q xx = q x : Infinitesimal Generator of process {X t } t R+ 1 Formally: Q = lim h 0 h [P(h) I ] where I : identity matrix

34 Structure of Markov jump processes Infinitesimal Generator 1 p x, y, y x E, limits q x := lim xx (t) p t 0 t, q xy = lim xy (t) t 0 t exist in R + and satisfy y x q xy = q x q xy : Jump rate from x to y Q := {q xy } x,y E where q xx = q x : Infinitesimal Generator of process {X t } t R+ 1 Formally: Q = lim h 0 h [P(h) I ] where I : identity matrix Structure of Markov jump processes Sequence {Y n } n N of visited states: Markov chain with transition q matrix p xy = I xy x y q x Conditionally on {Y n } n N, sojourn times {τ n } n N in successive states Y n : independent, with distributions Exp(q Yn )

35 Examples for Poisson process (λ): only non-zero jump rate q x,x+1 = λ = q x, x N

36 Examples for Poisson process (λ): only non-zero jump rate q x,x+1 = λ = q x, x N For FIFO M/M/1/ queue, non-zero rates: q x,x+1 = λ, q x,x 1 = µi x>0, x N hence q x = λ + µi x>0

37 Examples for Poisson process (λ): only non-zero jump rate q x,x+1 = λ = q x, x N For FIFO M/M/1/ queue, non-zero rates: q x,x+1 = λ, q x,x 1 = µi x>0, x N hence q x = λ + µi x>0 For M/M/ / queue, non-zero rates: q x,x+1 = λ, q x,x 1 = µx, x N hence q x = λ + µx

38 Structure of Markov jump processes (continued) Let T n := n 1 k=0 τ k: time of n-th jump. If T = + almost surely: trajectory determined on R +, hence generator Q determines law of process {X t } t R+

39 Structure of Markov jump processes (continued) Let T n := n 1 k=0 τ k: time of n-th jump. If T = + almost surely: trajectory determined on R +, hence generator Q determines law of process {X t } t R+ Process is called explosive if instead T < + with positive probability. Then process not completely characterized by generator

40 Structure of Markov jump processes (continued) Let T n := n 1 k=0 τ k: time of n-th jump. If T = + almost surely: trajectory determined on R +, hence generator Q determines law of process {X t } t R+ Process is called explosive if instead T < + with positive probability. Then process not completely characterized by generator Sufficient conditions for non-explosiveness: sup x E q x < + Recurrence of induced chain {Y n } n N For Birth and Death processes (i.e. E = N, only non-zero rates: β n = q n,n+1, birth rate; δ n = q n,n 1, death rate), non-explosiveness holds if 1 = + β n + δ n n>0

41 Kolmogorov s forward and backward equations Formal differentiation of P(t + h) = P(t)P(h) = P(h)P(t) yields d dt P(t) = P(t)Q d dt p xy(t) = z E p xz(t)q zy d dt d P(t) = QP(t) dt p xy(t) = z E q xzp zy (t) Kolmogorov s forward equation Kolmogorov s backward equation

42 Kolmogorov s forward and backward equations Formal differentiation of P(t + h) = P(t)P(h) = P(h)P(t) yields d dt P(t) = P(t)Q d dt p xy(t) = z E p xz(t)q zy d dt d P(t) = QP(t) dt p xy(t) = z E q xzp zy (t) Kolmogorov s forward equation Kolmogorov s backward equation 1 Follow directly from Q = lim h 0 h [P(h) I ] for finite E, in which case P(t) = exp(tq), t 0 Hold more generally in particular for non-explosive processes with a more involved proof (justifying exchange of summation and differentiation)

43 Stationary distributions and measures Definition Measure {π x } x E is stationary if it satisfies π T Q = 0, or equivalently the global balance equations x E, π x y x q xy = y x π y q yx flow out of x flow into x,

44 Stationary distributions and measures Definition Measure {π x } x E is stationary if it satisfies π T Q = 0, or equivalently the global balance equations x E, π x y x q xy = y x π y q yx flow out of x flow into x Kolmogorov s equations suggest that, if X 0 π for stationary π then X t π for all t 0,

45 Stationary distributions and measures Definition Measure {π x } x E is stationary if it satisfies π T Q = 0, or equivalently the global balance equations x E, π x y x q xy = y x π y q yx flow out of x flow into x Kolmogorov s equations suggest that, if X 0 π for stationary π then X t π for all t 0, Example: stationarity for birth and death processes π 0 β 0 = π 1 δ 1, π x (β x + δ x ) = π x 1 β x 1 + π x+1 δ x+1, x 1

46 Irreducibility, recurrence, invariance Definition Process {X t } t R+ is irreducible (respectively, irreducible recurrent) if induced chain {Y n } n N is. State x is positive recurrent if E x (R x ) < +, where R x = inf{t > τ 0 : X t = x}. Measure π is invariant for process {X t } t R+ if for all t > 0, π T P(t) = π T, i.e. x E, y E π y p yx (t) = π x.

47 Limit theorems 1 Theorem For irreducible recurrent {X t } t R+, invariant measure π, unique up to some scalar factor. It can be defined as, for any x E: Rx y E, π y = E x I Xt=y dt, or alternatively with T x := inf{n > 0 : Y n = x}, y E, π y = 1 T x E x I Yn=y. q y 0 n=1

48 Limit theorems 1 Theorem For irreducible recurrent {X t } t R+, invariant measure π, unique up to some scalar factor. It can be defined as, for any x E: Rx y E, π y = E x I Xt=y dt, or alternatively with T x := inf{n > 0 : Y n = x}, y E, π y = 1 T x E x I Yn=y. q y 0 n=1 Corollaries {ˆπ y } invariant for {Y n } n N {ˆπ y /q y } invariant for {X t } t R+. For irreducible recurrent {X t } t R+, either all or no state x E is positive recurrent.

49 Limit theorems 2 Theorem {X t } t R+ is ergodic (i.e. irreducible, positive recurrent) iff it is irreducible, non-explosive and such that π satisfying global balance equations. Then π is also the unique invariant probability distribution.

50 Limit theorems 2 Theorem {X t } t R+ is ergodic (i.e. irreducible, positive recurrent) iff it is irreducible, non-explosive and such that π satisfying global balance equations. Then π is also the unique invariant probability distribution. Theorem For ergodic {X t } t R+ with stationary distribution π, any initial distribution for X 0 and π-integrable f, 1 t almost surely lim f (X s )ds = π x f (x) (ergodic theorem) t t 0 x E and in distribution X t D π as t.

51 Limit theorems 3 Theorem For irreducible, non-ergodic {X t } t R+, any initial distribution for X 0, then for all x E, lim P(X t = x) = 0. t

52 Time reversal and reversibility For stationary ergodic {X t } t R with stationary distribution π, time-reversed process X t = X t : πy qyx Markov with transition rates q xy = π x

53 Time reversal and reversibility For stationary ergodic {X t } t R with stationary distribution π, time-reversed process X t = X t : πy qyx Markov with transition rates q xy = π x Definition Stationary ergodic {X t } t R with stationary distribution π reversible iff distributed as time-reversal { X t } t R, i.e. x y E, π x q xy = π y q yx, flow from x to y flow from y to x detailed balance equations.

54 Time reversal and reversibility For stationary ergodic {X t } t R with stationary distribution π, time-reversed process X t = X t : πy qyx Markov with transition rates q xy = π x Definition Stationary ergodic {X t } t R with stationary distribution π reversible iff distributed as time-reversal { X t } t R, i.e. x y E, π x q xy = π y q yx, flow from x to y flow from y to x detailed balance equations. Detailed balance, i.e. reversibility for π implies global balance for π. Example: for birth and death processes, detailed balance always holds for stationary measure.

55 Reversibility and truncation Proposition Let generator Q on E admit reversible measure π. Then for subset F E, truncated generator ˆQ: ˆQ xy = Q xy, x y F, ˆQ xx = ˆQ y x xy, x F admits {π x } x F as reversible measure.

56 Erlang s model of telephone network Call types s S: type-s calls arrive at instants of Poisson (λ s ) process, last (if accepted) for duration Exponential (µ s ) type-s calls require one circuit (unit of capacity) per link l s Link l has capacity C l circuits

57 Erlang s model of telephone network Call types s S: type-s calls arrive at instants of Poisson (λ s ) process, last (if accepted) for duration Exponential (µ s ) type-s calls require one circuit (unit of capacity) per link l s Link l has capacity C l circuits Stationary probability distribution: π x = 1 Z s S ρ xs s x s! I s l xs C, l where: ρ s = λ s /µ s, Z: normalizing constant. l

58 Erlang s model of telephone network Call types s S: type-s calls arrive at instants of Poisson (λ s ) process, last (if accepted) for duration Exponential (µ s ) type-s calls require one circuit (unit of capacity) per link l s Link l has capacity C l circuits Stationary probability distribution: π x = 1 Z s S ρ xs s x s! I s l xs C, l where: ρ s = λ s /µ s, Z: normalizing constant. Basis for dimensioning studies of telephone networks (prediction of call rejection probabilities) More recent application: performance analysis of peer-to-peer systems for video streaming. l

59 Jackson networks Stations i I receive external arrivals at Poisson rate λ i Station i when processing x i customers completes service at rate µ i φ i (x i ) (e.g.: φ i (x) = min(x i, n i ): queue with n i servers and service times Exponential (µ i )) After completing service at station i, customer joins station j with probability p ij, j I, and leaves system with probability 1 j I p ij Matrix P = (p ij ): sub-stochastic, such that (I P) 1

60 Jackson networks Stations i I receive external arrivals at Poisson rate λ i Station i when processing x i customers completes service at rate µ i φ i (x i ) (e.g.: φ i (x) = min(x i, n i ): queue with n i servers and service times Exponential (µ i )) After completing service at station i, customer joins station j with probability p ij, j I, and leaves system with probability 1 j I p ij Matrix P = (p ij ): sub-stochastic, such that (I P) 1 Traffic equations i I, λ i = λ i + j I λ j p ji or λ = (I P T ) 1 λ

61 Jackson networks (continued) Stationary measure: π x = i I xi ρ x i i m=1 φ i(m), where ρ i = λ i /µ i, and λ i : solutions of traffic equations

62 Jackson networks (continued) Stationary measure: π x = i I xi ρ x i i m=1 φ i(m), where ρ i = λ i /µ i, and λ i : solutions of traffic equations Application: process ergodic when π has finite mass. e.g. for φ i (x) = min(x, n i ), ergodicity iff i I, ρ i < n i.

63 Jackson networks (continued) Stationary measure: π x = i I xi ρ x i i m=1 φ i(m), where ρ i = λ i /µ i, and λ i : solutions of traffic equations Application: process ergodic when π has finite mass. e.g. for φ i (x) = min(x, n i ), ergodicity iff i I, ρ i < n i. Proof: verify partial balance equations for all x N I : i I, π x [ j i q x,x e i +e j + q x,x ei ] = j i π x e i +e j q x ei +e j,x + π x ei q x ei,x, π x i I q x,x+e i = i I π x+e i q x+ei,x, which imply global balance equations π x [ i I (q x,x e i + q x,x+ei + j i q x,x e i +e j )] = i I (π x e i q x ei,x + π x+ei q x+ei,x + j i π x e i +e j q x ei +e j,x)

64 Takeaway messages Poisson process a fundamental continuous-time process, adequate model for aggregate of infrequent independent events Markov jump processes: i) generator Q characterizes distribution if not explosive ii) Balance equation characterizes invariant distribution if irreducible non-explosive iii) Limit theorems: stationary distribution reflects long-term performance Exactly solvable models include reversible processes, plus several other important classes (e.g. Jackson networks)

Part I Stochastic variables and Markov chains

Part I Stochastic variables and Markov chains Part I Stochastic variables and Markov chains Random variables describe the behaviour of a phenomenon independent of any specific sample space Distribution function (cdf, cumulative distribution function)

More information

IEOR 6711, HMWK 5, Professor Sigman

IEOR 6711, HMWK 5, Professor Sigman IEOR 6711, HMWK 5, Professor Sigman 1. Semi-Markov processes: Consider an irreducible positive recurrent discrete-time Markov chain {X n } with transition matrix P (P i,j ), i, j S, and finite state space.

More information

Lecture 20: Reversible Processes and Queues

Lecture 20: Reversible Processes and Queues Lecture 20: Reversible Processes and Queues 1 Examples of reversible processes 11 Birth-death processes We define two non-negative sequences birth and death rates denoted by {λ n : n N 0 } and {µ n : n

More information

Stochastic process. X, a series of random variables indexed by t

Stochastic process. X, a series of random variables indexed by t Stochastic process X, a series of random variables indexed by t X={X(t), t 0} is a continuous time stochastic process X={X(t), t=0,1, } is a discrete time stochastic process X(t) is the state at time t,

More information

Markov chains. 1 Discrete time Markov chains. c A. J. Ganesh, University of Bristol, 2015

Markov chains. 1 Discrete time Markov chains. c A. J. Ganesh, University of Bristol, 2015 Markov chains c A. J. Ganesh, University of Bristol, 2015 1 Discrete time Markov chains Example: A drunkard is walking home from the pub. There are n lampposts between the pub and his home, at each of

More information

Recap. Probability, stochastic processes, Markov chains. ELEC-C7210 Modeling and analysis of communication networks

Recap. Probability, stochastic processes, Markov chains. ELEC-C7210 Modeling and analysis of communication networks Recap Probability, stochastic processes, Markov chains ELEC-C7210 Modeling and analysis of communication networks 1 Recap: Probability theory important distributions Discrete distributions Geometric distribution

More information

Lecture 7: Simulation of Markov Processes. Pasi Lassila Department of Communications and Networking

Lecture 7: Simulation of Markov Processes. Pasi Lassila Department of Communications and Networking Lecture 7: Simulation of Markov Processes Pasi Lassila Department of Communications and Networking Contents Markov processes theory recap Elementary queuing models for data networks Simulation of Markov

More information

Statistics 150: Spring 2007

Statistics 150: Spring 2007 Statistics 150: Spring 2007 April 23, 2008 0-1 1 Limiting Probabilities If the discrete-time Markov chain with transition probabilities p ij is irreducible and positive recurrent; then the limiting probabilities

More information

Data analysis and stochastic modeling

Data analysis and stochastic modeling Data analysis and stochastic modeling Lecture 7 An introduction to queueing theory Guillaume Gravier guillaume.gravier@irisa.fr with a lot of help from Paul Jensen s course http://www.me.utexas.edu/ jensen/ormm/instruction/powerpoint/or_models_09/14_queuing.ppt

More information

STAT 380 Continuous Time Markov Chains

STAT 380 Continuous Time Markov Chains STAT 380 Continuous Time Markov Chains Richard Lockhart Simon Fraser University Spring 2018 Richard Lockhart (Simon Fraser University)STAT 380 Continuous Time Markov Chains Spring 2018 1 / 35 Continuous

More information

Figure 10.1: Recording when the event E occurs

Figure 10.1: Recording when the event E occurs 10 Poisson Processes Let T R be an interval. A family of random variables {X(t) ; t T} is called a continuous time stochastic process. We often consider T = [0, 1] and T = [0, ). As X(t) is a random variable

More information

Performance Evaluation of Queuing Systems

Performance Evaluation of Queuing Systems Performance Evaluation of Queuing Systems Introduction to Queuing Systems System Performance Measures & Little s Law Equilibrium Solution of Birth-Death Processes Analysis of Single-Station Queuing Systems

More information

Introduction to Markov Chains, Queuing Theory, and Network Performance

Introduction to Markov Chains, Queuing Theory, and Network Performance Introduction to Markov Chains, Queuing Theory, and Network Performance Marceau Coupechoux Telecom ParisTech, departement Informatique et Réseaux marceau.coupechoux@telecom-paristech.fr IT.2403 Modélisation

More information

Chapter 5. Continuous-Time Markov Chains. Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan

Chapter 5. Continuous-Time Markov Chains. Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan Chapter 5. Continuous-Time Markov Chains Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan Continuous-Time Markov Chains Consider a continuous-time stochastic process

More information

The Transition Probability Function P ij (t)

The Transition Probability Function P ij (t) The Transition Probability Function P ij (t) Consider a continuous time Markov chain {X(t), t 0}. We are interested in the probability that in t time units the process will be in state j, given that it

More information

1 Continuous-time chains, finite state space

1 Continuous-time chains, finite state space Université Paris Diderot 208 Markov chains Exercises 3 Continuous-time chains, finite state space Exercise Consider a continuous-time taking values in {, 2, 3}, with generator 2 2. 2 2 0. Draw the diagramm

More information

ELEMENTS OF PROBABILITY THEORY

ELEMENTS OF PROBABILITY THEORY ELEMENTS OF PROBABILITY THEORY Elements of Probability Theory A collection of subsets of a set Ω is called a σ algebra if it contains Ω and is closed under the operations of taking complements and countable

More information

GI/M/1 and GI/M/m queuing systems

GI/M/1 and GI/M/m queuing systems GI/M/1 and GI/M/m queuing systems Dmitri A. Moltchanov moltchan@cs.tut.fi http://www.cs.tut.fi/kurssit/tlt-2716/ OUTLINE: GI/M/1 queuing system; Methods of analysis; Imbedded Markov chain approach; Waiting

More information

A review of Continuous Time MC STA 624, Spring 2015

A review of Continuous Time MC STA 624, Spring 2015 A review of Continuous Time MC STA 624, Spring 2015 Ruriko Yoshida Dept. of Statistics University of Kentucky polytopes.net STA 624 1 Continuous Time Markov chains Definition A continuous time stochastic

More information

Introduction to Queuing Networks Solutions to Problem Sheet 3

Introduction to Queuing Networks Solutions to Problem Sheet 3 Introduction to Queuing Networks Solutions to Problem Sheet 3 1. (a) The state space is the whole numbers {, 1, 2,...}. The transition rates are q i,i+1 λ for all i and q i, for all i 1 since, when a bus

More information

Chapter 2. Poisson Processes. Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan

Chapter 2. Poisson Processes. Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan Chapter 2. Poisson Processes Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan Outline Introduction to Poisson Processes Definition of arrival process Definition

More information

Continuous Time Processes

Continuous Time Processes page 102 Chapter 7 Continuous Time Processes 7.1 Introduction In a continuous time stochastic process (with discrete state space), a change of state can occur at any time instant. The associated point

More information

STATS 3U03. Sang Woo Park. March 29, Textbook: Inroduction to stochastic processes. Requirement: 5 assignments, 2 tests, and 1 final

STATS 3U03. Sang Woo Park. March 29, Textbook: Inroduction to stochastic processes. Requirement: 5 assignments, 2 tests, and 1 final STATS 3U03 Sang Woo Park March 29, 2017 Course Outline Textbook: Inroduction to stochastic processes Requirement: 5 assignments, 2 tests, and 1 final Test 1: Friday, February 10th Test 2: Friday, March

More information

Stability and Rare Events in Stochastic Models Sergey Foss Heriot-Watt University, Edinburgh and Institute of Mathematics, Novosibirsk

Stability and Rare Events in Stochastic Models Sergey Foss Heriot-Watt University, Edinburgh and Institute of Mathematics, Novosibirsk Stability and Rare Events in Stochastic Models Sergey Foss Heriot-Watt University, Edinburgh and Institute of Mathematics, Novosibirsk ANSAPW University of Queensland 8-11 July, 2013 1 Outline (I) Fluid

More information

CDA5530: Performance Models of Computers and Networks. Chapter 3: Review of Practical

CDA5530: Performance Models of Computers and Networks. Chapter 3: Review of Practical CDA5530: Performance Models of Computers and Networks Chapter 3: Review of Practical Stochastic Processes Definition Stochastic ti process X = {X(t), t T} is a collection of random variables (rvs); one

More information

Contents Preface The Exponential Distribution and the Poisson Process Introduction to Renewal Theory

Contents Preface The Exponential Distribution and the Poisson Process Introduction to Renewal Theory Contents Preface... v 1 The Exponential Distribution and the Poisson Process... 1 1.1 Introduction... 1 1.2 The Density, the Distribution, the Tail, and the Hazard Functions... 2 1.2.1 The Hazard Function

More information

LECTURE #6 BIRTH-DEATH PROCESS

LECTURE #6 BIRTH-DEATH PROCESS LECTURE #6 BIRTH-DEATH PROCESS 204528 Queueing Theory and Applications in Networks Assoc. Prof., Ph.D. (รศ.ดร. อน นต ผลเพ ม) Computer Engineering Department, Kasetsart University Outline 2 Birth-Death

More information

Irreducibility. Irreducible. every state can be reached from every other state For any i,j, exist an m 0, such that. Absorbing state: p jj =1

Irreducibility. Irreducible. every state can be reached from every other state For any i,j, exist an m 0, such that. Absorbing state: p jj =1 Irreducibility Irreducible every state can be reached from every other state For any i,j, exist an m 0, such that i,j are communicate, if the above condition is valid Irreducible: all states are communicate

More information

Continuous-Time Markov Chain

Continuous-Time Markov Chain Continuous-Time Markov Chain Consider the process {X(t),t 0} with state space {0, 1, 2,...}. The process {X(t),t 0} is a continuous-time Markov chain if for all s, t 0 and nonnegative integers i, j, x(u),

More information

CDA6530: Performance Models of Computers and Networks. Chapter 3: Review of Practical Stochastic Processes

CDA6530: Performance Models of Computers and Networks. Chapter 3: Review of Practical Stochastic Processes CDA6530: Performance Models of Computers and Networks Chapter 3: Review of Practical Stochastic Processes Definition Stochastic process X = {X(t), t2 T} is a collection of random variables (rvs); one rv

More information

Queueing Networks and Insensitivity

Queueing Networks and Insensitivity Lukáš Adam 29. 10. 2012 1 / 40 Table of contents 1 Jackson networks 2 Insensitivity in Erlang s Loss System 3 Quasi-Reversibility and Single-Node Symmetric Queues 4 Quasi-Reversibility in Networks 5 The

More information

1 IEOR 4701: Continuous-Time Markov Chains

1 IEOR 4701: Continuous-Time Markov Chains Copyright c 2006 by Karl Sigman 1 IEOR 4701: Continuous-Time Markov Chains A Markov chain in discrete time, {X n : n 0}, remains in any state for exactly one unit of time before making a transition (change

More information

Quantitative Model Checking (QMC) - SS12

Quantitative Model Checking (QMC) - SS12 Quantitative Model Checking (QMC) - SS12 Lecture 06 David Spieler Saarland University, Germany June 4, 2012 1 / 34 Deciding Bisimulations 2 / 34 Partition Refinement Algorithm Notation: A partition P over

More information

M/G/1 and M/G/1/K systems

M/G/1 and M/G/1/K systems M/G/1 and M/G/1/K systems Dmitri A. Moltchanov dmitri.moltchanov@tut.fi http://www.cs.tut.fi/kurssit/elt-53606/ OUTLINE: Description of M/G/1 system; Methods of analysis; Residual life approach; Imbedded

More information

IEOR 6711: Stochastic Models I, Fall 2003, Professor Whitt. Solutions to Final Exam: Thursday, December 18.

IEOR 6711: Stochastic Models I, Fall 2003, Professor Whitt. Solutions to Final Exam: Thursday, December 18. IEOR 6711: Stochastic Models I, Fall 23, Professor Whitt Solutions to Final Exam: Thursday, December 18. Below are six questions with several parts. Do as much as you can. Show your work. 1. Two-Pump Gas

More information

Chapter 1. Introduction. 1.1 Stochastic process

Chapter 1. Introduction. 1.1 Stochastic process Chapter 1 Introduction Process is a phenomenon that takes place in time. In many practical situations, the result of a process at any time may not be certain. Such a process is called a stochastic process.

More information

Exact Simulation of the Stationary Distribution of M/G/c Queues

Exact Simulation of the Stationary Distribution of M/G/c Queues 1/36 Exact Simulation of the Stationary Distribution of M/G/c Queues Professor Karl Sigman Columbia University New York City USA Conference in Honor of Søren Asmussen Monday, August 1, 2011 Sandbjerg Estate

More information

TCOM 501: Networking Theory & Fundamentals. Lecture 6 February 19, 2003 Prof. Yannis A. Korilis

TCOM 501: Networking Theory & Fundamentals. Lecture 6 February 19, 2003 Prof. Yannis A. Korilis TCOM 50: Networking Theory & Fundamentals Lecture 6 February 9, 003 Prof. Yannis A. Korilis 6- Topics Time-Reversal of Markov Chains Reversibility Truncating a Reversible Markov Chain Burke s Theorem Queues

More information

Intro Refresher Reversibility Open networks Closed networks Multiclass networks Other networks. Queuing Networks. Florence Perronnin

Intro Refresher Reversibility Open networks Closed networks Multiclass networks Other networks. Queuing Networks. Florence Perronnin Queuing Networks Florence Perronnin Polytech Grenoble - UGA March 23, 27 F. Perronnin (UGA) Queuing Networks March 23, 27 / 46 Outline Introduction to Queuing Networks 2 Refresher: M/M/ queue 3 Reversibility

More information

STAT STOCHASTIC PROCESSES. Contents

STAT STOCHASTIC PROCESSES. Contents STAT 3911 - STOCHASTIC PROCESSES ANDREW TULLOCH Contents 1. Stochastic Processes 2 2. Classification of states 2 3. Limit theorems for Markov chains 4 4. First step analysis 5 5. Branching processes 5

More information

Lecture 4a: Continuous-Time Markov Chain Models

Lecture 4a: Continuous-Time Markov Chain Models Lecture 4a: Continuous-Time Markov Chain Models Continuous-time Markov chains are stochastic processes whose time is continuous, t [0, ), but the random variables are discrete. Prominent examples of continuous-time

More information

Class 11 Non-Parametric Models of a Service System; GI/GI/1, GI/GI/n: Exact & Approximate Analysis.

Class 11 Non-Parametric Models of a Service System; GI/GI/1, GI/GI/n: Exact & Approximate Analysis. Service Engineering Class 11 Non-Parametric Models of a Service System; GI/GI/1, GI/GI/n: Exact & Approximate Analysis. G/G/1 Queue: Virtual Waiting Time (Unfinished Work). GI/GI/1: Lindley s Equations

More information

Poisson Processes. Stochastic Processes. Feb UC3M

Poisson Processes. Stochastic Processes. Feb UC3M Poisson Processes Stochastic Processes UC3M Feb. 2012 Exponential random variables A random variable T has exponential distribution with rate λ > 0 if its probability density function can been written

More information

STA 624 Practice Exam 2 Applied Stochastic Processes Spring, 2008

STA 624 Practice Exam 2 Applied Stochastic Processes Spring, 2008 Name STA 624 Practice Exam 2 Applied Stochastic Processes Spring, 2008 There are five questions on this test. DO use calculators if you need them. And then a miracle occurs is not a valid answer. There

More information

SMSTC (2007/08) Probability.

SMSTC (2007/08) Probability. SMSTC (27/8) Probability www.smstc.ac.uk Contents 12 Markov chains in continuous time 12 1 12.1 Markov property and the Kolmogorov equations.................... 12 2 12.1.1 Finite state space.................................

More information

Birth and Death Processes. Birth and Death Processes. Linear Growth with Immigration. Limiting Behaviour for Birth and Death Processes

Birth and Death Processes. Birth and Death Processes. Linear Growth with Immigration. Limiting Behaviour for Birth and Death Processes DTU Informatics 247 Stochastic Processes 6, October 27 Today: Limiting behaviour of birth and death processes Birth and death processes with absorbing states Finite state continuous time Markov chains

More information

Statistics 253/317 Introduction to Probability Models. Winter Midterm Exam Monday, Feb 10, 2014

Statistics 253/317 Introduction to Probability Models. Winter Midterm Exam Monday, Feb 10, 2014 Statistics 253/317 Introduction to Probability Models Winter 2014 - Midterm Exam Monday, Feb 10, 2014 Student Name (print): (a) Do not sit directly next to another student. (b) This is a closed-book, closed-note

More information

STOCHASTIC PROCESSES Basic notions

STOCHASTIC PROCESSES Basic notions J. Virtamo 38.3143 Queueing Theory / Stochastic processes 1 STOCHASTIC PROCESSES Basic notions Often the systems we consider evolve in time and we are interested in their dynamic behaviour, usually involving

More information

Simple queueing models

Simple queueing models Simple queueing models c University of Bristol, 2012 1 M/M/1 queue This model describes a queue with a single server which serves customers in the order in which they arrive. Customer arrivals constitute

More information

Continuous-time Markov Chains

Continuous-time Markov Chains Continuous-time Markov Chains Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ October 23, 2017

More information

Continuous time Markov chains

Continuous time Markov chains Continuous time Markov chains Alejandro Ribeiro Dept. of Electrical and Systems Engineering University of Pennsylvania aribeiro@seas.upenn.edu http://www.seas.upenn.edu/users/~aribeiro/ October 16, 2017

More information

MARKOV PROCESSES. Valerio Di Valerio

MARKOV PROCESSES. Valerio Di Valerio MARKOV PROCESSES Valerio Di Valerio Stochastic Process Definition: a stochastic process is a collection of random variables {X(t)} indexed by time t T Each X(t) X is a random variable that satisfy some

More information

Stochastic Processes (Week 6)

Stochastic Processes (Week 6) Stochastic Processes (Week 6) October 30th, 2014 1 Discrete-time Finite Markov Chains 2 Countable Markov Chains 3 Continuous-Time Markov Chains 3.1 Poisson Process 3.2 Finite State Space 3.2.1 Kolmogrov

More information

Solution: The process is a compound Poisson Process with E[N (t)] = λt/p by Wald's equation.

Solution: The process is a compound Poisson Process with E[N (t)] = λt/p by Wald's equation. Solutions Stochastic Processes and Simulation II, May 18, 217 Problem 1: Poisson Processes Let {N(t), t } be a homogeneous Poisson Process on (, ) with rate λ. Let {S i, i = 1, 2, } be the points of the

More information

PROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS

PROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS PROBABILITY: LIMIT THEOREMS II, SPRING 15. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please

More information

PROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS

PROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS PROBABILITY: LIMIT THEOREMS II, SPRING 218. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please

More information

Exponential Distribution and Poisson Process

Exponential Distribution and Poisson Process Exponential Distribution and Poisson Process Stochastic Processes - Lecture Notes Fatih Cavdur to accompany Introduction to Probability Models by Sheldon M. Ross Fall 215 Outline Introduction Exponential

More information

Other properties of M M 1

Other properties of M M 1 Other properties of M M 1 Přemysl Bejda premyslbejda@gmail.com 2012 Contents 1 Reflected Lévy Process 2 Time dependent properties of M M 1 3 Waiting times and queue disciplines in M M 1 Contents 1 Reflected

More information

Performance Modelling of Computer Systems

Performance Modelling of Computer Systems Performance Modelling of Computer Systems Mirco Tribastone Institut für Informatik Ludwig-Maximilians-Universität München Fundamentals of Queueing Theory Tribastone (IFI LMU) Performance Modelling of Computer

More information

Queuing Networks: Burke s Theorem, Kleinrock s Approximation, and Jackson s Theorem. Wade Trappe

Queuing Networks: Burke s Theorem, Kleinrock s Approximation, and Jackson s Theorem. Wade Trappe Queuing Networks: Burke s Theorem, Kleinrock s Approximation, and Jackson s Theorem Wade Trappe Lecture Overview Network of Queues Introduction Queues in Tandem roduct Form Solutions Burke s Theorem What

More information

Time Reversibility and Burke s Theorem

Time Reversibility and Burke s Theorem Queuing Analysis: Time Reversibility and Burke s Theorem Hongwei Zhang http://www.cs.wayne.edu/~hzhang Acknowledgement: this lecture is partially based on the slides of Dr. Yannis A. Korilis. Outline Time-Reversal

More information

ECE-517: Reinforcement Learning in Artificial Intelligence. Lecture 4: Discrete-Time Markov Chains

ECE-517: Reinforcement Learning in Artificial Intelligence. Lecture 4: Discrete-Time Markov Chains ECE-517: Reinforcement Learning in Artificial Intelligence Lecture 4: Discrete-Time Markov Chains September 1, 215 Dr. Itamar Arel College of Engineering Department of Electrical Engineering & Computer

More information

The exponential distribution and the Poisson process

The exponential distribution and the Poisson process The exponential distribution and the Poisson process 1-1 Exponential Distribution: Basic Facts PDF f(t) = { λe λt, t 0 0, t < 0 CDF Pr{T t) = 0 t λe λu du = 1 e λt (t 0) Mean E[T] = 1 λ Variance Var[T]

More information

Part II: continuous time Markov chain (CTMC)

Part II: continuous time Markov chain (CTMC) Part II: continuous time Markov chain (CTMC) Continuous time discrete state Markov process Definition (Markovian property) X(t) is a CTMC, if for any n and any sequence t 1

More information

Lectures 16-17: Poisson Approximation. Using Lemma (2.4.3) with θ = 1 and then Lemma (2.4.4), which is valid when max m p n,m 1/2, we have

Lectures 16-17: Poisson Approximation. Using Lemma (2.4.3) with θ = 1 and then Lemma (2.4.4), which is valid when max m p n,m 1/2, we have Lectures 16-17: Poisson Approximation 1. The Law of Rare Events Theorem 2.6.1: For each n 1, let X n,m, 1 m n be a collection of independent random variables with PX n,m = 1 = p n,m and PX n,m = = 1 p

More information

Queueing Theory. VK Room: M Last updated: October 17, 2013.

Queueing Theory. VK Room: M Last updated: October 17, 2013. Queueing Theory VK Room: M1.30 knightva@cf.ac.uk www.vincent-knight.com Last updated: October 17, 2013. 1 / 63 Overview Description of Queueing Processes The Single Server Markovian Queue Multi Server

More information

Stochastic Modelling Unit 1: Markov chain models

Stochastic Modelling Unit 1: Markov chain models Stochastic Modelling Unit 1: Markov chain models Russell Gerrard and Douglas Wright Cass Business School, City University, London June 2004 Contents of Unit 1 1 Stochastic Processes 2 Markov Chains 3 Poisson

More information

Continuous Time Markov Chains

Continuous Time Markov Chains Continuous Time Markov Chains Stochastic Processes - Lecture Notes Fatih Cavdur to accompany Introduction to Probability Models by Sheldon M. Ross Fall 2015 Outline Introduction Continuous-Time Markov

More information

Lecture 7. We can regard (p(i, j)) as defining a (maybe infinite) matrix P. Then a basic fact is

Lecture 7. We can regard (p(i, j)) as defining a (maybe infinite) matrix P. Then a basic fact is MARKOV CHAINS What I will talk about in class is pretty close to Durrett Chapter 5 sections 1-5. We stick to the countable state case, except where otherwise mentioned. Lecture 7. We can regard (p(i, j))

More information

Queueing. Chapter Continuous Time Markov Chains 2 CHAPTER 5. QUEUEING

Queueing. Chapter Continuous Time Markov Chains 2 CHAPTER 5. QUEUEING 2 CHAPTER 5. QUEUEING Chapter 5 Queueing Systems are often modeled by automata, and discrete events are transitions from one state to another. In this chapter we want to analyze such discrete events systems.

More information

reversed chain is ergodic and has the same equilibrium probabilities (check that π j =

reversed chain is ergodic and has the same equilibrium probabilities (check that π j = Lecture 10 Networks of queues In this lecture we shall finally get around to consider what happens when queues are part of networks (which, after all, is the topic of the course). Firstly we shall need

More information

T. Liggett Mathematics 171 Final Exam June 8, 2011

T. Liggett Mathematics 171 Final Exam June 8, 2011 T. Liggett Mathematics 171 Final Exam June 8, 2011 1. The continuous time renewal chain X t has state space S = {0, 1, 2,...} and transition rates (i.e., Q matrix) given by q(n, n 1) = δ n and q(0, n)

More information

TMA4265 Stochastic processes ST2101 Stochastic simulation and modelling

TMA4265 Stochastic processes ST2101 Stochastic simulation and modelling Norwegian University of Science and Technology Department of Mathematical Sciences Page of 7 English Contact during examination: Øyvind Bakke Telephone: 73 9 8 26, 99 4 673 TMA426 Stochastic processes

More information

Name of the Student:

Name of the Student: SUBJECT NAME : Probability & Queueing Theory SUBJECT CODE : MA 6453 MATERIAL NAME : Part A questions REGULATION : R2013 UPDATED ON : November 2017 (Upto N/D 2017 QP) (Scan the above QR code for the direct

More information

Statistics 992 Continuous-time Markov Chains Spring 2004

Statistics 992 Continuous-time Markov Chains Spring 2004 Summary Continuous-time finite-state-space Markov chains are stochastic processes that are widely used to model the process of nucleotide substitution. This chapter aims to present much of the mathematics

More information

Markov Processes Cont d. Kolmogorov Differential Equations

Markov Processes Cont d. Kolmogorov Differential Equations Markov Processes Cont d Kolmogorov Differential Equations The Kolmogorov Differential Equations characterize the transition functions {P ij (t)} of a Markov process. The time-dependent behavior of the

More information

Quality of Real-Time Streaming in Wireless Cellular Networks : Stochastic Modeling and Analysis

Quality of Real-Time Streaming in Wireless Cellular Networks : Stochastic Modeling and Analysis Quality of Real-Time Streaming in Wireless Cellular Networs : Stochastic Modeling and Analysis B. Blaszczyszyn, M. Jovanovic and M. K. Karray Based on paper [1] WiOpt/WiVid Mai 16th, 2014 Outline Introduction

More information

Multi Stage Queuing Model in Level Dependent Quasi Birth Death Process

Multi Stage Queuing Model in Level Dependent Quasi Birth Death Process International Journal of Statistics and Systems ISSN 973-2675 Volume 12, Number 2 (217, pp. 293-31 Research India Publications http://www.ripublication.com Multi Stage Queuing Model in Level Dependent

More information

QUEUING SYSTEM. Yetunde Folajimi, PhD

QUEUING SYSTEM. Yetunde Folajimi, PhD QUEUING SYSTEM Yetunde Folajimi, PhD Part 2 Queuing Models Queueing models are constructed so that queue lengths and waiting times can be predicted They help us to understand and quantify the effect of

More information

Stochastic Processes. Theory for Applications. Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS

Stochastic Processes. Theory for Applications. Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS Stochastic Processes Theory for Applications Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS Contents Preface page xv Swgg&sfzoMj ybr zmjfr%cforj owf fmdy xix Acknowledgements xxi 1 Introduction and review

More information

Queueing Theory I Summary! Little s Law! Queueing System Notation! Stationary Analysis of Elementary Queueing Systems " M/M/1 " M/M/m " M/M/1/K "

Queueing Theory I Summary! Little s Law! Queueing System Notation! Stationary Analysis of Elementary Queueing Systems  M/M/1  M/M/m  M/M/1/K Queueing Theory I Summary Little s Law Queueing System Notation Stationary Analysis of Elementary Queueing Systems " M/M/1 " M/M/m " M/M/1/K " Little s Law a(t): the process that counts the number of arrivals

More information

Introduction to queuing theory

Introduction to queuing theory Introduction to queuing theory Claude Rigault ENST claude.rigault@enst.fr Introduction to Queuing theory 1 Outline The problem The number of clients in a system The client process Delay processes Loss

More information

Lecture 10: Semi-Markov Type Processes

Lecture 10: Semi-Markov Type Processes Lecture 1: Semi-Markov Type Processes 1. Semi-Markov processes (SMP) 1.1 Definition of SMP 1.2 Transition probabilities for SMP 1.3 Hitting times and semi-markov renewal equations 2. Processes with semi-markov

More information

LIMITS FOR QUEUES AS THE WAITING ROOM GROWS. Bell Communications Research AT&T Bell Laboratories Red Bank, NJ Murray Hill, NJ 07974

LIMITS FOR QUEUES AS THE WAITING ROOM GROWS. Bell Communications Research AT&T Bell Laboratories Red Bank, NJ Murray Hill, NJ 07974 LIMITS FOR QUEUES AS THE WAITING ROOM GROWS by Daniel P. Heyman Ward Whitt Bell Communications Research AT&T Bell Laboratories Red Bank, NJ 07701 Murray Hill, NJ 07974 May 11, 1988 ABSTRACT We study the

More information

Example: physical systems. If the state space. Example: speech recognition. Context can be. Example: epidemics. Suppose each infected

Example: physical systems. If the state space. Example: speech recognition. Context can be. Example: epidemics. Suppose each infected 4. Markov Chains A discrete time process {X n,n = 0,1,2,...} with discrete state space X n {0,1,2,...} is a Markov chain if it has the Markov property: P[X n+1 =j X n =i,x n 1 =i n 1,...,X 0 =i 0 ] = P[X

More information

Stochastic Proceses Poisson Process

Stochastic Proceses Poisson Process Stochastic Proceses 2014 1. Poisson Process Giovanni Pistone Revised June 9, 2014 1 / 31 Plan 1. Poisson Process (Formal construction) 2. Wiener Process (Formal construction) 3. Infinitely divisible distributions

More information

Chapter 5. Chapter 5 sections

Chapter 5. Chapter 5 sections 1 / 43 sections Discrete univariate distributions: 5.2 Bernoulli and Binomial distributions Just skim 5.3 Hypergeometric distributions 5.4 Poisson distributions Just skim 5.5 Negative Binomial distributions

More information

Queueing Review. Christos Alexopoulos and Dave Goldsman 10/6/16. (mostly from BCNN) Georgia Institute of Technology, Atlanta, GA, USA

Queueing Review. Christos Alexopoulos and Dave Goldsman 10/6/16. (mostly from BCNN) Georgia Institute of Technology, Atlanta, GA, USA 1 / 24 Queueing Review (mostly from BCNN) Christos Alexopoulos and Dave Goldsman Georgia Institute of Technology, Atlanta, GA, USA 10/6/16 2 / 24 Outline 1 Introduction 2 Queueing Notation 3 Transient

More information

Examination paper for TMA4265 Stochastic Processes

Examination paper for TMA4265 Stochastic Processes Department of Mathematical Sciences Examination paper for TMA4265 Stochastic Processes Academic contact during examination: Andrea Riebler Phone: 456 89 592 Examination date: December 14th, 2015 Examination

More information

Poisson Processes. Particles arriving over time at a particle detector. Several ways to describe most common model.

Poisson Processes. Particles arriving over time at a particle detector. Several ways to describe most common model. Poisson Processes Particles arriving over time at a particle detector. Several ways to describe most common model. Approach 1: a) numbers of particles arriving in an interval has Poisson distribution,

More information

Non Markovian Queues (contd.)

Non Markovian Queues (contd.) MODULE 7: RENEWAL PROCESSES 29 Lecture 5 Non Markovian Queues (contd) For the case where the service time is constant, V ar(b) = 0, then the P-K formula for M/D/ queue reduces to L s = ρ + ρ 2 2( ρ) where

More information

Queueing Theory II. Summary. ! M/M/1 Output process. ! Networks of Queue! Method of Stages. ! General Distributions

Queueing Theory II. Summary. ! M/M/1 Output process. ! Networks of Queue! Method of Stages. ! General Distributions Queueing Theory II Summary! M/M/1 Output process! Networks of Queue! Method of Stages " Erlang Distribution " Hyperexponential Distribution! General Distributions " Embedded Markov Chains M/M/1 Output

More information

Networking = Plumbing. Queueing Analysis: I. Last Lecture. Lecture Outline. Jeremiah Deng. 29 July 2013

Networking = Plumbing. Queueing Analysis: I. Last Lecture. Lecture Outline. Jeremiah Deng. 29 July 2013 Networking = Plumbing TELE302 Lecture 7 Queueing Analysis: I Jeremiah Deng University of Otago 29 July 2013 Jeremiah Deng (University of Otago) TELE302 Lecture 7 29 July 2013 1 / 33 Lecture Outline Jeremiah

More information

arxiv: v2 [math.pr] 24 Mar 2018

arxiv: v2 [math.pr] 24 Mar 2018 Exact sampling for some multi-dimensional queueing models with renewal input arxiv:1512.07284v2 [math.pr] 24 Mar 2018 Jose Blanchet Yanan Pei Karl Sigman October 9, 2018 Abstract Using a recent result

More information

Exercises Stochastic Performance Modelling. Hamilton Institute, Summer 2010

Exercises Stochastic Performance Modelling. Hamilton Institute, Summer 2010 Exercises Stochastic Performance Modelling Hamilton Institute, Summer Instruction Exercise Let X be a non-negative random variable with E[X ]

More information

Northwestern University Department of Electrical Engineering and Computer Science

Northwestern University Department of Electrical Engineering and Computer Science Northwestern University Department of Electrical Engineering and Computer Science EECS 454: Modeling and Analysis of Communication Networks Spring 2008 Probability Review As discussed in Lecture 1, probability

More information

The story of the film so far... Mathematics for Informatics 4a. Continuous-time Markov processes. Counting processes

The story of the film so far... Mathematics for Informatics 4a. Continuous-time Markov processes. Counting processes The story of the film so far... Mathematics for Informatics 4a José Figueroa-O Farrill Lecture 19 28 March 2012 We have been studying stochastic processes; i.e., systems whose time evolution has an element

More information

Continuous time Markov chains

Continuous time Markov chains Chapter 2 Continuous time Markov chains As before we assume that we have a finite or countable statespace I, but now the Markov chains X {X(t) : t } have a continuous time parameter t [, ). In some cases,

More information

STAT 380 Markov Chains

STAT 380 Markov Chains STAT 380 Markov Chains Richard Lockhart Simon Fraser University Spring 2016 Richard Lockhart (Simon Fraser University) STAT 380 Markov Chains Spring 2016 1 / 38 1/41 PoissonProcesses.pdf (#2) Poisson Processes

More information