Stochastic modelling of epidemic spread
|
|
- Alexis Thompson
- 5 years ago
- Views:
Transcription
1 Stochastic modelling of epidemic spread Julien Arino Department of Mathematics University of Manitoba Winnipeg Julien 19 May 2012
2 1 Introduction 2 Stochastic processes 3 The SIS model used as an example 4 DTMC 5 CTMC
3 1 Introduction 2 Stochastic processes 3 The SIS model used as an example 4 DTMC 5 CTMC
4 Deterministic SIR Use the standard KMK as in Fred s lectures: S 0 = SI I 0 = SI I R 0 = I Basic reproduction number R 0 = S 0 If R 0 < 1, no outbreak If R 0 > 1, outbreak Use =1/4, N = S + I + R = 100, I (0) = 1, R 0 = {0.5, 1.5} and so that this is true
5 R 0 =0.5 R 0 = S S 90 I 90 I R R Time (days) Time (days) Number infectious Number infectious Time (days) Time (days)
6 Stochastic DTMC SIR Consider S(t) andi (t) (R(t) = N S(t) I (t)). Define transition probabilities p (s 0,i 0 ),(s,i)( t) = P{(S(t + t), I (t + t)) = (s + s 0, i + i 0 ) (S(t), I (t)) = (s, i)} by 8 si t (s 0, i 0 )=( 1, 1) >< i t (s 0, i 0 )=(0, 1) p (s 0,i 0 ),(s,i)( t) = 1 [ si + i] t (s 0, i 0 )=(0, 0) >: 0 otherwise Parameters as previously and t = 0.01
7 7 R 0 =0.5 R 0 = Number infectious 4 3 Number infectious Time (days) Time (days) Number infectious Number infectious Time (days) Time (days)
8 1000 Number of outbreaks 900 Number of simulations with an outbreak (/1000) R 0 =0.5 R 0 = Value of I 0 outbreak: 9t s.t. I (t) > I (0)
9 Why stochastic models? Important to choose right type of model for given application Deterministic models Mathematically easier Perfect reproducibility For given parameter set, one sim su ces Stochastic models Often harder Each realization di erent Needs many simulations In early stage of epidemic, very few infective individuals: Deterministic systems: behaviour entirely governed by parameters Stochastic systems: allows chance to play a role
10 In the context of the SIR model Deterministic R 0 strict threshold R 0 < 1 ) I (t)! 0 monotonically R 0 > 1 ) I (t)! 0 after a bump (outbreak) Stochastic R 0 threshold for mean R 0 < 1 ) I (t)! 0 monotonically (roughly) on average but some realizations have outbreaks R 0 > 1 ) I (t)! 0 after a bump on average (outbreak) but some realizations have no outbreaks
11 1 Introduction 2 Stochastic processes 3 The SIS model used as an example 4 DTMC 5 CTMC
12 Theoretical setting of stochastic processes Stochastic processes form a well defined area of probability, a subset of measure theory, itself a well established area of mathematics Strong theoretical content Very briefly presented here for completeness, will not be used in what follows Used here: A. Klemke, Probability Theory, Springer 2008 DON T PANIC!!!
13 -algebras Definition 1 ( -algebra) Let 6= ; be a set and A 2, the set of all subsets of, bea class of subsets of. A is called a -algebra if: 1 2 A 2 A is closed under complements, i.e., A c := \ A 2 A for any A 2 A 3 A is closed under countable unions, i.e., [ 1 n=1 A n 2 A for any choice of countably many sets A 1, A 2,...2 A is the space of all elementary events and A the system of observable events
14 Probability space Definition 2 1 Apair(, A), with a nonempty set and A 2 a -algebra is a measurable space, withsetsa 2 A measurable sets. If is at most countably infinite and if A =2,thenthemeasurablespace(, 2 )isdiscrete 2 Atriple(, A,µ)isameasure space if (, A) isa measurable space and µ is a measure on A 3 If in addition, µ( ) = 1, then (, A,µ)isaprobability space and the sets A 2 A are events. µ is then usually denoted P (or Prob)
15 Topological spaces and Borel Definition 3 (Topology) -algebras Let 6= ; be an arbitrary set. A class of sets is a topology on if: 1 ;, 2 2 A \ B 2 for any A, B 2 3 ([ A2F A) 2 for any F 2 (, ) isatopological space, setsa 2 are open and A 2 with A c 2 are closed Definition 4 (Borel -algebra) Let (, ) be a topological space. The -algebra B(, ) generated by the open sets is the Borel -algebra on, with elements A 2 B(, ) theborel sets
16 Polish spaces Definition 5 A separable topological space whose topology is induced by a complete metric is a Polish space R d, Z d, R N,(C([0, 1]), k k 1 ) are Polish spaces
17 Stochastic process Definition 6 (Stochastic process) Let (, F, P) be a probability space, (E, ) be a Polish space with Borel -algebra E and T R be arbitrary (typically, T = N, T = Z, T = R + or T an interval). A family of random variables X =(X t, t 2 T ) on (, F, P) with values in (E, E) iscalledastochastic process with index set (or time set) T and range E.
18 OKAY, DONE WITH HARD THEORY!!!
19 Types of processes So a stochastic process is a collection of random variables (r.v.) {X (t,!), t 2 T,! 2 } The index set T and r.v. can be discrete or continuous Discrete time Markov chain (DTMC) hast and X discrete (e.g., T = N and X (t) 2 N) Continuous time Markov chain (CTMC) hast continuous and X discrete (e.g., T = R + and X (t) 2 N) In this course, we focus on these two. Also exist Stochastic di erential equations (SDE), with both T and X (t) continuous
20 Definition 7 (Markov property) Markov property Let E be a Polish space with Borel -algebra B(E), T R and (X t ) t2t an E-valued stochastic process. Assume that (F t ) t2t = (X ) is the filtration generated by X. X has the Markov property if for every A 2 B(E) andalls, t 2 T with s apple t, P[X t 2 A F s ]=P[X t 2 A X s ] Theorem 8 (The usable one) If E is countable then X has the Markov property if for all n 2 N, all s 1 < < s n < tandalli 1,...,i n, i 2 Ewith P[X s1 = i 1,...,X sn = i n ] > 0, there holds P[X t = i X s1 = i 1,...,X sn = i n ]=P[X t = i X sn = i n ]
21 1 Introduction 2 Stochastic processes 3 The SIS model used as an example 4 DTMC 5 CTMC
22 SIS model We consider an SIS model with demography. In ODE, S 0 = b ds SI + I I 0 = SI ( + d)i Total population N 0 = b dn asymptotically constant (N(t)! b/d =: N as t!1). Can work with asymptotically autonomous system where N = N (and S = N I )
23 S 0 = b SI ds + I I 0 = SI ( + d)i DFE: (S, I )=(b/d, 0). EEP: (S, I )= +d, b d +d VdD&W: F = D I [ S] DFE = N, V = D I [( + d)i ] DFE = + d ) R 0 = (FV 1 )= + d N If R 0 < 1, lim t!1 (S(t), I (t)) = (N, 0) If R 0 > 1, lim t!1 (S(t), I (t)) = N R 0, N N R 0
24 Reduction to one equation As N is asymptotically constant, we can write S = N only need I 0 = (N I )I ( + d)i I and so We can then reconstruct dynamics of S from that of I
25 1 Introduction 2 Stochastic processes 3 The SIS model used as an example 4 DTMC 5 CTMC
26 Discrete time Markov chain Definition 9 (DTMC Simple definition) An experiment with finite number of possible outcomes S 1,...,S N is repeated. The sequence of outcomes is a discrete time Markov chain if there is a set of N 2 numbers {p ij } such that the conditional probability of outcome S j on any experiment given outcome S i on the previous experiment is p ij,i.e.,for1apple i, j apple N, t =1,..., p ji =Pr(S j on experiment t +1 S i on experiment t). The outcomes S 1,...,S N are the states, andthep ij are the transition probabilities. ThematrixP =[p ij ]isthetransition matrix.
27 Homogeneity One major distinction: If p ij does not depend on t, thechainishomogeneous If p ij (t), i.e, transition probabilities depend on t, thechainis nonhomogeneous
28 Markov chains operate on two level: Description of probability that the system is in a given state Description of individual realizations of the process Because we assume the Markov property, we need only describe how system switches (transitions) from one state to the next We say that the system has no memory, thenextstatedepends only on the current one
29 States The states are here the di erent values of I. I can take integer values from 0 to N The chain links these states Infection Infection Infection I=0 I=1 I=2 I=3 I=N-1 I=N Recovery or Death Recovery or Death Recovery or Death Recovery or Death
30 Probability vector Let p i (t) =P(I (t) =i) is the probability distribution We must have P i p i(t) =1 0 1 p 0 (t) p 1 (t) p(t) = B A p N (t) Initial condition at time t = 0: p(0)
31 Transition probabilities We then need description of probability of transition In general, suppose that S i is the current state, then one of S 1,...,S N must be the next state. If then we must have p ji = P{X (t + 1) = S j X (t) =S i } p i1 + p i2 + + p in =1, 1 apple i apple N Some of the p ij can be zero
32 Transition matrix The matrix 0 1 p 11 p 12 p 1N P = Bp 21 p 22 p 2N A p N1 p N2 p NN has nonnegative entries, p ij 0 entries less than 1, p ij apple 1 column sum 1, which we write NX p ij =1, i=1 j =1,...,N or, using the notation 1l =(1,...,1) T, 1l T P = 1l T
33 In matrix form p(t + 1) = Pp(t), t =1, 2, 3,... where p(t) =(p 1 (t), p 2 (t),...,p N (t)) T is a (column) probability vector and P =(p ij )isan N transition matrix, 0 1 p 11 p 12 p 1N P = Bp 21 p 22 p 2N A p N1 p N2 p NN
34 Stochastic matrices Definition 10 (Stochastic matrix) The nonnegative N N matrix M is stochastic if P N i=1 a ij =1for all j =1, 2,...,N. In other words, 1l T M = 1l T Theorem 11 Let M be a stochastic matrix M. Then the spectral radius (M) apple 1, i.e., all eigenvalues of M are such that apple 1. Furthermore, =1is an eigenvalue of M M has all column sums 1, 1l T M = 1l T e.value of M with associated left e.vector v, v T M = v T ) 1l T M = 1l T :lefte.vector1l T and e.value 1..
35 Long time behavior Let p(0) be the initial distribution (column) vector. Then p(1) = Pp(0) p(2) = Pp(1) = P(Pp(0)) = P 2 p(0) Iterating, we get that for any t 2 N, Therefore, lim p(t) = t!+1 p(t) =P t p(0) lim t!+1 Pt p(0) = lim t!+1 Pt p(0) if this limit exists (it does if P is constant and chain is regular)
36 Regular Markov chain Regular Markov chains not covered here, as most chains in the demographic context are absorbing. For information, though.. Definition 12 (Regular Markov chain) A regular Markov chain is one in which P k is positive for some integer k > 0, i.e., P k has only positive entries, no zero entries. Theorem 13 If P is the transition matrix of a regular Markov chain, then 1 the powers P t approach a stochastic matrix W, 2 each column of W is the same (column) vector w =(w 1,...,w N ) T, 3 the components of w are positive. So if the Markov chain is regular, lim p(t) =Wp(0) t!+1
37 Interesting properties of matrices Definition 14 (Irreducibility) The square nonnegative matrix M is reducible if there exists a permutation matrix P such that P T MP is block triangular. If M is not reducible, then it is irreducible Definition 15 (Primitivity) A nonnegative matrix M is primitive if and only if there is an integer k > 0 such that M k is positive
38 Definition 16 (Digraph) Directed graphs (digraphs) A directed graph (or digraph) G =(V, A) consists in a set V of vertices and a set A of arcs linking these vertices
39 Connecting graphs and matrices Adjacency matrix M has m ij = 1 if arc from vertex j to vertex i, 0 otherwise ) M = B A Column of zeros (except maybe diagonal entry): vertex is a sink (nothing leaves)
40 Paths of length exactly 2 Given adjacency matrix M, M 2 has number of paths of length exactly two between vertices ) M = B A Write m k ij the (i, j) entryinm k m 2 32 =2since2! 1! 3and2! 4! 3..
41 Connecting matrices, graphs and regular chains Theorem 17 A matrix M is irreducible i the associated connection graph is strongly connected, i.e.,thereisapathbetweenanypair(i, j) of vertices Irreducibility: 8i, j, 9k, m k ij > 0 Theorem 18 A matrix M is primitive if it is irreducible and there is at least one positive entry on the diagonal of M Primitivity: 9k, 8i, j, m k ij > 0 Theorem 19 Markov chain regular, transition matrix P primitive
42 Strong connectedness (1) Not strongly connected
43 Strong connectedness (2) Strongly connected
44 Strong components Strong component 1 Strong component 2
45 Absorbing states, absorbing chains Definition 20 AstateS i in a Markov chain is absorbing if whenever it occurs on the n th generation of the experiment, it then occurs on every subsequent step. In other words, S i is absorbing if p ii =1and p ij =0fori 6= j Definition 21 A Markov chain is said to be absorbing if it has at least one absorbing state, and if from every state it is possible to go to an absorbing state In an absorbing Markov chain, a state that is not absorbing is called transient
46 Here, we focus on absorbing strong components containing only one vertex Absorbing strong components Absorbing strong component Transient states
47 Some questions on absorbing chains 1 Does the process eventually reach an absorbing state? 2 Average number of times spent in a transient state, if starting in a transient state? 3 Average number of steps before entering an absorbing state? 4 Probability of being absorbed by a given absorbing state, when there are more than one, when starting in a given transient state?
48 Reaching an absorbing state Answer to question 1: Theorem 22 In an absorbing Markov chain, the probability of reaching an absorbing state is 1
49 Standard form of the transition matrix For an absorbing chain with k absorbing states and N transient states, the transition matrix can be written as Ik 0 P = R Q k with following meaning, Absorbing states Transient states Absorbing states I k 0 Transient states R Q with I k the k k identity matrix, 0 an k (N k) matrix of zeros, R an (N k) k matrix and Q an (N k) (N k) matrix.
50 The matrix I N k Q is invertible. Let F =(I N k Q) 1 be the fundamental matrix of the Markov chain T i be the sum of the entries on row i of F B = FR Answers to our remaining questions: 2 F ij is the average number of times the process is in the jth transient state if it starts in the ith transient state 3 T i is the average number of steps before the process enters an absorbing state if it starts in the ith transient state 4 B ij is the probability of eventually entering the jth absorbing state if the process starts in the ith transient state
51 Back to the SIS model ODE: I 0 = (N I )I ( + d)i This equation has 3 components: 1 (N I )I new infection 2 I recovery 3 di death In the DTMC, want to know what transitions mean and what their probabilities are Infection Infection Infection I=0 I=1 I=2 I=3 I=N-1 I=N Recovery or Death Recovery or Death Recovery or Death Recovery or Death
52 Consider small amount of time t, su ciently small that only one event occurs: new infection, loss of infectious individual, nothing.. Let B(i) := (N i)i D(i) :=( + d)i be rates of birth and death of infectious. Then p ji ( t) =P {I (t + )=j I(t) =i} 8 B(i) t j = i +1 >< D(i) t j = i 1 = 1 [B(i)+D(i)] t j = i >: 0 otherwise β( N 1)Δ t β( N 2)2 Δ t β( N 1)Δ t Infection Infection Infection I=0 I=1 I=2 I=3 I=N-1 I=N Recovery Recovery Recovery Recovery or Death or Death or Death or Death (γ+d )Δ t (γ+d )2 Δt (γ+d )3 Δ t (γ+d ) N Δ t
53 Transition matrix D(1) t 0 1 (B(1) + D(1)) t 0 B(1) t B.... A D(N) t B(N 1) t 1 D(N) t Note that indices in the matrix range from 0 to N here
54 Properties p 0 is an absorbing state For any initial distribution p(0) = (p 0 (t), p 1 (0),...,p N (0)), lim p(t) =(1, t!1 0,...,0)T lim p 0 (t) =1 t!1 Exists quasi-stationary distribution conditioned on non-extinction q i (t) = p i(t) 1 p 0 (t), i =1,...,N
55 Consequence of absorption We have lim p(t) =(1, t!1 0,...,0)T lim p 0 (t) =1 t!1 but time to absorption can be exponential
56 1 Introduction 2 Stochastic processes 3 The SIS model used as an example 4 DTMC 5 CTMC
57 Several ways to formulate CTMC s A continuous time Markov chain can be formulated in terms of infinitesimal transition probabilities branching process time to next event Here, time is in R +
58 For small t, p ji ( t) =P {I (t + )=j I(t) =i} 8 B(i) t + o( t) j = i +1 >< D(i) t + o( t) j = i 1 = 1 [B(i)+D(i)] t + o( t) j = i >: o( t) otherwise with o( t)! 0as t! 0
59 Forward Kolmogorov equations Assume we know I (0) = k. Then p i (t + t) =p i 1 (t)b(i 1) t + p i+1 (t)d(i + 1) t + p i (t)[1 (B(i)+D(i)) t]+o( t) Compute (p i (t + t) p i (t))/ t and take lim t!0, giving d dt p 0 = p 1 D(1) d dt p i = p i 1 B(i 1) + p i+1 D(i + 1) p i [B(i)+D(i)] i =1,...,N Forward Kolmogorov equations associated to the CTMC
60 Write previous system as In vector form p 0 = Qp with D(1) (B(1) + D(1)) D(2) 0 Q = 0 B(1) (B(2) + D(2)) 0 B D(N) A D(N) Q generator matrix. Of course, p(t) =e Qt p(0)
61 Linking DTMC and CTMC for small t DTMC: p(t + t) =P( t)p(t) for transition matrix P( t). Let t! 0, obtain Kolmogorov equations for CTMC d dt p = Qp where P( t) Q = lim t!0 t I = P 0 (0)
62 Going ODE to CTMC Easy to convert compartmental ODE model to CTMC!!
63 The general setting: flow diagram b S βsi γ I I ds di
64 ODE approach b S βsi γ I I ds di Focus on compartments b S βsi γ I βsi γ I I ds di
65 CTMC approach b S βsi γ I I ds di b Focus on arrows (transitions) S ds βsi γ I I di
66 Focus on transitions b S βsi γ I I ds di b: S! S +1 birth ds: S! S 1 death of susceptible SI : S! S 1, I! I + 1 new infection I : S! S + 1, I! I 1 recovery di : I! I 1 death of infectious Will use S = N I and omit S dynamics
67 Gillespie s algorithm (SIS model) 1: Set t t 0 and I (t) I (t 0 ) 2: while t apple t f do 3: t (N i)i + i + di 4: Draw t from T s E( t ) 5: v [ (N i)i/ t, (N i)i/ t + i/ t, 1] 6: Draw t from U([0, 1]) 7: Find pos such that t v(pos) 8: switch (pos) 9: case 1: New infection, I (t + t )=I(t)+1 10: case 2: End of infectious period, I (t + t )=I(t) 1 11: case 3: Death, I (t + t )=I(t) 1 12: end switch 13: t t + t 14: end while
68 IMPORTANT: in Gillespie s algorithm, we do not consider p ii,the event nothing happens
69 Drawing at random from an exponential distribution Want t from T s E( t ), i.e., T has probability density function f (x, t )= t e tx 1 x 0 Use cumulative distribution function F (x, t )= R x1 f (s, t) ds F (x, t )=(1 e tx )1 x 0 which has values in [0, 1]. So draw from U([0, 1]) and solve F (x, t )= for x: F (x, t )=, 1 e tx =, e tx =1, t x = ln(1 ), x = ln(1 ) t
Stochastic modelling of epidemic spread
Stochastic modelling of epidemic spread Julien Arino Centre for Research on Inner City Health St Michael s Hospital Toronto On leave from Department of Mathematics University of Manitoba Julien Arino@umanitoba.ca
More informationAn Introduction to Stochastic Epidemic Models
An Introduction to Stochastic Epidemic Models Linda J. S. Allen Department of Mathematics and Statistics Texas Tech University Lubbock, Texas 79409-1042, U.S.A. linda.j.allen@ttu.edu 1 Introduction The
More informationAN INTRODUCTION TO STOCHASTIC EPIDEMIC MODELS-PART I
AN INTRODUCTION TO STOCHASTIC EPIDEMIC MODELS-PART I Linda J. S. Allen Department of Mathematics and Statistics Texas Tech University Lubbock, Texas U.S.A. 2008 Summer School on Mathematical Modeling of
More informationStochastic process. X, a series of random variables indexed by t
Stochastic process X, a series of random variables indexed by t X={X(t), t 0} is a continuous time stochastic process X={X(t), t=0,1, } is a discrete time stochastic process X(t) is the state at time t,
More informationIrreducibility. Irreducible. every state can be reached from every other state For any i,j, exist an m 0, such that. Absorbing state: p jj =1
Irreducibility Irreducible every state can be reached from every other state For any i,j, exist an m 0, such that i,j are communicate, if the above condition is valid Irreducible: all states are communicate
More informationMARKOV PROCESSES. Valerio Di Valerio
MARKOV PROCESSES Valerio Di Valerio Stochastic Process Definition: a stochastic process is a collection of random variables {X(t)} indexed by time t T Each X(t) X is a random variable that satisfy some
More informationAARMS Homework Exercises
1 For the gamma distribution, AARMS Homework Exercises (a) Show that the mgf is M(t) = (1 βt) α for t < 1/β (b) Use the mgf to find the mean and variance of the gamma distribution 2 A well-known inequality
More information2. Transience and Recurrence
Virtual Laboratories > 15. Markov Chains > 1 2 3 4 5 6 7 8 9 10 11 12 2. Transience and Recurrence The study of Markov chains, particularly the limiting behavior, depends critically on the random times
More informationThe SIS and SIR stochastic epidemic models revisited
The SIS and SIR stochastic epidemic models revisited Jesús Artalejo Faculty of Mathematics, University Complutense of Madrid Madrid, Spain jesus_artalejomat.ucm.es BCAM Talk, June 16, 2011 Talk Schedule
More informationMarkov Chains, Stochastic Processes, and Matrix Decompositions
Markov Chains, Stochastic Processes, and Matrix Decompositions 5 May 2014 Outline 1 Markov Chains Outline 1 Markov Chains 2 Introduction Perron-Frobenius Matrix Decompositions and Markov Chains Spectral
More informationThursday. Threshold and Sensitivity Analysis
Thursday Threshold and Sensitivity Analysis SIR Model without Demography ds dt di dt dr dt = βsi (2.1) = βsi γi (2.2) = γi (2.3) With initial conditions S(0) > 0, I(0) > 0, and R(0) = 0. This model can
More informationLIMITING PROBABILITY TRANSITION MATRIX OF A CONDENSED FIBONACCI TREE
International Journal of Applied Mathematics Volume 31 No. 18, 41-49 ISSN: 1311-178 (printed version); ISSN: 1314-86 (on-line version) doi: http://dx.doi.org/1.173/ijam.v31i.6 LIMITING PROBABILITY TRANSITION
More informationOutlines. Discrete Time Markov Chain (DTMC) Continuous Time Markov Chain (CTMC)
Markov Chains (2) Outlines Discrete Time Markov Chain (DTMC) Continuous Time Markov Chain (CTMC) 2 pj ( n) denotes the pmf of the random variable p ( n) P( X j) j We will only be concerned with homogenous
More informationDerivation of Itô SDE and Relationship to ODE and CTMC Models
Derivation of Itô SDE and Relationship to ODE and CTMC Models Biomathematics II April 23, 2015 Linda J. S. Allen Texas Tech University TTU 1 Euler-Maruyama Method for Numerical Solution of an Itô SDE dx(t)
More informationStatistics 992 Continuous-time Markov Chains Spring 2004
Summary Continuous-time finite-state-space Markov chains are stochastic processes that are widely used to model the process of nucleotide substitution. This chapter aims to present much of the mathematics
More informationStochastic processes. MAS275 Probability Modelling. Introduction and Markov chains. Continuous time. Markov property
Chapter 1: and Markov chains Stochastic processes We study stochastic processes, which are families of random variables describing the evolution of a quantity with time. In some situations, we can treat
More informationTransmission in finite populations
Transmission in finite populations Juliet Pulliam, PhD Department of Biology and Emerging Pathogens Institute University of Florida and RAPIDD Program, DIEPS Fogarty International Center US National Institutes
More informationLecture 20 : Markov Chains
CSCI 3560 Probability and Computing Instructor: Bogdan Chlebus Lecture 0 : Markov Chains We consider stochastic processes. A process represents a system that evolves through incremental changes called
More information6 Continuous-Time Birth and Death Chains
6 Continuous-Time Birth and Death Chains Angela Peace Biomathematics II MATH 5355 Spring 2017 Lecture notes follow: Allen, Linda JS. An introduction to stochastic processes with applications to biology.
More informationStochastic2010 Page 1
Stochastic2010 Page 1 Extinction Probability for Branching Processes Friday, April 02, 2010 2:03 PM Long-time properties for branching processes Clearly state 0 is an absorbing state, forming its own recurrent
More informationLecture 11: Introduction to Markov Chains. Copyright G. Caire (Sample Lectures) 321
Lecture 11: Introduction to Markov Chains Copyright G. Caire (Sample Lectures) 321 Discrete-time random processes A sequence of RVs indexed by a variable n 2 {0, 1, 2,...} forms a discretetime random process
More informationExample: physical systems. If the state space. Example: speech recognition. Context can be. Example: epidemics. Suppose each infected
4. Markov Chains A discrete time process {X n,n = 0,1,2,...} with discrete state space X n {0,1,2,...} is a Markov chain if it has the Markov property: P[X n+1 =j X n =i,x n 1 =i n 1,...,X 0 =i 0 ] = P[X
More informationCopyright is owned by the Author of the thesis. Permission is given for a copy to be downloaded by an individual for the purpose of research and
Copyright is owned by the Author of the thesis. Permission is given for a copy to be downloaded by an individual for the purpose of research and private study only. The thesis may not be reproduced elsewhere
More informationhttp://www.math.uah.edu/stat/markov/.xhtml 1 of 9 7/16/2009 7:20 AM Virtual Laboratories > 16. Markov Chains > 1 2 3 4 5 6 7 8 9 10 11 12 1. A Markov process is a random process in which the future is
More informationAPM 541: Stochastic Modelling in Biology Discrete-time Markov Chains. Jay Taylor Fall Jay Taylor (ASU) APM 541 Fall / 92
APM 541: Stochastic Modelling in Biology Discrete-time Markov Chains Jay Taylor Fall 2013 Jay Taylor (ASU) APM 541 Fall 2013 1 / 92 Outline 1 Motivation 2 Markov Processes 3 Markov Chains: Basic Properties
More informationMarkov Processes Hamid R. Rabiee
Markov Processes Hamid R. Rabiee Overview Markov Property Markov Chains Definition Stationary Property Paths in Markov Chains Classification of States Steady States in MCs. 2 Markov Property A discrete
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Simulated Annealing Barnabás Póczos & Ryan Tibshirani Andrey Markov Markov Chains 2 Markov Chains Markov chain: Homogen Markov chain: 3 Markov Chains Assume that the state
More informationMarkov Chains. X(t) is a Markov Process if, for arbitrary times t 1 < t 2 <... < t k < t k+1. If X(t) is discrete-valued. If X(t) is continuous-valued
Markov Chains X(t) is a Markov Process if, for arbitrary times t 1 < t 2
More informationMarkov Chains CK eqns Classes Hitting times Rec./trans. Strong Markov Stat. distr. Reversibility * Markov Chains
Markov Chains A random process X is a family {X t : t T } of random variables indexed by some set T. When T = {0, 1, 2,... } one speaks about a discrete-time process, for T = R or T = [0, ) one has a continuous-time
More informationSection 9.2: Matrices. Definition: A matrix A consists of a rectangular array of numbers, or elements, arranged in m rows and n columns.
Section 9.2: Matrices Definition: A matrix A consists of a rectangular array of numbers, or elements, arranged in m rows and n columns. That is, a 11 a 12 a 1n a 21 a 22 a 2n A =...... a m1 a m2 a mn A
More informationCDA6530: Performance Models of Computers and Networks. Chapter 3: Review of Practical Stochastic Processes
CDA6530: Performance Models of Computers and Networks Chapter 3: Review of Practical Stochastic Processes Definition Stochastic process X = {X(t), t2 T} is a collection of random variables (rvs); one rv
More informationPart I Stochastic variables and Markov chains
Part I Stochastic variables and Markov chains Random variables describe the behaviour of a phenomenon independent of any specific sample space Distribution function (cdf, cumulative distribution function)
More informationLecture 20: Reversible Processes and Queues
Lecture 20: Reversible Processes and Queues 1 Examples of reversible processes 11 Birth-death processes We define two non-negative sequences birth and death rates denoted by {λ n : n N 0 } and {µ n : n
More informationSTOCHASTIC PROCESSES Basic notions
J. Virtamo 38.3143 Queueing Theory / Stochastic processes 1 STOCHASTIC PROCESSES Basic notions Often the systems we consider evolve in time and we are interested in their dynamic behaviour, usually involving
More informationPerron Frobenius Theory
Perron Frobenius Theory Oskar Perron Georg Frobenius (1880 1975) (1849 1917) Stefan Güttel Perron Frobenius Theory 1 / 10 Positive and Nonnegative Matrices Let A, B R m n. A B if a ij b ij i, j, A > B
More informationProbability Distributions
Lecture : Background in Probability Theory Probability Distributions The probability mass function (pmf) or probability density functions (pdf), mean, µ, variance, σ 2, and moment generating function (mgf)
More informationSMSTC (2007/08) Probability.
SMSTC (27/8) Probability www.smstc.ac.uk Contents 12 Markov chains in continuous time 12 1 12.1 Markov property and the Kolmogorov equations.................... 12 2 12.1.1 Finite state space.................................
More informationLecture 21. David Aldous. 16 October David Aldous Lecture 21
Lecture 21 David Aldous 16 October 2015 In continuous time 0 t < we specify transition rates or informally P(X (t+δ)=j X (t)=i, past ) q ij = lim δ 0 δ P(X (t + dt) = j X (t) = i) = q ij dt but note these
More information6.207/14.15: Networks Lectures 4, 5 & 6: Linear Dynamics, Markov Chains, Centralities
6.207/14.15: Networks Lectures 4, 5 & 6: Linear Dynamics, Markov Chains, Centralities 1 Outline Outline Dynamical systems. Linear and Non-linear. Convergence. Linear algebra and Lyapunov functions. Markov
More informationMATH 56A: STOCHASTIC PROCESSES CHAPTER 1
MATH 56A: STOCHASTIC PROCESSES CHAPTER. Finite Markov chains For the sake of completeness of these notes I decided to write a summary of the basic concepts of finite Markov chains. The topics in this chapter
More informationDiscrete Time Markov Chain of a Dynamical System with a Rest Phase
Discrete Time Markov Chain of a Dynamical System with a Rest Phase Abstract A stochastic model, in the form of a discrete time Markov chain, is constructed to describe the dynamics of a population that
More informationThree Disguises of 1 x = e λx
Three Disguises of 1 x = e λx Chathuri Karunarathna Mudiyanselage Rabi K.C. Winfried Just Department of Mathematics, Ohio University Mathematical Biology and Dynamical Systems Seminar Ohio University November
More informationHomework 4 due on Thursday, December 15 at 5 PM (hard deadline).
Large-Time Behavior for Continuous-Time Markov Chains Friday, December 02, 2011 10:58 AM Homework 4 due on Thursday, December 15 at 5 PM (hard deadline). How are formulas for large-time behavior of discrete-time
More informationProbability Distributions
Lecture 1: Background in Probability Theory Probability Distributions The probability mass function (pmf) or probability density functions (pdf), mean, µ, variance, σ 2, and moment generating function
More informationIEOR 6711, HMWK 5, Professor Sigman
IEOR 6711, HMWK 5, Professor Sigman 1. Semi-Markov processes: Consider an irreducible positive recurrent discrete-time Markov chain {X n } with transition matrix P (P i,j ), i, j S, and finite state space.
More information8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains
8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains 8.1 Review 8.2 Statistical Equilibrium 8.3 Two-State Markov Chain 8.4 Existence of P ( ) 8.5 Classification of States
More information18.175: Lecture 30 Markov chains
18.175: Lecture 30 Markov chains Scott Sheffield MIT Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup Outline Review what you know
More informationMean-field dual of cooperative reproduction
The mean-field dual of systems with cooperative reproduction joint with Tibor Mach (Prague) A. Sturm (Göttingen) Friday, July 6th, 2018 Poisson construction of Markov processes Let (X t ) t 0 be a continuous-time
More informationIntroduction to SEIR Models
Department of Epidemiology and Public Health Health Systems Research and Dynamical Modelling Unit Introduction to SEIR Models Nakul Chitnis Workshop on Mathematical Models of Climate Variability, Environmental
More informationMath Homework 5 Solutions
Math 45 - Homework 5 Solutions. Exercise.3., textbook. The stochastic matrix for the gambler problem has the following form, where the states are ordered as (,, 4, 6, 8, ): P = The corresponding diagram
More informationAvailability. M(t) = 1 - e -mt
Availability Availability - A(t) the probability that the system is operating correctly and is available to perform its functions at the instant of time t More general concept than reliability: failure
More informationIEOR 6711: Professor Whitt. Introduction to Markov Chains
IEOR 6711: Professor Whitt Introduction to Markov Chains 1. Markov Mouse: The Closed Maze We start by considering how to model a mouse moving around in a maze. The maze is a closed space containing nine
More informationDefinition A finite Markov chain is a memoryless homogeneous discrete stochastic process with a finite number of states.
Chapter 8 Finite Markov Chains A discrete system is characterized by a set V of states and transitions between the states. V is referred to as the state space. We think of the transitions as occurring
More informationRecap. Probability, stochastic processes, Markov chains. ELEC-C7210 Modeling and analysis of communication networks
Recap Probability, stochastic processes, Markov chains ELEC-C7210 Modeling and analysis of communication networks 1 Recap: Probability theory important distributions Discrete distributions Geometric distribution
More informationThe Transition Probability Function P ij (t)
The Transition Probability Function P ij (t) Consider a continuous time Markov chain {X(t), t 0}. We are interested in the probability that in t time units the process will be in state j, given that it
More informationNo class on Thursday, October 1. No office hours on Tuesday, September 29 and Thursday, October 1.
Stationary Distributions Monday, September 28, 2015 2:02 PM No class on Thursday, October 1. No office hours on Tuesday, September 29 and Thursday, October 1. Homework 1 due Friday, October 2 at 5 PM strongly
More informationProbability, Random Processes and Inference
INSTITUTO POLITÉCNICO NACIONAL CENTRO DE INVESTIGACION EN COMPUTACION Laboratorio de Ciberseguridad Probability, Random Processes and Inference Dr. Ponciano Jorge Escamilla Ambrosio pescamilla@cic.ipn.mx
More information2 Discrete-Time Markov Chains
2 Discrete-Time Markov Chains Angela Peace Biomathematics II MATH 5355 Spring 2017 Lecture notes follow: Allen, Linda JS. An introduction to stochastic processes with applications to biology. CRC Press,
More informationEpidemics in Networks Part 2 Compartmental Disease Models
Epidemics in Networks Part 2 Compartmental Disease Models Joel C. Miller & Tom Hladish 18 20 July 2018 1 / 35 Introduction to Compartmental Models Dynamics R 0 Epidemic Probability Epidemic size Review
More information1.3 Convergence of Regular Markov Chains
Markov Chains and Random Walks on Graphs 3 Applying the same argument to A T, which has the same λ 0 as A, yields the row sum bounds Corollary 0 Let P 0 be the transition matrix of a regular Markov chain
More informationMarkov Chains. As part of Interdisciplinary Mathematical Modeling, By Warren Weckesser Copyright c 2006.
Markov Chains As part of Interdisciplinary Mathematical Modeling, By Warren Weckesser Copyright c 2006 1 Introduction A (finite) Markov chain is a process with a finite number of states (or outcomes, or
More informationMatrix analytic methods. Lecture 1: Structured Markov chains and their stationary distribution
1/29 Matrix analytic methods Lecture 1: Structured Markov chains and their stationary distribution Sophie Hautphenne and David Stanford (with thanks to Guy Latouche, U. Brussels and Peter Taylor, U. Melbourne
More informationIEOR 6711: Stochastic Models I Professor Whitt, Thursday, November 29, Weirdness in CTMC s
IEOR 6711: Stochastic Models I Professor Whitt, Thursday, November 29, 2012 Weirdness in CTMC s Where s your will to be weird? Jim Morrison, The Doors We are all a little weird. And life is a little weird.
More informationBirth and Death Processes. Birth and Death Processes. Linear Growth with Immigration. Limiting Behaviour for Birth and Death Processes
DTU Informatics 247 Stochastic Processes 6, October 27 Today: Limiting behaviour of birth and death processes Birth and death processes with absorbing states Finite state continuous time Markov chains
More informationNonnegative Matrices I
Nonnegative Matrices I Daisuke Oyama Topics in Economic Theory September 26, 2017 References J. L. Stuart, Digraphs and Matrices, in Handbook of Linear Algebra, Chapter 29, 2006. R. A. Brualdi and H. J.
More informationLecture 15 Perron-Frobenius Theory
EE363 Winter 2005-06 Lecture 15 Perron-Frobenius Theory Positive and nonnegative matrices and vectors Perron-Frobenius theorems Markov chains Economic growth Population dynamics Max-min and min-max characterization
More informationStochastic Models. John M. Drake & Pejman Rohani
Epidemiological data are noisy Two types of noise: Observation error: the data are probabilistically related to the true state of the system Process noise: the system progresses probabilistically Environmental
More informationMarkov Chains and Stochastic Sampling
Part I Markov Chains and Stochastic Sampling 1 Markov Chains and Random Walks on Graphs 1.1 Structure of Finite Markov Chains We shall only consider Markov chains with a finite, but usually very large,
More informationSection 9.2: Matrices.. a m1 a m2 a mn
Section 9.2: Matrices Definition: A matrix is a rectangular array of numbers: a 11 a 12 a 1n a 21 a 22 a 2n A =...... a m1 a m2 a mn In general, a ij denotes the (i, j) entry of A. That is, the entry in
More informationData analysis and stochastic modeling
Data analysis and stochastic modeling Lecture 7 An introduction to queueing theory Guillaume Gravier guillaume.gravier@irisa.fr with a lot of help from Paul Jensen s course http://www.me.utexas.edu/ jensen/ormm/instruction/powerpoint/or_models_09/14_queuing.ppt
More informationMarkov Chains and Pandemics
Markov Chains and Pandemics Caleb Dedmore and Brad Smith December 8, 2016 Page 1 of 16 Abstract Markov Chain Theory is a powerful tool used in statistical analysis to make predictions about future events
More informationIntroduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 Markov Chain Monte Carlo Methods Barnabás Póczos & Aarti Singh Contents Markov Chain Monte Carlo Methods Goal & Motivation Sampling Rejection Importance Markov
More informationSocial network analysis: social learning
Social network analysis: social learning Donglei Du (ddu@unb.edu) Faculty of Business Administration, University of New Brunswick, NB Canada Fredericton E3B 9Y2 October 20, 2016 Donglei Du (UNB) AlgoTrading
More informationSystem theory and system identification of compartmental systems Hof, Jacoba Marchiena van den
University of Groningen System theory and system identification of compartmental systems Hof, Jacoba Marchiena van den IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF)
More informationLet (Ω, F) be a measureable space. A filtration in discrete time is a sequence of. F s F t
2.2 Filtrations Let (Ω, F) be a measureable space. A filtration in discrete time is a sequence of σ algebras {F t } such that F t F and F t F t+1 for all t = 0, 1,.... In continuous time, the second condition
More informationMarkov Chains, Random Walks on Graphs, and the Laplacian
Markov Chains, Random Walks on Graphs, and the Laplacian CMPSCI 791BB: Advanced ML Sridhar Mahadevan Random Walks! There is significant interest in the problem of random walks! Markov chain analysis! Computer
More informationStatistics 253/317 Introduction to Probability Models. Winter Midterm Exam Monday, Feb 10, 2014
Statistics 253/317 Introduction to Probability Models Winter 2014 - Midterm Exam Monday, Feb 10, 2014 Student Name (print): (a) Do not sit directly next to another student. (b) This is a closed-book, closed-note
More informationMarkov Chains Absorption (cont d) Hamid R. Rabiee
Markov Chains Absorption (cont d) Hamid R. Rabiee 1 Absorbing Markov Chain An absorbing state is one in which the probability that the process remains in that state once it enters the state is 1 (i.e.,
More information10. Linear Systems of ODEs, Matrix multiplication, superposition principle (parts of sections )
c Dr. Igor Zelenko, Fall 2017 1 10. Linear Systems of ODEs, Matrix multiplication, superposition principle (parts of sections 7.2-7.4) 1. When each of the functions F 1, F 2,..., F n in right-hand side
More information1 Stochastic Dynamic Programming
1 Stochastic Dynamic Programming Formally, a stochastic dynamic program has the same components as a deterministic one; the only modification is to the state transition equation. When events in the future
More informationMarkov chains. Randomness and Computation. Markov chains. Markov processes
Markov chains Randomness and Computation or, Randomized Algorithms Mary Cryan School of Informatics University of Edinburgh Definition (Definition 7) A discrete-time stochastic process on the state space
More informationFinite-Horizon Statistics for Markov chains
Analyzing FSDT Markov chains Friday, September 30, 2011 2:03 PM Simulating FSDT Markov chains, as we have said is very straightforward, either by using probability transition matrix or stochastic update
More informationCensoring Technique in Studying Block-Structured Markov Chains
Censoring Technique in Studying Block-Structured Markov Chains Yiqiang Q. Zhao 1 Abstract: Markov chains with block-structured transition matrices find many applications in various areas. Such Markov chains
More informationMarkov chains. 1 Discrete time Markov chains. c A. J. Ganesh, University of Bristol, 2015
Markov chains c A. J. Ganesh, University of Bristol, 2015 1 Discrete time Markov chains Example: A drunkard is walking home from the pub. There are n lampposts between the pub and his home, at each of
More informationThe Theory behind PageRank
The Theory behind PageRank Mauro Sozio Telecom ParisTech May 21, 2014 Mauro Sozio (LTCI TPT) The Theory behind PageRank May 21, 2014 1 / 19 A Crash Course on Discrete Probability Events and Probability
More informationMarkov-modulated interactions in SIR epidemics
Markov-modulated interactions in SIR epidemics E. Almaraz 1, A. Gómez-Corral 2 (1)Departamento de Estadística e Investigación Operativa, Facultad de Ciencias Matemáticas (UCM), (2)Instituto de Ciencias
More informationLectures on Markov Chains
Lectures on Markov Chains David M. McClendon Department of Mathematics Ferris State University 2016 edition 1 Contents Contents 2 1 Markov chains 4 1.1 The definition of a Markov chain.....................
More informationMarkov Model. Model representing the different resident states of a system, and the transitions between the different states
Markov Model Model representing the different resident states of a system, and the transitions between the different states (applicable to repairable, as well as non-repairable systems) System behavior
More informationNumerical solution of stochastic epidemiological models
Numerical solution of stochastic epidemiological models John M. Drake & Pejman Rohani 1 Introduction He we expand our modeling toolkit to include methods for studying stochastic versions of the compartmental
More informationAn Introduction to Entropy and Subshifts of. Finite Type
An Introduction to Entropy and Subshifts of Finite Type Abby Pekoske Department of Mathematics Oregon State University pekoskea@math.oregonstate.edu August 4, 2015 Abstract This work gives an overview
More information(b) What is the variance of the time until the second customer arrives, starting empty, assuming that we measure time in minutes?
IEOR 3106: Introduction to Operations Research: Stochastic Models Fall 2006, Professor Whitt SOLUTIONS to Final Exam Chapters 4-7 and 10 in Ross, Tuesday, December 19, 4:10pm-7:00pm Open Book: but only
More informationLAW OF LARGE NUMBERS FOR THE SIRS EPIDEMIC
LAW OF LARGE NUMBERS FOR THE SIRS EPIDEMIC R. G. DOLGOARSHINNYKH Abstract. We establish law of large numbers for SIRS stochastic epidemic processes: as the population size increases the paths of SIRS epidemic
More informationModelling in Systems Biology
Modelling in Systems Biology Maria Grazia Vigliotti thanks to my students Anton Stefanek, Ahmed Guecioueur Imperial College Formal representation of chemical reactions precise qualitative and quantitative
More informationLTCC. Exercises solutions
1. Markov chain LTCC. Exercises solutions (a) Draw a state space diagram with the loops for the possible steps. If the chain starts in state 4, it must stay there. If the chain starts in state 1, it will
More informationSTAT STOCHASTIC PROCESSES. Contents
STAT 3911 - STOCHASTIC PROCESSES ANDREW TULLOCH Contents 1. Stochastic Processes 2 2. Classification of states 2 3. Limit theorems for Markov chains 4 4. First step analysis 5 5. Branching processes 5
More informationMath 443/543 Graph Theory Notes 5: Graphs as matrices, spectral graph theory, and PageRank
Math 443/543 Graph Theory Notes 5: Graphs as matrices, spectral graph theory, and PageRank David Glickenstein November 3, 4 Representing graphs as matrices It will sometimes be useful to represent graphs
More informationStatistics 150: Spring 2007
Statistics 150: Spring 2007 April 23, 2008 0-1 1 Limiting Probabilities If the discrete-time Markov chain with transition probabilities p ij is irreducible and positive recurrent; then the limiting probabilities
More informationChapter 1: Systems of linear equations and matrices. Section 1.1: Introduction to systems of linear equations
Chapter 1: Systems of linear equations and matrices Section 1.1: Introduction to systems of linear equations Definition: A linear equation in n variables can be expressed in the form a 1 x 1 + a 2 x 2
More informationCDA5530: Performance Models of Computers and Networks. Chapter 3: Review of Practical
CDA5530: Performance Models of Computers and Networks Chapter 3: Review of Practical Stochastic Processes Definition Stochastic ti process X = {X(t), t T} is a collection of random variables (rvs); one
More informationSTA 624 Practice Exam 2 Applied Stochastic Processes Spring, 2008
Name STA 624 Practice Exam 2 Applied Stochastic Processes Spring, 2008 There are five questions on this test. DO use calculators if you need them. And then a miracle occurs is not a valid answer. There
More information