Stochastic Modelling Unit 1: Markov chain models
|
|
- Julius Hodges
- 6 years ago
- Views:
Transcription
1 Stochastic Modelling Unit 1: Markov chain models Russell Gerrard and Douglas Wright Cass Business School, City University, London June 2004 Contents of Unit 1 1 Stochastic Processes 2 Markov Chains 3 Poisson Processes 4 Markov Jump Processes 5 Martingales 1 Stochastic processes 1.1 The Stochastic Process A stochastic process is just a sequence of random variables. It involves a random element and a time element. Any collection {X α : α A} of random variables may be considered as a stochastic process. Examples: rainfall since midnight the number of cars passing a traffic census stock market index, observed day by day (or minute by minute) a football team s points score, match by match the size of the hole in the ozone layer An unpredictable (random) observable process may be modelled to predict its future behaviour: short-term movements (exchange rates) 1
2 medium-term extrema (storm damage claims, derivatives) long-term asymptotics (steady-state costs, eg health insurance) Usefulness for modelling is determined by the nature of the dependence of X n (or X(t)) on preceding X values. The set of all values X can ever take is the state space. 1.2 Classification of stochastic processes Based on time parameter: discrete-time or continuous-time. We write {X 1, X 2,...} in discrete time, {X(t) : t 0} in continuous Based on state space: discrete or continuous. A counting process is discrete, taking only integer values; a no-claims bonus scheme is another example of a discrete state space; the size of an insurance claim is treated as continuous. 1.3 The History H t the history of X up until time t is the collection of answers to all questions about the behaviour of X in [0, t]. We can write Ht X to avoid confusion. In formal notation, Ht X = σ({x s : 0 s t}). IE(X t H t ) = X t, but IE(X t+s H t ) X t+s. {H X t : t 0} is the filtration generated by X. A filtration may contain the histories of many processes at once. (The same things are true in discrete time.) 1.4 Stationarity A stochastic process X is stationary if the distribution of X t is the same for all t for any k the distribution of the vector (X t, X t+1,..., X t+k ) is the same for all t. Example: we might assume that the process of interest rates is stationary. 1.5 Stationary, independent increments An increment of a process X is a quantity of the form X t+s X t. X has stationary increments if the distribution of X t+s X t is the same as that of X s X 0 for all s, t. X has independent increments if X t+s X t and X u+v X u are independent r.v.s whenever the intervals (t, t + s) and (u, u + v) do not overlap. 2
3 1.6 The Markov property This is about memorylessness. The future is independent of the past, given the present Formally, X has the Markov property if the distribution of X t+s given H t is the same as the distribution of X t+s given X t. 1.7 The Martingale property (Named after a gambling system, in turn named after a horse harness.) On average, the process stays where it is. Like the no-arbitrage principle. Formally, IE(X t+s H t ) = X t. (H is usually, but does not have to be, H X.) 2 Markov Chains 2.1 Introduction A discrete-time process {X n : n 0} with a discrete state space is a Markov Chain if it possesses the Markov property: The future is independent of the past, given the present Example: Weather. State space {rainy, sunny}. Example: Random walk, binomial lattice model. Example: A no-claims discount scheme with four levels: Level 1 Level 2 Level 3 Level 4 no discount 10% discount 25% discount 40% discount After a year without making a claim, move up to the next discount level (unless already in 4). After a year with one claim, move down to the next discount level (unless already in 1). 3
4 Example: An actor s career. The actor is employed, unemployed or doing temporary work. A month of employment is followed by another such month, with prob. 0.8, or by unemployment otherwise. After a month of unemployment, the actor finds temporary work with probability 0.4, employment as an actor with probability 0.2. After a month of temporary work the probability of finding employment is 0.2, or the temporary work continues with probability 0.7; otherwise unemployment looms. 2.2 Transition probabilities Define the transition probability from i to j as p ij = P(X n+1 = j X n = i), independent of events before time n by the Markov property. We assemble the p ij into a matrix P, the transition matrix. Note: entries 0, row sums = 1. Example (actor) P = Chapman-Kolmogorov Equations The k-step transition probability p (k) ij is defined as P(X n+k = j X n = i). Chapman-Kolmogorov: P (k) = P k, the kth power of matrix p. 2.4 Long-term behaviour In many cases we observe that for large n the distribution of X n converges to a limit π, ie. P(X n = j X n = i) π j, the limit being the same regardless of the starting point. We can find π by solving πp = π with the constraint that π j = 1. 4
5 2.5 Fitting a Markov Chain model Observe the process for an extended period, or several copies of the process (one actor for several years, or several actors for one year). n i = number of times the process is in state i, n ij the number of transitions from i to j. Then estimate p ij by p ij = n ij /n i. 2.6 Time-inhomogeneous Markov Chains Transition probabilities may change with time. Young drivers and very old drivers may have more accidents than middle-aged drivers, for example. P would therefore depend on n, giving P n. The analogous form of the Chapman-Kolmogorov Equations continues to hold, but there is no long-term limit. Even if individual behaviour is not time-homogeneous, the behaviour of the population as a whole could be treated as time-homogeneous because of statistical equilibrium. 3 Poisson Processes 3.1 The standard Poisson Process Events can happen at any time. The probability of an event in any time interval (t, t + dt) is λdt + o(dt), independently of other intervals. Suppose T 1, T 1 + T 2, T 1 + T 2 + T 3,... are the times of the first, second, third,... events. The T i are independent of one another, and all have exponential distribution with rate λ. Therefore X(t), the number of events in (0, t), is a Poisson random variable with mean λt. X is a stochastic process in continuous time (t > 0). 3.2 Time-dependent Poisson Process The underlying rate at which events occur changes in some deterministic way. For example, more domestic storm damage claims occur in autumn than in summer. X(t) is still a Poisson r.v., but the mean is now more complicated. 3.3 Age-dependent Poisson Processes The transition rate depends on the time since the last transition ( age ). 5
6 If h(t) is the transition rate at age t, we have where S is the survivor function, S(t) = P(time to next transition> t). The solution is S(t + dt) = S(t) {1 h(t)dt}, ( S(t) = exp t 0 ) h(u) du. X(t) is no longer a Poisson r.v.: instead, X is a simple renewal process. 4 Markov Jump Processes 4.1 The Markov property Same as in discrete time: P(X(t + s) = j H t ) depends only on X(t), j, s and possibly t. Initially we only consider cases where this probability is independent of t (time-homogeneous cases); use the notation p ij (s) = P(X(t + s) = j X(t) = i). Later there will be variations involving time-inhomogeneity or age-dependence. Example: reversionary annuity: states are {both partners alive, only husband alive, only wife alive, neither alive} Example: marriage model: states {never married, married, divorced, remarried, widowed} Example: long-term care model: states {healthy, short-term sick, long-term sick} 4.2 The Behaviour of Markov Processes The length of time spent in the current state x must have a memoryless (ie, exponential) distribution, with rate parameter (λ x ) which depends only on the state itself. This implies that the mean time spent in state x on any one visit is 1/λ x. Once the jump occurs, the probability that it takes the chain to state y is r xy, regardless of duration in x. The matrix R is a discrete-time transition matrix. The associated chain is the jump chain of X. A state x for which λ x = 0 is absorbing: if the process hits the state, it will never leave. 6
7 4.3 The Chapman-Kolmogorov Equations As in discrete time the Chapman-Kolmogorov Equations are Notice that P (0) = I, the identity matrix. P (s + t) = P (s) P (t) 4.4 The Kolmogorov Differential Equations Setting t = ds in the Chapman-Kolmogorov equations gives or which gives P (s + ds) = P (s) P (ds) P (s + ds) P (s) = P (s){p (ds) I}, P (s) = P (s) Q, where Q = P (0), the generator matrix of the chain. This is called the Kolmogorov Forward Equation. By putting s = dt in the Chapman-Kolmogorov equations, we obtain the Kolmogorov Backward Equation, P (t) = Q P (t). The formal solution of the Kolmogorov DEs is P (t) = exp(tq), but this exponential may be hard to evaluate. Example: Linear birth-and-death process. The rate of transitions from x to x + 1 is the birth rate xβ; from x to x 1 the death rate xδ. Therefore q x,x 1 = xδ, q x,x+1 = xβ, q x,x = x(β + δ) with all other entries in the Q-matrix being zero. Thus the Q-matrix is δ (β + δ) β 0... Q = 0 2δ 2(β + δ) 2β δ 3(β + δ)
8 4.5 Long-term behaviour As for the discrete-time version, a time-homogeneous Markov jump process converges (under certain conditions), in the sense that P(X(t) = j) π j as t regardless of the starting position, where π is the solution to πq = 0. We can therefore calculate the long-run proportion of time spent in the states. 4.6 Fitting a Markov jump model The elements of R, the transition matrix of the jump chain, are estimated as in discrete time. Estimates for the parameters λ x are based on average durations in state x. Testing goodness of fit can take many forms, such as is exponential a good distribution to fit? are destinations independent of durations? do 2nd and subsequent visits have different durations? 4.7 Time-inhomogeneous Markov models Used when there is an underlying reason for transition rates to change with time. Example: care model: rates of falling sick are higher in winter, recovery rates lower. Also used for a single individual with age-varying transition rates. The transition rates are: P(X(t + dt) = j X(t) = i) = q ij (t) dt + o(dt). The formal solution P (t) = exp(tq) no longer holds. The only possibility is to attempt to solve the DEs by hand. Example: Poisson process with rate λ(t) = 2λt/(1 + t 2 ): d dt p 0,j(t) = 2λt 1 + t p 0,j(t) + 2λt t p 0,j 1(t). 2 Even in this case it is not obvious that there is an explicit solution. In more complicated cases numerical approximation is required. It is unlikely that X converges to a limiting distribution 8
9 4.8 Semi-Markov models Two restrictions imposed by the Markov format are relaxed: the durations need not be exponential the destinations may depend on the durations Much more flexible. But difficult to fit because there are so many parameters. Only practical when there is a prior idea of the distribution of the durations. Long-run proportion of time spent in the various states may be found as above. 5 Martingales 5.1 Basic properties Formally, IE(X t+s H t ) = X t. (H is usually, but does not have to be, H X.) Example: random walk with zero-mean increments, binomial lattice, even a (non-markov) process with increased volatility after a jump. It follows from the definition that IEX t = IEX Martingale convergence The usefulness of martingales arises from convergence theorems: A martingale bounded above or below converges, X t X. If X is bounded above and below, or has bounded variance, then IEX = IEX Stopping times A random time T is a stopping time if you can always tell whether or not it has happened yet. Example: the first time something occurs, but not the last time something occurs. Formally: the answer to the question Is T t? must lie in H t. 9
10 5.4 Stopped Martingales If X is a martingale and T a stopping time, then Z, defined by { Xt if T > t Z t = if T t is a stopped martingale. A stopped martingale is also a martingale. X T 5.5 Optional Stopping Suppose X is a martingale, certain to hit 0 or a eventually, and T is the first time of hitting one of them. Let q be the probability that X hits 0 first. Then Z is a bounded martingale, so converges, and Z = 0 (with prob. q) or a (with prob. 1 q), so IEZ = a(1 q). But IEZ = IEZ 0 = X 0. Hence q = 1 X 0 /a. 5.6 Practical issues First find your martingale. Commonly the process being studied is not a martingale. But transformations can help (such as Y t = h Xt ). 10
Recap. Probability, stochastic processes, Markov chains. ELEC-C7210 Modeling and analysis of communication networks
Recap Probability, stochastic processes, Markov chains ELEC-C7210 Modeling and analysis of communication networks 1 Recap: Probability theory important distributions Discrete distributions Geometric distribution
More informationMarkov Chains. X(t) is a Markov Process if, for arbitrary times t 1 < t 2 <... < t k < t k+1. If X(t) is discrete-valued. If X(t) is continuous-valued
Markov Chains X(t) is a Markov Process if, for arbitrary times t 1 < t 2
More informationCDA5530: Performance Models of Computers and Networks. Chapter 3: Review of Practical
CDA5530: Performance Models of Computers and Networks Chapter 3: Review of Practical Stochastic Processes Definition Stochastic ti process X = {X(t), t T} is a collection of random variables (rvs); one
More informationCDA6530: Performance Models of Computers and Networks. Chapter 3: Review of Practical Stochastic Processes
CDA6530: Performance Models of Computers and Networks Chapter 3: Review of Practical Stochastic Processes Definition Stochastic process X = {X(t), t2 T} is a collection of random variables (rvs); one rv
More information18.175: Lecture 30 Markov chains
18.175: Lecture 30 Markov chains Scott Sheffield MIT Outline Review what you know about finite state Markov chains Finite state ergodicity and stationarity More general setup Outline Review what you know
More informationStochastic process. X, a series of random variables indexed by t
Stochastic process X, a series of random variables indexed by t X={X(t), t 0} is a continuous time stochastic process X={X(t), t=0,1, } is a discrete time stochastic process X(t) is the state at time t,
More information(b) What is the variance of the time until the second customer arrives, starting empty, assuming that we measure time in minutes?
IEOR 3106: Introduction to Operations Research: Stochastic Models Fall 2006, Professor Whitt SOLUTIONS to Final Exam Chapters 4-7 and 10 in Ross, Tuesday, December 19, 4:10pm-7:00pm Open Book: but only
More informationSTOCHASTIC PROCESSES Basic notions
J. Virtamo 38.3143 Queueing Theory / Stochastic processes 1 STOCHASTIC PROCESSES Basic notions Often the systems we consider evolve in time and we are interested in their dynamic behaviour, usually involving
More information18.600: Lecture 32 Markov Chains
18.600: Lecture 32 Markov Chains Scott Sheffield MIT Outline Markov chains Examples Ergodicity and stationarity Outline Markov chains Examples Ergodicity and stationarity Markov chains Consider a sequence
More information18.440: Lecture 33 Markov Chains
18.440: Lecture 33 Markov Chains Scott Sheffield MIT 1 Outline Markov chains Examples Ergodicity and stationarity 2 Outline Markov chains Examples Ergodicity and stationarity 3 Markov chains Consider a
More informationProbability, Random Processes and Inference
INSTITUTO POLITÉCNICO NACIONAL CENTRO DE INVESTIGACION EN COMPUTACION Laboratorio de Ciberseguridad Probability, Random Processes and Inference Dr. Ponciano Jorge Escamilla Ambrosio pescamilla@cic.ipn.mx
More informationMarkov chains. 1 Discrete time Markov chains. c A. J. Ganesh, University of Bristol, 2015
Markov chains c A. J. Ganesh, University of Bristol, 2015 1 Discrete time Markov chains Example: A drunkard is walking home from the pub. There are n lampposts between the pub and his home, at each of
More informationLecture Notes 7 Random Processes. Markov Processes Markov Chains. Random Processes
Lecture Notes 7 Random Processes Definition IID Processes Bernoulli Process Binomial Counting Process Interarrival Time Process Markov Processes Markov Chains Classification of States Steady State Probabilities
More informationPart I Stochastic variables and Markov chains
Part I Stochastic variables and Markov chains Random variables describe the behaviour of a phenomenon independent of any specific sample space Distribution function (cdf, cumulative distribution function)
More informationSMSTC (2007/08) Probability.
SMSTC (27/8) Probability www.smstc.ac.uk Contents 12 Markov chains in continuous time 12 1 12.1 Markov property and the Kolmogorov equations.................... 12 2 12.1.1 Finite state space.................................
More informationMarkov processes and queueing networks
Inria September 22, 2015 Outline Poisson processes Markov jump processes Some queueing networks The Poisson distribution (Siméon-Denis Poisson, 1781-1840) { } e λ λ n n! As prevalent as Gaussian distribution
More informationLecture 7: Simulation of Markov Processes. Pasi Lassila Department of Communications and Networking
Lecture 7: Simulation of Markov Processes Pasi Lassila Department of Communications and Networking Contents Markov processes theory recap Elementary queuing models for data networks Simulation of Markov
More information1. Stochastic Process
HETERGENEITY IN QUANTITATIVE MACROECONOMICS @ TSE OCTOBER 17, 216 STOCHASTIC CALCULUS BASICS SANG YOON (TIM) LEE Very simple notes (need to add references). It is NOT meant to be a substitute for a real
More informationIEOR 6711, HMWK 5, Professor Sigman
IEOR 6711, HMWK 5, Professor Sigman 1. Semi-Markov processes: Consider an irreducible positive recurrent discrete-time Markov chain {X n } with transition matrix P (P i,j ), i, j S, and finite state space.
More informationProblem Points S C O R E Total: 120
PSTAT 160 A Final Exam December 10, 2015 Name Student ID # Problem Points S C O R E 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 10 10 10 11 10 12 10 Total: 120 1. (10 points) Take a Markov chain with the
More informationFigure 10.1: Recording when the event E occurs
10 Poisson Processes Let T R be an interval. A family of random variables {X(t) ; t T} is called a continuous time stochastic process. We often consider T = [0, 1] and T = [0, ). As X(t) is a random variable
More informationMarkov Processes Hamid R. Rabiee
Markov Processes Hamid R. Rabiee Overview Markov Property Markov Chains Definition Stationary Property Paths in Markov Chains Classification of States Steady States in MCs. 2 Markov Property A discrete
More informationIrreducibility. Irreducible. every state can be reached from every other state For any i,j, exist an m 0, such that. Absorbing state: p jj =1
Irreducibility Irreducible every state can be reached from every other state For any i,j, exist an m 0, such that i,j are communicate, if the above condition is valid Irreducible: all states are communicate
More informationContinuous-Time Markov Chain
Continuous-Time Markov Chain Consider the process {X(t),t 0} with state space {0, 1, 2,...}. The process {X(t),t 0} is a continuous-time Markov chain if for all s, t 0 and nonnegative integers i, j, x(u),
More informationDiscrete Markov Chain. Theory and use
Discrete Markov Chain. Theory and use Andres Vallone PhD Student andres.vallone@predoc.uam.es 2016 Contents 1 Introduction 2 Concept and definition Examples Transitions Matrix Chains Classification 3 Empirical
More informationThe Transition Probability Function P ij (t)
The Transition Probability Function P ij (t) Consider a continuous time Markov chain {X(t), t 0}. We are interested in the probability that in t time units the process will be in state j, given that it
More informationData analysis and stochastic modeling
Data analysis and stochastic modeling Lecture 7 An introduction to queueing theory Guillaume Gravier guillaume.gravier@irisa.fr with a lot of help from Paul Jensen s course http://www.me.utexas.edu/ jensen/ormm/instruction/powerpoint/or_models_09/14_queuing.ppt
More informationLecture 4a: Continuous-Time Markov Chain Models
Lecture 4a: Continuous-Time Markov Chain Models Continuous-time Markov chains are stochastic processes whose time is continuous, t [0, ), but the random variables are discrete. Prominent examples of continuous-time
More informationChapter 2 SOME ANALYTICAL TOOLS USED IN THE THESIS
Chapter 2 SOME ANALYTICAL TOOLS USED IN THE THESIS 63 2.1 Introduction In this chapter we describe the analytical tools used in this thesis. They are Markov Decision Processes(MDP), Markov Renewal process
More informationMATH 56A: STOCHASTIC PROCESSES CHAPTER 1
MATH 56A: STOCHASTIC PROCESSES CHAPTER. Finite Markov chains For the sake of completeness of these notes I decided to write a summary of the basic concepts of finite Markov chains. The topics in this chapter
More information8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains
8. Statistical Equilibrium and Classification of States: Discrete Time Markov Chains 8.1 Review 8.2 Statistical Equilibrium 8.3 Two-State Markov Chain 8.4 Existence of P ( ) 8.5 Classification of States
More informationOptional Stopping Theorem Let X be a martingale and T be a stopping time such
Plan Counting, Renewal, and Point Processes 0. Finish FDR Example 1. The Basic Renewal Process 2. The Poisson Process Revisited 3. Variants and Extensions 4. Point Processes Reading: G&S: 7.1 7.3, 7.10
More informationT. Liggett Mathematics 171 Final Exam June 8, 2011
T. Liggett Mathematics 171 Final Exam June 8, 2011 1. The continuous time renewal chain X t has state space S = {0, 1, 2,...} and transition rates (i.e., Q matrix) given by q(n, n 1) = δ n and q(0, n)
More informationPage 0 of 5 Final Examination Name. Closed book. 120 minutes. Cover page plus five pages of exam.
Final Examination Closed book. 120 minutes. Cover page plus five pages of exam. To receive full credit, show enough work to indicate your logic. Do not spend time calculating. You will receive full credit
More informationMarkov Chains CK eqns Classes Hitting times Rec./trans. Strong Markov Stat. distr. Reversibility * Markov Chains
Markov Chains A random process X is a family {X t : t T } of random variables indexed by some set T. When T = {0, 1, 2,... } one speaks about a discrete-time process, for T = R or T = [0, ) one has a continuous-time
More informationMATH HOMEWORK PROBLEMS D. MCCLENDON
MATH 46- HOMEWORK PROBLEMS D. MCCLENDON. Consider a Markov chain with state space S = {0, }, where p = P (0, ) and q = P (, 0); compute the following in terms of p and q: (a) P (X 2 = 0 X = ) (b) P (X
More informationMarkov Chains. Chapter 16. Markov Chains - 1
Markov Chains Chapter 16 Markov Chains - 1 Why Study Markov Chains? Decision Analysis focuses on decision making in the face of uncertainty about one future event. However, many decisions need to consider
More informationLecture 1: Brief Review on Stochastic Processes
Lecture 1: Brief Review on Stochastic Processes A stochastic process is a collection of random variables {X t (s) : t T, s S}, where T is some index set and S is the common sample space of the random variables.
More informationSTAT STOCHASTIC PROCESSES. Contents
STAT 3911 - STOCHASTIC PROCESSES ANDREW TULLOCH Contents 1. Stochastic Processes 2 2. Classification of states 2 3. Limit theorems for Markov chains 4 4. First step analysis 5 5. Branching processes 5
More informationContinuous-time Markov Chains
Continuous-time Markov Chains Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ October 23, 2017
More informationStochastic modelling of epidemic spread
Stochastic modelling of epidemic spread Julien Arino Centre for Research on Inner City Health St Michael s Hospital Toronto On leave from Department of Mathematics University of Manitoba Julien Arino@umanitoba.ca
More informationStatistics 150: Spring 2007
Statistics 150: Spring 2007 April 23, 2008 0-1 1 Limiting Probabilities If the discrete-time Markov chain with transition probabilities p ij is irreducible and positive recurrent; then the limiting probabilities
More informationStochastic processes. MAS275 Probability Modelling. Introduction and Markov chains. Continuous time. Markov property
Chapter 1: and Markov chains Stochastic processes We study stochastic processes, which are families of random variables describing the evolution of a quantity with time. In some situations, we can treat
More informationPoissonprocessandderivationofBellmanequations
APPENDIX B PoissonprocessandderivationofBellmanequations 1 Poisson process Let us first define the exponential distribution Definition B1 A continuous random variable X is said to have an exponential distribution
More informationIEOR 6711: Stochastic Models I, Fall 2003, Professor Whitt. Solutions to Final Exam: Thursday, December 18.
IEOR 6711: Stochastic Models I, Fall 23, Professor Whitt Solutions to Final Exam: Thursday, December 18. Below are six questions with several parts. Do as much as you can. Show your work. 1. Two-Pump Gas
More informationPROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS
PROBABILITY: LIMIT THEOREMS II, SPRING 15. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please
More informationSince D has an exponential distribution, E[D] = 0.09 years. Since {A(t) : t 0} is a Poisson process with rate λ = 10, 000, A(0.
IEOR 46: Introduction to Operations Research: Stochastic Models Chapters 5-6 in Ross, Thursday, April, 4:5-5:35pm SOLUTIONS to Second Midterm Exam, Spring 9, Open Book: but only the Ross textbook, the
More informationChapter 5. Continuous-Time Markov Chains. Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan
Chapter 5. Continuous-Time Markov Chains Prof. Shun-Ren Yang Department of Computer Science, National Tsing Hua University, Taiwan Continuous-Time Markov Chains Consider a continuous-time stochastic process
More informationStat 150 Practice Final Spring 2015
Stat 50 Practice Final Spring 205 Instructor: Allan Sly Name: SID: There are 8 questions. Attempt all questions and show your working - solutions without explanation will not receive full credit. Answer
More informationContinuous Time Processes
page 102 Chapter 7 Continuous Time Processes 7.1 Introduction In a continuous time stochastic process (with discrete state space), a change of state can occur at any time instant. The associated point
More informationMATH37012 Week 10. Dr Jonathan Bagley. Semester
MATH37012 Week 10 Dr Jonathan Bagley Semester 2-2018 2.18 a) Finding and µ j for a particular category of B.D. processes. Consider a process where the destination of the next transition is determined by
More informationModelling data networks stochastic processes and Markov chains
Modelling data networks stochastic processes and Markov chains a 1, 3 1, 2 2, 2 b 0, 3 2, 3 u 1, 3 α 1, 6 c 0, 3 v 2, 2 β 1, 1 Richard G. Clegg (richard@richardclegg.org) November 2016 Available online
More informationLTCC. Exercises. (1) Two possible weather conditions on any day: {rainy, sunny} (2) Tomorrow s weather depends only on today s weather
1. Markov chain LTCC. Exercises Let X 0, X 1, X 2,... be a Markov chain with state space {1, 2, 3, 4} and transition matrix 1/2 1/2 0 0 P = 0 1/2 1/3 1/6. 0 0 0 1 (a) What happens if the chain starts in
More informationMarkov Processes Cont d. Kolmogorov Differential Equations
Markov Processes Cont d Kolmogorov Differential Equations The Kolmogorov Differential Equations characterize the transition functions {P ij (t)} of a Markov process. The time-dependent behavior of the
More informationDynamic Risk Measures and Nonlinear Expectations with Markov Chain noise
Dynamic Risk Measures and Nonlinear Expectations with Markov Chain noise Robert J. Elliott 1 Samuel N. Cohen 2 1 Department of Commerce, University of South Australia 2 Mathematical Insitute, University
More informationExample: physical systems. If the state space. Example: speech recognition. Context can be. Example: epidemics. Suppose each infected
4. Markov Chains A discrete time process {X n,n = 0,1,2,...} with discrete state space X n {0,1,2,...} is a Markov chain if it has the Markov property: P[X n+1 =j X n =i,x n 1 =i n 1,...,X 0 =i 0 ] = P[X
More informationPROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS
PROBABILITY: LIMIT THEOREMS II, SPRING 218. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please
More informationINVERSE PROBLEMS FOR MARKOV MODELS IN HEALTH INSURANCE. Semyen I. Spivak. The Bashkir State University, Ufa, Russia
INVERSE PROBLEMS FOR MARKOV MODELS IN HEALTH INSURANCE Semyen I. Spivak The Bashkir State University, Ufa, Russia The paper deals with inverse problems for Markov models in health insurance. Two opposite
More informationThe exponential distribution and the Poisson process
The exponential distribution and the Poisson process 1-1 Exponential Distribution: Basic Facts PDF f(t) = { λe λt, t 0 0, t < 0 CDF Pr{T t) = 0 t λe λu du = 1 e λt (t 0) Mean E[T] = 1 λ Variance Var[T]
More informationCS145: Probability & Computing Lecture 18: Discrete Markov Chains, Equilibrium Distributions
CS145: Probability & Computing Lecture 18: Discrete Markov Chains, Equilibrium Distributions Instructor: Erik Sudderth Brown University Computer Science April 14, 215 Review: Discrete Markov Chains Some
More informationArrivals and waiting times
Chapter 20 Arrivals and waiting times The arrival times of events in a Poisson process will be continuous random variables. In particular, the time between two successive events, say event n 1 and event
More informationDiscrete random structures whose limits are described by a PDE: 3 open problems
Discrete random structures whose limits are described by a PDE: 3 open problems David Aldous April 3, 2014 Much of my research has involved study of n limits of size n random structures. There are many
More informationReadings: Finish Section 5.2
LECTURE 19 Readings: Finish Section 5.2 Lecture outline Markov Processes I Checkout counter example. Markov process: definition. -step transition probabilities. Classification of states. Example: Checkout
More informationMAS275 Probability Modelling Exercises
MAS75 Probability Modelling Exercises Note: these questions are intended to be of variable difficulty. In particular: Questions or part questions labelled (*) are intended to be a bit more challenging.
More informationLecture 4: Introduction to stochastic processes and stochastic calculus
Lecture 4: Introduction to stochastic processes and stochastic calculus Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London
More informationAPPM 4/5560 Markov Processes Fall 2019, Some Review Problems for Exam One
APPM /556 Markov Processes Fall 9, Some Review Problems for Exam One (Note: You will not have to invert any matrices on the exam and you will not be expected to solve systems of or more unknowns. There
More informationThe story of the film so far... Mathematics for Informatics 4a. Continuous-time Markov processes. Counting processes
The story of the film so far... Mathematics for Informatics 4a José Figueroa-O Farrill Lecture 19 28 March 2012 We have been studying stochastic processes; i.e., systems whose time evolution has an element
More informationSTAT 380 Continuous Time Markov Chains
STAT 380 Continuous Time Markov Chains Richard Lockhart Simon Fraser University Spring 2018 Richard Lockhart (Simon Fraser University)STAT 380 Continuous Time Markov Chains Spring 2018 1 / 35 Continuous
More informationExercises. (a) Prove that m(t) =
Exercises 1. Lack of memory. Verify that the exponential distribution has the lack of memory property, that is, if T is exponentially distributed with parameter λ > then so is T t given that T > t for
More informationhttp://www.math.uah.edu/stat/markov/.xhtml 1 of 9 7/16/2009 7:20 AM Virtual Laboratories > 16. Markov Chains > 1 2 3 4 5 6 7 8 9 10 11 12 1. A Markov process is a random process in which the future is
More informationLecture 11: Introduction to Markov Chains. Copyright G. Caire (Sample Lectures) 321
Lecture 11: Introduction to Markov Chains Copyright G. Caire (Sample Lectures) 321 Discrete-time random processes A sequence of RVs indexed by a variable n 2 {0, 1, 2,...} forms a discretetime random process
More informationContinuous time Markov chains
Continuous time Markov chains Alejandro Ribeiro Dept. of Electrical and Systems Engineering University of Pennsylvania aribeiro@seas.upenn.edu http://www.seas.upenn.edu/users/~aribeiro/ October 16, 2017
More informationModelling data networks stochastic processes and Markov chains
Modelling data networks stochastic processes and Markov chains a 1, 3 1, 2 2, 2 b 0, 3 2, 3 u 1, 3 α 1, 6 c 0, 3 v 2, 2 β 1, 1 Richard G. Clegg (richard@richardclegg.org) December 2011 Available online
More informationOutlines. Discrete Time Markov Chain (DTMC) Continuous Time Markov Chain (CTMC)
Markov Chains (2) Outlines Discrete Time Markov Chain (DTMC) Continuous Time Markov Chain (CTMC) 2 pj ( n) denotes the pmf of the random variable p ( n) P( X j) j We will only be concerned with homogenous
More informationContinuous Time Markov Chains
Continuous Time Markov Chains Stochastic Processes - Lecture Notes Fatih Cavdur to accompany Introduction to Probability Models by Sheldon M. Ross Fall 2015 Outline Introduction Continuous-Time Markov
More informationEXAM IN COURSE TMA4265 STOCHASTIC PROCESSES Wednesday 7. August, 2013 Time: 9:00 13:00
Norges teknisk naturvitenskapelige universitet Institutt for matematiske fag Page 1 of 7 English Contact: Håkon Tjelmeland 48 22 18 96 EXAM IN COURSE TMA4265 STOCHASTIC PROCESSES Wednesday 7. August, 2013
More informationSTA 624 Practice Exam 2 Applied Stochastic Processes Spring, 2008
Name STA 624 Practice Exam 2 Applied Stochastic Processes Spring, 2008 There are five questions on this test. DO use calculators if you need them. And then a miracle occurs is not a valid answer. There
More informationMARKOV PROCESSES. Valerio Di Valerio
MARKOV PROCESSES Valerio Di Valerio Stochastic Process Definition: a stochastic process is a collection of random variables {X(t)} indexed by time t T Each X(t) X is a random variable that satisfy some
More informationPerformance Modelling of Computer Systems
Performance Modelling of Computer Systems Mirco Tribastone Institut für Informatik Ludwig-Maximilians-Universität München Fundamentals of Queueing Theory Tribastone (IFI LMU) Performance Modelling of Computer
More informationLecture 21. David Aldous. 16 October David Aldous Lecture 21
Lecture 21 David Aldous 16 October 2015 In continuous time 0 t < we specify transition rates or informally P(X (t+δ)=j X (t)=i, past ) q ij = lim δ 0 δ P(X (t + dt) = j X (t) = i) = q ij dt but note these
More informationSubject CT4 Models. October 2015 Examination INDICATIVE SOLUTION
Institute of Actuaries of India Subject CT4 Models October 2015 Examination INDICATIVE SOLUTION Introduction The indicative solution has been written by the Examiners with the aim of helping candidates.
More informationBirth and Death Processes. Birth and Death Processes. Linear Growth with Immigration. Limiting Behaviour for Birth and Death Processes
DTU Informatics 247 Stochastic Processes 6, October 27 Today: Limiting behaviour of birth and death processes Birth and death processes with absorbing states Finite state continuous time Markov chains
More informationIntroduction to Queuing Networks Solutions to Problem Sheet 3
Introduction to Queuing Networks Solutions to Problem Sheet 3 1. (a) The state space is the whole numbers {, 1, 2,...}. The transition rates are q i,i+1 λ for all i and q i, for all i 1 since, when a bus
More informationReading: Karlin and Taylor Ch. 5 Resnick Ch. 3. A renewal process is a generalization of the Poisson point process.
Renewal Processes Wednesday, December 16, 2015 1:02 PM Reading: Karlin and Taylor Ch. 5 Resnick Ch. 3 A renewal process is a generalization of the Poisson point process. The Poisson point process is completely
More informationChapter 16 focused on decision making in the face of uncertainty about one future
9 C H A P T E R Markov Chains Chapter 6 focused on decision making in the face of uncertainty about one future event (learning the true state of nature). However, some decisions need to take into account
More informationStochastic Modelling
Stochastic Modelling Simulating Random Walks and Markov Chains This lab sheets is available for downloading from www.staff.city.ac.uk/r.j.gerrard/courses/dam/, as is the spreadsheet mentioned in section
More informationstochnotes Page 1
stochnotes110308 Page 1 Kolmogorov forward and backward equations and Poisson process Monday, November 03, 2008 11:58 AM How can we apply the Kolmogorov equations to calculate various statistics of interest?
More informationA review of Continuous Time MC STA 624, Spring 2015
A review of Continuous Time MC STA 624, Spring 2015 Ruriko Yoshida Dept. of Statistics University of Kentucky polytopes.net STA 624 1 Continuous Time Markov chains Definition A continuous time stochastic
More informationQ = (c) Assuming that Ricoh has been working continuously for 7 days, what is the probability that it will remain working at least 8 more days?
IEOR 4106: Introduction to Operations Research: Stochastic Models Spring 2005, Professor Whitt, Second Midterm Exam Chapters 5-6 in Ross, Thursday, March 31, 11:00am-1:00pm Open Book: but only the Ross
More informationLECTURE #6 BIRTH-DEATH PROCESS
LECTURE #6 BIRTH-DEATH PROCESS 204528 Queueing Theory and Applications in Networks Assoc. Prof., Ph.D. (รศ.ดร. อน นต ผลเพ ม) Computer Engineering Department, Kasetsart University Outline 2 Birth-Death
More informationMath 416 Lecture 11. Math 416 Lecture 16 Exam 2 next time
Math 416 Lecture 11 Math 416 Lecture 16 Exam 2 next time Birth and death processes, queueing theory In arrival processes, the state only jumps up. In a birth-death process, it can either jump up or down
More informationCOPYRIGHTED MATERIAL CONTENTS. Preface Preface to the First Edition
Preface Preface to the First Edition xi xiii 1 Basic Probability Theory 1 1.1 Introduction 1 1.2 Sample Spaces and Events 3 1.3 The Axioms of Probability 7 1.4 Finite Sample Spaces and Combinatorics 15
More informationStochastic Processes. Theory for Applications. Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS
Stochastic Processes Theory for Applications Robert G. Gallager CAMBRIDGE UNIVERSITY PRESS Contents Preface page xv Swgg&sfzoMj ybr zmjfr%cforj owf fmdy xix Acknowledgements xxi 1 Introduction and review
More informationAny live cell with less than 2 live neighbours dies. Any live cell with 2 or 3 live neighbours lives on to the next step.
2. Cellular automata, and the SIRS model In this Section we consider an important set of models used in computer simulations, which are called cellular automata (these are very similar to the so-called
More informationLatent voter model on random regular graphs
Latent voter model on random regular graphs Shirshendu Chatterjee Cornell University (visiting Duke U.) Work in progress with Rick Durrett April 25, 2011 Outline Definition of voter model and duality with
More informationLectures on Markov Chains
Lectures on Markov Chains David M. McClendon Department of Mathematics Ferris State University 2016 edition 1 Contents Contents 2 1 Markov chains 4 1.1 The definition of a Markov chain.....................
More informationLet (Ω, F) be a measureable space. A filtration in discrete time is a sequence of. F s F t
2.2 Filtrations Let (Ω, F) be a measureable space. A filtration in discrete time is a sequence of σ algebras {F t } such that F t F and F t F t+1 for all t = 0, 1,.... In continuous time, the second condition
More informationStochastic Processes
Stochastic Processes Stochastic Process Non Formal Definition: Non formal: A stochastic process (random process) is the opposite of a deterministic process such as one defined by a differential equation.
More information1 Delayed Renewal Processes: Exploiting Laplace Transforms
IEOR 6711: Stochastic Models I Professor Whitt, Tuesday, October 22, 213 Renewal Theory: Proof of Blackwell s theorem 1 Delayed Renewal Processes: Exploiting Laplace Transforms The proof of Blackwell s
More informationCAS Exam MAS-1. Howard Mahler. Stochastic Models
CAS Exam MAS-1 Howard Mahler Stochastic Models 2019 Stochastic Models, HCM 12/31/18, Page 1 The slides are in the same order as the sections of my study guide. Section # Section Name 1 Introduction 2 Exponential
More information