Countable state discrete time Markov Chains

Size: px

Start display at page:

Download "Countable state discrete time Markov Chains"

Brooke Johnston
5 years ago
Views:

1 Countable state discrete time Markov Chains Tuesday, March 18, :12 PM Readings: Lawler Ch. 2 Karlin & Taylor Chs. 2 & 3 Resnick Ch. 1 Countably infinite state spaces are of practical utility in situations where the state of the system does not have a natural known upper limit. In particular useful in situations where there may be a practical upper limit, but it's fuzzy and/or not really known a priori Queueing theory without any maximum size limit Population dynamics and epidemic modeling Financial modeling Spatial processes away from boundaries random walks on lattices (atomic physics) We will revisit the theoretical development for finite state discrete time Markov chains and comment on what formulas need generalization, and which just carry over and which require modification. Model formulation The setup is almost exactly the same. Stochastic update rule works the same way, except the iid noise stream might have countable state space. One can still define a formal probability transition matrix via: along with a initial probability distribution vector where And these suffice to fully characterize the Markov chain. The only difference is that these vectors and matrices have "infinite dimension." So might be awkward to exhibit it as an actual matrix, but one can certainly specify it by simply giving a formula for all and. Simulation of countable state Markov chain The basic idea is still the same; given the current state i, look at the ith row of the probability transition matrix P and take that as the probability distribution for the state of the system at the next epoch. Key difference is that sampling from this probability distribution may involve sampling from a countable state random variable. The finite state random variable simulation procedure generalizes in principle to Stoch14 Page 1

2 countable state case, but somewhat of a pain. There are however, two mitigating factors regarding this hassle. In practice, most countable state Markov chains typically have rather simple rules for how they change state, typically: in one epoch, there are only a finite number of possible new states to reach (in which case a finite state space simulation works just fine) or, when the state change is not restricted to a finite set (for a given starting state), then often the update to the state involves the Poisson distribution Here is an alternative method for simulating Poisson random variables, based on an observation we will discuss a bit later. How many points generated in this way fall in the interval [0,a]? Answer: It is a random variable which has Poisson distribution with mean. So one can use this idea to cheaply code and simulate Poisson random variables because exponentially distributed random variables are very easy to simulate. from the inverse transform method. Finite Horizon Statistics Stoch14 Page 2

3 All the previous formulas carry over, except the matrix-vector multiplications are now involving infinite sums, so maybe think about the matrices as operators. From a mathematical standpoint, one may worry about whether these infinite sums converge. One can in fact guarantee convergence using the following facts: These intrinsic bounds on the probability transition matrix and initial probability distribution will guarantee that the operations will converge, using simple versions of Hölder inequalities. Stoch14 Page 3

4 Long-Time Properties of Countable State Markov Chains Recall that one of the key steps in describing long time properties of FSDT Markov chains was identifying which communication classes are transient and recurrent. For FSDT MC this classification was entirely topological; all closed communication classes are recurrent. But this is no longer true for infinite state Markov chains; it's possible in particular to have transient irreducible Markov chains, for example. Therefore, we will need to add an analytical component to classifying infinite state Markov chains. This classification procedure will be discussed in the next lecture; for now we will simply discuss the consequences of what it means for a class of a countable state Markov chain to belong to the various classes. As before, we say that a state j is recurrent provided that: where State j is transient otherwise. For countable state MCs, one needs to subdivide recurrent states into two subcategories: A state is said to be positive recurrent provided that it is recurrent and. A state is said to be null recurrent provided that it is recurrent and not positive recurrent. How can null recurrence even be possible? Look at the definition of expectation: Stoch14 Page 4

5 So a null recurrent situation might be one in which If and the state is recurrent, then it will be positive recurrent. If then the state is null recurrent (if it's also recurrent) It turns out, for example, that the standard random walk on is null recurrent in 1 and 2 dimensions and transient in 3 or more dimensions. This calculation can be found in the textbooks (Karlin and Taylor Section 2.6); it's a technical special-purpose calculation. Deferring the more general process of classification to the next lecture, we'll focus on describing the long-time properties of the countable state MC once it's classified. Positive and null recurrence are also class properties. First of all, if the MC is reducible, then as for the finite state case, one can break the calculation up into the various classes. Recurrent classes are necessarily closed, so once a recurrent class is entered it stays there, and we can compute what happens in the long run as if that class were a irreducible MC in its own right. For a transient class which is not closed, then one can treat it with the exact same absorption probability and accumulated cost/reward formulas as before, but now the matrices might be infinite dimensional so inverting them might be awkward. Typically the recurrence relation form of the formulas is more useful for infinite state MCs; all infinite sums converge by similar Stoch14 Page 5

6 arguments as above. For a transient which is closed, I can treat it as its own irreducible MC. Now we'll use this reasoning to focus on irreducible MCs (i.e., a particular closed communication class). Positive recurrent Markov chain: The stationary distribution exists and is unique by the proof we gave/sketched for FSDT MCs; note how Assumption A in that proof is the definition of positive recurrence. Stationary distribution is also a limit distribution when the MC is aperiodic: Even with the aperiodicity assumption, the LLN for MCs holds for positive recurrent irreducible MCs. Proofs of these statements can be found in Resnick Secs. 2.12, Null recurrent Markov chains: These do not possess a stationary distribution (ever). But they do have an invariant measure which satisfies: (unnormalized stationary distribution). So the previous two sentences imply that the invariant measure must have an infinite sum for a null recurrent MC. And the invariant measure turns out to be unique (under irreducibilty) up to constant multiple. (Resnick Sec. 2.12) What are the long time properties? Invariant measure doesn't necessarily tell you anything useful Stoch14 Page 6

7 Invariant measure doesn't necessarily tell you anything useful For any In Feller's An Introduction to Probability Theory and its Applications Ch. 3, he discusses the null recurrence properties of random walks in terms of their counterintuitive notion of "fairness." Transient MCs and moreover by the definition of transience, not guaranteed to come back. Stoch14 Page 7

Transience: Whereas a finite closed communication class must be recurrent, an infinite closed communication class can be transient:

Transience: Whereas a finite closed communication class must be recurrent, an infinite closed communication class can be transient: Stochastic2010 Page 1 Long-Time Properties of Countable-State Markov Chains Tuesday, March 23, 2010 2:14 PM Homework 2: if you turn it in by 5 PM on 03/25, I'll grade it by 03/26, but you can turn it in