Finite-Horizon Statistics for Markov chains

Size: px

Start display at page:

Download "Finite-Horizon Statistics for Markov chains"

Gwen Lawrence
5 years ago
Views:

1 Analyzing FSDT Markov chains Friday, September 30, :03 PM Simulating FSDT Markov chains, as we have said is very straightforward, either by using probability transition matrix or stochastic update rule. So it may seem that to answer questions about Markov chain models, one can just run the easy simulations and just take statistics of the results and get the answers one wants. Indeed in many situations, one does this. But we will discuss some analytical tools that complement direct numerical simulations, and the virtue of these analytical tools is the following: When you do direct (Monte Carlo) simulations of a Markov chain, then the quality of the answers that you get from taking statistical averages only increases slowly, namely like the square root of the computational effort. (half order accuracy). (Inescapable consequence of probability laws for the variance of sums of independent random variables.) This issue is particularly relevant for systems where the number of states in state space is large. The analytical formulas we'll develop are deterministic, which means that they need only be solved once (rather than averaging over many realizations), and one can use efficient computational linear algebra techniques to solve them. And the answers will be exact (up to roundoff error). Some of the questions we can address with analytical techniques refer to long-time properties of Markov chains and these can be especially expensive to simulate numerically. Finite-Horizon Statistics for Markov chains How can we describe what the statistics of a Markov chain is over some fixed finite time horizon? These questions will abstractly reduce to questions of the following form: Stochastic11 Page 1

2 That is, the rule that the probability of the future state given a past history is the same as the probability of the future state given the information about the last epoch of the past history still holds true when the epochs involved have gaps. By repeating this process inductively for a finite number of steps, we obtain: Stochastic11 Page 2

3 From this inductive argument, we see that any question involving the statistics of the Markov chain over a specified finite set of epochs (finite-dimensional distribution of the Markov chain) can be reduced to computing the following two quantities: To compute these quantities, we will employ the fundamental Chapman-Kolmogorov equation For any n < n' < m Stochastic11 Page 3

4 The proof of the Chapman-Kolmogorov equation uses another important tool: Law of Total Probability Given a partition Of sample space, we have: The art of using the law of total probability in stochastic processes is typically in choosing partitions that simplify the calculation for P(A). The way in which we use the law of total probability in proving the Chapman-Kolmogorov equation is to use the state of the Markov chain at the intermediate time n' to define the partition. We will actually use a conditional extension of the law of total probability using the common principle that you can take a probability theorem and it remains true if you condition every probability on the same event: Stochastic11 Page 4

5 To compute the key quantities of interest for finite-time statistics of Markov chains, we will employ the Chapman- Kolmogorov equation. Continuing by induction Stochastic11 Page 5

6 For the case of time-homogenous Markov chains: The other key quantity of interest is: Apply the law of total probability, partitioning on the value of X 0. Stochastic11 Page 6

7 So the above calculations show that computing any finite-time statistic of a Markov chain can be done deterministically by simply multiplying together the probability transition matrix(ces) and initial probability distribution vector in a suitable way. Now we will begin discussing tools to analyze how Markov chains behave over indefinite time horizons. Stationary Distribution This quantity only makes sense when we talk about time-homogenous Markov chains so we will implicitly restrict our discussion to this case. A stationary distribution of a Markov chain is a probability distribution typically encoded as a row vector: Which satisfies the following conditions: The last two conditions are just conditions to be a probability distribution. The first condition is the most meaningful one, and it implies the following: If one takes the stationary distribution as the initial probability distribution for the Markov chain, then the Markov chain Stochastic11 Page 7

8 will always be in that stationary distribution. Now suppose we take as initial probability distribution: This doesn't mean that the Markov chain doesn't change state. It means that the probability to observe the Markov chain in a given state doesn't change with time. Markov chains aren't typically initialized with a stationary distribution, but the stationary distribution plays a very important role because in some sense it is a natural distribution associated to the Markov chain. More precisely, as we will discuss, the stationary distribution is often a limit distribution, meaning that at long times, the probability distribution for the state of the Markov chain will approach the stationary distribution even if the initial distribution was arbitrary. First, let us address some technical mathematical issues regarding stationary distributions: When are we guaranteed that a stationary Stochastic11 Page 8

9 distribution exists for a Markov chain? When are we guaranteed that the stationary distribution is unique? When are we guaranteed that the stationary distribution is in fact a limit distribution, meaning that all initial probability distributions will give rise to Markov chains that eventually the stationary distribution? Existence: Any finite-state Markov chain has a stationary distribution. Reason: Since the sums of rows in the probability transition matrix are all 1, it follows that: So since P has a right eigenvector with eigenvalue 1, it must have a left eigenvector with eigenvalue 1. How are we guaranteed that this left eigenvector can be chosen to have all nonnegative entries? This follows from an advanced theorem from linear algebra called the Perron-Frobenius Theorem (Appendix B of Karlin and Taylor) which has to do with matrices with all nonnegative entries. Stochastic11 Page 9

No class on Thursday, October 1. No office hours on Tuesday, September 29 and Thursday, October 1.

Stationary Distributions Monday, September 28, 2015 2:02 PM No class on Thursday, October 1. No office hours on Tuesday, September 29 and Thursday, October 1. Homework 1 due Friday, October 2 at 5 PM strongly