Mathematical Foundations of Finite State Discrete Time Markov Chains

Size: px

Start display at page:

Download "Mathematical Foundations of Finite State Discrete Time Markov Chains"

Doris Stevens
6 years ago
Views:

1 Mathematical Foundations of Finite State Discrete Time Markov Chains Friday, February 07, :04 PM Stochastic update rule for FSDT Markov Chain requires an initial condition. Most generally, this can be expressed through an initial probability distribution for the state of the system, expressed via: for This can be prescribed arbitrarily subject to the usual constraints on a probability distribution: An important special case of an initial probability distribution is when the initial condition is known precisely, i.e., with probability 1. Then the initial probability distribution is a discrete delta distribution: We always demand that is independent of the noise stream. An alternative (and in fact the most common) way to describe FSDT Markov chains is based on the Markov property, which we state, for now, for discrete-time stochastic processes. A discrete-time stochastic process property provided that: is said to have the Markov Stoch14 Page 1

2 for all A Markov process is a stochastic process with the Markov property, and intuitively, it means that the future evolution of the Markov process given the current state is independent of further information about the past. In other words, for computing statistics of the future, only the present state matters, not past information. A discrete-time, finite-state Markov process is called an FSDT Markov chain. In the homework you will see that there are two other equivalent ways of describing the Markov property, and you'll be asked to show it. Intuitively, it says the following: Stoch14 Page 2

3 In words, a Markov process is one for which the past is conditionally independent of the future, given the present. An important observation about this version of the Markov property is it shows that the Markov property is not directional in time. The Markov property tells us that to determine the evolution of a FSDT Markov chain, we only need to prescribe the following conditional probabilities: Therefore, to define a FSDT Markov chain model, we can prescribe: 1. Probability Transition Matrix (Matrices) for and A widely used simplification is a time-homogenous FSDT Markov chain, meaning that the statistics regarding the evolution of the FSDT Markov chain are invariant under time translation. Under this condition, we can simply write a single probability transition matrix: for all (analogous to concept of autonomous dynamical system) 2. Initial Probability Distribution But in this matrix formulation, it's convenient to encode the initial probability distribution as a vector: (not too fussy about row vs column vector). To specify a FSDT Markov chain model using this probability transition matrix framework, one needs to prescribe the probability transition matrices which can be arbitrary up to the conditions: Stoch14 Page 3

4 together with an initial probability distribution, encoded via the vector subject to the conditions: The rows of the probability transition matrix have important meaning: The ith row of is the probability distribution for the state of the system at the next epoch, given that Equivalence of Stochastic Update Rule and Probability Transition Matrix Representations First, we show that starting with a stochastic update rule description, the resulting stochastic process must satisfy the Markov property and show that the probability transition matrix can be described in terms of the stochastic update rule. Suppose we are given a stochastic update rule where the is a stream of iid random variables. And we're given some initial probability distribution for but this looks the same in both descriptions. Let's show that the Markov property holds. Stoch14 Page 4

5 Intuitively, using known information to simplify the unknown quantity, but you have to be careful in how you do this. Here is a valid rule for doing this: P(g(X,Y) = c X=x) = P(g(x,Y)=c X=x) with the caveat that if X is a continuous random variable, then g better be continuous as well, otherwise this might not work (Borel paradox). Note carefully that in making the substitution of known information, one should not erase the condition. This is one rule that's used to simplify complicated conditional probability expressions. Another key rule that's used to simplify such expressions is: P(g(Y)=c X=x) = P(g(Y)=c) if Y is independent of X. Both of these rules also work if X, Y are replaced by collections of random variables, i.e., random vectors. To use this rule that independence allows the forgetting of conditions, we'd want to show that is independent of The intuitive reason this is true is that is the new noise that enters at epoch n, and the past and current values have been determined before ever needed to be generated. To show this mathematically, just iterate the following argument: is independent of all so in particular with deterministic and since are independent of so is n>1), so is with deterministic and since are independent of (assuming Continue inductively up to so is with deterministic and since are independent of Stoch14 Page 5

6 We can therefore use this independence of the unknown variables to write: from the conditioned known This establishes the Markov property. So what's the probability transition matrix? By knowing the probability distribution for probability transition matrix. we can compute this Now we'll show that if we're given a FSDT Markov chain defined in terms of a probability transition matrix, then we can also derive a stochastic update rule description. The derivation of this property is the same idea as how one can simulate finite state random variables. Let's consider the "classical" situation where the given pseudorandom number generator generates uniformly distributed random variables on the unit interval. Stoch14 Page 6

7 How would one be able to use this "built-in" random number generator to simulate random variables on a finite state space {1,..,M} with probabilities. Mathematically, one write this as: This is a special case of what's known as the Inverse Transform Method which says that you can use the CDF to define a function g that will map a uniform random variable U to the random variable with that CDF. Another ingredient we will need for our derivation is the concept of an Indicator Function This is a binary random variable defined as follows: So now we're ready for the derivation. We are given the probability transition matrices for and the initial probability distribution, but again the initial probability distribution looks the same in both formulations. We just need to show that we can define a stochastic update rule, and how it's derived from these probability transition matrices. Stoch14 Page 7

8 We'll construct the stochastic update rule that gets the job done. Declare a sequence of iid random variables to all be. Next define the function: Then the stochastic update rule generates a stochastic process with probability transition matrix given. Stoch14 Page 8

Note special lecture series by Emmanuel Candes on compressed sensing Monday and Tuesday 4-5 PM (room information on rpinfo)

Formulation of Finite State Markov Chains Friday, September 23, 2011 2:04 PM Note special lecture series by Emmanuel Candes on compressed sensing Monday and Tuesday 4-5 PM (room information on rpinfo)