A Brief Introduction to Markov Chains and Hidden Markov Models

Transcription

1 A Brief Introduction to Markov Chains and Hidden Markov Modes Aen B MacKenzie Notes for December 1, 3, &8, 2015 Discrete-Time Markov Chains You may reca that when we first introduced random processes, we defined a Markov process to be a process with the foowing property: For arbitrary times t 1 < t 2 < < t k < t k+1, P[X(t k+1 )=x k+1 X(t 1 )=x 1, X(t 2 )=x 2,,X(t k )=x k ]=P[X(t k+1 )=x k+1 X(t k )=x k ] In other words, given the present, the future of the process is independent of the past 1 The consequence of this, the Markov property, is that given any conditiona probabiity mass function or probabiity density function that is conditioned on severa time instants, we can aways reduce to a conditiona pmf/pdf that is ony conditioned on the most recent time instant Because a information about the future evoution of the process is summarized by the current vaue, X(t), we refer to X(t) as the state of the process We have seen many Markov processes aready: the sum process, the Poisson process, the Random teegraph process, and the Wiener process are a exampes of Markov processes 2 In this set of notes, we wi focus excusivey on Markov processes in which the vaues of X(t) come from a discrete set, which wi usuay be mapped to the integers Such a Markov process is commony caed a Markov chain In some cases, X(t) may take on vaues from a finite set, in which case we say that it is a finite-state Markov chain Furthermore, our interest wi be on discrete-time processes, hence we wi adopt the notation {X n } to refer to the process from this point forward For Markov chains, we typicay assume that time starts at n = 0 In genera, a Markov chain can be characterized by giving the initia pmf, p j (0) =P[X 0 = j], 1 When we presented this property before, we were focused primariy on continuous-vaued processes, hence we used pdfs In this deveopment, we wi mosty focus on discrete-vaued processes, hence we wi use pmfs 2 The iid process is a trivia exampe of a Markov process, too and the one-step state-transition probabiities, p (n) ij = P[X n+1 = j X n = i] However, we wi assume that p ij (n) =p ij is constant for a time n A Markov chain with this property is said to be time homogenous These one-step transition probabiities are often represented as a

2 abriefintroductiontomarkovchainsandhiddenmarkovmodes 2 matrix, known as the transition probabiity matrix 2 P = 6 4 p 00 p 0 02 p 10 p 1 12 p i0 p i1 Note that each row of this matrix must add up to 1, as the ith row represents the conditiona pmf over the next state, given that the current state is i If the Markov chain is finite with n states, then P wi be an n n matrix Some Probabiity Computations From this point, it is easy to evauate the joint pmf for the first n time instants of the process: P[X 0 = i 0, X 1 = i 1,,X n 1 = i n 1 ]=P[X 0 = i 0 ]P[X 1 = i 1 X 0 = i 0 ] P[X n 1 = i n 1 X n 2 = i n 2 ] = p i0 (0)p i0 i 1 p i1 i 2 p in 2 i n 1 However, to find the joint pmf for arbitrary time instants, we need to find the transition probabiity for an arbitrary number of steps Let p ij (n) be the probabiity that the chain moves from state i to state j in n steps, that is P[X k+n = j X k = i] First, consider the two-step transition probabiity, p ij (2) By the theorem on tota probabiity, we have p ij (2) =P[X k+2 = j X k = i] = Â P[X k+2 = j, X k+1 = X k = i] = Â P[X k+2 = j X k+1 =, X k = i]p[x k+1 = X k = i] = Â P[X k+2 = j X k+1 = ]P[X k+1 = X k = i] = Â p j p i This is a we and good, but it is much more convient to write this in matrix form Let P(2) be the matrix of two-step transition probabiities Then the equation above impies that P(2) =P P = P 2 A simiar argument can be constructed to find P(n), the n-step transition probabiity Foowing the same argument as above, we

3 abriefintroductiontomarkovchainsandhiddenmarkovmodes 3 have that p ij (m + n)) = P[X k+m+n = j X k = i] = Â P[X k+m+n = j, X k+m = X k = i] = Â p i (m)p j (n) This equation, which is extremey important in the anaysis of Markov chains, is caed the Chapman-Komogorov equation Note that in matrix form, it tes us that P(m + n) =P(m) P(n) We can combine this, using induction, with the fact that P(1) =P, by definition, to obtain that P(n) =P n For finite state Markov chains, P n can be computed numericay to determine n-state transition behavior We can now find the pmf for arbitrary time instants If n 1 < n 2 < < n k, then P[X n1 = i 1, X n2 = i 2,,X nk = i k ]=p i1 (n 1 ) p i1 i 2 (n 2 n 1 ) p i2 i 3 (n 3 n 2 ) p ik 1 i k (n k n k 1 ) where p i1 (n 1 ) is the state probabiity at time n 1 Often, it is these state probabiities at time n, p i (n), in which we are most interested Given the initia state distribution, ~p(0), these are easy to compute as p i (n) =Â p k (0)p ki (n) k Moreover, in matrix notation, we have that the probabiity distribution over states at time n is given by ~p(n) =~p(0) P n For exampe, Figure 1 shows a two-state Markov chain This simpe chain has a surprisingy arge number of appications, particuary in communications It is commony used to represent channe conditions (which are modeed as switching between a "good" state and a "bad" state) and primary user activity (in which a primary user switches between "active" and "ide" states) It has aso been used to mode the activity of human speaker on a teephone ca, who moves between periods of speaking and periods of sience a 1 a b Figure 1: A simpe two-state Markov chain b

4 abriefintroductiontomarkovchainsandhiddenmarkovmodes 4 The transition probabiity matrix of this Markov chain is given by: " # 1 a a P = b 1 b As discussed above, we can find the n-step probabiity transition matrix by finding P n And, moreover, if we know the initia state distribution, ~p(0), then we can find the state probabiity distribution at time n given by ~p(n) =~p(0)p n For exampe, if we take a = 01 and b = 02, then we can use Matab to compute the reevant probabiities above In particuar, in this case the one-step transition probabiity matrix is P = " Using Matab (or equivaent), we can find arbitrary powers of P But, in particuar, we might see that " # P = By this point, it appears that P(n) =P n is converging to a matrix in which each row is (2/3, 1/3) More on this shorty # Stationary and Limiting Probabiity Distributions Suppose that we are given a Markov chain with transition probabiity matrix P A stationary distribution is a distribution ~p over the state space with the property that ~p = ~pp Such a distribution is very specia because if the initia distribution is ~p, that is, if ~p(0) =p, then we wi have that the distribution is ~p at every time instant n, ~p(n)p In other words, if the state distribution of the Markov chain is a stationary distribution, then the state distribution wi remain the same forevermore Indeed, a Markov chain whose initia state distribution is a stationary distribution of the chain wi be a stationary random process For a finite-state Markov chain with n states, we can find the stationary distribution(s) by soving the system of n inear equations given by ~p = ~pp However, this system of equations wi be underconstrained 3 We need one additiona equation, Â k p k = 1, in order to find a soution of ~p = ~pp that is a probabiity distribution A homogeneous inear system of equations, such as the one given by ~p = ~pp aways has a trivia soution, which is the a-zero soution, 3 Specificay, it is a system of homogeneous inear equations

5 abriefintroductiontomarkovchainsandhiddenmarkovmodes 5 but this obviousy cannot be scaed to be probabiity distribution If we can find a nonzero soution ~p to the system of homogeneous equations ~p = ~pp with  k p k <, then q k = p k /  j p j wi be a stationary distribution of the Markov chain Returning to the exampe two-state chain above, the homogeneous system of equations for a stationary distribution is given in matrix form as h i h p 0 p 1 = p 0 p 1 i " 1 a a b 1 b Or, if we write out the system of homogeneous equations, then we have # p 0 =(1 a)p 0 + bp 1 p 1 = ap 0 +(1 b)p 1 These two equations are redundant (both can be rewritten as ap 0 = bp 1 ) To sove the system, we need the additiona equation p 0 + p 1 = 1 Soving the system with this additiona equation produces: p 0 = b a + b p 1 = a a + b If we substitute in the vaues a = 01 and b = 02, then we find the soution is p 0 = 2/3 and p 1 = 1/3 Reca that these appeared to be the vaues to which the rows of P(n) =P n were converging A reasonabe question to ask at this point is: When wi a Markov chain with an arbitrary initia distribution converge to a stationary state distribution? That is, when can we say that im n! ~p(n) = p for arbitrary initia distributions ~p(0)? This turns out to be a somewhat compicated question to answer in genera So, in the interest of time, we wi answer it for a specia case 4 To answer this question, we first need to define some additiona properties of Markov chains We say that state j is accessibe from state i, written i! j, if for some n 0, p ij (n) > 0 That is, if there is some sequence of transitions from i to j that has nonzero probabiity We say that states i and j communicate, written i $ j, if they are accessibe to each other, that is if i! j and j! i A Markov chain in which every state communicates with every other state is said to be irreducibe 5 4 This specia case turns out to be the core of the genera soution, too 5 In the event that a Markov chain is not irreducibe, it can be decomposed into communicating casses The anaysis of genera Markov chains begins with such a decomposition

6 abriefintroductiontomarkovchainsandhiddenmarkovmodes 6 Suppose that we start a Markov chain in state i State i is said to be recurrent if the probabiity that the process returns to state i is 1 That is, state i is recurrent if f i = P[returning to state i X 0 = i] =1 It is easy to show that a state is recurrent if and ony if  p ii (n) = n=1 So, of course, a state is transient if and ony if  n= ii(n) < Furthermore, we can show that if i $ j, then i is recurrent if and ony if j is recurrent Thus, we can concude that if a Markov chain is irreducibe then either a of its states are recurrent or a of its states are transient Moreover, if an irreducibe Markov chain is finite then a states must be recurrent Suppose that we start a Markov chain in a recurrent state, i, at time 0, X 0 = i Let T i (1) be the first time n 1 > 0 that the chain returns to state i That is, T i (1) =n 1 is the smaest number such that X n1 = i Subsequenty, et T i (k) be the time between the k 1th return to state i and the kth return to state i In other words, the chain returns to state i at times T i (1), T i (1) +T i (2), T i (1) +T i (2) +T i (3), The T i form an iid sequence, since each return time is independent of previous return times The proportion of time spent in state i after k returns to the state is given by k T i (1)+T i (2)+T i (3)+ + T i (k) By the strong aw of arge numbers, though, we know that T i (1)+T i (2)+T i (3)+ + T i (k) k! E[T i ] Hence, the ong term proportion of time spent in state i is given by q i = 1/E[T i ] If q i > 0, then we say that the state i is positive recurrent If q i = 0, then we say that the state is nu recurrent Note that nu recurrent states are a bit odd: The probabiity of returning to them is 1, but the expected time between visits is infinity! If i $ j, then i is positive recurrent if and ony if j is positive recurrent Thus, if {X n } is an irreducibe Markov chain, we can say that the ong term proportion of time spent in state i is q i Further, for any irreducibe Markov chain, ~q wi satisfy ~q = ~qp And, if the Markov chain is positive recurrent, then we wi have  i q i = 1, and so ~q wi be a stationary distribution of the chain The structure of the state transition probabiities can impose periodicity in the Markov chain We say that a state i has period d if it can

7 abriefintroductiontomarkovchainsandhiddenmarkovmodes 7 ony reoccur at times that are mutipes of d and d is the argest integer with this property That is p ii (n) =0 whenever n is not a mutipe of d We say that state i is aperiodic if it has period 1 (and periodic if it has period d > 1) Further, we can show that if i $ j, then i and j wi have the same period Thus, we say that an irreducibe Markov chain is aperiodic if any of its states are aperiodic (because this wi impy that a of its states are aperiodic) This brings us to the key resut of this section, and arguaby the key resut of Markov chain theory: If {X n } is an irreducibe, aperiodic Markov chain, then exacty one of the foowing assertions hods: A states are transient or a states are nu recurrent; p ij (n)! 0 as n! for a i and j; and no stationary distribution over the states exists A states are positive recurrent; there exists a unique stationary distribution ~p; and p ij (n)! p j as n! for a i and j Returning to our prior exampe, the two-state exampe is an irreducibe, aperiodic Markov chain Thus, we can immediatey concude that, because it is irreducibe and finite, it is positive recurrent In this case, the theorem tes us that there exists a unique stationary distribution, ~p, which we aready found to be ~p =(2/3, 1/3), and, moreover, that P(n) converges to the matrix in which each row is ~p This is what our numerica exampe (with a = 01 and b = 02) suggested earier For a more compex exampe, consider the one-sided random wak exampe shown in figure 2 This chain is ceary irreducibe and aperiodic, thus it is appropriate to appy the theorem above In this case, we proceed to appy the theorem by first attempting to find a stationary distribution In particuar, we can show that a soution to the system of homogeneous equations ~p = ~pp must satisfy p k p k = p 0 for k = 1, 2, 3, For ~p to be a stationary distribution, though, it must aso be the case that  k=0 p k = 1 But we have aready seen that {p k } forms a geometric sequence We can concude that it is possibe to find a stationary distribution ony if p/() < 1; that is, ony if p < 1/2 Thus, by appying the theorem, we have that when p < 1/2, the Markov chain is positive recurrent and the imiting distribution is the unique stationary distribution When p 1/2, according to the theorem the the Markov chain is either transient or nu recurrent 6 6 It is possibe to show that the chain is nu recurrent if and ony if p = 1/2 This represents the somewhat genera concusion that nu recurrence occurs on the knife edge between positive recurrence and transience

8 abriefintroductiontomarkovchainsandhiddenmarkovmodes 8 p p p p Figure 2: A one-sided random wak Hidden Markov Modes (HMMs) Suppose that {X n } is a time-homogenous, discrete-time Markov chain, ike those discussed in the previous two sections, with transition probabiity matrix P and initia probabiity distribution ~p(0) However, suppose that instead of observing the Markov chain itsef, we observe a sequence of signas {Y n }, which are correated with the underying chain, but that we are not abe to observe the chain directy In particuar, suppose that the Markov chain has N s states and that there are N o different possibe signas (observations) that can be observed Furthermore, suppose that the conditiona probabiity of observing a particuar signa is P(Y n = X n = x) =b x,, where B is an N s N o observation matrix and where Y n is conditionay independent of X k and Y k for a k 6= n The parameters of this mode are ~p(0), P, and B If these parameters are known, then it is possibe to write down a compete pmf for the mode In particuar, T 1 T p(~x,~y ~p(0), P, B) =p x0 p xt,x t+1 b xt,y t n=0 n=0 The ony thing about this pmf that is even moderatey interesting is the fact that we have expicity written the dependence on the parameters; otherwise, it is exacty what you woud expect given the description of the mode above There are three primary probems that are of interest with regards to HMMs: Given the observed data and the parameters ~p(0), P, and B, compute the conditiona distribution of the state Given the observed data and the parameters ~p(0), P, and B, compute the most ikey sequence for hidden states Given the observed data, compute the maximum ikeihood (ML) estimate of ~p(0), P, and B 7 The first and second probems are basic probabiity computations They coud be soved using conventiona methods for computing 7 In this probem, it is assumed that N s and N o are known

9 abriefintroductiontomarkovchainsandhiddenmarkovmodes 9 probabiities that we have discussed a semester However, such approaches wi have high computationa compexity We-known recursive agorithms provide a much more efficient soution In the case of the first probem, the objective is to compute, given the observations, either the probabiity distribution over states or the probabiity associated with a particuar state transition from time t to time t These probabiities may be computed either based on past observations (simiar to the case of causa fitering, seen previousy) or based on both past and future observations (simiar to the case of smoothing, described previousy) Interestingy, the computation in this case can be computed using a we-known agorithm know as the forward-backward agorithm or (especiay in coding theory, where this probem aso arises) as the BCJR agorithm The second probem asks a sighty different question Namey, it asks what the most ikey sequence of states {X n } is, given the observations {Y n } The Viterbi agorithm is a we-known, computationayefficient soution to this probem Note that, athough this probem is reated to the first, the desired end-resut is quite different Namey, one wants to know the singe most ikey sequence of states that ed to a given set of observations The third probem is actuay the most interesting (and the one most associated with HMMs) In this case, it assumed that ~p(0), P, and B are unknown The objective is to use the data to find the best estimate of these system parameters The agorithm proceeds in a very interesting way, and makes use of the soution to the first probem Suppose that we coud observe X n, in addition to Y n Then, one approach to estimating ~p(0), P, and B woud be to find the vaues of ~p(0), P, and B that woud maximize p(~x,~y ~p(0), P, B) This is maximum ikeihood estimation, as we discussed earier in the term Since the vaues of X n are not actuay avaiabe, we estimate this probabiity (actuay, the og of this probabiity, the og-ikeihood) using the conditiona expectation, given an estimate of the parameters That is, given estimates ~p (k) (0), P (k), and B (k), we compute 8 Obviousy, if we can compute the probabiity distribution over states at time t given the observations, ~p(t), then the probabiity of the transition from state i to state j from time t to time t + 1 wi be simpy p i (t)p ij Q(~p(0), P, B ~p (k) (0), P (k), B (k) )=E[og p(x, y ~p(0), P, B) ~p (k) (0), P (k), B (k) ] Then, we find the vaues of ~p(0), P, and B that maximize Q(~p(0), P, B ~p (k) (0), P (k), B (k) ), and we et ~p (k+1) (0), P (k+1), B (k+1) equa these vaues Then, we iterate Finding Q(~p(0), P, B ~p (k) (0), P (k), B (k) ) essentiay invoves using the forward-backward agorithm to compute the state and transition probabiities that we discussed with respect to the first probem above Once those probabiities are computed, the form of

10 a brief introduction to markov chains and hidden markov modes 10 Q(~p(0), P, B ~p (k) (0), P (k), B (k) ) is then actuay quite simpe, so that finding ~p (k+1) (0), P (k+1), B (k+1) is easy This particuar agorithm for finding the parameters of a HMM is known as the Baum-Wech agorithm Athough it is computationay quite efficient, it can sti be quite onerous to compute if N s or N o is arge or if the number of observations T is arge On the other hand, it is difficut to get good estimates for the parameters is T is sma For more information, see Chapter 5 of the course notes of Bruce Hajek, avaiabe on the web (See syabus for detais) His notation is not identica to mine, but he works out the agorithms here in some further detai