Chapter 8 Markov Chains and Some Applications ( 馬哥夫鏈 )

Transcription

1 Chater 8 arkov Chains and oe Alications ( 馬哥夫鏈 Consider a sequence of rando variables,,, and suose that the set of ossible values of these rando variables is {,,,, }, which is called the state sace. It will be helful to interret n as being the state of soe syste at tie n (or at the n-th ste, and, in accordance with this interretation, we say the syste is in state i at tie n if n i. The sequence of rando variables is said to for a arkov chain if each tie the syste is in state i there is soe fixed robability, say, that it will next be in state j. That is, for all i, i,, i n-, i, j, P( n j n i, n- i n-,, i, i. The nubers are called the transition robabilities of the arkov chain and they satisfy and j, i,,,,. It is convenient to arrange the transition robabilities in a square atrix as follows: K K ( (x( which is called the transition robability atrix.(ust be square Exale 8.: uose that whether or not it rains toorrow deends on revious weather conditions only through whether or not it is raining today. uose further that if it is raining today, then it will rain toorrow with robability, and, if it is not raining today, then it will rain toorrow with robability q. Can we regard this syste as a arkov chain? If the answer is yes lease find its transition robability atrix. The syste is in state when it does not rain; state when it rains. n rains/ sunny after n days {, } q, q,,. q q Exale 8.: Consider a gabler who at each lay of the gae either wins unit with robability or loses unit with robability. If we suose that the gabler will quit laying when his fortune hits either or, then the gabler s sequence of fortunes is a arkov chain. What are the transition robabilities? {,,,, N} where NN... N i, i, ( i,,..., N... N i, i, ( i,,..., N. N N... NN All zeros otherwise

2 Knowledge of the transition robability atrix and the distribution of enables us, in theory, to coute all robabilities of interest. For instance, the joint robability density function of,,, n is given by P( n i n, n- i n-,, i, i P( n i n n- i n-,, i, i P( n- i n-,, i, i i n, i n P( n- i n-,, i, i i n, i n i n, i n P( i. i, i i, i For a arkov chain, reresents the robability that a syste in state i will enter j at the next transition. We can also define the two-stage transition robability, ij, that a syste resently in state i will be in state j after two additional transitions. That is, P( n j n i. The can be couted fro the as follows: P j i ( k k k P( j, k i P( j k, i P( k i In general, we define the k-stage transition robabilities, denoted as (k, by (k P( nk j n i. The value of (k ij can be evaluated by couting the (i,j entry of the atrix k, where is the transition robability atrix. (k the (i, j entry of the atrix k The k th ower of

3 Exercise 8.: (Rando walk Assue that a an is getting drunk and he is standing at a osition shown in the following figure. His house is situated eters away fro the sea. The robability that he is walking towards his house by one eter is / and the robability that he is walking towards the sea by one eter is /. Let us also assue that he will reain his state once he is in his house or in the sea. (a Let the sale sace be {,,,, }, in which i reresents that the an is sitting at a location which is i eters fro his house. Find the corresonding transition robability atrix (. P({ fro i to j } to the left i, i-, i,,, to the right i, i, i,,,,, otherwise. (b Let be the robability that the an oves fro the state i by two stes to the state j. Prove that k. P E P( E F P( F P( E F P( F... P( E F P( F ( N N Use Baye s theore, ({ initially at i and finally at j after stes} {at k after the first ste} ij k P({ at k after the first ste} k k

4 (c Let (n be the (i, j entry of the atrix n, where is the transition robability atrix. Prove that the robability that the an oves fro the state i by n stes to the state j is given by (n. By (b, P({ initially at i and finally at j after stes} k the (i, j the entry of the atrix Use.I., assue that P ({ after n stes } the (i, j entry of n. n Then P ({ after n stes} P ({ initially at i and finally are j after n stes} { at k after n stes} P({ at k after n stes} k " the( i, k entry of the (i, j entry of " " k (d Find the robability that the an will eventually fall into the sea at the -th ste. P ( ( Target: The dot roduct of the rd row of and the th colun of ( 8.. 8

5 : original osition Exercise 8.: In this odel a frog can ju only to neighboring rocks as shown in the above figure. Initially the frog stays on one of the six rocks and going to ju randoly either to the left or to the right. The robability that the frog will ju to the left is /; it will ju to the right is / and will reain its osition (juing u and down is /. (a Use a arkov chain to reresent this odel and also find its state sace and the corresonding transition robability atrix. {,,,,, } P({fro stable i to state j } ii (i,,,,,, to the left i, i - (i,,,, to the right i, i (i,,,,, otherwise.

6 (b Find the robability that the frog will be at the original rock after juing six ties. ( the (, entry of, ( ( ( P.. (c Find the robability that the frog will not be at the original rock after juing ties. ( ( P i, ( the (, entry of i.

7 Exercise 8.: uose that there are DVDs of action ovies, of horror ovies, and of coedy ovies. A erson firstly watches the action ovie and will randoly select a DVD to watch aong the reaining two ovie tyes. (a Construct a odel of arkov chain and find the corresonding transition robability atrix. tate sace {,, }, action, horror, coedy (b Find the robability that the fifth watched ovie is not of horror tye. P ({ fifth watched ovie is of horror tye} ( the (, entry of P({fifth watch ovie is not of horror tye} ( 88..