REPRESENTING MARKOV CHAINS WITH TRANSITION DIAGRAMS

Transcription

1 Joural o Mathematics ad Statistics, 9 (3): 49-54, 3 ISSN Sciece Publicatios doi:.38/jmssp Published Olie 9 (3) 3 ( REPRESENTING MARKOV CHAINS WITH TRANSITION DIAGRAMS Farida Kachapova School o Computig ad Mathematical Scieces, Faculty o Desig ad Creative Techology, Aucklad Uiversity o Techology, New Zealad Received 3-5-, Revised 3-5-6; Accepted ABSTRACT Stochastic processes have may useul applicatios ad are taught i several uiversity programmes. Studets ote ecouter diiculties i learig stochastic processes ad Markov chais, i particular. I this article we describe a teachig strategy that uses trasitio diagrams to represet a Markov chai ad to re-deie properties o its states i simple terms o directed graphs. This strategy utilises the studets ituitio ad makes the learig o complex cocepts about Markov chais aster ad easier. The method is illustrated by worked examples. The described strategy helps studets to master properties o iite Markov chais, so they have a solid basis or the study o iiite Markov chais ad other stochastic processes. Keywords: Trasitio Diagram, Trasitio Matrix, Markov Chai, First Passage Time, Persistet State, Trasiet State, Periodic State, Iter-Commuicatig States Sciece Publicatios. INTRODUCTION Stochastic processes are importat or modellig may atural ad social pheomea ad have useul applicatios i computer sciece, physics, biology, ecoomics ad iace. There are may textbooks o stochastic processes, rom itroductory to advaced oes (Cilar, 3; Grimmett ad Stirzaker, ; Hsu, ; Krylov, ; Lawler, 6; Revuz, 8). Discrete Markov processes are the simplest ad most importat class o stochastic processes. There are oly ew publicatios o teachig stochastic processes ad Markov chais, such as Chag- Xig (9), Wag (a; b), ad Wag ad Ko (3). More research eeds to be carried out o how to teach stochastic processes to researchers (Wag, a). While tryig to uderstad Markov chais models, studets usually ecouter may obstacles ad diiculties (Wag, b). May lecturers use visual displays such as sample paths ad trasitio diagrams to illustrate Markov chais. I this article we utilise trasitio diagrams urther or teachig several importat cocepts o Markov chais. We explai i details how 49 these cocepts ca be deied i terms o trasitio diagrams (treated as directed weighted graphs) ad we accompay this with worked examples. Trasitio diagrams provide a good techiques or solvig some problems about Markov chais, especially or studets with poor mathematical backgroud.. TRANSITION DIAGRAM OF A MARKOV CHAIN: DEFINITIONS A homogeeous iite Markov chai is etirely deied by its iitial state distributio ad its trasitio matrix S [p ij ], where p ij P(X i X j) is the trasitio probability rom state j to state i. The graphical represetatio o a Markov chai is a trasitio diagram, which is equivalet to its trasitio matrix. The trasitio diagram o a Markov chai X is a sigle weighted directed graph, where each vertex represets a state o the Markov chai ad there is a directed edge rom vertex j to vertex i i the trasitio probability p ij >; this edge has the weight/probability o p ij.

2 Farida Kachapova / Joural o Mathematics ad Statistics 9 (3): 49-54, 3 It equals the probability o gettig rom state j to state i i exactly steps. It ca be calculated as the correspodig elemet o the matrix S () but it is usually easier to id it rom the trasitio diagram as a sum o the probabilities o all edge sequeces o legth rom j to i. Example I the chai rom Example, the 3-step trasitio probability rom to equals: Fig.. The trasitio diagram o the Markov chai rom Example Example A Markov chai has states,, 3, 4, 5, 6 ad the ollowig trasitio matrix: Sciece Publicatios S..3 This is its trasitio diagram. I the diagram i Fig. the probability o each edge is show ext to it. For example, the loop rom state to state has probability.4 p P(X X ) ad the edge rom state to state 3 has probability.5 p 3 P(X 3 X ). I the graph termiology, a edge sequece o legth is a ordered sequece o edges e, e,, e, where e i ad e i+ are adjacet edges or all i,,,. A path is a edge sequece, where all edges are distict. A simple path is a path, where all vertices are distict (except possibly the start ad ed vertices). A cycle is a simple path there the start vertex ad the ed vertex are the same. I a trasitio diagram the probability o a edge sequece equals a product o the probabilities o its edges. 3. PROPERTIES OF A MARKOV CHAIN IN TERMS OF TRANSITION DIAGRAMS 3.. N-Step Trasitio Probability A -step trasitio probability is: ij p P X i X j. 5 p a + a, where: a is the probability o the path 3 ad a is the probability o the edge sequece. These probabilities are easy to id rom the diagram i Fig.. So: p Probability o Visitig a State or the First Time Let us cosider a radom variable: T i mi { : X i}. It represets the umber o steps to visit a state i or the irst time. It is called the irst passage time o the state i. Related probabilities are: ij i ij i P T m X j ad P T < X j. Clearly: ij ij. ( m ) ij m These probabilities ca be iterpreted as ollows: the probability to visit i o step m or the irst time startig rom j; ij the probability to visit i i iite umber o steps startig rom j.

3 Farida Kachapova / Joural o Mathematics ad Statistics 9 (3): 49-54, 3 I terms o trasitio diagrams, ij equals a sum o the probabilities o all edge sequeces rom j to i that do ot iclude the vertex i betwee the start ad ed vertices. ij equals a similar sum or the edge sequeces o legth m oly. For iite Markov chais these probabilities are easier to id rom their trasitio diagrams tha with other methods. Example 3 From the trasitio diagram i Fig. we ca calculate the ollowig probabilities: 64 as the probability o the path 456; 64 or ad 64. For vertices ad we have: So: ;.3 as the probability o the path 3;.4.3. as the probability o the path 3; ( + ) ad i geeral, or ay,.4.3 as the probability o the edge sequece loops aroud. Sciece Publicatios... 3 with ( + ) times m Persistet ad Trasiet States. A state i o a Markov chai is called persistet i ii ad trasiet otherwise. Thus, i the chai starts at a persistet state, it will retur to this state almost surely. I the chai starts at a trasiet state, there is a positive probability o ever returig to this state. From the trasitio diagram we ca evaluate the probability ii ad thereore determie whether the state i is persistet or trasiet. Example 4 For each o the states ad 4 o the Markov chai i Example determie whether the state is persistet or trasiet. 5 Solutio as the probability o the cycle 456. So the state 4 is persistet. ( ).4 as the probability o the loop aroud. ( ). ( 3 ) as the probability o the cycle 3. + More geerally, or ay, ad.3.5 as the probability o the edge sequece So: times ( m ) m Sice.7 <, the state is trasiet. Lemma.7. Suppose i ad j are two dieret states o a Markov chai. I p ji > ad ij, the the state i is trasiet. This lemma is easily derived rom the deiitio o ij. The lemma ca be rephrased i terms o trasitio diagrams: i the chai ca reach state j rom state i i oe step (p ji > ) but caot come back ( ij ), the the state i is trasiet. Lemma gives a method o idig trasiet states rom a trasitio diagram without ay calculatios. For example, rom Fig. we ca see that p 4.3 > ad 4 because the chai caot retur rom state 4 to state. Thereore by Lemma the state is trasiet. This is cosistet with the result o Example Mea Recurrece Time The mea recurrece time o a persistet state i is deied as µ i m m ii. I i is a trasiet state, µ i by the deiitio. Thus, µ i is the expected time o returig to the state i i the chai starts at i. Example 5 For each o the states ad 4 o the Markov chai i Example id its mea recurrece time.

4 Farida Kachapova / Joural o Mathematics ad Statistics 9 (3): 49-54, 3 Solutio Sice the state is trasiet, µ. For the state 4, Sciece Publicatios ad So µ Periodic States or ay 3. The period o a state i is the greatest commo divisor o all with pii >. The state i is periodic i its period is greater tha ; otherwise it is aperiodic. I terms o trasitio diagrams, a state i has a period d i every edge sequece rom i to i has the legth, which is a multiple o d. Example 6 For each o the states ad 4 o the Markov chai i Example id its period ad determie whether the state is periodic. Solutio The trasitio diagram i Fig. has a cycle 3 o legth ad a cycle 3 o legth 3. The greatest commo divisor o ad 3 equals. Thereore the period o the state equals ad the state is aperiodic. Ay edge sequece rom 4 to 4 is a cycle 456 or its repetitio, so its legth is a multiple o 3. Hece the state 4 is periodic with period Commuicatig States ji State i commuicates with state j (otatio i j) i p > or some. I terms o trasitio diagrams, a state i commuicates with state j i there is a path rom i to j. State i iter-commuicates with state j (otatio i j) i the two states commuicate with each other. Theorem (Grimmett ad Stirzaker, ) Suppose i ad j are two states o a Markov chai ad i j. The: i ad j have the same period; i is persistet j is persistet; i is trasiet j is trasiet. 5 Iter-commuicatio is a equivalece relatio o the set Q o all states o a Markov chai. So the set Q ca be partitioed ito equivalece classes; all states i oe equivalece class share the same properties, accordig to Theorem. Example 7 Let us cosider the Markov chai rom Example ad its trasitio diagram i Fig.. Clearly, the states ad 3 iter-commuicate. Also, sice p > ad, sice there is a path 3 rom to. Next, 4 but ot 4 (there is o path rom 4 to ). States 4, 5 ad 6 all iter-commuicate. Thereore, the equivalece class o is: [] {,, 3} ad the equivalece class o 4 is: [4] {4, 5, 6}. Accordig to Theorem ad Examples 4 ad 6, the states, ad 3 are all trasiet ad aperiodic; the states 4, 5 ad 6 are all persistet ad periodic with period SECOND EXAMPLE OF TRANSITION DIAGRAM Next example illustrates that it is easier to partitio the state set ito equivalece classes irst ad the classiy the states. Example 8 Use a trasitio diagram to describe properties o a Markov chai with the ollowig trasitio matrix: Solutio S This is the chai s trasitio diagram:

5 Farida Kachapova / Joural o Mathematics ad Statistics 9 (3): 49-54, 3 Thereore the state 5 is persistet ad so is the state 6. Each state i has a loop aroud it correspodig to a path o legth rom i to i. Thereore each state is aperiodic. Next we calculate the mea recurrece time or each state. Fig.. The trasitio diagram o the Markov chai rom Example 8 First we id equivalece classes o itercommuicatig states: [] {, }; [3] {3, 4}; [5] {5, 6}. The vertices i Fig. correspodig to itercommuicatig states are marked with the same colour. Next we id persistet ad trasiet states. Accordig to Theorem, we just eed to check oe state rom each equivalece class. State. aroud. For, Sciece Publicatios.5 as the probability o the loop ( + ) as the probability o the edge sequece.... So: ( + ) times ( m ) m Thereore the state is persistet ad so is the state. State 4. p 64.5 > ad 46. So by Lemma the state 4 is trasiet ad so is the state State 5..5 as the probability o the loop aroud 5. For, ( + ) as the probability o the edge sequece ( + ) times. So µ m m ( + ) ( -.75) r Here we used the ormula r. ( r) For state,.75 ad or,.5.5. So: µ ( + ) ( -.5) + ( + ) Sice states 3 ad 4 are trasiet, µ 3 µ 4. For state 5: µ ( + ).5.5 ( -.5) Similarly, µ 6. From the values o µ i calculated above, we ca see that ulike other properties, the mea recurrece time ca be dieret or iter-commuicatig states.

6 Farida Kachapova / Joural o Mathematics ad Statistics 9 (3): 49-54, 3 Let us cosider vector π µ i made o reciprocals o the mea recurrece times. Clearly, i this case 4 π. Multiplyig it by the trasitio matrix S we ca easily check that π is a statioary distributio o the Markov chai: S π π. Thus, i π is the iitial state distributio, the the chai has this distributio at every step. I other words, π is the equilibrium distributio. With the distributio π the probability o each state is iversely proportioal to its mea recurrece time. I other words, whe the chai is i the equilibrium, it has a lower chace o beig i a state i i it takes loger o average to make a retur trip rom i to i. 5. CONCLUSION I this article the trasitio diagram o a iite Markov chai is treated as a directed weighted graph. Several properties o the chai s states are re-deied i terms o the trasitio diagram, which makes these properties more ituitive ad easy to uderstad. The author has bee usig the described teachig strategy i a course o stochastic processes i the Aucklad Uiversity o Techology, New Zealad, or several years. Case studies show that trasitio diagrams help the studets to master importat cocepts o iite Markov chais, so they have a solid basis or the studies o iiite Markov chais ad other stochastic processes. This teachig strategy re-iorces the itial mathematical deiitios; it uses the graphical represetatio o a Markov chai to make the complex cocepts clearer ad easier to assimilate, sice there is a eed to make a itroductory course i Markov chais as simple as possible (Wag, b). With trasitio diagrams the studets ca classiy the states o a Markov chais with miimal calculatios ad eve use their ituitio, which is ot ote possible i the studies o probability ad stochastic processes. 6. REFERENCES Chag-Xig, L., 9. Probe ito the teachig o probability theory ad stochastic processes. Proceedigs o the Iteratioal Coerece o Computatioal Itelligece ad Sotware Egieerig, Dec. -3, IEEE Xplore Press, Wuha, pp: -4. DOI:.9/CISE Cilar, E., 3. Itroductio to Stochastic Processes. st Ed., Elsevier, ISBN-: , pp: 46. Grimmett, G. ad D. Stirzaker,. Probability ad Radom Processes. 3rd Ed., Oxord Uiversity Press, New York, ISBN-: 9857, pp: 596. Hsu, H.P.,. Schaum s Outlie o Probability, Radom Variables, ad Radom Processes. d Ed., McGraw-Hill, New York, ISBN-: 76389, pp: 43. Krylov, N.V.,. Itroductio to the Theory o Radom Processes. st Ed., America Mathematical Society, ISBN-: , pp: 3. Lawler, G.F., 6. Itroductio to Stochastic Processes. d Ed., Chapma ad Hall/CRC, ISBN-: X, pp: 34. Revuz, D., 8. Markov Chais. st Ed., Elsevier, ISBN-: 8883, pp: 388. Wag, A.L., a. How much ca be taught about stochastic processes ad to whom. I: Traiig Researchers i the Use o Statistics, Bataero, C. (Ed.), pp: Wag, A.L., b. Itroducig Markov chais models to udergraduates. Iteratioal Statistical Istitute, 53rd Sessio, Seoul, pp:-4. Wag, A.L. ad S.H. Ko, 3. Should simple Markov processes be taught by mathematics teachers? Iteratioal Statistical Istitute, 54 th Sessio, Berli, pp:-4. Sciece Publicatios 54