Steven R. Dunbar Department of Mathematcs 203 Avery Hall Unversty of Nebraska-Lncoln Lncoln, NE 68588-0130 http://www.math.unl.edu Voce: 402-472-3731 Fax: 402-472-8466 Topcs n Probablty Theory and Stochastc Processes Steven R. Dunbar Classes of States and Statonary Dstrbutons Ratng Mathematcally Mature: proofs. may contan mathematcs beyond calculus wth 1
Secton Starter Queston Randomly dstrbute three balls between two urns, labeled A and B. Each perod, select an urn at random, and f t s not empty, remove a ball from that urn and put t n the other urn. How could you determne what fracton of tme s urn A empty? What s that fracton of tme? Does ths depend on how the balls are ntally dstrbuted between the two urns? Key Concepts 1. The states of a Markov chan partton nto equvalence classes accordng to ther type: communcatng, perodc versus aperodc, recurrent versus transent, and ergodc. 2. Every rreducble ergodc Markov chan has a unque non-negatve soluton of π P j = π j π = 1 called a statonary dstrbuton. 2
Vocabulary 1. The states of a Markov chan partton nto equvalence classes accordng to ther type: communcatng, perodc versus aperodc, recurrent versus transent, and ergodc. 2. Any vector (π) j satsfyng the set of the equatons π P j = π j π = 1 s a statonary probablty dstrbuton of the Markov chan. Mathematcal Ideas Classfcaton of States and Statonary Dstrbutons Defnton. State j s accessble from state f P n j > 0 for some n 1. That s, state j s accessble from state f and only f t s possble that the process wll enter state j. Defnton. Two states communcate f state j s accessble from state and state s accessble from j. Remark. Note that t s possble that j s accessble from but that and j do not communcate. Ths would happen f j s an absorbng state accessble from. Proposton 1. Beng able to communcate s an equvalence relaton, so that the states of a Markov chan partton nto dsjont classes. Proof. Left as an exercse. 3
Defnton. The perod of a state, wrtten as d(), s the greatest common dvsor of all ntegers n 1 for whch P n > 0. If P n = 0 for all n, defne d() = 0. Proposton 2. 1. If communcates wth j, then d() = d(j). 2. If state has perod d(), then there exsts an nteger N dependng on such that for all ntegers n N, P nd() > 0. That s, return to state can occur at all suffcently large multples of the perod d(). 3. If P m j > 0, then P m+nd() j > 0 for all suffcently large n. Proof. Left as an exercse. Defnton. A Markov chan n whch each state has perod 1 s aperodc. Defnton. The Markov chan s rreducble f t has only one class. Defnton. For any state, let f denote the probablty that startng n the process wll ever reenter state. State s recurrent f f = 1 and transent f f < 1. Proposton 3. If state s recurrent, then the process wll reenter nfntely often. Proof. Left as an exercse. Proposton 4. 1. State s recurrent f n=1 P n j =. 2. State s transent f Proof. Left as an exercse. n=1 P n j <. Defnton. If a state s recurrent, then t s postve recurrent f, startng n, the expected tme untl the process returns to state s fnte. Proposton 5. In a fnte-state Markov chan all recurrent states are postve recurrent. 4
Proof. Left as an exercse. Defnton. Postve recurrent and aperodc states are ergodc. Theorem 6. For an rreducble ergodc Markov chan lm n P n j exsts and s ndependent of. Furthermore, lettng π j = lm Pj n n then π j s the unque non-negatve soluton of π P j = π j π = 1 Defnton. Any vector (π) j satsfyng the set of the equatons s a statonary probablty dstrbuton of the Markov chan. A Markov chan started accordng to a statonary dstrbuton wll have ths dstrbuton for all future tmes. Remark. A lmtng dstrbuton s always a statonary dstrbuton, but the converse s not true. A Markov chan may have a statonary dstrbuton but no lmtng dstrbuton. For example, the perodc Markov chan whose transton probablty matrx s ( ) 0 1 P = 1 0 has no lmtng dstrbuton but π = (, ) s a statonary dstrbuton. Notce that ths Markov chan s not ergodc. Remark. Note that ths theorem says that a probablty transton matrx for an rreducble ergodc Markov chan has a left egenvector wth correspondng egenvalue 1. Ths s a specal case of the more general Perron-Frobenus Theorem. 5
Amusng Example of a Statonary Dstrbuton The followng example s adapted from the soluton n Lessard based on the problem orgnally posed n The Rddler. The nterest s partly n the humorous context and partly that the context obscures that the Markov chan has more states than s at frst obvous. There s a bathroom n your offce buldng that has only one tolet. There s a small sgn stuck to the outsde of the door that you can slde from Vacant to Occuped so that no one else wll try the door handle (theoretcally) when you are nsde. Unfortunately, people often forget to slde the sgn to Occuped when enterng, and they often forget to slde t to Vacant when extng. Assume that 1/3 of bathroom users don t notce the sgn upon enterng or extng. Therefore, whatever the sgn reads before ther vst, t stll reads the same thng durng and after ther vst. Another 1/3 of the users notce the sgn upon enterng and make sure that t says Occuped as they enter. However, half the tme they forget to slde t to Vacant when they ext. The remanng 1/3 of the users are very conscentous: They make sure the sgn reads Occuped when they enter, and then they slde t to Vacant when they ext. Fnally, assume that the bathroom s occuped exactly half of the tme, all day, every day. Two questons about ths workplace stuaton: 1. If you go to the bathroom and see that the sgn on the door reads Occuped, what s the probablty that the bathroom s actually occuped? 2. If the sgn reads Vacant, what s the probablty that the bathroom actually s vacant? 3. Extra credt: What happens as the percentage of conscentous bathroom users changes? The frst step s to defne the states and transton probabltes. We mght thnk that because the bathroom can be ether occuped or vacant, and the sgn n front can ether read Vacant or Occuped, that there should be four states, one for each possble par of occupaton state and sgn. However, 6
ths s not the case. Consder the state bathroom s occuped and the sgn says ts occuped. We must dstngush between the cases where the person occupyng the bathroom s conscentous (they wll defntely slde the sgn to Vacant when they leave) or not (they wll leave the sgn as Occuped after they leave). We must therefore add more states to our Markov chan that correspond to the dfferent ways n whch the bathroom can be occuped. Consder that there are three types of users: oblvous, forgetful, and conscentous, each of proporton 1/3. Imagne a sequence of short tmes, say every mnute. Usng the assumpton that the bathroom s occuped exactly half of the tme, all day, every day we can generalze slghtly to assume that the transton from state to state each mnute s tself a Markov chan wth transton probablty ( O V ) O V Note that the statonary dstrbuton s (, ) whch s consstent wth the assumpton that the bathroom s occuped exactly half of the tme, all day, every day. Now the possble transtons at each mnute are: 1. A Conscentous person s n the bathroom, and t s Occuped and the sgn says Occuped. We wll name ths state as OCO. Ether the person stays and the state remans Occuped wth sgn Occuped wth probablty, or the conscentous person leaves and the state s V V wth probablty. 2. A Forgetful person s n the bathroom, and t s Occuped and the sgn says Occuped. We wll name ths state as OF O. Ether the person stays and the state remans Occuped wth sgn Occuped wth probablty, or the forgetful person leaves and the state s V O wth probablty = 1/4 or the forgetful person leaves, changng the sgn and the state s V V wth probablty = 1/4. 3. An Oblvous person s n the bathroom, and t s Occuped and the sgn says Occuped. We wll name ths state as OOO. Ether the person stays and the state remans Occuped wth sgn Occuped wth 7
probablty, or the oblvous person leaves and the state s V O wth probablty. 4. A Oblvous person s n the bathroom, and t s Occuped but the sgn says Vacant. We wll name ths state as OOV. Ether the person stays and the state remans Occuped wth sgn Vacant wth probablty, or the oblvous person leaves and the state s V O wth probablty. 5. The bathroom s Vacant, but the sgn says Occuped. Wth probablty some person wll approach, nevertheless try the door and determne the true state of vacancy and enter, dealng wth the sgn accordng to ther type. Then the transton s to OCO wth probablty 1/6, OF O wth probablty 1/6 and OOO wth probablty 1/6. Wth probablty the state remans V O. 6. The bathroom s Vacant, and the sgn says Vacant. Wth probablty some person wll approach, nevertheless try the door and easly determne the true state of vacancy and enter, dealng wth the sgn accordng to ther type. Then the transton s to OCO wth probablty 1/6, OF O wth probablty 1/6 and OOV wth probablty 1/6. Wth probablty the state remans V V. A state transton dagram s n Fgure 1. Then the transton probablty matrx s P = OCO OF O OOO OOV V O V V OCO 0 0 0 0 OF O 0 0 0 1/4 1/4 OOO 0 0 0 0 OOV 0 0 0 0 V O 1/6 1/6 1/6 0 0 V V 1/6 1/6 0 1/6 0 The statonary dstrbuton s then the soluton of πp = π, π = 1 8
1/6 1/6 OCO VO OOO 1/6 1/6 1/6 1/4 1/4 OFO VV OOV 1/6 Fgure 1: The state transton dagram for the bathroom occupancy. wth soluton π OCO = 1 6, π OF O = 1 6, π OOO = 1 12, π OOV = 1 12, π V O = 1 4, π V V = 1 4. Note that the bathroom s stll Vacant half the tme, although the tme s splt equally over the 2 sgn states. π V V P [Vacant says Vacant ] = = π V V + π OOV P [Occuped says Occuped ] = Sources 1/4 1/4 + 1/12 = 3 4 π OCO + π OF O + π OOO π OCO + π OF O + π OOO + π V O = 5 8. Ths secton s adapted from: Ross, Introducton to Probablty Models, Taylor and Karln, An Introducton to Stochastc Modelng, Karln and Taylor, A Second Course n Stochastc Processes and Lessard Lessard. 9
Algorthms, Scrpts, Smulatons Algorthm Scrpts Problems to Work for Understandng 1. Provde examples of the classfcatons of states: (a) A trval two-state Markov chan n whch nether state s accessble from the other. (b) A trval two-state Markov chan n whch both states communcate. (c) A Markov chan n whch all states have perod 2. (d) A three-state Markov chan wth two states absorbng and one transent state. What states are communcatng n ths example? (e) An ergodc Markov chan. 2. Randomly dstrbute three balls between two urns, labeled A and B. Each perod, select an urn at random, and f t s not empty, remove a ball from that urn and put t n the other urn. Make a Markov chan model of ths stuaton and classfy all states. What fracton of tme s urn A empty? Does ths depend on how the balls are ntally dstrbuted between the two urns? 10
3. Assume the fracton of oblvous, forgetful, and conscentous users are p, q, and r respectvely, wth 0 p, q, r 1 and p + q + r = 1. Solve the bathroom problem under ths general assumpton. 4. Assume the fracton of oblvous, forgetful, and conscentous users are p, q, and r respectvely, wth 0 p, q, r 1 and p + q + r = 1. Further assume the bathroom s occuped 2/3 of the tme, and vacant 1/3 of the tme. Solve the bathroom problem under ths general assumpton. 5. Assume the fracton of oblvous, forgetful, and conscentous users are p, q, and r respectvely, wth 0 p, q, r 1 and p + q + r = 1. Further assume the bathroom s occuped 1 v of the tme, and vacant v of the tme, where 0 < v < 1. Solve the bathroom problem under ths general assumpton. What happens as v 0 or v 1? 6. Assume the fracton of oblvous, forgetful, and conscentous users are p, q, and r respectvely, wth 0 p, q, r 1 and p + q + r = 1. Also assume that the oblvous people spend twce as long n the bathroom as the conscentous or forgetful people. Assume the bathroom s vacant v of the tme, where 0 < v < 1. Are all these assumptons consstent? Solve the bathroom problem under ths general assumpton. Readng Suggeston: References [1] S. Karln and H. Taylor. A Second Course n Stochastc Processes. Academc Press, 1981. [2] Sheldon M. Ross. Introducton to Probablty Models. Elsever, 6th edton, 1997. 11
[3] H. M. Taylor and Samuel Karln. An Introducton to Stochastc Modelng. Academc Press, thrd edton, 1998. Outsde Readngs and Lnks: 1. 2. 3. 4. I check all the nformaton on each page for correctness and typographcal errors. Nevertheless, some errors may occur and I would be grateful f you would alert me to such errors. I make every reasonable effort to present current and accurate nformaton for publc use, however I do not guarantee the accuracy or tmelness of nformaton on ths webste. Your use of the nformaton from ths webste s strctly voluntary and at your rsk. I have checked the lnks to external stes for usefulness. Lnks to external webstes are provded as a convenence. I do not endorse, control, montor, or guarantee the nformaton contaned n any external webste. I don t guarantee that the lnks are actve at all tmes. Use the lnks here wth the same cauton as you would all nformaton on the Internet. Ths webste reflects the thoughts, nterests and opnons of ts author. They do not explctly represent offcal postons or polces of my employer. Informaton on ths webste s subject to change wthout notce. Steve Dunbar s Home Page, http://www.math.unl.edu/~sdunbar1 Emal to Steve Dunbar, sdunbar1 at unl dot edu Last modfed: Processed from L A TEX source on September 25, 2017 12