CSCI-6971 Lecture Notes: Stochastic processes

Size: px

Start display at page:

Download "CSCI-6971 Lecture Notes: Stochastic processes"

Luke Jacobs
6 years ago
Views:

1 CSCI-6971 Lecture Notes: Stochastic processes Kristopher R. Beevers Departet of Coputer Sciece Resselaer Polytechic Istitute February 2, Overview Defiitio 1.1. A stochastic process is a collectio of rado variables X 1, X 2,..., X t paraeterized by tie. There are a uber of aspects of a stochastic process that we ca exaie. Aog the: depedecies betwee variables i the sequece various kids of log-ter averages the frequecy at which boudary evets occur Soe exaples of stochastic processes iclude the followig: Defiitio 1.2. Rado walks ca be thought of as cuulative stochastic processes. For exaple, S r1 X r + S 0 is a rado walk o the lie. Defiitio 1.3. Arrival processes such as the Beroulli ad Poisso processes odel the frequecy of arrivals or successes i soe tie-depedet odel. I geeral, we cosider odels i which iterarrival ties are idepedet rado variables. Defiitio 1.4. Markov processes are sequeces where the ext value depeds o the past oly through the curret value, i.e.: P X t+1 k X 1:t ) P X t+1 k X t ) 1) Defiitio 1.5. Martigales are processes where the expectatio of the ext value is exactly the sae as the curret value, i.e.: E [X t+1 X 1:t ] X t 2) Martigales frequetly arise i gablig, for exaple, ad they ca be thought of as a odel of a truly fair gae. The priary sources for ost of this aterial are: Itroductio to Probability, D.P. Bertsekas ad J.N. Tsitsiklis, Athea Scietific, Belot, MA, 2002; ad Radoized Algoriths, R. Motwai ad P. Raghava, Cabridge Uiversity Press, Cabridge, UK, 1995; ad Stochastic Processes & Models, D. Stirzaker, Oxford Uiversity Press, New York, 2005; ad the author s ow otes. 1

2 2 Arrival processes We ow discuss two arrival processes, the Beroulli process ad the Poisso process, which are discrete- ad cotiuous-tie aalogs of each other, respectively. 2.1 Beroulli process The Beroulli process is a arrival process cosistig of a sequece of i.i.d. Beroulli trails 1, each of which takes uit tie. The key property of a Beroulli process is that it is eoryless, i.e.: P T t T > ) P T t) 3) This forulatio is derived fro the defiitio of coditioal probability.) There are several iterestig PDFs that we ca exaie for a Beroulli process: The uber S of arrivals i trials: ) f S k) BINOMIAL[, p] p k 1 p) k 4) k E [S] p 5) var S) p1 p) 6) The uber T of trials up to ad icludig the first success : f T ) GEOMETRIC[p] p1 p) 1 7) E [T] 1 p 8) var T) 1 p p 2 9) The tie Y k util the kth success: this is just the su of the first k iterarrival ties T 1,..., T k which are i.i.d. geoetric rado variables. Y k is distributed accordig to the Pascal PMF probability ass fuctio) of order k : f Yk ) PASCAL[p, k] [ ] k E [Y k ] E T i k p i1 t 1 k 1 ) p k 1 p) t k 10) 11) ) k k1 p) var Y k ) var T i p i1 2 12) 2.2 Poisso approxiatio to the bioial distributio Give a rado variable Z POISSON[λ]: λ λk f Z k) POISSON[λ] e 13) k! E [Z] λ 14) var Z) λ 15) 1 Beroulli trials ad Poisso trials: for a sequece of Beroulli trials, P X i 1) p ad P X i 0) 1 p for soe fixed p. A sigle Poisso trial is itself just a Beroulli trial, but whe grouped ito a sequece, Poisso trials are differet i that each trial ay have differet probabilities, i.e. P X i 1) p i ad P X i 0) 1 p i. 2

3 Clai 2.1. If is large ad p is sall, the POISSON[p] BINOMIAL[, p]. Proof: Let p λ/. We begi with soe aipulatio of the bioial distributio: ) BINOMIAL[, p] p k 1 p) k 16) k 1 λ ) k 17) Exaiig each copoet, we see that: li li k 1 i0 i 1 λ ) k li! λ k k)!k! k k 1 i λk k! i0 1 λ ) k 18) 19) 1 20) 1 λ ) 1 λ ) k 21) 1 e λ 22) The last equality follows fro the idetity li 1 + x ) e x.) So we have that λk li BINOMIAL[, p] e λ k! POISSON[λ] 23) I geeral, the Poisso approxiatio is valid to withi several decial places if 100 ad p Poisso process The Poisso process is the cotiuous-tie aalog of the Beroulli process. Let P k, τ) be the probability that k arrivals occur durig a iterval of legth τ. The a process is Poisso if it eets the followig properties: Tie-hoogeeity: P k, τ) is the sae for ay iterval of legth τ Idepedece: the uber of arrivals durig a iterval is idepedet of the history of arrivals outside the iterval Sall iterval probabilities: P k > 1, τ) is egligible i copariso to P 0, τ) ad P 1, τ) as τ decreases. We exaie the last property further. If a process is Poisso, the the PDF describig the uber of arrivals i a legth-τ iterval is: λτ λτ)k P k, τ) POISSON[λτ] e k! Takig the Taylor expasio for differet values of k, we see that: 24) λτ Taylor P 0, τ) e 1 λτ + Oτ 2 ) 25) P 1, τ) e λτ λτ Taylor λτ + Oτ 2 ) 26) P k > 1, τ) Oτ 2 ) 27) 3

4 which verifies the fial property stated above. There are soe other iterestig PDFs, siilar to those we exaied for the Beroulli process: The tie T util the first arrival. Note that T > t if ad oly if there were o arrivals i [0, t], so: F T t) P P t) 1 P T > t) 1 P 0, t) 1 e λt 28) f T t) df T dt E [T] 1 λ λe λt EXPONENTIAL[λ] 29) 30) var T) 1 λ 2 31) The tie Y k util the kth arrival. This is agai just the su of the first k iterarrival ties, so Y k follows the Erlag PDF of order k, i.e.: f Yk y) ERLANG[λ, k] λk y k 1 e λy E [Y k ] k λ k 1)! 32) 33) var Y k ) k λ 2 34) 2.4 Rado icidece paradox Suppose we have a Poisso process ad we fix soe tie t. The what is the legth L of the iterarrival iterval 2 that cotais t? Note that t is ot a rado varaible, but L is. We ll assue that the process has bee ruig log eough that there have bee previous arrivals. Ituitively, it sees that we d expect L to be the legth of a typical iterarrival iterval, i.e. L EXPONENTIAL[λ]. However, this is ot the case! Cosider the followig graphical depictio: arrival arrival U t V tie At tie U, the previous arrival occurs, ad at tie V, the ext arrival occurs. The tie we ve fixed, t, falls betwee U ad V. Clearly, the legth of the iterarrival iterval i which t falls is the: L t U) + V t ) 35) Because of the eorylessess property of a Poisso process, t U) ad V t ) are idepedet rado variables, each distributed accordig to a expoetial distributio with paraeter λ. Note that whe ruig a Poisso process backwards i tie, it 2 Iterarrival iterval: the iterval betwee two cosecutive arrivals. 4

5 also reais Poisso.) Thus, L is the su of two expoetial rado variables, i.e. L ERLANG[λ, 2] ad E [L] 2/λ. The ituitio behid this apparet paradox is that a arrival at a arbitrary tie is ore likely to fall i a large iterval rather tha a sall oe, so the expected legth of the iterval i which it falls is loger tha the average iterarrival tie. 3 Markov chais We ow cosider stochastic processes where the value at soe tie takes o oe of a fiite set of states. Together with a odel of the trasitios betwee states ad a few other properties, we have processes kow as Markov chais. Defiitio 3.1. I a discrete-tie Markov chai, trasitios require uit tie. Such a chai has the followig properties: A set of states S {1,..., } A atrix P of trasitio probabilities, with: P ij P X +1 j X i) j1 P ij 1 for all i The Markov property is et: P X +1 x X 0: ) P X +1 x X ) 36) Note that give a distributio for the iitial state, we ca copute a distributio for the history X 0: of the Markov chai. For soe history x 0: : P X 0 x 0,..., X x ) i0 P xi x i+1 37) Defiitio 3.2. We defie the -step trasitio probability as the distributio of the state at soe future tie, coditioed o the curret state. We specify this as: r ij ) P X j X 0 i) 38) Total prob. Markov k1 k1 P X 1 k X 0 i) P X j X 1 k, X 0 i) 39) P X 1 k X 0 i) P X j X 1 k) 40) r ik 1)P kj 41) k1 Equatio 41 is kow as the Chapa-Kologorov Equatio. 3.1 Properties of states ad chais Defiitio 3.3. A state j is accessible fro i if there is soe such that r ij > 0. It is possible to reach j fro i i steps.) We defie the set Ai) as: Ai) {j S j is accessible fro i} 42) 5

6 Defiitio 3.4. A state i is recurret if for all j Ai) it is true that i Aj). I other words, give eough tie, we will always retur to i; it is oly possible fro i to reach states where there is soe probability of returig to i. Defiitio 3.5. A state i is trasiet if it is ot recurret. Give eough tie, we will evetually eter a state fro which it is ipossible to retur to i. Thus, i will be visited a fiite uber of ties. The followig picture depicts recurret ad trasiet states: States 1, 3, ad 4 are recurret, while state 2 is trasiet. Defiitio 3.6. If state i is recurret, Ai) fors a recurret class. The, all states i Ai) are accessible fro each other ad o states outside Ai) are accessible fro a state i Ai). A Markov chai ca be decoposed ito oe or ore recurret classes, plus soe trasiet states. Oce the process eters a recurret class, the process reais i that class forever ad all states i the class are visited a ifiite uber of ties. If X 0 is trasiet, the for soe k, X 0:k are all trasiet ad X k+1: are all i the sae recurret class. Defiitio 3.7. Let R be soe recurret class. R is periodic if its states ca be grouped ito d > 1 disjoit subsets S 1,..., S d such that all trasitios fro S k lead to S k+1 ad all trasitios fro S d lead to S 1. I other words, the states i R ca be partitioed like the followig: A recurret class that is ot periodic is tered aperiodic. Defiitio 3.8. A ergodic state is oe that is both aperiodic ad recurret. Defiitio 3.9. A ergodic Markov chai is oe i which all states are ergodic. 6

7 3.2 Steady-state behavior It is frequetly useful to aalyze the log-ter state occupacy behavior of a Markov chai. I other words, what is the behavior of r ij ) as becoes large? First, ote that if there are ultiple recurret classes, li r ij ) depeds o the iitial state. Furtherore, eve if there is a sigle recurret class, r ij ) ay ot coverge if the class is periodic, as see i the followig siple exaple: 1 2 Here, regardless of the legth of the process, we have: r ii ) { 1 if is eve 0 if is odd 43) Defiitio For these reasos, we restrict our study of covergece to Markov chais with a sigle aperiodic recurret class. For such a Markov chai, we have: Here, π j is the steady-state probability of j. li r ij) π j P X j) 44) Defiitio Cobiig the above with the Chapa-Kologorov Equatio: li k1 The equatios 46 are kow as the balace equatios. r ik 1)P kj π j 45) π k P kj π j 46) k1 We ca cosider the balace equatios together with the oralizatio equatio k1 π k 1 ad for a syste of equatios, which ca be solved to obtai the π j s. A property of the solutio is that π j 0 if j is trasiet, ad π j > 0 otherwise. The steady-state probabilities ca be thought of as expected state frequecies. For a Markov chai with a sigle aperiodic recurret class, π j li v ij ) 47) where v ij ) is the expected value of the uber of visits to j withi the first trasitios, startig fro i. We ca also exaie the frequecy of trasitios. Let q jk ) be the expected uber of ties the trasitio fro j to k is take i the first steps. The: q jk ) li π j P jk 48) 7

8 3.3 Absorptio It is also useful to aalyze the short-ter behavior of Markov chais. A iportat questio is: if we start i a trasiet state, how log will it be util we eter the first recurret state? Sice what happes afterward does ot atter, we focus o the case where every recurret state is absorbig, i.e. P kk 1. Defiitio Fix soe absorbig state s. The probability that s is evetually reached, startig fro state i, is tered the absorptio probability: a i P X evetually becoes s X 0 i) 49) Clearly, we have a s 1. Furtherore, for all absorbig i s, a i 0. The reaiig case is that i which i is trasiet. Let A be the evet that s is evetually reached. For a trasiet state i, we have: a i P A X 0 i) 50) P A X 0 i, X 1 j) P X 1 j X 0 j) 51) j1 Markov P A X 1 j) P ij 52) j1 a j P ij 53) j1 The equatios for a i for a syste which we ca solve to obtai the absorptio probabilities. There are several other related properties that we ca exaie. Defiitio Expected tie to absorptio: the tie util we eter soe recurret state. Let µ i E [u. trasitios util a recurret state is etered X 0 i] 54) E [i{ 0 X is recurret} X 0 i] 55) The: µ i { 0 if i is recurret 1 + j1 P ijµ j if i is trasiet 56) These equatios ca be solved to fid the uique expected tie to absorptio. Defiitio Mea first passage tie t i : the tie util we reach recurret state s, startig fro i. { 0 if i s t i 1 + j1 P 57) ijt j otherwise These equatios ca agai be solved to fid the uique ea first passage tie. Defiitio Mea recurrece tie t s : the uber of trasitios up to the first retur to s, startig fro s: t s 1 + j1 P sj t j 58) 8

9 3.4 Cotiuous-tie Markov chais Defiitio I a cotiuous-tie Markov chai, the tie betwee trasitios is a cotiuous rado variable. We defie the followig rado variables: X state after the th trasitio 59) Y tie of the th trasitio 60) T tie betwee the 1)st ad th trasitios 61) Defiitio For soe state i, the tie T util the ext trasitio is distributed accordig to EXPONENTIAL[ν i ] where ν i is the trasitio rate, i.e. the average uber of trasitios out of i per uit tie spet i i. We assue that T is idepedet of the past history of the process ad the ext state. Defiitio A cotiuous-tie Markov chai defies the trasitio probability atrix as before. Cobiig this with the trasitio rate, we defie q ij ν i P ij to be the trasitio rate fro i to j, i.e. the average uber of trasitios fro i to j per uit tie spet i i. As essetially the sae steady-state covergece results apply for cotiuous-tie Markov chais as for discrete-tie Markov chais, we oit further discussio. 9

Generalized Semi- Markov Processes (GSMP)

Generalized Semi- Markov Processes (GSMP) Geeralized Semi- Markov Processes (GSMP) Summary Some Defiitios Markov ad Semi-Markov Processes The Poisso Process Properties of the Poisso Process Iterarrival times Memoryless property ad the residual