8 STOCHASTIC PROCESSES

Transcription

1 8 STOCHASTIC PROCESSES The word stochastic is derived from the Greek στoχαστικoς, meaning to aim at a target. Stochastic rocesses involve state which changes in a random way. A Markov rocess is a articular kind of stochastic rocess. Using discrete time the state of the rocess at time n + 1 deends only on its state at time n. The classic examle of a stochastic rocess is the random walk... Random Walk The simlest form of the random walk roblem imagines a line marked out in unit stes or aces from some origin: A erson or other object starts at the origin and then makes a sequence of stes, some to the right and some to the left, at random. It is reasonable to think of a sequence of turns. At each turn a weighted coin is tossed and if it lands heads one ste is taken to the right and if it lands tails one ste is taken to the left. In the analysis below assume: Probability of a left-ste (tails) is q Probability of a right-ste (heads) is where + q = 1 Consider a walk which consists of a total of n stes or turns. Let X be a random variable whose value, r, is the number of those n stes which are to the right. Given a total of n stes, each of which has a robability of being a right-ste, the robability of there being r right-stes is given by the Binomial distribution: P(X = r) = ( n r ) r q n r (8.1) Usually one is interested in the net dislacement. Call this k measured in net stes to the right of the origin. Clearly: Net dislacement to the right = Total right-stes Total left-stes Since the total number of left-stes is n r the net dislacement k can be exressed algebraically: k = r (n r) = 2r n hence r = 1 2 (n + k) Rewriting (8.1): P ( ( X = 1 2 (n + k)) = 1 2 ) n 1 2 (n+k) q 1 2 (n k) (8.2) (n + k) An incomlete interretation is the following: for fixed n, this is the robability that the number of right-stes is such as to give a net dislacement of k stes to the right. 8.1

2 Interretation of the Probability Since X is the number of right-stes, its value must be an integer. Therefore the robability P ( X = 1 2 (n + k)) requires the term 1 2 (n + k) to be an integer. Accordingly n + k (and hence n k) are even in the exression for the robability: P ( ( ) X = 1 2 (n + k)) n = 1 2 (n+k) q 1 2 (n k) (n + k) 1 2 Further, n k n. This accords with common sense. If the total number of stes is 2 the net dislacement must be one of the three ossibilities: two stes to the left, back to the start, or two stes to the right. These corresond to values of k = 2, 0, +2. Clearly it is imossible to get more than two units away from the origin if you take only two stes and it is equally imossible to end u exactly one unit from the origin if you take two stes. The following table shows the robabilities associated with the different ossible values of k for n = 1, 2, 3, 4: n k P(net = k) 1 1 q q 2 0 2q q 3 1 3q q q 4 2 4q q q 4 4 For given n, P(net = k) is the robability that the net dislacement is k units to the right of the origin. For each n any missing value of k (such as k = 2 when n = 3) is imossible and P(net = k) = 0. Notice that for each n the tabulated robabilities total 1. Thus for n = 3 the sum of the robabilities is q 3 + 3q q + q = (q + ) 3 =

3 Exected Dislacement and Drift Given that X is distributed Binomial(n, ), the exectation E(X) = n. This is also the exectation E ( 1 2 (n + k)), so: Hence: n = E(X) = E ( 1 2 (n + k)) = 1 2 ( n + E(k) ) E(k) = 2n n = n(2 1) = n ( 2 ( + q) ) = n( q) If = q the exected dislacement is zero but if q the exected dislacement is non-zero and the walk is not exected to end at the starting oint. This henomenon is known as drift. The exected net dislacement is roortional to the number of stes so the longer the walk the greater the drift. The term recurrent random walk is used to describe a random walk which is certain to return to the starting oint in a finite number of stes. In the resent case, the random walk is recurrent if and only if = q = 1 2. The term transient random walk is used to describe a random walk which has a non-zero robability of never returning to the starting oint. In the resent case, the random walk is transient if q. Corollary A footnote to the random walk analysis is to consider the robability of landing on the origin at ste n. Clearly n must be even and k = 0 so, from (8.2): ( ) P ( n X = 1 2 n) 1 = 2 n 1 2 n q 1 2 n, if n even 0, otherwise Remember that X is the number of right-stes. When this is 1 2n the number of right-stes is obviously the same as the number of left-stes; thus P(X = 1 2n) is exactly equivalent to P(return to origin at ste n). The Gambler s Ruin Problem Many stochastic rocesses are disguised variants of the random walk roblem. One of the best-known variants is the Gambler s Ruin roblem. You suose there are two gamblers, A and B, and they each have a ile of ound coins: A has initial caital of n B has initial caital of (a n) Play then roceeds by a sequence of turns. At each turn: Probability(A wins the turn) = Probability(B wins the turn) = q where + q = 1 At the end of each turn one ound is transferred from the loser s ile of ound coins to that of the winner. Note that the total caital is a and that stays constant. The game ends when one layer is ruined and has no money left. 8.3

4 A diagrammatic reresentation of the game at the start is: artition A s initial caital B s initial caital { }} {{ }} { u 0 u n 1 u n u n+1 u a 0 1 n 1 n n + 1 a 1 a The horizontal line is a units long and is marked off at unit intervals numbered 0, 1, 2,..., a. Point n is marked as the artition. The n units of line to the left of the artition reresent A s initial caital and the a n units of line to the right of the artition reresent B s initial caital. At each turn the artition moves one lace to the right if A wins (this outcome has robability ) and one lace to the left if B wins (this outcome has robability q). Let u n = robability that A ultimately wins starting from n Let v n = robability that B ultimately wins starting from n It is imortant to note that u n + v n may be less than unity. In this kind of roblem there is often the ossibility of there being no winner. There could be a non-zero robability of the game going on for ever with the artition moving backwards and forwards but never reaching 0 or a. The only resectable way of tackling this roblem is to determine u n and v n searately and check whether or not they sum to 1. Probability that A wins First, extend the notation u n so that, for examle: Let u n+1 = robability that A ultimately wins starting from n + 1 Let u n 1 = robability that A ultimately wins starting from n 1 Consider the osition of the artition after the first turn: The artition is at n+1 with robability and from n+1 the robability of ultimately winning is u n+1. The artition is at n 1 with robability q and from n 1 the robability of ultimately winning is u n 1. Now + q necessarily sum to 1 (unlike u n + v n about which the sum is still in doubt) so after one turn the artition must be at n + 1 or n 1 with robabilities u n+1 and u n 1 resectively so: u n = u n+1 + q u n 1 (8.3) 8.4

5 This is a homogeneous difference equation. In words: The robability of A ultimately winning from n = (robability of first turn landing on n + 1) (robability of winning from n + 1) + (robability of first turn landing on n 1) (robability of winning from n 1) The difference equation (8.3) holds for n = 1, 2,..., (a 1) but since the game ends when n = 0 or n = a the equation does not hold for u 0 or u a. Either leads to an invalid right-hand side. Points 0 and a on a random walk are known as absorbing barriers. No walk can ass these barriers. The absorbing barriers lead to the boundary conditions: u 0 = 0 u a = 1 robability that A wins when B has all the caital robability that A wins when B is out of caital From (7.4), the general solution to the difference equation is: u n = A 1 (1) n + A 2 ) n rovided q (8.4) Given the boundary conditions u 0 = 0 and u a = 1: A 1 + A 2 = 0 and A 1 + A 2 ) a = 1 Solve for A 1 and A 2 and back substitute in (8.4) to give: u n = ) n 1 a (8.5) ) 1 Further discussion of this will be ostoned until after a solution has been found for v n... Probability that B wins As with u n, first extend the notation v n so that, for examle: Let v n+1 = robability that B ultimately wins starting from n + 1 Let v n 1 = robability that B ultimately wins starting from n 1 The difference equation is set u in exactly the same way as for u n and is now: v n = v n+1 + q v n 1 8.5

6 The absorbing barriers now lead to different boundary conditions: v 0 = 1 v a = 0 robability that B wins when A is out of caital robability that B wins when A has all the caital The general solution has exactly the same form as (8.4): v n = A 1 (1) n + A 2 ) n rovided q (8.6) Given the boundary conditions v 0 = 1 and v a = 0: A 1 + A 2 = 1 and A 1 + A 2 ) a = 0 Solve for A 1 and A 2 and back substitute in (8.6) to give: v n = ) n ) a 1 ( ) q a (8.7) Now that both u n and v n are determined, their sum can be comuted: u n + v n = ) n ( 1 + q ) a ( q n ) a = 1 ) 1 Haily the sum is unity so there really is a winner. Observation about the Solutions If both layers have a reasonable amount of caital to start with the outcome is very sensitive to the sign of the difference between and q... The assumtions are that the total caital a is fairly large and that 0 n a (so that the artition is initially not close to 0 or a). If > q then q < 1 and when q is raised to the owers n and a small values result. By insection of (8.5) and (8.7) u n 1 and v n 0. A small rightward bias makes it very likely that the artition will end u at the right-hand end. If < q then q < 1 and when q is raised to the owers n and a small values result. By insection of (8.5) and (8.7) u n 0 and v n 1. A small leftward bias makes it very likely that the artition will end u at the left-hand end. The fair Case If = q the solutions found for u n and v n are invalid because the constants A 1 and A 2 in the general solutions (8.4) and (8.6) are not indeendent. 8.6

7 From (7.5), the aroriate general solution for u n now is: u n = (A 1 + A 2 n) (1) n Using the same boundary conditions u 0 = 0 and u a = 1: A 1 = 0 and A 2 a = 1 Solving and substituting gives: u n = n a The aroriate general solution for v n is likewise: v n = (A 1 + A 2 n) (1) n Using the boundary conditions v 0 = 1 and v a = 0: A 1 = 1 and 1 + A 2 a = 0 Solving and substituting gives: v n = a n a Again, haily, u n + v n = 1 and there really is a winner. Notice that both solutions show that the robability of each layer winning is equal to that layer s share of the caital. Gamblers say that the robability of winning is roortional to the initial share of the stake. The Exected Length of a Game I Assume that the length of a game is finite and: Let d n turns be the exected duration of lay when starting from n Extend this notation so that: d n+1 turns is the exected duration of lay when starting from n + 1 d n 1 turns is the exected duration of lay when starting from n 1 Consider the situation after one turn. The artition is either at n + 1 (and the robability of this outcome is ) where there are exected to be d n+1 further turns or at n 1 (and the robability of this outcome is q) where there are exected to be d n 1 further turns. Thus the exected number of turns from n is the very first turn lus either d n+1 more or d n 1 more. This gives the inhomogeneous difference equation: d n = 1 + d n+1 + q d n 1 (8.8) 8.7

8 Notice the secial case = 1 and q = 0 when the exected duration from n is simly the first turn lus d n+1 more. The boundary conditions now are d 0 = 0 and d a = 0. No further turns are to be exected if the artition has reached an absorbing barrier. From (7.6), the general solution to the new difference equation is: d n = A 1 + A 2 ) n + n q rovided q (8.9) Given the boundary conditions d 0 = 0 and d a = 0: A 1 + A 2 = 0 and A 1 + A 2 Solve for A 1 and A 2 and back substitute in (8.9) to give: d n = n q a q. 1 n ) 1 ( ) q a This is the exected duration of lay when q. The Exected Length of a Game II ) a + a q = 0 When = q the general solution (8.9) is invalid. From (7.7), the aroriate solution is: Given the boundary conditions d 0 = 0 and d a = 0: d n = A 1 + A 2 n n 2 (8.10) A 1 = 0 and A 2 a a 2 = 0 Solve for A 1 and A 2 and back substitute in (8.10) to give: d n = n(a n) This is a remarkable result. Suose the total caital a is 1000 but layer A starts with just 1 (so n = 1) whereas layer B starts with 999. Since the artition starts off just one unit from the left-hand end, there is a robability of 1 2 that the game will be over at the first turn. Nevertheless the exected duration of lay is 1.(1000 1) or 999 turns. 1 Although the robability of layer A winning is only 1000 a mere 1 investment rovides entertainment that is exected to last 999 turns! Glossary The following technical terms have been introduced: stochastic rocesses random walk drift recurrent random walk transient random walk artition absorbing barrier boundary conditions stake 8.8

9 Exercises VIII 1. Consider a variant of the Gambler s Ruin roblem. To decide the outcome of each turn, the layers are using a fat coin which sometimes lands on its edge. Such an outcome is deemed a draw for the turn and no money changes hands. There are now three robabilities relating to the outcome of each turn: Probability(A wins the turn) = Probability(B wins the turn) = q Probability(Turn is a draw) = r Necessarily + q + r = 1 and, in this question, assume that q. It is ossible that = r or that q = r but such coincidences turn out not to matter. It is not immediately clear whether the introduction of turns that have no effect alters the robabilities that A ultimately wins or that B ultimately wins but it seems likely that the duration of lay (measured in turns) will be increased. Comlete the following tasks: (a) Modify equation (8.3) and determine the robability u n that A ultimately wins starting from n. (b) In a similar manner determine the robability v n that B ultimately wins starting from n. (c) Could the two results have been redicted at the outset? In what circumstances will the game never finish? (d) Modify equation (8.8) and determine d n the duration of lay starting from n. If the ratio : q is ket constant while the value of r is increased steadily, does the duration of lay lengthen in a way that might have been redicted at the outset? 2. Suose that at each turn the two layers lay one round of the aer-scissors-stone game. The outcome may be a win for A, a win for B or a draw and money changes hands only when there is clear win. Comlete the following tasks: (a) Determine the robabilities, q and r for a round of the aer-scissors-stone game and note that = q. (b) Rework question 1 for the case = q. (c) Use the values of, q and r aroriate for the aer-scissors-stone game in the exressions derived for u n, v n and d n. Could the results have been redicted at the outset? 8.9