Mathematica reimbursement

Size: px

Start display at page:

Download "Mathematica reimbursement"

Ethelbert Joseph
6 years ago
Views:

1 Mathematica reimbursement Please hand in your Expense Approval forms (for purchase of copies of Mat hemat ica). Today will be your last chance to do this if you want to reimbursed anytime soon. Name of Person or Business To Be Reimbursed: <your name> Date: <today's date > Department: Mathematical Sciences Remit To Address: <your address > Purpose for Incurring the Expense: Purchase of soft - ware related to PI's research Total: [make entries on separate lines for purchase price, sales tax, and total] Signature (Person Incurring Expense): <your signature>

2 Lec0.nb Ergodic Markov chains (sections. and.) Ergodicity Definition.4: A Markov chain is called an er godic chain if it is possible to go from every state to every state in a finite number of steps. (Note that no absorb - ing Markov chain is ergodic, aside from the trivial absorbing Markov chains with one absorbing state and no transient states.) Example: Random walk on {,,,4} with reflection on the ends ( with probability and 4 with probabil - ity ) is ergodic. Ref 0,, 0, 0,, 0,, 0, 0,, 0,, 0, 0,, MatrixForm MatrixPower Ref,

3 Lec0.nb We say a Markov chain is r egular if there exists an N such that for all i and j, it is possible to go from s i to s j in exactly N steps; that is, the Nth power of the transition matrix P has all its entries positive (see Defi - nition.). Note that every regular Markov chain is er godic. The reflecting random walk described above is ergodic but not regular. A slightly different reflecting random walk that is regular can be obtained from our non-regular example by changing the behavior at the ends so that reflection is only partial: ParRef,, 0, 0,, 0,, 0, 0,, 0,, 0, 0,, MatrixForm MatrixPower ParRef,

4 4 Lec0.nb Recall that a function f on the state space of a Markov chain with transition matrix P = (p ij ) is called harmonic if (numbering the states,...,n) we have f (i) = j p ij f (j ) for all i, or equivalently, Pf = f (where f denotes the column vector with entries f (),...,f (n)). Theorem: If the Markov chain with transition matrix P is ergodic, the only harmonic functions are the constant functions. Proof: Let f be a harmonic function, and let M be the maximum value of f, and take x 0 such that f (x 0 ) = M. By the same reasoning that we used in the proof of the Maximum Principle for absorbing Markov chains, we can show that every successor y of x 0 satisfies f (y) = M, and that every successor z of every successor of x 0 satisfies f (z) = M, and so on. Since the chain is ergodic, every state can be reached from x 0, so we conclude that f (s) = M for EVERY state s. Hence f is a constant function.

5 Lec0.nb Since the only harmonic functions are the constant functions, the only column eigenvectors for P with eigenvalue are the multiples of the all-'s column vector. It follows that the space of row eigenvectors with eigenvalue is also -dimensional. It turns out that there is a (necessarily unique) row vector w that is both a probability vector and a -eigenvector for P. The components of w are all strictly positive. (It is a good exercise to prove via general linear algebra that if is a simple eigenvalue of the stochastic matrix P, i.e. if it has multiplicity, then the product of a row-eigenvector with eigenvalue and a columneigenvector with eigenvalue is non-zero. So there is a unique row-eigenvector whose entries sum to. But proving that the entries are positive requires more than simple linear algebra.)

6 6 Lec0.nb In the case where the chain is regular, w admits a natural interpretation: w j is the limit of p ij as n. That n is, w j is the limit, as n, of the probability that a Markov chain started in state s i that evolves via the transition matrix P will be in state s j after n steps. It is not a priori obvious that this limit exists, and indeed, for non-regular chains it does not. Regular Markov chains Theorem.7: If P is the transition matrix for a regular Markov chain, then as n, P n W where W is a square matrix whose rows are all equal to the same row-vector w, where w is both a probability vector and a solution to wp = w. Two examples: MatrixForm N MatrixPower ParRef, MatrixForm N MatrixPower,,,, , 0.47.,,, 0.749, 0.47

7 Lec0.nb P n up n

8 Lec0.nb (where I is the identity matrix of the appropriate size). When we run the Markov chain starting from an initial state governed by the probability distribution w, its state at each later time-step is also governed by w. (That is, if Pr ob(x 0 = i) = w i for all i, with X n denoting the state of the chain at time n, then we also have Pr ob(x = i) = w i for all i, Pr ob(x = i) = w i for all i, etc.) We call this a stationary Markov chain, and call w the equilibrium measure or stationary distribution. Theorem.9: Let P be the transition matrix for a regular Markov chain and v an arbitrary probability vector. Then vp n w as n. The content of Theorem.9 is that w is a stable equilibrium (everything converges towards it).

9 Lec0.nb Non-regular ergodic chains If an ergodic chain is non-regular then there is a unique positive integer d > (called the period of the chain) and an essentially unique (i.e., unique modulo cyclic relabelling) partition of the state-space of the chain into disjoint classes C, C,..., C d such that states in C can only be followed by states in C, which can only be followed by states in C, etc., and states in C d can only be followed by states in C. There is still a unique probability measure w such that wp=w, but it is best thought of as (w +w +...+w d )/ d, where w,w,...,w d are probabil - i ity measures such that w j is positive if state sj is in class C i and is zero otherwise, and where w P = w, w P = w,..., and w d P = w. For instance, our original model of reflecting random walk on {,,,4} has period, with classes {,} and {,4}; the uniform distribution on {,,,4} should be viewed as the average of the uniform distribution on {,} and the uniform distribution on {,4}.

10 0 Lec0.nb The stationary distribution

11 Lec0.nb s i s i s s s n s i s i

12 Lec0.nb walk on {,,,4} had edges joining to, to, and to 4. So deg() =, deg() =, deg() =, and deg(4) =, and (,,,) is an invariant vector. To turn it into a probability vector, divide by 6. Check: w 6, 6, 6, 6 6,,, 6 w.ref 6,,, 6 MatrixForm w 6 6 If an array is not a list of lists but just a list, Mathematica will treat it as a column vector. MatrixForm w 6 6 Compare: MatrixForm 6,,, MatrixForm 6,,, 6 6 6

13 Lec0.nb Example : Our example of partially reflecting random walk on {,,,4} had additional edges joining to itself and 4 to itself. So deg() =, deg() =, deg() =, and deg(4) =, and (,,,) is an invariant vector. To turn it into a probability vector, divide by. Check: 4, 4, 4, 4.ParRef 4, 4, 4, 4 Note also that the associated transition matrix ParRef is doubly stochastic: ParRef Proof of Claim: Use mass-flow. If we put mass deg(s i ) at state i for all i, and let it flow, then unit of mass flows along every edge in both directions, so the total mass at each site doesn't change.

14 4 Lec0.nb Sometimes the transition matrix P is sparse (many entries are zeroes), so we can find a fixed vector of P by solving the associated system of linear equations by hand. Example: Sparse 0,,, 0,, 0, 0,, 0,,, 0,, 0, 0, Clear a, b, c, d MatrixForm a, b, c, d.sparse MatrixForm a, b, c, d b d a c a c b a b c d b a c

15 Lec0.nb b Set d =. b b = d = d, so b =. + d = a, so a =. a + c = c, so a = c Consistency check: a + c = b., so a = c, so c =. Hence the invariant probability vector is ( ). Or you can let Mat hemat ica do it for you: NullSpace Transpose Sparse IdentityMatrix 4 The pinned stepping stone model The stepping stone model, an example of an absorbing Markov chain, becomes an example of a regular Markov chain if we "pin" some of the sites, making at least one site permanently black and at least one site permanently white. Here's some code for the pinned stepping stone model in which the top row and bottom row of a 0-by-0 grid are pinned to opposite colors: Size : 0

16 6 Lec0.nb Board Table Table Which i, 0, i Size,, True, RandomInteger, j,, Size, i,, Size MatrixPlot Board, ColorFunction "Monochrome" 0 0 RandDir : random direction in grid, 0, 0,,, 0, 0, RandomInteger, 4 Wrap x_ : wrap coordinates Which x 0, Size, x Size,, True, x Recolor : recolor board Module NewDir, a, b, NewDir RandDir ; a RandomInteger, Size, RandomInteger, Size ; b Wrap a NewDir, Wrap a NewDir ; Board a, a Board b, b ; Return Board ; BoardHistory : Table Recolor, n,, 000 ;

17 Lec0.nb Animate MatrixPlot BoardHistory n, ColorFunction If 0, Red, Blue &, ColorFunctionScaling False, n, Range, 000 n

18 Lec0.nb N Sum BoardHistory n 00, n,, MatrixPlot Sum BoardHistory n 00, n,, 00, ColorFunction "Grayscale" ArrayPlot::cfun : Value of option ColorFunction Grayscale is not a valid color function, or a gradient ColorData entity. 0 0 Why is it so slow? And what should I use instead of "Grayscale"?

19 Lec0.nb Laws of large numbers Theorem. (Weak Law of Large Numbers for Ergodic Markov Chains): Let H j n be the proportion of times in the first n steps that an ergodic chain started from state s i is in state s j ; i.e., it's / n times the number of visits to state s j in the first n steps (assuming the chain started in s i ). Then for any Ε > 0, Pr ob( H j n - wj > Ε) 0 regardless of i. Strong Law of Large Numbers for Ergodic Markov n Chains: With the random variable H j defined as above, we have H j n wj with probability. Special case: If our ergodic Markov chain has the transition matrix then this is just the strong law of large numbers for coin-tossing: if you toss a fair coin forever, then with probability, the proportion of heads up to time n approaches as n.

20 0 Lec0.nb But what does this really mean? For simplicity, let's choose one particular value of j and suppress it from the notation, writing H n w. Recall that each H n is a random variable, that is, a function H n (Ω) for Ω in. Recall that H n (Ω) w is defined as the assertion H n w

21 Lec0.nb (we'll see in a minute why we want to restrict to Ε's that are reciprocals of integers). Hence we can write the event {Ω: H n (Ω) w} as k m n m {Ω: H n (Ω) - w < / k }. So how does probability theory work with infinite unions and infinite intersections? "Fact": If E, E,... are events in, () Prob( n E n ) = lim N Prob( N n E n ) and () Prob( n E n ) = lim N Prob( N n E n ). (I call it a "Fact" with quotation marks because in a sense it's part of the way we DEFINE the probabili - ties of events like these. Then one needs a theorem that says that using these formulas can never lead to a cont r adict ion.)

22 Lec0.nb X 0 X X n q ij 0 q ij q ij n X 0 X q ij 0 q ij n ij H n X 0 X X 0 X

23 Lec0.nb Example: Toss a coin until it comes up Tails. This is a Markov chain with transition matrix 0 Let X(Ω) be the time until the first occurrence of Tails. X(Ω) is undefined if Ω is the outcome Heads, Heads, Heads,... but this outcome has probability ( )( )( )... = 0. Ε 0 k

24 4 Lec0.nb n

25 Lec0.nb number of coin-tosses ending in infinitely many Heads or infinitely many Tails is likewise countable, so it has probability 0; so if we look at where the correspon - dence between real numbers and coin-toss experi - ments fails, we're dealing with a leftover set of reals of measure 0 that fails to correspond to a set of cointoss outcomes with probability 0. For purposes of computing probabilities, expectations, etc., sets of measure or probability 0 can be safely ignored. Probabilists say "For a set of Ω of probability, H n (Ω) w" or "For almost every Ω, H n (Ω) w" or "H n (Ω) w almost surely", often abbreviating "almost every" with "a.e." and "almost surely" with "a.s.", to mean "outside of a set of Ω's of probability 0".

26 6 Lec0.nb S j n nhj n Σ j Σ j nw j r S n j nwj Σ j n s Π s e x r H j n w j Σ j

27 Lec0.nb Pr ob[r H n j wj Σ j n s] converges to that same integral. If we try to use the random quantity H j n as an esti - mate of the non-random but unknown quantity w j, we expect error on the order of C/sqrt( n) for some C.

28 Lec0.nb First passage and recurrence times Suppose P is the stochastic matrix of transition probabilities for an ergodic Markov chain, and w is the unique stationary measure in vector form, satisfying the three conditions w > 0 (that is, each component of w is positive), i w i = (that is, w is a probability vector), and wp = w (that is, w is invariant under P). Note that the second condition can also be written as w =, where on the left side of the equation is the all-'s column vector (not to be confused with the scalar on the right side of the equation). Let W be the square matrix each of whose rows is w. Then wp = w implies WP = W, and also WP n = W for all n. P,,,

29 Lec0.nb Eigenvalues P, 6 Eigenvectors P What Mathematica is trying to give us here is a list of vectors forming a basis for the column eigenspaces. These vectors are, and, construed as column vectors. Mathematica unhelpfully displays this list of vectors as a matrix whose rows correspond to the desired column vectors Check:,,,.,,,,,., 4, 6 If that last one doesn't scream "eigenvector" at you try this:,,,.,, 6, 6 Is there a way to force Mathematica to display a list of lists as a list of lists like,,,? Anyway we want the row eigenvectors not the column eigenvectors. Eigenvectors Transpose P this is the one we want The first row is the one that corresponds to the eigenvalue, and we rescale it by to get a probability vector. w,, w.p, W w, w ; MatrixForm W

30 0 Lec0.nb MatrixForm W.P MatrixForm P.W Claim: MW = W for every stochastic matrix M. Proof: Every column c of W is a multiple of the all 's vector and hence satisfies Mc = c. Consequences:PW = W and WW = W. Claim: (P-W) n = P n - W. Proof: By induction. It's clearly true for n=, and if it's true for n, then (P-W) n = (P-W)(P-W) n = (P-W)(P n -W) = P n - WP n - PW + WW = P n - W - W + W = P n - W, as was to be proved.

31 Lec0.nb The first passage time from i to j is a random vari - able, namely, the time it takes for a chain started in state i to first reach state j. (If i =j, the first passage time is taken to be 0.) The mean first passage time is the expected value of the first passage time, and is denoted by m i,j or m ij. This is always finite (since we are assuming in this part of the course that our Markov chain has finitely many st at es). One way to compute the mean first passage time is to turn j into an absorbing state (by giving the transition j j probability and every other transition j k probability 0) and then compute the expected absorption time starting from i. Example: Random walk on {,,,4} with complete reflec - tion at the ends (i.e., with probability and 4 with probability ). What is m, (the expected time from to )?

32 Lec0.nb Let h(x) be the expected time until absorption at. We have h() = 0, h() = a, h() = b, and h(4) = c. Familiar considerations lead us to the equations a = + (0+b)/, b = + (a+c)/, c = + b. Solve a 0 b, b a c, c b, a, b, c a, b, c 9 Trick: Note that the expected time until absorption for random walk on {,,,4} with absorption at and reflection at 4 is the same as the expected time until absorption for random walk on {,,,4,,6,7} with absorption at and 7. (A random walk on {,,,4,,6,7} turns into a random walk on {,,,4} if we collapse to, 6 to, and 7 to.) So we can reduce this problem to ordinary gambler's ruin and use the formulas you found for problem..6: a =, b = 4, c =. Here's the Q-matrix for the 4-state chain, if we make the state absorbing:

33 Lec0.nb Q 0,, 0,, 0,, 0,, 0 ; MatrixForm Q Inverse IdentityMatrix Q 4 4.,, 9 r i

34 4 Lec0.nb One way to see this is to show that (.4) r i = + k p ik m ki (hint: where are you after step?) and then use the fact that every mean first passage time is finite. A helpful companion to the above formula is (.) m ij = + k p ik m kj for i j. (Note that for both (.4), when k=j the term p ik m kj vanishes since m jj = 0. Likewise for (.), when k=i the term p ik m ki vanishes since m jj = 0.) We will use these formulas to solve for r i and the m ij 's, using matrix equations. Mean first passage matrix and mean recurrence matrix m ii

35 Lec0.nb r i m ii k p ik m ki r i m ij k p ik m kj s i r i w i

36 6 Lec0.nb Hence wc (a row vector all of whose entries are equal to ) equals wd (a row vector whose ith entry is equalto w i r i ), so =w i r i as claimed. With this information we can solve for D, and then we can find M = (I -P) (C-D). Or can we?... Is I -P invertible, for every ergodic Markov chain? Quite the contrary: (I -P) = - = 0 (where is the column-eigenvector for the eigenvalue ), so I -P has non-trivial kernel and hence is not invertible. What to do? The fundamental matrix for an ergodic chain

37 Lec0.nb we have 0 = w (I - P + W) x = w x. Therefore, W x = 0 and (I - P) x = 0. But the second of these implies that x = Px, which can only happen if x is a constant vector (the space of harmonic functions for an ergodic Markov chain with finite state-space is -dimensional). Since wx = 0, and w has strictly positive entries, we see that x = 0. This completes the proof. We write Z = (I - P + W) and call it the fundamental matrix of the ergodic Markov chain. Grinstead and Snell prove that m ij = (z jj - z ij ) r j with r j = / w j. Our two-state example: MatrixForm P MatrixForm W

38 Lec0.nb Z Inverse IdentityMatrix P W ; MatrixForm Z 7 r,, Z, Z, r Z, Z, r Check: The transit time from to is distributed like a geometric random variable with p = /, so its expected value is. Likewise, the transit time from to is distributed like a geometric random variable with p = /, so its expected value is. Let's return to random walk on {,,,4} and computing the mean transit time from to using Z: P4 0,, 0, 0,, 0,, 0, 0,, 0,, 0, 0,, 0 ; MatrixForm P Eigenvalues P4,,, The second eigenvalue is so the second eigenvector will be the one we want.

39 Lec0.nb Eigenvectors Transpose P4 That is the eigenvectors are,,,,,,,,,,,,,,, w4 6 6,,, 6 W4 w4, w4, w4, w4 ; MatrixForm W Z4 Inverse IdentityMatrix 4 P4 W4 ; MatrixForm Z Z4, Z4, w4

40 40 Lec0.nb = (I - W) + (P - W) + ( P - W) +... but we won't be exploring those sorts of applications. (With this definition, z ij has a natural interpretation: it's the expected excess number of times you visit j when you start at i, as compared with starting from equilibrium.) For the homework, use the definition of Z given in the book.

Absorbing Markov chains (sections 11.1 and 11.2)

Absorbing Markov chains (sections 11.1 and 11.2) What is a Markov chain, really? That is, what kind of mathematical object is it? It's NOT a special kind of stochastic matrix (although we do use stochastic