Stochastic processes and

Size: px

Start display at page:

Download "Stochastic processes and"

John Burns
6 years ago
Views:

Stochastic processes and Markov chains (part I) Wessel van Wieringen w.n.van.wieringen@vu.

1 Stochastic processes and Markov chains (part I) Wessel van Wieringen nl Department of Epidemiology and Biostatistics, VUmc & Department of Mathematics, VU University Amsterdam, The Netherlands

2 Stochastic processes

3 Stochastic processes Example 1 The intensity of the sun. Measured every day by the KNMI. Stochastic variable X t represents the sun s intensity at day t, 0 t T. Hence, X t assumes values in R + (positive values only).

Stochastic processes Example 2 DNA sequence of 11

4 Stochastic processes Example 2 DNA sequence of 11 bases long. At each base position there is an A, C, G or T. Stochastic variable X i is the base at position i, i = 1,,11. In case the sequence has been observed, say: (x 1, x 2,, x 11 ) = ACCCGATAGCT, then A is the realization of X 1, C that of X 2, et cetera.

5 Stochastic processes position t t+1 t+2 t+3 base A A T C A A A A G G G G C C C C T T T T position t position t+1 position t+2 position t+3

6 Stochastic processes Example 3 A patient s heart pulse during surgery. Measured continuously during interval [0, T]. Stochastic variable X t represents the occurrence of a heartbeat at time t, 0 t T. Hence, X t assumes only the values 0 (no heartbeat) and 1 (heartbeat).

7 Stochastic processes Example 4 Brain activity of a human under experimental conditions. Measured continuously during interval [0, T]. Stochastic variable X t represents the magnetic field at time t, 0 t T. Hence, X t assumes values on R.

8 Stochastic processes Differences between examples Discrete X t Continuous Discret te Example 2 Example 1 ntinuou Con Time s Example 3 Example 4

9 Stochastic processes The state space S is the collection of all possible values that the random variables of the stochastic process may assume. If S = {E 1, E 2 2,, E s s} } discrete, then X t is a discrete stochastic variable. examples 2 and 3. If S = [0, ) ) discrete, then X t is a continuous stochastic variable. examples 1 and 4.

10 Stochastic processes Stochastic process: Discrete time: t = 0, 1, 2, 3,. S = {-3, -2, -1, 0, 1, 2,..} P(one step up) = ½ = P(one step down) Absorbing state at

11 Stochastic processes The first passage time of a certain state E i in S is the timetatwhichx t X t = E i for the first time since the start of the process. What is the first passage 7 6 time of 2? And of -1?

12 Stochastic processes The absorbing state is a state E i for which the following holds: if X t = E i than X t+1 = E i for all s t. The process will never leave state E i The absorbing state is

13 Stochastic processes The time of absorption of an absorbing state is the first passage time of that state The time of absorption is

14 Stochastic processes The recurrence time is the first time t at which the process has returned to its initial state The recurrence time is

15 Stochastic processes A stochastic process is described by a collection of time points, the state space and the simultaneous distribution of the variables X t, i.e., the distributions of all X t and their dependency. There are two important types of processes: Poisson process: all variables are identically and independently distributed. Examples: queues for counters, call centers, servers, et cetera. Markov process: the variables are dependent in a simple manner.

16 Markov processes

17 Markov processes A 1 st order Markov process in discrete time is a sto- chastic process {X t } t=1,2, for which the following holds: P(X t+1 =x t+1 X t =x t,, X 1 =x 1 ) = P(X t+1 =x t+1 X t =x t ). In other words, only the present determines the future, the past is irrelevant. From every state with probability 0.5 one step up or down (except for state -3)

18 Markov processes The 1 st order Markov property, i.e. P(X t+1 =x t+1 X t =x t,, X 1 =x 1 ) = P(X t+1 =x t+1 X t =x t ), does not imply independence between X t-1 and X t+1. In fact, often P(X t+1 =x t+1 X t-1 =x t-1 ) is unequal to zero. For instance, P(X t+1 =2 X t-1 =0) = P(X t+1 =2 2 X t =1) * P(X t =1 X t-1 =0) = ¼ ¼

19 Markov processes An m-order Markov process in discrete time is a stochastic process {X t } t=1,2, for which the following holds: P(X t+1 =x t+1 X t =x t,, X 1 =x 1 ) = P(X t+1 =x t+1 X t =x t,,x t-m+1 = x t-m+1). Loosely, the future depends on the most recent past. Suppose m=9: 9 preceeding bases t+1 CTTCCTCAAGATGCGTCCAATCCCTCAAAC Distribution of the t+1-th base depends on the 9 preceeding ones.

20 Markov processes A Markov process is called a Markov chain if the state space is discrete, i.e., is finite or countable. In these lecture series we consider Markov chains in discrete time. Recall the DNA example.

21 Markov processes The transition probabilities are the but also the P(X t+1 =x t+1 X t =x t ), P(X t+1 =x t+1 X s =x s ) for s < t, where x t+1, x t in {E 1,, E s }=S S. P(X t+1 =3 X t =2) = 0.5 P(X t+1 =0 X t =1) = 0.5 P(X t+1 =5 X t =7) =

22 Markov processes A Markov process is called time homogeneous if the transition probabilities are independent of t: For example: P(X t+1 =x 1 X t =x 2 ) = P(X s+1 =x 1 X s =x 2 ). P(X 2 =-1 1 X X 1 =2) = P(X 6 =-1 1 X X 5 =2) P(X 584 =5 X 583 =4) = P(X 213 =5 X 212 =4)

23 Markov processes Example of a time inhomogeneous process h(t) Bathtub curve t Infant mortality Random failures Wearout failures

24 Markov processes Consider a DNA sequence of 11 bases. Then, S={a, c, g, t}, X i is the base of position i, and {X i } i=1,, 11 is a Markov chain if the base of position i only depends on the base of position i-1, and not on those before i-1. If this is plausible, a Markov chain is an acceptable model for base ordering in DNA sequences. A C G T state diagram

25 Markov processes In remainder, only time homogeneous Markov processes. We then denote the transition probabilities of a finite time homogeneous Markov chain in discrete time {X t } t=1,2, with S={E 1,, E s } as: P(X t+1 =E j X t =E i ) = p ij (does not depend on t). Putting the p ij in a matrix yields the transition matrix: The rows of this matrix sum to one.

26 Markov processes In the DNA example: where p AA = P(X t+1 = A X t = A) and: p AA + p AC + p AG + p AT = 1 p AA A p AC p CA p GC p GC p GA p AG p CA + p CC + p CG + p CT = 1 G T p TG et cetera p GG p GA p AG p GT p TC C p CC p CT p TT

27 Markov processes The initial distribution π = (π 1,,π s ) T gives the probabilities of the initial state: π i = P(X 1 = E i ) for i = 1,, s, and π π s = 1. Together the initial distribution π and the transition g matrix P determine the probability distribution of the process, the Markov chain {X t } t=1,2,

28 Markov processes We now show how the couple (π, P) determines the probability distribution (transition probabilities) of time steps larger than one. Hereto define for n = 2 : general n :

29 Markov processes For n = 2 : just the definition

30 Markov processes For n = 2 : use the fact that P(A, B) +P(A P(A, B C )=P(A)

31 Markov processes For n = 2 : use the definition of conditional probability: P(A, B) = P(A B) P(B)

32 Markov processes For n = 2 : use the Markov property

33 Markov processes For n = 2 :

34 Markov processes In summary, we have shown that: In a similar fashion we can show that: I d th t iti t i f t i th In words: the transition matrix for n steps is the onestep transition matrix raised to the power n.

35 Markov processes The general case: is proven by induction to n. One needs the Kolmogorov-Chapman equations: for all n, m 0 and i, j =1,,S.

36 Markov processes Proof of the Kolmogorov-Chapman equations:

37 Markov processes Induction proof of Assume Then:

38 Markov processes A numerical example: Then: matrix multiplication ( rows times columns )

39 Markov processes Thus: = P(X t+2 = I X t = I) is composed of two probabilities: probability of this route plus probability of this route I I I I I I II II II II II II position t position t+1 position t+2 position t position t+1 position t+2

40 Markov processes In similar fashion we may obtain: Sum over probs. of all possible paths between 2 states: I I I I I I II II II pos. t pos. t+1 pos. t+2 II II II pos. t+3 pos. t+4 pos. t+5 Similarly:

41 Markov process How do we sample from a Markov process? Reconsider the DNA example. A C G T p AA A p AC p CA p GC p GC p GA p AG p GA p AG p TC C p CT p CC f r o m A C G T p GT to G T A C G T p GG p TG p TT

42 Markov process 1 Sample the first base from the initial distribution π: P(X 1 =A) = 0.45, P(X 1 =C) = 0.10, et cetera Suppose we have drawn an A. Our DNA sequence now consists of one base, namely A. After the first step, our sequence looks like: A 2 Sample the next base using P given the previous base. Here X 1 =A, thus we sample from: P(X 2 =A X 1 =A) = 0.1, P(X 2 =C X 1 =A) = 0.1, et cetera. In case X 1 =C, we would have sampled from: P(X 2 =A X 1 =C) = 0.2, P(X 2 =C X 1 =C) = 0.3, et cetera.

43 Markov process 2 After the second step: AT 3 The last step is iterated until the last base. After the third step: ATC 9 After nine steps: ATCCGATGC GC

44 Andrey Markov

45 Andrey Markov

46 Andrey Markov In the famous first application of Markov chains, Andrey Markov studied the sequence of 20,000 letters in Pushkin s poem Eugeny Onegin, discovering that - the stationary vowel probability is 0.432, - the probability of a vowel following a vowel is 0.128, and - the probability of a vowel following a consonant is Markov also studied the sequence of 100,000 letters in Aksakov s novel The Childhood of Bagrov, the Grandson. This was in 1913, long before the computer age! Basharinl et al. (2004)

47 Parameter estimation

48 Parameter estimation T A G T T C G G A A G C A C G T G G A T G A G G A A C C C G C C A C C C C A T C C G C C T G G T G T T T G A T A G C T G T A T T C G G G C A T G G C G G C C C A C G C T A T T C G T C T G T A C G C A A G C C T C G T T T T A T C C C C G C C G C C A G A G T A C C G C A G C A C C C G G C G C G A C G G G A C C G C G C C G C G G C C A A C T A G C G C A G C C G C A G A C C G C C A C G T G C G C A G G A C T T C C C A C C C T C A C C C T T C C G C T C A C C C T T A T C G G A G G G C A C T C G C A C G G A T A G T T G G T A G G G G G A C A T C C C A G G G T A G C C G C A C G A T G C G G A T A G G G G T A T T T A T G G C G C C C T C A C C C A G G C G A T C T C G G G C C C G G C T G G G C G C C G G G C G G A C C C A G A T A G A T C G G T G C C C C G G C C C C A A C C G A A G G G T A G G A C T G T C G G A C C T G C G G C G C C C G C G A C G A T A C G A C A G A C A G G A A A G A A A G C A T C C A G G G A C G G C G C G C T T G C T G G A C T G A C C A G C C A C T C C A T G G G G T C C G T A C C C G C C C G G G G G C A C T T T C G C C C C C G C G G A G A G C T G A G A C A A T C G A T G T G C C C A G C C G A G G G G G C C C G C C A G C C C A G G G G A C C G G A C T A A G G G C C A C C G T A G G G T A C A G C A A C C C C G G A C A C T G G C G A C C C C G C A C C A C C C C C A C C C G T A G C T C A G T A C A C G A G G A G A G G A A C C C G C G C G A A G C C C C T G G G C G C T C T T G C T G G T C A G A A A T A C C G T C G T T T A T C C G T G G A C T C G T A C G A C C T G C C A C C C T G G C A G G A C T G A C T G C C G C C A C C C C C C G T C C A C T A C G G A A A G T G A G G G G G C C A A C C G C G T A G G A C A A G A G G G T C T C G C C T C G G G G G G T T G C C G G C C G G G G C A G G T T C A G G A G T T A T T T C C A G G C C C C A C G G A C T G T T T T T C C G T C C C A G A T G G G A T G G A A T T A A C G C A C G C C A T C A C A C C T G C A C G C C G G C C C C C T G G A G A C C G T G C G C A A How G A G G G do G G C we T A C estimate C C A C T T T C T the G C G T parameters A T A T A C G T T A G G A C T C G A A G C C C A G G G T A G T G A C G G A C G C G C T C G A T C C C C C G T T C A C C G G G A C C C T G C C G G A T T T A A G G C G C G T G C C G G G C C C G T G C T C A C C C C C G G A C of A G a C C Markov A G A C A C A T chain T T C T C C from T G C G A the G T G G data? T G C C G C T T A C C G G G T C T G A T A C G G A G T A G A G G C C C C C C G G C T A T A G C C A C A T G C C T A C A T C C T C G C G T C A C A C G G G C T C A G T T C G C C T G G G G A C G G A T G C C C C G A T C G C G T G G G A T A C C C G C G G C T T A C G A A T C A A A C A C G C C G C G G C G T G C G G G C G G G G A G G C C T A G G C T T C G G G A G G G G C A T T G C G C C C T C G A T A C G A C C G G G T C C C C G C C C A C G C T C T T T G G G T G T C G G C G C C G C A C T Maximum C C C T G C G T G C likelihood! C G G T C T C C T T G T G T A A C G G T G A T A C G G G G G T A C C G G T G T G C C A G C A G G C C T C T A A G C T A T A G C C C C T G C C C A C C T T C G T G A C A C C C G G G T T G C C C C C A C G T C C C G G T A G T C A A A G G G C T A C C C A G C G C C G C A T C G A C G T C C C T T G G G T G G C G G T T A A C A G C A G A T C T T T G C C A G A A G T C G C G C A A C A A A G G C A C T C T G C A G C C T G G C C A C C A G G C G G C C C T C T C G A G C G G C C A G C A G G A C C C G G G C A C G G C A C T C T A T A G C A C G G T A G C C A C T G C A G T A C C G T A G G A C C A C C C G G T G A C A C A G T G G A G A C C G G G G T T A C A C C A C T C C C C C G A G G T T A G C C A T C C T C G A G A A C G A C T A A G C G C G G G A A C G A G G C G A G C A G C C A A T A C G T G C G A A G A A C G T C T G C G G A G C G C C A T A C A C G A T A A C T A C G G T T T A T A G G T A C G T C G C A T G G A G G T T C G C G G C T G T A A A A C C C A C A C A T A C T C G G T C C C A C A C G G G T G T A A G C A C T C G T C A G T C C G C A G C C T C G T C T C G C C G A G G A T C A C G A G T C G G T C G C C C T C A A A G G A G C T T C A A G T T G G A C C C T G A C C G G G G A C T T G A G A A G T A A G C A C G C C C G A C G A A G C G G G G C C A G C A C C G G T A A A C C T G C G A C A A G T C C T T T A C C G C T A G G G T A C T G C G T G T C A A C C A G T C A C G A G T G C C C C A C C C T G G C G A C C C G G C G A G G A C C G C C C C C T T C C G G G A A G G T C A G A C C T G T G A G C A G A G A G C C A G G A T G C C A C T A A C G C T C A C C G T A C C T T C C G C C T G A C T C G C C G G T C C G C G C G C G A G C A A C C C C T A A C C A C C A A C C G G C G G A G G A C C C A G C A G A A T A T G G C G A A C C G G C C C C C G C G A C T G C G C G A T C G G C A G A C T C C T G T G C G G G G A G C A T A C A G G C T C T C C T C G G A G C G T G G C T C A G T G C C G C G T G G G C C G C T T T C T C C A G C C C C T A C A C G C C C A C T A C T C G A A C G T T T T T G T G G G C C T G A C T T G C C G C G T A C C C G C G C C T T T C G C C C T G A C G A C C C G C G T T A C C T A T G A G G G A T C A T G T G G G G T G G G G C C C C G G G T C C A C G C T A T C G A C G T A T G G T T G C T G C G T C A A C T C A C G C T C G A T G T A C A C G C C G T C C T C G T A A C A G C G A C C A C G C T C C C G A A G C T G G A T A A A C G A C T T C A G C A A T A A G G G A C G G T C A G G G G G G A G G A C G T A C G T A C T G G G C A G G C A G G C G G C T G G C A C C C T G A G G C A A A T G C G G A C G C G A A C C A T C C A A G A C G G G C G A C A A A A C G A C C G G A A G A G C G A G G A G A C G T A G C A C C G T T T C C G C G G C T T C C G A A C G G A G C T A A C A G A G A C C C G C C G G A T C T C G T T G G T A C C G A G C C G C C G T A T A C A T C A C A C C T G C A C G G C C A G T G C C T G C C T T A G T T C G G G C G C T A G C C C G T C G T C C C T A T T T A A C T G C G C G A G T G G C G A G C T C G G C

49 Parameter estimation Using the definition of conditional probability, the likelihood can be decomposed as follows: where x 1,, x T in S = {E 1,, E S }.

50 Parameter estimation Using the Markov property, we obtain:

51 Parameter estimation Furthermore, e.g.: where only one transition probability at the time enters the likelihood, due to the indicator function.

52 Parameter estimation The likelihood then becomes: where, e.g.,

53 Parameter estimation Now note that, e.g., Or, p AA + p AC + p AG + p AT = 1. p AT = 1 - p AA - p AC - p AG. AT AA AC AG Substitute this in the likelihood, and take the logarithm to arrive at the log-likelihood. likelihood Note: it is irrelevant which transition probability is substituted.

54 Parameter estimation The log-likelihood: likelihood:

55 Parameter estimation Differentiation the log-likelihood likelihood yields, e.g.: Equate the derivatives to zero. This yields four systems of equations to solve, e.g.:

56 Parameter estimation The transition probabilities are then estimated by: Verify 2 nd order part. derivatives of log-likehood are negative!

57 Parameter estimation If one may assume the observed data is a realization of a stationary Markov chain, the initial distribution is estimated by the stationary distribution. If only one realization of a stationary Markov chain is available and stationarity cannot be assumed, the initial distribution is estimated by:

58 Parameter estimation Thus, for the following sequence: ATCGATCGCA, tabulation yields The maximum likelihood estimates thus become:

59 Parameter estimation In R: > DNAseq <- c("a", "T", "C", "G", "A", "T", "C", "G", "C" "A") > table(dnaseq) DNAseq A C G T > table(dnaseq[1:9], DNAseq[2:10]) A C G T A C G T

60 Testing the order of the Markov chain

61 Testing the order of a Markov chain Often the order of the Markov chain is unknown and needs to be determined from the data. This is done using a Chi-square test for independence. Consider a DNA-sequence. To assess the order, one first tests whether the sequence consists of independent letters. Hereto count the nucleotides, di-nucleotides and tri-nucleotides in the sequence:

62 Testing the order of a Markov chain The null hypothesis of independence between the letters 2 of the DNA sequence evaluated by the χ 2 statistic: which has (4-1) x (4-1) = 9 degrees of freedom. If the null hypothesis cannot be rejected one would fit a If the null hypothesis cannot be rejected, one would fit a Markov model of order m=0. However, if H 0 can be rejected one would test for higher order dependence.

63 Testing the order of a Markov chain To test for 1 st order dependence, consider how often a di- nucleotide, e.g., CT, is followed by, e.g., T. The 1 st order dependence hypothesis is evaluated by: where This is χ 2 distributed with (16-1) 1) x(41)=45dof (4-1) 45 d.o.f..

64 Testing the order of a Markov chain A 1 st order Markov chain provides a reasonable description of the sequence of the hlye gene of the E.coli bacteria. Independence test: Chi-sq stat: , p-value: st order dependence test: Chi-sq stat: , p-value: The sequence of the prra gene of the E.coli bacteria requires a higher order Markov chain model. Independence test: Chi-sq stat: , p-value: e-04 1 st order dependence test: Chi-sq stat: , p-value: e e-08

65 Testing the order of a Markov chain In R (the independence case, assuming the DNAseq-object is a character-object j containing the sequence): > # calculate nucleotide and dimer frequencies > nuclfreq <- matrix(table(dnaseq), ncol=1) > dimerfreq <- table(dnaseq[1:(length(dnaseq)-1)], DNAseq[2:length(DNAseq)]) > # calculate expected dimer frequencies > dimerexp <- nuclfreq %*% t(nuclfreq)/(length(dnaseq)-1) > # calculate test statistic and p-value > teststat <- sum((dimerfreq - dimerexp)^2 / dimerexp) > pval <- exp(pchisq(teststat, 9, lower.tail=false, log.p=true)) Exercise: modify the code above for the 1 st order test. t

66 Application: sequence discriminationi i

67 Application: sequence discrimination We have fitted 1 st order Markov chain models to a representative coding and noncoding sequence of the E.coli bacteria. P coding A C G T A C G T P noncoding A C G T A C G T These models can used to discriminate between coding and noncoding sequences.

68 Application: sequence discrimination For a new sequence with unknown function calculate the likelihood under the coding and noncoding model, e.g.: These likelihoods lih are compared by means of their log ratio: the log-likelihood ratio.

69 Application: sequence discrimination If the log likelihood LR(X) of a new sequence exceeds a certain threshold, the sequence is classified as coding and non-coding otherwise. Back to E.coli To illustrate the potential of the discrimination approach, we calculate the log likelihood ratio for large set of known coding and noncoding sequences. The distribution of the two sets of LR(X) s are compared.

70 Application: sequence discrimination noncoding sequences coding sequences frequenc cy log likelihood ratio

71 Application: sequence discrimination flip mirror violinplot noncoding coding

72 Application: sequence discrimination Conclusion E.coli example Comparison of the distributions indicate that one could discriminate reasonably between coding and noncoding sequences on the basis of simple 1 st order Markov. Improvements: - Higher order Markov model. - Incorporate more structure information of the DNA.

73 References & further reading

74 References and further reading Basharin, G.P., Langville, A.N., Naumov, V.A (2004), The life and work of A.A. Markov, Linear Algebra and its Applications, 386, Ewens, W.J, Grant, G (2006), Statistical Methods for Bioinformatics, Springer, New York. Reinert, G., Schbath, S., Waterman, M.S. (2000), Probabilistic and statistical properties of words: an overview, Journal of Computational Biology, 7, 1-46.

75 This material is provided under the Creative Commons Attribution/Share-Alike/Non-Commercial License. See for details.

Stochastic processes and

Stochastic processes and Markov chains (part II) Wessel van Wieringen w.n.van.wieringen@vu.nl wieringen@vu nl Department of Epidemiology and Biostatistics, VUmc & Department of Mathematics, VU University