CS 4491/CS 7990 SPECIAL TOPICS IN BIOINFORMATICS

Transcription

1 CS 4491/CS 7990 SPECIAL TOPICS IN BIOINFORMATICS * The contents are adapted from Dr. Jean Gao at UT Arlington Mingon Kang, Ph.D. Computer Science, Kennesaw State University

2 Primer on Probability Random Variable Probability Distribution Independence Bayes Theorem Means and Variances

3 Definition of Probability frequentist interpretation: the probability of an event is proportion of the time events of same kind will occur in the long run eamples the probability my flight to Chicago will be on time the probability this ticket will win the lottery the probability it will rain tomorrow always a number in the interval [0,1] 0 means never occurs 1 means always occurs

4 Sample Spaces sample space: a set of possible outcomes for some event eamples flight to Chicago: {on time, late, cancelled, accident} lottery:{ticket 1 wins, ticket 2 wins,,ticket n wins} weather tomorrow: {rain, not rain} or {sun, rain, snow} or {sun, clouds, rain, snow, sleet} or

5 Random Variables random variable: a variable representing the outcome of an eperiment eample X represents the outcome of my flight to Chicago we write the probability of my flight being on time as Pr(X = on-time) or when it s clear which variable we re referring to, we may use the shorthand Pr(on-time)

6 Notation Uppercase letters and capitalized words denote random variables Lowercase letters and uncapitalized words denote values We ll denote a particular value for a variable as follows Pr( X ) Pr( Fever true) we ll also use the shorthand form Pr( ) for Pr( X ) for Boolean random variables, we ll use the shorthand Pr( fever ) for Pr( Fever true) Pr( fever ) for Pr( Fever false )

7 Probability Distributions if X is a random variable, the function given by Pr(X = ) for each is the probability distribution function p(x) of X (also called probability mass function, pmf.) requirements: Pr( ) 0 for every 0.3 Pr( )

8 Joint Distributions joint probability distribution of two RVs: the function given by Pr(X =, Y = y) read X equals and Y equals y eample, y Pr(X =, Y = y) sunny, on-time 0.20 rain, on-time 0.20 probability that it s sunny and my flight is on time snow, on-time 0.05 sunny, late 0.10 rain, late 0.30 snow, late 0.15

9 Marginal Distributions the marginal distribution of X is defined by Pr( ) Pr(, y) y the distribution of X ignoring other variables this definition generalizes to more than two variables, e.g. Pr( ) Pr(, y, z) y z

10 Marginal Distribution Eample joint distribution marginal distribution for X, y Pr(X =, Y = y) sunny, on-time 0.20 rain, on-time 0.20 snow, on-time 0.05 Pr(X = ) sunny 0.3 rain 0.5 snow 0.2 sunny, late 0.10 rain, late 0.30 snow, late 0.15

11 Joint/Marginal Distributions Two variable case The values of the joint distribution are in the 4 4 square, and the values of the marginal distributions are along the right and bottom margins. Ref:

12 Conditional Distributions the conditional distribution of X given Y is defined as: Pr( X Y y) Pr( X P( Y, Y y) y) the distribution of X given that we know Y

13 Conditional Distribution Eample joint distribution conditional distribution for X given Y=on-time, y Pr(X =, Y = y) sunny, on-time 0.20 rain, on-time 0.20 snow, on-time 0.05 Pr(X = Y=on-time) sunny 0.20/0.45 = rain 0.20/0.45 = snow 0.05/0.45 = sunny, late 0.10 rain, late 0.30 snow, late 0.15

14 Independence two random variables, X and Y, are independent if Pr(, y) Pr( ) Pr( y) for all and y

15 Independence Eample #1 joint distribution, y Pr(X =, Y = y) sunny, on-time 0.20 rain, on-time 0.20 snow, on-time 0.05 sunny, late 0.10 rain, late 0.30 snow, late 0.15 marginal distributions Pr(X = ) sunny 0.3 rain 0.5 snow 0.2 y Pr(Y = y) on-time 0.45 late 0.55 Are X and Y independent here? NO.

16 Independence Eample #2 joint distribution, y Pr(X =, Y = y) sunny, fly-united 0.27 rain, fly-united 0.45 snow, fly-united 0.18 sunny, fly-northwest 0.03 rain, fly-northwest 0.05 snow, fly-northwest 0.02 marginal distributions Pr(X = ) sunny 0.3 rain 0.5 snow 0.2 y Pr(Y = y) fly-united 0.9 fly-northwest 0.1 Are X and Y independent here? YES.

17 Conditional Independence two random variables X and Y are conditionally independent given Z if Pr( X Y, Z) Pr( X Z) once you know the value of Z, knowing Y doesn t tell you anything about X alternatively Pr(, y z) Pr( z) Pr( y z) for all, y, z

18 Conditional Independence Eample Flu Fever Vomit Pr true true true 0.04 true true false 0.04 true false true 0.01 true false false 0.01 false true true false true false false false true false false false Fever and Vomit are not independent: e.g. Pr( fever, vomit) Pr( fever ) Pr( vomit) Fever and Vomit are conditionally independent given Flu: Pr( fever, vomit flu) Pr( fever flu) Pr( vomit flu) Pr( fever, vomit flu) Pr( fever flu) Pr( vomit flu) etc.

19 Bayes Theorem Pr( y) Pr( y ) Pr( ) Pr( y) Pr( y ) Pr( ) Pr( y )Pr( ) this theorem is etremely useful there are many cases when it is hard to estimate Pr( y) directly, but it s not too hard to estimate Pr(y ) and Pr()

20 Bayes Theorem Eample MDs usually aren t good at estimating Pr(Disorder Symptom) they re usually better at estimating Pr(Symptom Disorder) if we can estimate Pr(Fever Flu) and Pr(Flu) we can use Bayes Theorem to do diagnosis Pr( flu fever ) Pr( fever Pr( fever flu)pr( flu) flu)pr( flu) Pr( fever flu)pr( flu)

21 Epected Values the epected value of a random variable that takes on numerical values is defined as: X E Pr( ) this is the same thing as the mean we can also talk about the epected value of a function of a random variable g( X ) g( ) E Pr( )

22 Epected Value Eamples E Shoesize 5 Pr(Shoesize 5) Pr( Shoesize 14) E gain( Lottery ) gain(winning) Pr(winning) gain(losing) Pr(losing) $ $ $0.899

23 Markov Models in Computational Biology there are many cases in which we would like to represent the statistical regularities of some class of sequences genes various regulatory sites in DNA (e.g. where RNA polymerase and transcription factors bind) proteins in a given family Markov models are well suited to this type of task

24 References Bioinformatics Classic: Krogh et. al. (1998) An Introduction to Hidden Markov Models for Biological Sequences Computational Methods in Mol. Biol Book: Eddy & Durbin, Chapter 3. Jones and Pevzner: Chapter 11. Tutorial: Rabiner, L. (1989) A tutorial on hidden Markov models and selected applications in speech recognition Proc IEEE 77 (2)

25 Eample X: 35 years old customer with an income of $4,000 and fair credit rating H: Hypothesis that the customer will buy a computer

26 P(H X): probability that the customer will buy a computer given that we know his age, credit rating and income (posterior) P(H): probability that the customer will buy a computer regardless of age, credit rating, and income P(X H): probability that the customer is 35 year old, have fair credit rating and earns $40,000, given that he has bought our computer (likelihood) P(X): probability that a person from our set of customers is 35 year old, have fair credit rating and earns $40,000.

27 Maimum Likelihood Estimation (MLE) Strategy of estimating the parameters of a statistical model given observations Choose a value that maimizes the probability of observed data θ = argma P( θ), where is observation and θ is a parameter

28 Maimum a posteriori estimation (MAP) Estimating unobserved population (or making a decision) Choose a value that is most probable give observed data and prior belief: θ = argma P y

29 transition probabilities 0.12 ) Pr( 0.38 ) Pr( 0.34 ) Pr( 0.16 ) Pr( g t g g g c g a i i i i i i i i A Markov Chain Model A T C G begin state transition

30 Markov Chain Models a Markov chain model (a triplicate) is defined by a set of states some states emit symbols other states (e.g. the begin state) are silent a set of transitions with associated probabilities the transitions emanating from a given state define a distribution over the possible net states Initial state probabilities

31 Markov Chain Models given some sequence of length L, we can ask how probable the sequence is given based on our model for any probabilistic model of sequences, we can write this probability as Pr( ) Pr( L Pr( L, L1 L1,...,,..., 1 1 ) ) Pr( L1 L2,..., 1 )...Pr( key property of a (1 st order) Markov chain: the probability of each depends only on the value of Pr( ) i i 1 Pr( L Pr( 1 ) L1 L i2 ) Pr( Pr( i L1 i1 ) L2 )...Pr( 2 1 ) Pr( 1 1 ) )

32 The Probability of a Sequence for a Given Markov Chain Model A G begin C T Pr( cggt ) Pr( c)pr( g c)pr( g g)pr(t g )

33 Markov Chain Models can also have an end state; allows the model to represent a distribution over sequences of different lengths preferences for ending sequences with certain symbols A G begin end C T

34 Markov Chain Notation the transition parameters can be denoted by where a similarly we can denote the probability of a sequence as a B L i2 a Pr( i i 1) i1i Pr( 1 ) Pr( i i 1) 1 i1i i2 where represents the transition from the begin state L

35 Eample Application CpG islands CpG is a pair of nucleotides C and G, appearing successively in this order, along one DNA strand. CpG dinucleotides are rarer in eukaryotic genomes than epected given the marginal probabilities of C and G but the regions upstream of genes (promoter regions) are richer in CpG dinucleotides than elsewhere CpG islands useful evidence for finding genes could predict CpG islands with Markov chains one to represent CpG islands one to represent the rest of the genome

36 Estimating the Model Parameters given some data (e.g. a set of sequences from CpG islands), how can we determine the probability parameters of our model? one approach: maimum likelihood estimation given a set of data D set the parameters Pr( D ) to maimize i.e. make the data D look likely under the model

37 Maimum Likelihood Estimation suppose we want to estimate the parameters Pr(a), Pr(c), Pr(g), Pr(t) and we re given the sequences accgcgctta gcttagtgac tagccgttac then the maimum likelihood estimates are Pr( a) Pr( c) Pr( g) Pr( t)

38 Maimum Likelihood Estimation suppose instead we saw the following sequences gccgcgcttg gcttggtggc tggccgttgc then the maimum likelihood estimates are Pr( a) Pr( c) Pr( g) Pr( t) do we really want to set this to 0?

39 A Bayesian Approach instead of estimating parameters strictly from the data, we could start with some prior belief for each for eample, we could use Laplace estimates where character i Pr( a) n i i na 1 ( n 1) i represents the number of occurrences of using Laplace estimates with the sequences gccgcgcttg gcttggtggc tggccgttgc Pr( a) Pr( c) pseudocount

40 A Bayesian Approach a more general form: m-estimates Pr( a) n i a n i p a m m prior probability of a number of virtual instances with m=8 and uniform priors gccgcgcttg gcttggtggc tggccgttgc Pr( c)

41 Estimation for 1 st Order Probabilities to estimate a 1 st order parameter, such as Pr(c g), we count the number of times that c follows the history g in our given sequences using Laplace estimates with the sequences gccgcgcttg Pr( a g) Pr( a c) gcttggtggc 7 1 tggccgttgc Pr( c g) Pr( g g) Pr( t g) 12 4

42 Markov Chains for Discrimination Question: Given a short stretch of genomic sequence, how could we decide if it comes from a CpG island or not? given sequences from CpG islands, and sequences from other regions, we can construct a model to represent CpG islands a null model to represent the other regions can then score a test sequence by: score( ) log Pr( CpG model) Pr( null model)

43 Markov Chains for Discrimination why use score( ) log Bayes rule tells us Pr( CpG) Pr( null ) Pr( CpG)Pr( CpG) Pr( CpG ) Pr( ) Pr( null )Pr( null ) Pr( null ) Pr( ) if we re not taking into account prior probabilities of two classes ( Pr(CpG) and Pr(null) ) then we just need to compare Pr( CpG) and Pr( null )

44 where Pr( CpGmodel) and a + st is the transition probability for the CpG island model from letter s to letter t, which is set as a st c L1 and c + st is the number of times letter s followed by t. So the score can be further written as Pr( st c t' st' L Pr( Pr( 1, 1 ) ) L 2 L 2,..., Pr( a i1 i 1 i model) i1 ) score( ) log Pr( CpGmodel) Pr( null model) L i log a a i1 1 i1 i i

45 Markov Chains for Discrimination parameters estimated for CpG and null models human sequences containing 48 CpG islands 60,000 nucleotides + A C G T A C G T A C G T A C G T

46 Markov Chains for Discrimination light bars represent negative sequences (non-cpg island) dark bars represent positive sequences

47 Higher Order Markov Chains the Markov property specifies that the probability of a state depends only on the probability of the previous state but we can build more memory into our states by using a higher order Markov model in an nth order Markov model Pr( i i, i2,..., 1 ) Pr( i i 1,..., 1 in )