Joint Probability Distributions, Correlations

Transcription

1 Joint Probability Distributions, Correlations

2 What we learned so far Events: Working with events as sets: union, intersection, etc. Some events are simple: Head vs Tails, Cancer vs Healthy Some are more complex: 10<Gene expression<100 Some are evcen more complex: Series of dice rolls: 1,3,5,3,2 Conditional probability: P(A B)=P(A B)/P(B) Independent events: P(A B)=P(A) or P(A B)= P(A)*P(B) Bayes theorem: relates P(A B) to P(B A) Random variables: Mean, Variance, Standard deviation. How to work with E(g(X)) Discrete (Uniform, Bernoulli, Binomial, Poisson, Geometric, Negative binomial, Hypergeometric, Power law); PMF: f(x)=prob(x=x); CDF: F(x)=Prob(X x); Continuous (Uniform, Exponential, Erlang, Gamma, Normal, Lognormal); PDF: f(x) such that Prob(X inside A)= A f(x)dx; CDF: F(x)=Prob(X x) Next step: work with multiple random variables

3 Concept of Joint Probabilities Biological systems are usually described not by a single random variable but by many random variables Example: The expression state of a human cell: 20,000 random variables X i for each of its genes A joint probability distribution describes the behavior of several random variables We will start with just two random variables X and Y and generalize when necessary Chapter 5 Introduction 3

4 Joint Probability Mass Function Defined The joint probability mass function of the discrete random variables X and Y, denoted as f x, y, satifies: XY (1) f x, y 0 All probabilities are non-negative XY (2) f x, y 1 The sum of all x y XY probabilities is 1 (3) f x, y P X x, Y y (5-1) XY Sec Joint Probability Distributions 4

5 Example 5 1: # Repeats vs. Signal Bars You use your cell phone to check your airline reservation. It asks you to speak the name of your departure city to the voice recognition system. Let Y denote the number of times you have to state your departure city. Let X denote the number of bars of signal strength on you cell phone. y = number of times city x = number of bars of signal strength name is stated Figure 5 1 Joint probability distribution of X and Y. The table cells are the probabilities. Observe that more bars relate to less repeating. Probability Bar Chart of Number of Repeats vs. Cell Phone Bars 1 2 Cell Phone Bars 3 4 Times 3 Times Twice Once Sec Joint Probability Distributions 5

6 Marginal Probability Distributions (discrete) For a discrete joint PDF, there are marginal distributions for each random variable, formed by summing the joint PMF over the other variable. f x f x, y X y XY f y f x, y Y x XY Called marginal because they are written in the margins y = number of times city name x = number of bars of signal strength is stated f Y ( y ) = f X (x ) = Figure 5 6 From the prior example, the joint PMF is shown in green while the two marginal PMFs are shown in purple. Sec Marginal Probability Distributions 6

7 Mean & Variance of X and Y are calculated using marginal distributions y = number of times city x = number of bars of signal strength name is stated f (y ) = y *f (y ) = y 2 *f (y ) = f (x ) = x *f (x ) = x 2 *f (x ) = μ X =E(X) = 2.35; σ X2 = V(X) = = = μ Y = E(Y) = 2.49; σ Y2 =V(Y) = = = Sec Marginal Probability Distributions 7

8 Conditional Probability Distributions Recall that P B A P A B P A P(Y=y X=x)=P(X=x,Y=y)/P(X=x)= =f(x,y)/f X (x) From Example 5 1 P(Y=1 X=3) = 0.25/0.55 = P(Y=2 X=3) = 0.20/0.55 = P(Y=3 X=3) = 0.05/0.55 = P(Y=4 X=3) = 0.05/0.55 = Sum = 1.00 y = number of times city name is stated x = number of bars of signal strength f Y (y ) = f X (x ) = Note that there are 12 probabilities conditional on X, and 12 more probabilities conditional upon Y. Sec Conditional Probability Distributions 8

9 Joint Random Variable Independence Random variable independence means that knowledge of the values of X does not change any of the probabilities associated with the values of Y. Opposite: Dependence implies that the values of X are influenced by the values of Y Sec Independence 9

10

11 Independence for Discrete Random Variables Remember independence of events (slide 21 lecture 3) : P(A B)=P(A B)/P(B)=P(A) or P(B A)= P(A B)/P(A)=P(B) or P(A B)=P(A) P(B) Random variables independent if any events A that Y=y and B that X=x are independent P(Y=y X=x)=P(Y=y) for any x or P(X=x Y=y)=P(X=x) for any y or P(X=x, Y=y)=P(X=x) P(Y=y) for any x and y

12 X and Y are Bernoulli variables Y=0 Y=1 X=0 2/6 1/6 X=1 2/6 1/6 What is the marginal P Y (Y=0)? A. 1/6 B. 2/6 C. 3/6 D. 4/6 E. I don t know Get your i clickers 12

13 X and Y are Bernoulli variables Y=0 Y=1 X=0 2/6 1/6 X=1 2/6 1/6 What is the marginal P Y (Y=0)? A. 1/6 B. 2/6 C. 3/6 D. 4/6 E. I don t know Get your i clickers 13

14 X and Y are Bernoulli variables Y=0 Y=1 X=0 2/6 1/6 X=1 2/6 1/6 What is the conditional P(X=0 Y=0)? A. 2/6 B. 1/2 C. 1/6 D. 4/6 E. I don t know Get your i clickers 14

15 X and Y are Bernoulli variables Y=0 Y=1 X=0 2/6 1/6 X=1 2/6 1/6 What is the conditional P(X=0 Y=0)? A. 2/6 B. 1/2 C. 1/6 D. 4/6 E. I don t know Get your i clickers 15

16 X and Y are Bernoulli variables Y=0 Y=1 X=0 2/6 1/6 X=1 2/6 1/6 Are they independent? A. yes B. no C. I don t know Get your i clickers 16

17 X and Y are Bernoulli variables Y=0 Y=1 X=0 2/6 1/6 X=1 2/6 1/6 Are they independent? A. yes B. no C. I don t know Get your i clickers 17

18 X and Y are Bernoulli variables Y=0 Y=1 X=0 1/2 0 X=1 0 1/2 Are they independent? A. yes B. no C. I don t know Get your i clickers 18

19 X and Y are Bernoulli variables Y=0 Y=1 X=0 1/2 0 X=1 0 1/2 Are they independent? A. yes B. no C. I don t know Get your i clickers 19

20 Joint Probability Density Function Defined The joint probability density function for the continuous random variables X and Y, denotes as f XY (x,y), satisfies the following properties: (1) f x, y 0 for all x, y XY (2) f x, y dxdy 1 XY XY (3) P X, Y R f x, y dxdy (5-2) R Figure 5 2 Joint probability density function for the random variables X and Y. Probability that (X, Y) is in the region R is determined by the volume of f XY (x,y) over the region R. Sec Joint Probability Distributions 20

21 Joint Probability Density Function Figure 5 3 Joint probability density function for the continuous random variables X and Y of expression levels of two different genes. Note the asymmetric, narrow ridge shape of the PDF indicating that small values in the X dimension are more likely to occur when small values in the Y dimension occur. Sec Joint Probability Distributions 21

22 Marginal Probability Distributions (continuous) Rather than summing a discrete joint PMF, we integrate a continuous joint PDF. The marginal PDFs are used to make probability statements about one variable. If the joint probability density function of random variables X and Y is f XY (x,y), the marginal probability density functions of X and Y are: f x f x, y dy X y XY f y f x, y dx (5-3) Y x XY X Y, f x f x y, Sec Marginal Probability Distributions 22 y x XY f y f x y XY

23 Conditional Probability Density Function Defined Given continuous random variables X and Y with joint probability density function f x, y, the conditional probability densiy function of Y given fxy x, y fxy x, y f y = if fx x0 Yx f x f x, y dy y X y XY Compare to discrete: P(Y=y X=x)=f XY (x,y)/f X (x) Yx which satifies the following properties: (1) f 0 Yx (2) f y dy 1 Yx XY Sec Conditional Probability Distributions X=x is (5-4) (3) PY BX x f ydyfor any set B in the range of Y B 23

24 Conditional Probability Distributions Conditional probability distributions can be developed for multiple random variables by extension of the ideas used for two random variables. Suppose p = 5 and we wish to find the distribution of X 1, X 2 and X 3 conditional on X 4 =x 4 and X 5 =x 5. f x, x, x XX X for f x, x 0. X X xx 4 5 f x, x, x, x, x XX XX X f x, x X X 4 5 Sec More Than Two Random Variables 24

25 Independence for Continuous Random Variables For random variables X and Y, if any one of the following properties is true, the others are also true. Then X and Y are independent. Y X X Y X A Y B PX APY B (1) f x, y f x f y XY X Y (2) f y f y for all x and y with f x 0 Yx (3) f y f x for all x and y with f y 0 Xy P(Y=y X=x)=P(Y=y) for any x or P(X=x Y=y)=P(X=x) for any y or P(X=x, Y=y)=P(X=x) P(Y=y) for any x and y (4) P, for any sets A and B in the range of X and Y, respectively. (5-7) Sec Independence 25

26 Covariation, Correlations

27

28

29 Covariance Defined Covariance is a number qunatifying between two random variables. average dependence The covariance between the random variables X and Y, denoted as cov XY, or is E X Y E XY XY 5-14) XY X Y X Y The units of are units of X times units of Y. XY Unlike the range of variance, -. XY ( Sec 5 2 Covariance & Correlation 29

30 Covariance and PMF tables y = number of times city name is stated x = number of bars of signal strength The probability distribution of Example 5 1 is shown. By inspection, note that the larger probabilities occur as X and Y move in opposite directions. This indicates a negative covariance. Sec 5 2 Covariance & Correlation 30

31 Covariance and Scatter Patterns Figure 5 13 Joint probability distributions and the sign of cov(x, Y). Note that covariance is a measure of linear relationship. Variables with non zero covariance are correlated. Sec 5 2 Covariance & Correlation 31

32 Independence Implies σ=ρ = 0 but not vice versa If X and Y are independent random variables, σ XY = ρ XY = 0 (5 17) ρ XY = 0 is necessary, but not a sufficient condition for independence. Independent covariance=0 NOT independent covariance=0 Sec 5 2 Covariance & Correlation 32

33 Correlation is normalized covariance Also called: Pearson correlation coefficient ρ XY =σ XY /σ X σ Y is the covariance normalized to be 1 ρ XY 1 Karl Pearson ( ) English mathematician and biostatistician

34

35

36 Spearman rank correlation Pearson correlation tests for linear relationship between X and Y Unlikely for variables with broad distributions nonlinear effects dominate Spearman correlation tests for any monotonic relationship between X and Y Calculate ranks (1 to n), r X (i) and r Y (i) of variables in both samples. Calculate Pearson correlation between ranks: Spearman(X,Y) = Pearson(r X, r Y ) Ties: convert to fractions, e.g. tie for 6s and 7s place both get 6.5. This can lead to artefacts. If lots of ties: use Kendall rank correlation (Kendall tau)

37 Matlab exercise: Correlation/Covariation Generate a sample with Stats=100,000 of two Gaussian random variables r1 and r2 which have mean 0 and standard deviation 2 and are: Uncorrelated Correlated with correlation coefficient 0.9 Correlated with correlation coefficient 0.5 Trick: first make uncorrelated r1 and r2. Then change r1 to: r1m=mix.*r2+(1 mix.^2)^0.5.*r1; where mix= corr. coeff. In each case calculate covariance and correlation coefficient In each case make scatter plot: plot(r1,r2, k. );

38 Matlab exercise: Correlation/Covariation 1. Stats=100000; 2. r1=2.*randn(stats,1); 3. r2=2.*randn(stats,1); 4. disp('covariance matrix='); disp(cov(r1,r2)); 5. disp('correlation=');disp(corr(r1,r2)); 6. figure; plot(r1,r2,'ko'); 7. mix=0.9; %Mixes r2 to r1 but keeps same variance 8. r1m=mix.*r2+sqrt(1 mix.^2).*r1; 9. disp('covariance matrix='); disp(cov(r1m,r2)); 10.disp('Correlation=');disp(corr(r1m,r2)); 11.figure; plot(r1m,r2,'ko'); 12.mix= 0.5; %REDO LINES 8 11

39

40 Linear Functions of Random Variables A function of random variables is itself a random variable. A function of random variables can be formed by either linear or nonlinear relationships. We start with linear functions. Given random variables X 1, X 2,,X p and constants c 1, c 2,, c p Y= c 1 X 1 + c 2 X c p X p (5 24) is a linear combination of X 1, X 2,,X p. Sec 5 4 Linear Functions of Random Variables 40

41

42 Mean & Variance of a Linear Function Y= c 1 X 1 + c 2 X c p X p p p E Y c E X c E X... c E X (5-25) Yc1V X1c2V X2 cpvx p cc i j XiX j V... 2 cov (5-26) If X, X,..., X are independent, then cov X X 0, 1 2 i j p i j 2 2 V Y c V X c V X... c V X p p (5-27) Sec 5 4 Linear Functions of Random Variables 42

43 Example 5 31: Error Propagation A semiconductor product consists of three layers. The variances of the thickness of each layer is 25, 40 and 30 nm 2. What is the variance of the finished product? Answer: V X V X SD X X X X X nm i nm i 2 Sec 5 4 Linear Functions of Random Variables 43

44

45

46 Mean & Variance of an Average X1 X2... X p If X and p E X i p Then EX (5-28a) p If the X are independent with V X i 2 2 p Then V X (5-28b) 2 p p i 2 Sec 5 4 Linear Functions of Random Variables 46

47 Credit: XKCD comics