Conditional distributions

Transcription

1 Conditional distributions Will Monroe July 6, 017 with materials by Mehran Sahami and Chris Piech

2 Independence of discrete random variables Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!). X Y iff x, y : P ( X = x, Y = y)=p ( X =x) P (Y = y) - or - p X,Y (x, y )= p X ( x) py ( y )

3 Independence of continuous random variables Two random variables are independent if knowing the value of one tells you nothing about the value of the other (for all values!). X Y iff x, y : f X,Y ( x, y )=f X ( x) f Y ( y) - or - f X,Y (x, y )=g(x)h( y ) - or - F X,Y (x, y)=f X (x) F Y ( y)

4 Review: Sum of independent binomials m flips n flips X: number of heads Y: number of heads in first n flips in next m flips X Bin (n, p) Y Bin (m, p) X +Y Bin (n+m, p) More generally: N X i Bin (ni, p) all X i independent ( N X i Bin ni, p i=1 i=1 )

5 Review: Sum of independent Poissons λ₁ chips/cookie X: number of chips in first cookie λ₂ chips/cookie Y: number of chips in second cookie X Poi (λ 1 ) Y Poi (λ ) X +Y Poi(λ 1 +λ ) More generally: N X i Poi (λ i ) all X i independent N ( ) X i Poi λ i i=1 i=1

6 Review: Convolution A convolution is the distribution of the sum of two independent random variables. f X +Y (a)= dy f X (a y) f Y ( y)

7 Review: Sum of independent normals X N (μ 1,σ 1 ) Y N (μ, σ ) X +Y N (μ 1+μ, σ 1 +σ ) More generally: X i N (μ i, σ i ) all X i independent N ( N N X i N μ i, σ i i=1 i=1 i=1 )

8 Virus infections 150 computers in a dorm: 50 Macs (each independently infected with probability 0.1) 100 PCs (each independently infected with probability 0.4) What is P( 40 machines infected)? M: # infected Macs M Bin (50, 0.1) X N (5, 4.5) P: # infected PCs P Bin (100, 0.4) Y N (40, 4) P( M + P 40) P( X +Y 39.5) W =X +Y N (5+40, 4.5+4)=N (45, 8.5) W P(W 39.5)=P 1 Φ( 1.03) ( )

9 Review: Conditional probability The conditional probability P(E F) is the probability that E happens, given that F has happened. F is the new sample space. P( EF) P(E F )= P( F) S E EF F

10 Discrete conditional distributions The value of a random variable, conditioned on the value of some other random variable, has a probability distribution. P ( X = x, Y = y) p X Y ( x, y)= P (Y = y) p X,Y (x, y ) = py ( y )

11 Conditionals from a joint PMF 0 R 1 1 Y P (R=1, Y =3) P( R=1 Y =3)= P(Y =3) 0.19 = = p R Y (1 3)

12 Conditionals from a joint PMF 0 R 1 1 Y P (R=r, Y = y) p R Y (r, y)= P (Y = y) 0 1 Y R 1

13 More web server hits Your web server gets X requests from humans and Y requests from bots in a day, independently. X ~ Poi(λ₁) Y ~ Poi(λ₂) so X + Y ~ Poi(λ₁ + λ₂) (independence) P( X =k,y =n k ) P ( X =k) P (Y =n k) P( X =k X +Y =n)= = P ( X +Y =n) P( X +Y =n) λ k λ n k e λ1 e λ n! = k! (n k)! e λ + λ ( λ1 +λ )n k n k λ λ n! 1 = k!(n k)! (λ 1 +λ )n k n k λ λ λ1 1 n = ( X X +Y ) Bin (X +Y, ) λ +λ k λ 1 +λ λ 1 +λ 1 =n 1 1 ( )( )( )

14 Continuous conditional distributions The value of a random variable, conditioned on the value of some other random variable, has a probability distribution. f X,Y (x, y ) f X Y ( x y)= f Y ( y)

15 Ratios of continuous probabilities The probability of an exact value for a continuous random variable is 0. But ratios of these probabilities are still well-defined! P ( X=a) f X (a) = P ( X=b) f X (b)

16 Defining the undefined P ( X =a) P( X a) = P ( X =b) P( X b) P (a ε X a+ε) =lim ε 0 P (b ε X b+ε) a+ε dx f X ( x) f (a) if ε is small X =lim ε 0 a ε b+ε dx f X ( x) f (b) X b ε ε f X (a) f X (a) = = ε f X (b) f X (b) ε fx(a) ε fx(b)

17 Conditioning on a continuous RV f X Y ( x y)=p( X =x Y = y) P ( X = x, Y = y) = P(Y = y) f X,Y (x, y ) = f Y ( y)

18 Mixing discrete and continuous a b P (a1 X a, b1 N b )= dx f X, N (x, n) a1 n=b1 f X, N (x, n) f X N ( x n)= p N (n) f X, N ( x, n) p N X (n x)= f X (x)

19 Discrete + Continuous Bayes p N X (n x) f X ( x) f X N (x n)= p N (n) f X N ( x n) p N (n) p N X (n x)= f X ( x) P(N X) N X

20 Break time!

21 The probability of a probability

22 Beta random variable An beta random variable models the probability of a trial s success, given previous trials. The PDF/CDF let you compute probabilities of probabilities! X Beta (a, b) f X ( x)= Cx { a 1 b 1 (1 x) 0 if 0< x <1 otherwise

23 Estimating an unknown probability You roll a loaded die N times, get A sixes (and N - A non-sixes). What s the probability that the die is loaded such that sixes come up less than 1/4 of the time? X: probability of getting a six A: number of sixes in N rolls A X ~ Bin(N, X) X ~ Uni(0, 1) ( I know nothing ) P( A=a X =x) f X ( x) f X A (x a)= P( A=a)??? 1 a n a N = x (1 x) 1 P ( A=a) a ( ) a n a =C x (1 x)

24 Beta: Fact sheet number of successes + 1 X Beta (a, b) number of failures + 1 probability of success PDF: expectation: variance: f X ( x)= Cx { a 1 b 1 (1 x) 0 a E[ X ]= a+b ab Var( X )= (a+b) (a+b+1) if 0< x <1 otherwise

25 Beta takes many forms

26 Conjugate distribution X Beta (1, 1) f X ( x)=c x (1 x) if 0< x <1 0 0 =C x (1 x) =C 1 =1 dx C=1 0 Beta (1, 1)=Uni(0, 1) X A ~ Beta(a + 1, N a + 1) posterior likelihood P( A=a X =x)f X (x) f X A ( x a)= P( A=a) normalizing constant X ~ Beta(1, 1) prior

27 Subjective priors X A ~ Beta(a + 1, N a + 1) posterior P( A=a X =x) f X (x) f X A ( x a)= P( A=a) X ~ Beta(1, 1) prior How did we decide on Beta(1, 1) for the prior? Beta(1, 1): we haven t seen any rolls yet. Beta(4, 1): we ve seen 3 sixes and 0 non-sixes. Beta(, 6): we ve seen 1 six and 5 non-sixes. Beta prior = imaginary previous trials

28 Advanced: Dirichlet distribution Beta is the distribution ( conjugate prior ) for the p in the Bernoulli and binomial. Dirichlet is the distribution for the p₁, p₂, in the multinomial. X 1, X, Dir (a1, a, ) fx 1, X, ( x 1, x, )= C x1 a1 1 x b 1 if 0<{x1, x, }<1, x 1 + x + =1 (0 otherwise)

29 Frequentists vs. Bayesians Frequentist Bayesian A probability is the (real or theoretical) result of a number of experiments. A probability is a belief. All probabilities are based on objective experiences. All probabilities are based on subjective priors. (It s not really a debate anymore real statisticians / data scientists / machine learning practitioners can and do think both ways!) image: Eric Kilby