A [somewhat] Quick Overview of Probability. Shannon Quinn CSCI 6900

Transcription

1 A [somewhat] Quick Overview of Probability Shannon Quinn CSCI 6900

2 [Some material pilfered from Probabilistic and Bayesian Analytics Note to other teachers and users of these slides. Andrew would be delighted if you found this source material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit your own needs. PowerPoint originals are available. If you make use of a significant portion of these slides in your own lecture, please include this message, or the following link to the source repository of Andrew s tutorials: tutorials. Comments and corrections gratefully received. Andrew W. Moore School of Computer Science Carnegie Mellon University awm@cs.cmu.edu Copyright Andrew W. Moore

3 Probability - what you need to really, really know Probabilities are cool

4 Probability - what you need to really, really know Probabilities are cool Random variables and events

5 Discrete Random Variables A is a Boolean-valued random variable if A denotes an event, there is uncertainty as to whether A occurs. Examples A = The US president in 2023 will be male A = You wake up tomorrow with a headache A = You have Ebola A = the 1,000,000,000,000 th digit of π is 7 Define P(A) as the fraction of possible worlds in which A is true We re assuming all possible worlds are equally probable

6 Discrete Random Variables A is a Boolean-valued random variable if A denotes an event, a possible outcome of an experiment there is uncertainty as to whether A occurs. the experiment is not deterministic Define P(A) as the fraction of experiments in which A is true We re assuming all possible outcomes are equiprobable Examples You roll two 6-sided die (the experiment) and get doubles (A=doubles, the outcome) I pick two students in the class (the experiment) and they have the same birthday (A=same birthday, the outcome)

7 Visualizing A Event space of all possible worlds Worlds in which A is true P(A) = Area of reddish oval Its area is 1 Worlds in which A is False

8 Probability - what you need to really, really know Probabilities are cool Random variables and events There is One True Way to talk about uncertainty: the Axioms of Probability

9 The Axioms of Probability 0 <= P(A) <= 1 P(True) = 1 P(False) = 0 P(A or B) = P(A) + P(B) - P(A and B) Dice Experiments Events, random variables,., probabilities

10 (This is Andrew s joke)

11 Interpreting the axioms 0 <= P(A) <= 1 P(True) = 1 P(False) = 0 P(A or B) = P(A) + P(B) - P(A and B) The area of A can t get any smaller than 0 And a zero area would mean no world could ever have A true

12 Interpreting the axioms 0 <= P(A) <= 1 P(True) = 1 P(False) = 0 P(A or B) = P(A) + P(B) - P(A and B) The area of A can t get any bigger than 1 And an area of 1 would mean all worlds will have A true

13 Interpreting the axioms 0 <= P(A) <= 1 P(True) = 1 P(False) = 0 P(A or B) = P(A) + P(B) - P(A and B) A B

14 Interpreting the axioms 0 <= P(A) <= 1 P(True) = 1 P(False) = 0 P(A or B) = P(A) + P(B) - P(A and B) A P(A or B) B P(A and B) B Simple addition and subtraction

15 Theorems from the Axioms 0 <= P(A) <= 1, P(True) = 1, P(False) = 0 P(A or B) = P(A) + P(B) - P(A and B) è P(not A) = P(~A) = 1-P(A) P(A or ~A) = 1 P(A and ~A) = 0 P(A or ~A) = P(A) + P(~A) - P(A and ~A) 1 = P(A) + P(~A) - 0

16 Elementary Probability in Pictures P(~A) + P(A) = 1 A ~A

17 Another important theorem 0 <= P(A) <= 1, P(True) = 1, P(False) = 0 P(A or B) = P(A) + P(B) - P(A and B) è P(A) = P(A ^ B) + P(A ^ ~B) A = A and (B or ~B) = (A and B) or (A and ~B) P(A) = P(A and B) + P(A and ~B) P((A and B) and (A and ~B)) P(A) = P(A and B) + P(A and ~B) P(A and A and B and ~B)

18 Elementary Probability in Pictures P(A) = P(A ^ B) + P(A ^ ~B) A ^ B B A ^ ~B ~B

19 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence

20 Independent Events Definition: two events A and B are independent if Pr(A and B)=Pr(A)*Pr(B). Intuition: outcome of A has no effect on the outcome of B (and vice versa). We need to assume the different rolls are independent to solve the problem. You frequently need to assume the independence of something to solve any learning problem.

21 Some practical problems You re the DM in a D&D game. Joe brings his own d20 and throws 4 critical hits in a row to start off DM=dungeon master D20 = 20-sided die Critical hit = 19 or 20 What are the odds of that happening with a fair die? Ci=critical hit on trial i, i=1,2,3,4 P(C1 and C2 and C4) = P(C1)* *P(C4) = (1/10)^4

22 Multivalued Discrete Random Variables Suppose A can take on more than 2 values A is a random variable with arity k if it can take on exactly one value out of {v 1,v 2,.. v k } Example: V={aaliyah, aardvark,., zymurge, zynga} Example: V={aaliyah_aardvark,, zynga_zymgurgy} Thus P( A = P v A = v ) = 0 if i j i j ( A = 2 k v1 A = v A = v ) = 1

23 Terms: Binomials and Multinomials Suppose A can take on more than 2 values A is a random variable with arity k if it can take on exactly one value out of {v 1,v 2,.. v k } Example: V={aaliyah, aardvark,., zymurge, zynga} Example: V={aaliyah_aardvark,, zynga_zymgurgy} The distribution Pr(A) is a multinomial For k=2 the distribution is a binomial

24 More about Multivalued Random Variables Using the axioms of probability and assuming that A obeys It s easy to prove that j i v A v A P j i = = = 0 if ) ( 1 ) ( 2 1 = = = = k v A v A v A P ) ( ) ( = = = = = = i j i v j A P v A v A v A P And thus we can prove 1 ) ( 1 = = = k j v j A P

25 Elementary Probability in Pictures k j= 1 P( A = v j ) = 1 A=2 A=3 A=5 A=4 A=1

26 Elementary Probability in Pictures k j= 1 P( A = v j ) = 1 A=aaliyah A= A=. A=zynga A=aardvark

27 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities

28 A practical problem I have lots of standard d20 die, lots of loaded die, all identical. Loaded die will give a 19/20 ( critical hit ) half the time. In the game, someone hands me a random die, which is fair (A) or loaded (~A), with P(A) depending on how I mix the die. Then I roll, and either get a critical hit (B) or not (~B). Can I mix the dice together so that P(B) is anything I want - say, p(b)= 0.137? P(B) = P(B and A) + P(B and ~A) = 0.1*λ + 0.5*(1- λ) = mixture model λ = ( )/0.4 =

29 Another picture for this problem It s more convenient to say if you ve picked a fair die then i.e. Pr(critical hit fair die)=0.1 if you ve picked the loaded die then. Pr(critical hit loaded die)=0.5 A (fair die) ~A (loaded) Conditional probability: Pr(B A) = P(B^A)/P(A) A and B ~A and B P(B A) P(B ~A)

30 Definition of Conditional Probability P(A ^ B) P(A B) = P(B) Corollary: The Chain Rule P(A ^ B) = P(A B) P(B)

31 Some practical problems marginalizing out A I have 3 standard d20 dice, 1 loaded die. Experiment: (1) pick a d20 uniformly at random then (2) roll it. Let A=d20 picked is fair and B=roll 19 or 20 with that die. What is P(B)? P(B) = P(B A) P(A) + P(B ~A) P(~A) = 0.1* *0.25 = 0.2

32 posterior P(A B) = P(B A) * P(A) P(B) prior Bayes rule P(B A) = P(A B) * P(B) P(A) Bayes, Thomas (1763) An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53: by no means merely a curious speculation in the doctrine of chances, but necessary to be solved in order to a sure foundation for all our reasonings concerning past facts, and what is likely to be hereafter. necessary to be considered by any that would give a clear account of the strength of analogical or inductive reasoning

33 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities Bayes Rule

34 Some practical problems Joe throws 4 critical hits in a row, is Joe cheating? A = Joe using cheater s die C = roll 19 or 20; P(C A)=0.5, P(C ~A)=0.1 B = C1 and C2 and C3 and C4 Pr(B A) = P(B ~A)= P( A B) = P( B P( B A) P( A) A) P( A) + P( B ~ A) P(~ A) P( A B) = * P( A) * P( A) *(1 P( A))

35 What s the experiment and outcome here? Outcome A: Joe is cheating Experiment: Joe picked a die uniformly at random from a bag containing 10,000 fair die and one bad one. Joe is a D&D player picked uniformly at random from set of 1,000,000 people and n of them cheat with probability p>0. I have no idea, but I don t like his looks. Call it P(A)=0.1

36 Some practical problems Joe throws 4 critical hits in a row, is Joe cheating? A = Joe using cheater s die C = roll 19 or 20; P(C A)=0.5, P(C ~A)=0.1 B = C1 and C2 and C3 and C4 Pr(B A) = P(B ~A)= ) ( ) / ( ) ( ) ( ) / ( ) ( ) ( ) ( B P A P A B P B P A P A B P B A P A B P = ) ( ) ( ) ( ) ( B P A P A B P A B P = ) ( ) ( ) ( ) ( A P A P A B P A B P = ) ( ) ( A P A P = ) ( ) ( 6,250 A P A P = Moral: with enough evidence the prior P(A) doesn t really matter.

37 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities Bayes Rule MLE s, smoothing, and MAPs

38 Some practical problems I bought a loaded d20 on EBay but it didn t come with any specs. How can I find out how it behaves? Frequency Face Shown 1. Collect some data (20 rolls) 2. Estimate Pr(i)=C(rolls of i)/c(any roll)

39 One solution I bought a loaded d20 on EBay but it didn t come with any specs. How can I find out how it behaves? Frequency P(1)=0 P(2)=0 P(3)=0 P(4)=0.1 MLE = maximum likelihood estimate Face Shown P(19)=0.25 P(20)=0.2 But: Do I really think it s impossible to roll a 1,2 or 3? Would you bet your house on it?

40 A better solution I bought a loaded d20 on EBay but it didn t come with any specs. How can I find out how it behaves? Frequency Face Shown 0. Imagine some data (20 rolls, each i shows up 1x) 1. Collect some data (20 rolls) 2. Estimate Pr(i)=C(rolls of i)/c(any roll)

41 A better solution I bought a loaded d20 on EBay but it didn t come with any specs. How can I find out how it behaves? Frequency Face Shown P(1)=1/40 P(2)=1/40 P(3)=1/40 P(4)=(2+1)/40 P(19)=(5+1)/40 P(20)=(4+1)/40=1/8 Pˆr( i) C( i) + 1 C( ANY ) + C( IMAGINED) = 0.25 vs really different! Maybe I should imagine less data?

42 A better solution? Frequency Face Shown P(1)=1/40 P(2)=1/40 P(3)=1/40 P(4)=(2+1)/40 P(19)=(5+1)/40 P(20)=(4+1)/40=1/8 Pˆr( i) C( i) + 1 C( ANY ) + C( IMAGINED) = 0.25 vs really different! Maybe I should imagine less data?

43 A better solution? Q: What if I used m rolls with a probability of q=1/20 of rolling any i? Pˆr( i) = C( ANY ) C( i) C( IMAGINED) Pˆr( i) = C( i) + mq C( ANY ) + m I can use this formula with m>20, or even with m<20 say with m=1

44 A better solution Q: What if I used m rolls with a probability of q=1/20 of rolling any i? Pˆr( i) = C( ANY ) C( i) C( IMAGINED) Pˆr( i) = C( i) + mq C( ANY ) + m If m>>c(any) then your imagination q rules If m<<c(any) then your data rules BUT you never ever ever end up with Pr(i)=0

45 Terminology more later This is called a uniform Dirichlet prior C(i), C(ANY) are sufficient statistics Pˆr( i) = C( i) + mq C( ANY ) + m MLE = maximum likelihood estimate MAP= maximum a posteriori estimate

46 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities Bayes Rule MLE s, smoothing, and MAPs The joint distribution

47 Some practical problems I have 1 standard d6 die, 2 loaded d6 die. Loaded high: P(X=6)=0.50 Loaded low: P(X=1)=0.50 Experiment: pick one d6 uniformly at random (A) and roll it. What is more likely rolling a seven or rolling doubles? Three combinations: HL, HF, FL P(D) = P(D ^ A=HL) + P(D ^ A=HF) + P(D ^ A=FL) = P(D A=HL)*P(A=HL) + P(D A=HF)*P(A=HF) + P(A A=FL)*P(A=FL)

48 Some practical problems I have 1 standard d6 die, 2 loaded d6 die. Loaded high: P(X=6)=0.50 Loaded low: P(X=1)=0.50 Experiment: pick one d6 uniformly at random (A) and roll it. What is more likely rolling a seven or rolling doubles? Three combinations: HL, HF, FL Roll 2 Roll D 7 2 D 7 3 D D 5 7 D 6 7 D

49 A brute-force solution A Roll 1 Roll 2 P Comment FL 1 1 1/3 * 1/6 * ½ doubles FL 1 2 1/3 * 1/6 * 1/10 A joint probability table shows P(X1=x1 and and Xk=xk) FL for 1 every possible combination of values x1,x2,., xk 1 6 seven FL With 2 this you 1 can compute any P(A) where A is any FL boolean 2 combination of the primitive events (Xi=Xk), e.g. P(doubles) FL 6 P(seven or 6 eleven) doubles HL 1 P(total is higher 1 than 5) HL 1. 2 HF 1 1 doubles

50 The Joint Distribution Example: Boolean variables A, B, C Recipe for making a joint distribution of M variables:

51 The Joint Distribution Example: Boolean variables A, B, C Recipe for making a joint distribution of M variables: 1. Make a truth table listing all combinations of values of your variables (if there are M Boolean variables then the table will have 2 M rows). A B C

52 The Joint Distribution Example: Boolean variables A, B, C Recipe for making a joint distribution of M variables: 1. Make a truth table listing all combinations of values of your variables (if there are M Boolean variables then the table will have 2 M rows). 2. For each combination of values, say how probable it is. A B C Prob

53 The Joint Distribution Example: Boolean variables A, B, C Recipe for making a joint distribution of M variables: 1. Make a truth table listing all combinations of values of your variables (if there are M Boolean variables then the table will have 2 M rows). 2. For each combination of values, say how probable it is. 3. If you subscribe to the axioms of probability, those numbers must sum to 1. A B C Prob A C 0.30 B 0.10

54 Using the Joint One you have the JD you can ask for the probability of any logical expression involving your attribute P( E) = rows matching E P(row) Abstract: Predict whether income exceeds $50K/yr based on census data. Also known as "Census Income" dataset. [Kohavi, 1996] Number of Instances: 48,842 Number of Attributes: 14 (in UCI s copy of dataset); 3 (here)

55 Using the Joint P(Poor Male) = P( E) = rows matching E P(row)

56 Using the Joint P(Poor) = P( E) = rows matching E P(row)

57 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities Bayes Rule MLE s, smoothing, and MAPs The joint distribution Inference

58 Inference with the Joint P( E 1 E 2 ) = P( E1 P( E 2 E ) 2 ) = rows matching E 1 rows matching E P(row) and E 2 2 P(row)

59 Inference with the Joint P( E 1 E 2 ) = P( E1 P( E 2 E ) 2 ) = rows matching E 1 rows matching E P(row) and E 2 2 P(row) P(Male Poor) = / = 0.612

60 Estimating the joint distribution Collect some data points Estimate the probability P(E1=e1 ^ ^ En=en) as #(that row appears)/#(any row appears). Gender Hours Wealth g1 h1 w1 g2 h2 w2.. gn hn wn

61 Estimating the joint distribution For each combination of values r: Total = C[r] = 0 Complexity? O(2 d ) d = #attributes (all binary) For each data row r i C[r i ] ++ Total ++ Complexity? O(n) n = total size of input data Gender Hours Wealth g1 h1 w1 = C[r i ]/Total g2 h2 w2.. r i is female,40.5+, poor gn hn wn

62 Estimating the joint distribution For each combination of values r: Total = C[r] = 0 Complexity? O( k i = arity of attribute i d i= 1 k i ) For each data row r i C[r i ] ++ Total ++ Complexity? O(n) n = total size of input data Gender Hours Wealth g1 h1 w1 g2 h2 w2.. gn hn wn

63 Estimating the joint distribution For each combination of values r: Total = C[r] = 0 For each data row r i C[r i ] ++ Total ++ Complexity? O( i= 1 k i = arity of attribute i Complexity? O(n) n = total size of input data d k i ) Gender Hours Wealth g1 h1 w1 g2 h2 w2.. gn hn wn

64 Estimating the joint distribution For each data row r i If r i not in hash tables C,Total: Insert C[r i ] = 0 C[r i ] ++ Total ++ Complexity? O(m) m = size of the model Complexity? O(n) n = total size of input data Gender Hours Wealth g1 h1 w1 g2 h2 w2.. gn hn wn

65 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities Bayes Rule MLE s, smoothing, and MAPs The joint distribution Inference Density estimation and classification

66 Density Estimation Our Joint Distribution learner is our first example of something called Density Estimation A Density Estimator learns a mapping from a set of attributes values to a Probability Input Attributes Density Estimator Probability Copyright Andrew W. Moore

67 Density Estimation Compare it against the two other major kinds of models: Input Attributes Input Attributes Classifier Density Estimator Prediction of categorical output or class One of a few discrete values Probability Input Attributes Regressor Prediction of real-valued output Copyright Andrew W. Moore

68 Density Estimation è Classification Input Attributes x Classifier Prediction of categorical output One of y1,., yk Input Attributes Class Density Estimator ^ P(x,y) To classify x ^ ^ 1. Use your estimator to compute P(x,y1),., P(x,yk) 2. Return the class y* with the highest predicted probability ^ ^ ^ ^ Ideally is correct with P(x,y*) = P(x,y*)/(P(x,y1) +. + P(x,yk)) Binary case: predict POS if ^ P(x)>0.5 Copyright Andrew W. Moore

69 Classification vs Density Estimation Classification Density Estimation

70 Bayes Classifiers If we can do inference over Pr(X 1,X d,y) in particular compute Pr(X 1 X d Y) and Pr(Y). And then we can use Bayes rule to compute Pr( Y X,..., X ) = 1 d Pr( X1,..., X d Y ) Pr( Y ) Pr( X,..., X ) 1 d

71 Summary: The Bad News Density estimation by directly learning the joint is trivial, mindless and dangerous Andrew s joke Density estimation by directly learning the joint is hopeless unless you have some combination of Very few attributes Attributes with low arity Lots and lots of data Otherwise you can t estimate all the row frequencies Copyright Andrew W. Moore

72 Part of a Joint Distribution A B C D E p is the effect of the is the effect of a The effect of this to this effect : be the effect of the not the effect of any does not affect the general does not affect the question any manner affect the principle

73 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities Bayes Rule MLE s, smoothing, and MAPs The joint distribution Inference Density estimation and classification Naïve Bayes density estimators and classifiers

74 Naïve Density Estimation The problem with the Joint Estimator is that it just mirrors the training data. (and is also really, really hard to compute) We need something which generalizes more usefully. Copyright Andrew W. Moore The naïve model generalizes strongly: Assume that each attribute is distributed independently of any of the other attributes.

75 Naïve Distribution General Case Suppose X 1,X 2,,X d are independently distributed. Pr( X = x1,..., X d = xd ) = Pr( X1 = x1 )... Pr( X d = 1 d x ) So if we have a Naïve Distribution we can construct any row of the implied Joint Distribution on demand. How do we learn this? Copyright Andrew W. Moore

76 Learning a Naïve Density Estimator #records with X i = xi P( X i = xi ) = #records MLE P( X i = x i ) = #records with X i = x #records + m i + mq Dirichlet (MAP) Another trivial learning algorithm! Copyright Andrew W. Moore

77 Probability - what you need to really, really know Probabilities are cool Random variables and events The Axioms of Probability Independence, binomials, multinomials Conditional probabilities Bayes Rule MLE s, smoothing, and MAPs The joint distribution Inference Density estimation and classification Naïve Bayes density estimators and classifiers Conditional independence more on this tomorrow!