Probability. Raul Queiroz Feitosa

Transcription

1 Probability These slides are mostly inspired on slides from Christofer Bishop Raul Queiroz Feitosa

2 Objective Recall some fundamentals of Probability Theory. 3/16/2016 Probability 2

3 Interpretation of Probability Frequentist: Limit of an infinite number of trials. Bayesian A way to quantify uncertainty. 3/16/2016 Probability 3

4 Discrete Random Variables A discrete random variable X can take on any value on a finite or countable set of values X. The probability that X=x is denoted by P X = x, or just, P x, whereby 0 P x 1 and P x p() is called the probability mass function (pmf). Example: for X={1,2,3,4} 1 x X p x x /16/2016 Probability 4

5 Murder mystery A murder has been committed. Two suspects: Butler Cook There are three possible murder weapons: Pistol Knife fireplace Poker 3/16/2016 Probability 5

6 Prior Distribution Prior Probability expresses the belief that an event might occur without taking any evidence into account. Butler has served the family for many years. Cook hired recently, rumors of dodgy history. P Culprit = Butler = 20% P Culprit = Cook = 80% This is called a factor graph P Culprit Probabilities add up to 100%. Culprit Butler, Cook 3/16/2016 Probability 6

7 Conditional Distribution Conditional distribution expresses the belief that an event might occur given some observation(s) or evidence(s). Butler is an ex-army and keeps a pistol in a locked drawer. Cook has access to lot of knives. Pistol Knife Poker Cook 5% 65% 30% Butler 80% 10% 10% =100% =100% P Weapon Culprit) 3/16/2016 Probability 7

8 Conditional Distribution Factor Graph P Culprit prior distribution Culprit Butler, Cook P Weapon Culprit conditional distribution Weapon Pistol, Knife, Poker 3/16/2016 Probability 8

9 Joint Distribution Joint Probability expresses the belief that multiple joint events occur. What is the probability that the Cook committed the murder with a Pistol? P Culprit = Cook = 20% P Weapon = Pistol Culprit = Cook = 80% P Weapon = Pistol, Culprit = Cook = 20% 80% = 16% Likewise for other combinations of Weapon and Culprit. 3/16/2016 Probability 9

10 Joint Distribution Pistol Knife Poker Cook 4% 52% 24% Butler 16% 2% 2% =100% P(Weapon, Culprit)=P Weapon Culprit) P(Culprit) P(y, x)=p y x) P(x) product rule joint distribution conditional distribution prior/marginal distribution 3/16/2016 Probability 10

11 Joint Distribution Factor Graph prior distribution P Culprit Culprit Butler, Cook P Weapon Culprit conditional distribution Generative Model Weapon Pistol, Knife, Poker P(Weapon, Culprit)=P Weapon Culprit) P(Culprit) 3/16/2016 Probability 11

12 Generative Viewpoint Murderer Weapon Cook Butler Cook Butler Cook Cook Cook Butler Knife Knife Pistol Poker Knife Pistol poker pistol knife Cook Cook Butler Cook Poker Knife Pistol Knife knife poker pistol 3/16/2016 Probability 12

13 Marginal Distribution Marginal Probability is the probability that an event occurred obtained by summing over the probabilities of all other events. Given the joint distribution (weapon, culprit), what is the probability distribution that murder was committed with a Pistol? P Weapon = Pistol, Culprit = Cook = 4% P Weapon = Pistol, Culprit = Butler = 16% P Weapon = Pistol = 4% + 16% = 20% Likewise for other Weapons. 3/16/2016 Probability 13

14 Marginal Distribution P(Culprit)=P Weapon = Pistol, Culprit + +P Weapon = Knife, Culprit + P Weapon = Poker, Butler joint distributions Pistol Knife Poker Total Cook 4% 52% 24% 80% Butler 16% 2% 2% 20% Total 20% 54% 26% 100% marginal distribution of weapon P(Weapon)=P Weapon, Culprit = Cook + +P(Weapon, Culprit = Butler) marginal distribution of culprit (=prior)! P x = P(x, y) y sum rule 3/16/2016 Probability 14

15 Reasoning Backwards P Culprit P Weapon Culprit 3/16/2016 Probability 15

16 Posterior Distribution Posterior Probability is the revised of prior after receiving additional information. A Pistol was found in the scene of the crime. Pistol Knife Poker Cook 4% 52% 24% Butler 16% 2% 2% P Culprit Knife) = P(Weapon=Knife,Culprit) P Weapon=Knife,Culprit=Cook +P(Weapon=Knife,Culprit=Butler) = P Weapon=Knife Culprit)p(Culprit) P(Weapon=Knife) 3/16/2016 Probability 16

17 Generative Viewpoint A Pistol was found in the scene of the crime. Murderer Weapon Cook Knife Butler Knife Cook Pistol Cook Poker Cook Knife Butler Pistol Cook Poker Cook Knife Butler Pistol Cook Knife 3/16/2016 Probability 17

18 Posterior Distribution. Joint distribution Pistol Knife Poker Cook 4% 52% 24% Butler 16% 2% 2% Posterior distribution p Culprit Weapon) Pistol Knife Poker Cook 20% 96% 92% Butler 80% 4% 8% P x y = P(x, y) P(y) P x y = P y x P(x) P(y) Bayes rule 3/16/2016 Probability 18

19 Bayes Theorem It follows from the product rule P(y, x)=p y x) P(x) =P x y) P(y) likelihood prior P x y) = P y x) P(x) P(y) posterior P y = P y x P(x) x marginal 3/16/2016 Probability 19

20 The Rules of Probability Sum Rule Product Rule P x = P(x, y) y P(x, y)=p x y)p(y)=p y x) P(x) Bayes Theorem P y x) = P x y) P(y) P(x) Denominator P x = P x y)p(y) y 3/16/2016 Probability 20

21 Continuous Random Variables A continuous random variable X can take any real value. The probability that X q, denoted by F q = P X q is called cumulative probability density or cdf. We define the probability density function - pdf as p x = dd x dd p x p x F x F x May take values greater than 1 x 3/16/2016 Probability 21

22 The Rules of Probability Assuming that x is continuous and y is discrete Sum Rule Product Rule p x = p(x, y) y p(x, y)=p x y) + P y = p x, y dd P(y)=P y x) p(x) Bayes Theorem P y x) = p x y) P(y) p(x) p x y) = P y x) p(x) P(y) Denominator + p x = p x y)p y P y = p x, y dd y 3/16/2016 Probability 22

23 Probability END 3/16/2016 Graphical Models 23