Statistika pro informatiku

Transcription

1 Statistika pro informatiku prof. RNDr. Roman Kotecký DrSc., Dr. Rudolf Blažek, PhD Katedra teoretické informatiky FIT České vysoké učení technické v Praze MI-SPI, ZS 2011/12, Přednáška 2 Evropský sociální fond. Praha & EU: Investujeme do vaší budoucnosti Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 1 / 23

2 Recapitulation Recapitulation Ω is the set of all possible outcomes. Events are subsets of Ω, A Ω. A probability P is a function P : P(Ω) A P(A) [0, 1] such that (N): P(Ω) = 1 and (A): P( l 1 A l ) = l 1 P(A l) for all pairwise disjoint A 1, A 2, Ω. For at most countable Ω the probability is determined by a function p : Ω [0, 1] such that ω Ω p(ω) = 1. Then P(A) = ω A p(ω). Probability rules: (i) P( ) = 0. (ii) Monotonicity. If A B then P(A) P(B). (iii) Additivity. P(A B) + P(A B) = P(A) + P(B) (and thus also, P(A) + P(A c ) = 1). (iv) Subadditivity. P( n 1 A n ) n 1 P(A n). (v) Inclusion-exclusion principle. P( n k=1 A k) = n k=1 P(A k) i<j P(A i A j ) + + ( 1) n+1 P(A 1 A n ). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 2 / 23

3 Recapitulation If P(B) > 0 then the conditional probability that A occurs under the condition B is defined to be P(A B) = P(A B). P(B) Conditional probability is a probability. The other properties of the probability law follow: P(Ω B) = 1. P(A 1 A 2 B) = P(A 1 B) + P(A 2 B). The probability P(A B) lives on B: for A B =, we have P(A B) = 0. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 3 / 23

4 Conditional probability Multiplicative law A multiplicative law Sometimes it is easier to gain a conditional information and the unconditional probability compute by turning the definition : P(A B) = P(A B)P(B). A multiplicative law is a generalization: P(A 1 A 2 A 3 ) = P(A 1 )P(A 2 A 1 )P(A 3 A 1 A 2 ). It follows by substituting into the definitions on the right hand side: P(A 1 )P(A 2 A 1 )P(A 3 A 1 A 2 ) =P(A 1 ) P(A2 A1) P(A 3 A 1 A 2) P(A 1) P(A 1 A 2) = =P(A 1 A 2 A 3 ). In general: P( n i=1 A i) = P(A 1 )P(A 2 A 1 )... P(A n n 1 i=1 A i). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 4 / 23

5 Conditional probability Multiplicative law Example What is the probability that in the sequence of 3 cards drawn one after another there are no hearts? A i = {i-th card is not hearts}, i = 1, 2, 3. P(A 1 A 2 A 3 ) = P(A 1 )P(A 2 A 1 )P(A 3 A 1 A 2 ) = Illustration of the computation with the help of the tree of conditional probabilities: P (A3 A1 A2) A3 P (A1 A2 A3) 37/ P (A2 A1) A2 P (A c 3 A1 A2) 38/51 13/50 P (A1 A2 A c 3) P (A1) P (A c 2 A1) 39/52 13/51 A1 P (A c 1) 13/52 The probability in a given vertex of the tree is the product of the corresponding values on the path stemming from the root. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 5 / 23

6 Conditional probability Case distinction formula Case distinction formula Věta o úplné pravděp. A family B 1, B 2,..., B n of events is called a partition of Ω if Ω = n i=1 B i (it means that Ω = n i=1 B i and B i B j = whenever i j). Proposition (Case distinction formula) Let B 1, B 2,..., B n be a partition of Ω such that P(B i ) > 0 for all i. Then P(A) = n P(A B i )P(B i ) i=1 for any event A. Proof. Since A = n i=1(a B i ), by definition of conditional probability and by aditivity, n i=1 P(A B i)p(b i ) = n i=1 P(A B i) = P(A). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 6 / 23

7 Conditional probability Bayes theorem Graphical illustration of the case distinction formula: P (B 1) B 1 P (A B 1) root P (B 2) B 2 P (A B 2) A P (B 3) P (A B 3) B 3 Theorem (Bayes theorem) Let B 1, B 2,..., B n be a partition of Ω such that P(B i ) > 0 for all i and let A be an event with P(A) > 0. Then P(B k A) = P(B k)p(a B k ) n i=1 P(B i)p(a B i ). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 7 / 23

8 Conditional probability Bayes formula Proof. P(B k )P(A B k ) n i=1 P(B i)p(a B i ) = P(A B k) = P(B k A) P(A) Thomas Bayes (c April 1761) was an English mathematician and Presbyterian minister, known for having formulated a specific case of the theorem that bears his name: Bayes theorem. Bayes never published what would eventually become his most famous accomplishment; his notes were edited and published after his death by Richard Price. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 8 / 23

9 Conditional probability Reliability of a drug test Example (Reliability of a drug test) Cyclists in Tour de France are tested on a usage of a particular drug. About 1% are known to use the drug and the test yields positive result for 95% of those positive and 10% of positive result for those negative. P(a randomly chosen positively tested cyclist is taking drug)? B 1 set of those using the drug. B 2 = Ω \ B 1. A is the set of those who reacted positively. P(B 1 ) = 0.01, P(B 2 ) = 0.99, P(A B 1 ) = 0.95, P(A B 2 ) = 0.1. P(B 1 A) = = P(A B 1 )P(B 1 ) P(A B 1 )P(B 1 ) + P(A B 2 )P(B 2 ) = The test is not effective in finding those using the drug. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 9 / 23

10 Conditional probability Reliability of a drug test Example (continued) P(B 1 A c ) = = P(A c B 1 )P(B 1 ) P(A c B 1 )P(B 1 ) + P(A c B 2 )P(B 2 ) = But, it is very effective for those testing negative (false accusation very unlikely). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 10 / 23

11 Conditional probability Monty Hall problem Monty Hall problem TV show Let s make a deal. After the game presenter shows three closed doors: there is a luxurious car behind one of them and a goat behind remaining two. You have to choose, so you pick up one, say #1. Now, he opens another one, say #3, there is a goat. He says to you: Do you want to change your mind and pick door #2? What would be your answer? What is the probability, at this moment, that the car is behind the door #1 and what is the probability that the car is behind the door #2? Correct answer is, yes, you should switch. The probabilities are 1/3 and 2/3, respectively. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 11 / 23

12 Conditional probability Monty Hall problem Important: all this is under assumption that the host is determined to show you the goat: if the car is behind #1, he opens randomly #2 or #3, if it is not behind #1, he opens the unique remaining doors with a goat. If he opens any door randomly (irrespectively of what I have chosen), the probabilities (for switching and nonswitching) are 1 2 and 1 2. Reasoning: C i the event that the car is behind the door #i with all 3 placements of the car being equally probable, P(C i ) = 1, i = 1, 2, 3. 3 G the event that I was shown the goat. P(C 2 G) = P(C 2 G C 1 )P(C 1 ) + P(C 2 G C c 1)P(C c 1) P(G C 1 )P(C 1 ) + P(G C c 1 )P(Cc 1 ) = = = 2 3. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 12 / 23

13 Conditional probability Independence Independence Nezávislost Intuitively: A and B independent if prob. one assigns to A is not influenced by the information that B occurred, P(A B) = P(A), and conversely. Definition Let (Ω, P) be a probability space. Two events A, B are called stochastically independent with respect to P if P(A B) = P(A)P(B). Example Notice that independence is not linked with causality: Two dice. A = {sum = 7}, B = {first roll is 6}. A = B = 6, A B = 1. Hence, P(A B) = = P(A)P(B). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 13 / 23

14 Conditional probability Independence Definition Let (Ω, P) be a probability space and I an arbitrary index set. A family (A i ) i I of events is called independent with respect to P if, for any finite subset J I, we have P( i J A i ) = i J P(A i ). Example (Dependence in spite of pairwise independence) Two coins. B 1 A = {first toss is head} = {1} {0, 1}, B = {second toss is head} = {0, 1} {1}, C = {both tosses give the same} = {(0, 0), (1, 1)}. Then, P(A B) = 1/4 = P(A)P(B), 0 P(A C) = 1/4 = P(A)P(C), C 0 1 and P(B C) = 1/4 = P(B)P(C), while P(A B C) = 1/4 1/8 = P(A)P(B)P(C). A Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 14 / 23

15 Random variables Random variables and their distributions Random variables Often, we are interested in quantities that depend on randomness. Their value depends on a random outcome of an experiment. After the experiment is done, the outcome ω Ω is known and X takes a particular value, X : Ω R. In general, some values are more likely than others. X Sample space Ω R Definition A random variable is a function X : Ω R. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 15 / 23

16 Random variables Random variables and their distributions Examples (of random variables) Tossing a coin n-times. ω = (ω 1,..., ω n ) Ω = {0, 1} n. X is the # of heads, X(ω) = n ω i=1 i {0, 1,..., n}. Rolling a die n times. ω = (ω(1),..., ω(n)) {1,..., 6} n, Y is the maximum, Y(ω) = max{ω(1),..., ω(n)}. Sequences of n zeros and ones. Z is the longest segment of ones, Z(ω) = k n. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 16 / 23

17 Random variables Random variables and their distributions Example (Minimum from two tosses of a 4-sided die) Two tosses of a fair 4-sided die (regular simplex). Ω = {1, 2, 3, 4} 2. Consider the random variable X(ω) = min{ω(1), ω(2)}: X R P(X = 1) = P({ω : X(ω) = 1}) = = P({(1, 1), (1, 2), (1, 3), (1, 4), (2, 1), (3, 1), (4, 1)}) = Similarly, P(X = 2) = P({(2, 2), (2, 3), (2, 4), (3, 2), (4, 2)}) = 5 16, P(X = 3) = P({(3, 3), (3, 4), (4, 3)}) = 3 16, and P(X = 4) = P({(4, 4)}) = Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 17 / 23

18 Random variables Probability mass function We are still assuming that Ω is at most countable. Thus, the image set X(Ω) = {x R : x = X(ω) for some ω Ω} is also at most countable and it makes sense to evaluate the probability of the event {X = x} := {ω Ω X(ω) = x}, Definition The probability mass function (PMF) of a random variable X is the function p X (x) = P({X = x}) = P(X = x) giving the probability of each numerical value that the random variable can take. Notice that x X(Ω) p X (x) = 1. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 18 / 23

19 Random variables Probability mass function Example (The number of heads for n fair coin tosses) X(ω) = n i=1 ω i {0, 1,..., n}. We have p X (k) = P(X = k) = ω: n i=1 ωi=k 1 = ( ) n 2 n k 2 n for any k {0, 1,..., n} and p X (x) = 0 for any x {0, 1,..., n}. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 19 / 23

20 Random variables Distribution function Distribution function To evaluate the likelihood of particular values (or subsets of values) of a random variable X we need to evaluate the probabilities of events of the form {X A } := {ω Ω X(ω) A } = X 1 (A ). For this it is sufficient to know its distribution function: Definition The (cumulative) distribution function (CDF) of a random variable X is the function F X : R [0, 1]: F X (x) = P(X x) = P(X 1 ((, x])). Roughly, p X (x) = P(X = x) = F(x + ɛ) F(x ɛ). Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 20 / 23

21 Random variables Distribution function Example Binomial distribution (tossing of an unfair coin) Ω = {0, 1} n, P(ω) = n i=1 pωi (1 p) 1 ωi. For X(ω) = ω i we have p X (k) = ( n k) pk (1 p) n k and F X (x) = ( ) n p i (1 p) n i i i x n = 20, p = 1/4: Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 21 / 23

22 Random variables Distribution function Properties of a distribution function Proposition A distribution function F X has the following properties: (a) lim x F X (x) = 0, lim x + F X (x) = 1. (b) x < y, implies F X (x) F X (y). (c) F is right continuous: lim y x+ F X (y) = F X (x). If X(Ω) is bounded, then F X (x) = 0 for x < min k Calculation of the probability mass function p X (x): x k and F X (x) = 1 for x max x k. k collect all ω for which X(ω) = x and sum their probabilities. Calculation of the distribution function F X (x): collect all ω for which X(ω) x and sum their probabilities. Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 22 / 23

23 Random variables Distribution function Example (Minimum from two tosses of a 4-sided die (continuation)) X 4 X(ω) = min{ω(1), ω(2)}: R Probability mass function: px /16 5/16 3/16 1/ x Distribution function: 4 FX x Roman Kotecký, Rudolf Blažek (FIT ČVUT) Statistika pro informatiku MI-SPI, ZS 2011/12, Přednáška 2 23 / 23