Quick Review of Probability

Size: px

Start display at page:

Download "Quick Review of Probability"

Alfred Franklin
5 years ago
Views:

1 Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter 2 & Teachig Material 2. D. P. Bertsekas, J. N. Tsitsiklis. Itroductio to Probability.

2 Basic Ideas Defiitio: A experimet is a process that results i a outcome that caot be predicted i advace with certaity Examples: Rollig a die Tossig a coi Weighig the cotets of a box of cereal Defiitio: The set of all possible outcomes of a experimet is called the sample space for the experimet Examples: For rollig a fair die, the sample space is {1, 2, 3, 4, 5, 6} For a coi toss, the sample space is {heads, tails} For weighig a cereal box, the sample space is (0, ), a more reasoable sample space is (12, 20) for a 16 oz. box (with a ifiite umber of outcomes) Statistics-Berli Che 2

3 More Termiology Defiitio: A subset of a sample space is called a evet The empty set Ø is a evet The etire sample space is also a evet A give evet is said to have occurred if the outcome of the experimet is oe of the outcomes i the evet. For example, if a die comes up 2, the evets {2, 4, 6} ad {1, 2, 3} have both occurred, alog with every other evet that cotais the outcome 2 Statistics-Berli Che 3

4 Combiig Evets The uio of two evets A ad B, deoted A B, is the set of outcomes that belog either to A, to B, or to both I words, A B meas A or B. So the evet A or B occurs wheever either A or B (or both) occurs Example: Let A = {1, 2, 3} ad B = {2, 3, 4} The A B = {1, 2, 3, 4} Statistics-Berli Che 4

5 Itersectios The itersectio of two evets A ad B, deoted by A B, is the set of outcomes that belog to A ad to B I words, A B meas A ad B. Thus the evet A ad B occurs wheever both A ad B occur Example: Let A = {1, 2, 3} ad B = {2, 3, 4} The A B = {2, 3} Statistics-Berli Che 5

6 Complemets The complemet of a evet A, deoted A c, is the set of outcomes that do ot belog to A I words, A c meas ot A. Thus the evet ot A occurs wheever A does ot occur Example: Cosider rollig a fair sided die. Let A be the evet: rollig a six = {6}. The A c = ot rollig a six = {1, 2, 3, 4, 5} Statistics-Berli Che 6

7 Mutually Exclusive Evets Defiitio: The evets A ad B are said to be mutually exclusive if they have o outcomes i commo A, A,..., A More geerally, a collectio of evets is said to 1 2 be mutually exclusive if o two of them have ay outcomes i commo Sometimes mutually exclusive evets are referred to as disjoit evets Statistics-Berli Che 7

8 Example Whe you flip a coi, you caot have the coi come up heads ad tails The followig Ve diagram illustrates mutually exclusive evets Statistics-Berli Che 8

9 Probabilities Defiitio: Each evet i the sample space has a probability of occurrig. Ituitively, the probability is a quatitative measure of how likely the evet is to occur Give ay experimet ad ay evet A: The expressio P(A) deotes the probability that the evet A occurs P(A) is the proportio of times that the evet A would occur i the log ru, if the experimet were to be repeated over ad over agai Statistics-Berli Che 9

10 Axioms of Probability 1. Let S be a sample space. The P(S) = 1 2. For ay evet A, 0 PA ( ) 1 3. If A ad B are mutually exclusive evets, the P( A B) = P( A) + P( B) A, A,... More geerally, if are mutually exclusive 1 2 evets, the PA ( A...) = PA ( ) + PA ( ) Statistics-Berli Che 10

11 A Few Useful Thigs For ay evet A, P(A c ) = 1 P(A) Let Ø deote the empty set. The P(Ø) = 0 If A is a evet, ad A = { } (ad are mutually exclusive), the P(A) = P(E1) + P(E2) +.+ P(E ). E1, E2,..., E E 1, E2,..., E Additio Rule (for whe A ad B are ot mutually exclusive): PA ( B) = PA ( ) + PB ( ) PA ( B) Statistics-Berli Che 11

12 Coditioal Probability ad Idepedece Defiitio: A probability that is based o a part of the sample space is called a coditioal probability E.g., calculate the probability of a evet give that the outcomes from a certai part of the sample space occur Let A ad B be evets with P(B) 0. The coditioal probability of A give B is PAB ( ) = PA ( B) PB ( ) Ve diagram Statistics-Berli Che 12

13 More Defiitios Defiitio: Two evets A ad B are idepedet if the probability of each evet remais the same whether or ot the other occurs If P(B) 0 ad P(B) 0, the A ad B are idepedet if P(B A) = P(B) or, equivaletly, P(A B) = P(A) If either P(A) = 0 or P(B) = 0, the A ad B are idepedet Are A ad B idepedet (?) Statistics-Berli Che 13

14 The Multiplicatio (Chai) Rule If A ad B are two evets ad P(B) 0, the P(A B) = P(B)P(A B) If A ad B are two evets ad P(A) 0, the P(A B) = P(A)P(B A) If P(A) 0, ad P(B) 0, the both of the above hold If A ad B are two idepedet evets, the P(A B) = P(A)P(B) This result ca be exteded to more tha two evets Statistics-Berli Che 14

15 Law of Total Probability If A 1,, A are mutually exclusive ad exhaustive evets, ad B is ay evet, the P(B) = P(A 1 B) + + P(A B) Exhaustive evets: The uio of the evets cover the sample space S= A 1 A 2 A Or equivaletly, if P(A i ) 0 for each A i, P(B) = P(B A 1 )P(A 1 )+ + P(B A )P(A ) Statistics-Berli Che 15

Example Customers who purchase a certai make of car ca order a egie i ay of three sizes. Of all the cars sold, 45% have the smallest egie, 35% have a medium-sized egie, ad 20% have the largest.

16 Example Customers who purchase a certai make of car ca order a egie i ay of three sizes. Of all the cars sold, 45% have the smallest egie, 35% have a medium-sized egie, ad 20% have the largest. Of cars with smallest egies, 10% fail a emissios test withi two years of purchase, while 12% of those with the medium size ad 15% of those with the largest egie fail. What is the probability that a radomly chose car will fail a emissios test withi two years? Statistics-Berli Che 16

17 Solutio Let B deote the evet that a car fails a emissios test withi two years. Let A 1 deote the evet that a car has a small egie, A 2 the evet that a car has a medium size egie, ad A 3 the evet that a car has a large egie. The P(A 1 ) = 0.45, P(A 2 ) = 0.35, ad P(A 3 ) = Also, P(B A 1 ) = 0.10, P(B A 2 ) = 0.12, ad P(B A 3 ) = By the law of total probability, P(B) = P(B A 1 ) P(A 1 ) + P(B A 2 )P(A 2 ) + P(B A 3 ) P(A 3 ) = 0.10(0.45) (0.35) (0.20) = Statistics-Berli Che 17

18 Statistics-Berli Che 18 Bayes Rule Let A 1,, A be mutually exclusive ad exhaustive evets, with P(A i ) 0 for each A i. Let B be ay evet with P(B) 0. The = = = i i i k k k k A P B A P A P B A P B P B A P B A P 1 ) ( ) ( ) ( ) ( ) ( ) ( ) (

19 Example The proportio of people i a give commuity who have a certai disease (D) is A test is available to diagose the disease. If a perso has the disease, the probability that the test will produce a positive sigal (+) is If a perso does ot have the disease, the probability that the test will produce a positive sigal is If a perso tests positive, what is the probability that the perso actually has the disease? Statistics-Berli Che 19

20 Solutio Let D represet the evet that a perso actually has the disease Let + represet the evet that the test gives a positive sigal We wish to fid P(D +) We kow P(D) = 0.005, P(+ D) = 0.99, ad P(+ D C ) = 0.01 Usig Bayes rule P( D + ) = P( + P( + D) P( D) D) P( D) + P( + D C ) P( D C ) = 0.99(0.005) 0.99(0.005) (0.995) = Statistics-Berli Che 20

21 Radom Variables Defiitio: A radom variable assigs a umerical value to each outcome i a sample space We ca say a radom variable is a real-valued fuctio of the experimetal outcome Defiitio: A radom variable is discrete if its possible values form a discrete set Statistics-Berli Che 21

22 Example The umber of flaws i a 1-ich legth of copper wire maufactured by a certai process varies from wire to wire. Overall, 48% of the wires produced have o flaws, 39% have oe flaw, 12% have two flaws, ad 1% have three flaws. Let be the umber of flaws i a radomly selected piece of wire The, P( = 0) = 0.48, P( = 1) = 0.39, P( = 2) = 0.12, ad P( = 3) = 0.01 The list of possible values 0, 1, 2, ad 3, alog with the probabilities of each, provide a complete descriptio of the populatio from which was draw Statistics-Berli Che 22

23 Probability Mass Fuctio The descriptio of the possible values of ad the probabilities of each has a ame: The probability mass fuctio Defiitio: The probability mass fuctio (deoted as pmf) of a discrete radom variable is the fuctio p(x) = P( = x). The probability mass fuctio is sometimes called the probability distributio Statistics-Berli Che 23

24 Cumulative Distributio Fuctio The probability mass fuctio specifies the probability that a radom variable is equal to a give value A fuctio called the cumulative distributio fuctio (cdf) specifies the probability that a radom variable is less tha or equal to a give value The cumulative distributio fuctio of the radom variable is the fuctio F(x) = P( x) Statistics-Berli Che 24

25 Example Recall the example of the umber of flaws i a radomly chose piece of wire. The followig is the pdf: P( = 0) = 0.48, P( = 1) = 0.39, P( = 2) = 0.12, ad P( = 3) = 0.01 For ay value x, we compute F(x) by summig the probabilities of all the possible values of x that are less tha or equal to x F(0) = P( 0) = 0.48 F(1) = P( 1) = = 0.87 F(2) = P( 2) = = 0.99 F(3) = P( 3) = = 1 Statistics-Berli Che 25

26 More o Discrete Radom Variables Let be a discrete radom variable. The The probability mass fuctio (cmf) of is the fuctio p(x) = P( = x) The cumulative distributio fuctio (cdf) of is the fuctio F(x) = P( x) F( x) = p( t) = P( = t) t x px ( ) = P ( = x) = 1, where the sum is over all the possible x values of x t x Statistics-Berli Che 26

27 Mea ad Variace for Discrete Radom Variables The mea (or expected value) of is give by μ = xp( = x) x where the sum is over all possible values of, also deoted as E[ ] The variace of is give by 2 2 σ = ( x μ ) P( = x) x = xp ( = x) μ. x 2 2 [( ) 2 ] μ, also deotedas E [ ] 2 E[ ] ( ) 2, also deoted as E The stadard deviatio is the square root of the variace Statistics-Berli Che 27

28 The Probability Histogram Whe the possible values of a discrete radom variable are evely spaced, the probability mass fuctio ca be represeted by a histogram, with rectagles cetered at the possible values of the radom variable The area of the rectagle cetered at a value x is equal to P( = x) Such a histogram is called a probability histogram, because the areas represet probabilities Statistics-Berli Che 28

29 Example The followig is a probability histogram for the example with umber of flaws i a radomly chose piece of wire P( = 0) = 0.48, P( = 1) = 0.39, P( = 2) = 0.12, ad P( = 3) = 0.01 Figure 2.8 Statistics-Berli Che 29

30 Cotiuous Radom Variables A radom variable is cotiuous if its probabilities are give by areas uder a curve The curve is called a probability desity fuctio (pdf) for the radom variable. Sometimes the pdf is called the probability distributio Let be a cotiuous radom variable with probability desity fuctio f(x). The f( x) dx= 1. Statistics-Berli Che 30

31 Computig Probabilities Let be a cotiuous radom variable with probability desity fuctio f(x). Let a ad b be ay two umbers, with a < b. The b Pa ( b) = Pa ( < b) = Pa ( < b) = f ( xdx ). I additio, P ( a) = P ( < a) = f( xdx ) P ( a) = P ( > a) = f( xdx ). a a a Statistics-Berli Che 31

32 More o Cotiuous Radom Variables Let be a cotiuous radom variable with probability desity fuctio f(x). The cumulative distributio fuctio (cdf) of is the fuctio x F( x) = P( x) = f( t) dt. The mea of is give by μ = xf ( xdx )., also deoted as E[ ] The variace of is give by σ 2 2 = x μ ( ) f( x) dx = x f( x) dx μ. 2 2 [( ) 2 ] μ 2 ( ) 2 E, also deotedas E E [ ] [ ], also deoted as Statistics-Berli Che 32

33 Media ad Percetiles Let be a cotiuous radom variable with probability mass fuctio f(x) ad cumulative distributio fuctio F(x) The media of is the poit x m that solves the equatio F( xm) = P( xm) = f( x) dx= 0.5. If p is ay umber betwee 0 ad 100, the pth percetile is the poit x p that solves the equatio F( xp) = P( xp) = f( x) dx= p/100. x m x p The media is the 50 th percetile Statistics-Berli Che 33

34 Liear Fuctios of Radom Variables If is a radom variable, ad a ad b are costats, the μ σ σ + = aμ + b a b a + b = a σ a + b = a σ Statistics-Berli Che 34

35 More Liear Fuctios If ad Y are radom variables, ad a ad b are costats, the μ μ μ μ μ. a + by = a + by = a + b Y More geerally, if 1,, are radom variables ad c 1,, c are costats, the the mea of the liear combiatio c 1 1,, c is give by μc + c c = c1μ + c2 μ cμ Statistics-Berli Che 35

36 Two Idepedet Radom Variables If ad Y are idepedet radom variables, ad S ad T are sets of umbers, the P( S ad Y T) = P( S) P( Y T). More geerally, if 1,, are idepedet radom variables, ad S 1,, S are sets, the P( S, S,..., S ) = P( S ) P( S )... P( S ) Statistics-Berli Che 36

37 Variace Properties If 1,, are idepedet radom variables, the the variace of the sum is give by σ = σ + σ σ If 1,, are idepedet radom variables ad c 1,, c are costats, the the variace of the liear combiatio c c is give by σ c + c c = c1 σ + c2 σ cσ Statistics-Berli Che 37

38 More Variace Properties If ad Y are idepedet radom variables with 2 2 variaces σ ad σ Y, the the variace of the sum + Y is σ σ σ Y= + Y. The variace of the differece Y is σ σ σ Y = + Y. Statistics-Berli Che 38

39 Idepedece ad Simple Radom Samples Defiitio: If 1,, is a simple radom sample, the 1,, may be treated as idepedet radom variables, all from the same populatio Phrased aother way, 1,, are idepedet, ad idetically distributed (i.i.d.) Statistics-Berli Che 39

40 Properties of (1/4) If 1,, is a simple radom sample from a populatio with mea μ ad variace σ 2, the the sample mea is a radom variable with mea of sample mea variace of sample mea μ σ = 2 = μ 2 σ. = L+ The stadard deviatio of is σ = σ. Statistics-Berli Che 40

41 Properties of (2/4) Populatio parameters ( 2 μ,σ ) simple radom sample of size sample mea = x 1 ( = 37.8) sample mea simple radom sample of size = x 2 ( = 40.2) sample mea simple radom sample of size = x 3 ( = 38.6) 1, 2, K, are i.i.d ad follow the same distributio sample mea cae be view x, x2, K, x,k 1 = ( L + ) as a radom variable with values 1 k ca be represeted as Statistics-Berli Che 41

42 Statistics-Berli Che 42 Properties of (3/4) [ ] ( ) μ μ μ μ μ μ μ μ μ = = = = = L L L E,, 2, 1 K are i.i.d ad follow the same distributio with mea μ ( ) ( ) σ σ σ σ σ σ σ σ μ σ = = = = = L L L E,, 2, 1 K are idetically distributed (follow the same distributio with variace ),, 2, 1 K 2 σ are idepedet

43 Properties of (4/4) μ mea of sample mea (equal to populatio mea ) μ sample mea = xi sample mea = x j The spread of sample mea is determied by the variace of sample 2 mea ( equal to 2 σ where is the populatio variace) σ 2 σ Statistics-Berli Che 43

44 Joitly Distributed Radom Variables If ad Y are joitly discrete radom variables: The joit probability mass fuctio of ad Y is the fuctio p( xy, ) = P ( = xad Y= y) The margial probability mass fuctios of ad Y ca be obtaied from the joit probability mass fuctio as follows: p ( x) = P ( = x) = pxy (, ) p( y) = PY ( = y) = pxy (, ) y where the sums are take over all the possible values of Y ad of, respectively The joit probability mass fuctio has the property that pxy (, ) = 1 x y where the sum is take over all the possible values of ad Y Y x Statistics-Berli Che 44

45 Joitly Cotiuous Radom Variables If ad Y are joitly cotiuous radom variables, with joit probability desity fuctio f(x,y), ad a < b, c < d, the P ( a b d b ad c Y d ) = f ( x, y ) dydx. a c The joit probability desity fuctio has the property that f( x, y) dydx= 1. Statistics-Berli Che 45

46 Margials of ad Y If ad Y are joitly cotiuous with joit probability desity fuctio f(x,y), the the margial probability desity fuctios of ad Y are give, respectively, by f ( x) = f( x, y) dy f ( y) = f( x, y) dx. Y Such a process is called margializatio Statistics-Berli Che 46

47 More Tha Two Radom Variables If the radom variables 1,, are joitly discrete, the joit probability mass fuctio is px (,..., x) = P ( = x,..., = x) If the radom variables 1,, are joitly cotiuous, they have a joit probability desity fuctio f(x 1, x 2,, x ), where P a b a b f x x dx dx 1 ( 1 1 1,..., ) = L ( 1,..., ) a a1 for ay costats a 1 b 1,, a b b b Statistics-Berli Che 47

48 Meas of Fuctios of Radom Variables (1/2) If the radom variables 1,, are joitly discrete, the joit probability mass fuctio is px (,..., x) = P ( = x,..., = x) If the radom variables 1,, are joitly cotiuous, they have a joit probability desity fuctio f(x 1, x 2,, x ), where P a b a b f x x dx dx 1 ( 1 1 1,..., ) = L ( 1,..., ) a a1 for ay costats a 1 b 1,, a b. b b Statistics-Berli Che 48

49 Meas of Fuctios of Radom Variables (2/2) Let be a radom variable, ad let h() be a fuctio of. The: If is a discrete with probability mass fuctio p(x), the mea of h() is give by μ h( x) = hxpx ( ) ( ). x, also deoted as E h where the sum is take over all the possible values of [ ( )] If is cotiuous with probability desity fuctio f(x), the mea of h(x) is give by μ h ( x ) = hx ( ) f( xdx )., also deoted as E h [ ( )] Statistics-Berli Che 49

50 Fuctios of Joit Radom Variables If ad Y are joitly distributed radom variables, ad h(,y) is a fuctio of ad Y, the If ad Y are joitly discrete with joit probability mass fuctio p(x,y), μ = h xy pxy h Y (, ) (, ) (, ). x y where the sum is take over all possible values of ad Y If ad Y are joitly cotiuous with joit probability mass fuctio f(x,y), μ h (, Y ) h( x, y) f ( x, y) dxdy. = Statistics-Berli Che 50

51 Discrete Coditioal Distributios Let ad Y be joitly discrete radom variables, with joit probability desity fuctio p(x,y), let p (x) deote the margial probability mass fuctio of ad let x be ay umber for which p (x) > 0. The coditioal probability mass fuctio of Y give = x is p Y pxy (, ) ( y x) =. px ( ) Note that for ay particular values of x ad y, the value of p Y (y x) is just the coditioal probability P(Y=y =x) Statistics-Berli Che 51

52 Cotiuous Coditioal Distributios Let ad Y be joitly cotiuous radom variables, with joit probability desity fuctio f(x,y). Let f (x) deote the margial desity fuctio of ad let x be ay umber for which f (x) > 0. The coditioal distributio fuctio of Y give = x is f Y f ( xy, ) ( y x) =. f ( x) Statistics-Berli Che 52

53 Coditioal Expectatio Expectatio is aother term for mea A coditioal expectatio is a expectatio, or mea, calculated usig the coditioal probability mass fuctio or coditioal probability desity fuctio The coditioal expectatio of Y give = x is deoted by E(Y = x) or μ Y Statistics-Berli Che 53

54 Idepedece (1/2) Radom variables 1,, are idepedet, provided that: If 1,, are joitly discrete, the joit probability mass fuctio is equal to the product of the margials: px (,..., x) = p ( x)... p ( x) If 1,, are joitly cotiuous, the joit probability desity fuctio is equal to the product of the margials: f ( x,..., x ) = f( x )... f( x ). 1 1 Statistics-Berli Che 54

55 Idepedece (2/2) If ad Y are idepedet radom variables, the: If ad Y are joitly discrete, ad x is a value for which p (x) > 0, the p Y (y x)= p Y (y) If ad Y are joitly cotiuous, ad x is a value for which f (x) > 0, the f Y (y x)= f Y (y) Statistics-Berli Che 55

56 Covariace Let ad Y be radom variables with meas μ ad μ Y The covariace of ad Y is Cov(, ). Y = μ( μ )( Y μ ) Y A alterative formula is Cov( Y, ) = μ μ μ. Y Y Statistics-Berli Che 56

57 Correlatio Let ad Y be joitly distributed radom variables with stadard deviatios σ ad σ Y The correlatio betwee ad Y is deoted ρ,y ad is give by ρ Y, Cov(, Y ). σ σ = Or, called correlatio coefficiet Y For ay two radom variables ad Y -1 ρ,y 1. Statistics-Berli Che 57

58 Covariace, Correlatio, ad Idepedece If Cov(,Y) = ρ,y = 0, the ad Y are said to be ucorrelated If ad Y are idepedet, the ad Y are ucorrelated It is mathematically possible for ad Y to be ucorrelated without beig idepedet. This rarely occurs i practice Statistics-Berli Che 58

59 Example The pair of radom variables (, Y ) takes the values (1, 0), (0, 1), ( 1, 0), ad (0, 1), each with probability ¼ Thus, the margial pmfs of ad Y are symmetric aroud 0, ad E[] = E[Y ] = 0 Furthermore, for all possible value pairs (x, y), either x or y is equal to 0, which implies that Y = 0 ad E[Y ] = 0. Therefore, cov(, Y ) = E[( E[] )(Y E[Y ])] = 0, ad ad Y are ucorrelated However, ad Y are ot idepedet sice, for example, a ozero value of fixes the value of Y to zero Statistics-Berli Che 59

60 Variace of a Liear Combiatio of Radom Variables (1/2) If 1,, are radom variables ad c 1,, c are costats, the μ μ μ c c = c c c c = c c 1 + cc i j i j i= 1 j= i+ 1 σ σ... σ 2 Cov(, ). For the case of two radom variables σ Y = σ + σy + 2 Cov, ( Y ) Statistics-Berli Che 60

61 Variace of a Liear Combiatio of Radom Variables (2/2) If 1,, are idepedet radom variables ad c 1,, c are costats, the σ c c = c1 σ cσ I particular, σ = σ σ. 1 1 Statistics-Berli Che 61

62 Summary (1/2) Probability ad rules Coutig techiques Coditioal probability Idepedece Radom variables: discrete ad cotiuous Probability mass fuctios Statistics-Berli Che 62

63 Summary (2/2) Probability desity fuctios Cumulative distributio fuctios Meas ad variaces for radom variables Liear fuctios of radom variables Mea ad variace of a sample mea Joitly distributed radom variables Statistics-Berli Che 63

Quick Review of Probability

Quick Review of Probability Berli Che Departmet of Computer Sciece & Iformatio Egieerig Natioal Taiwa Normal Uiversity Refereces: 1. W. Navidi. Statistics for Egieerig ad Scietists. Chapter & Teachig Material.