Review of Probability Axioms and Laws

Size: px

Start display at page:

Download "Review of Probability Axioms and Laws"

Dora Watkins
5 years ago
Views:

1 Review of robabilit ioms ad Laws Berli Che Deartmet of Comuter ciece & Iformatio Egieerig Natioal Taiwa Normal Uiversit Referece:. D.. Bertsekas, J. N. Tsitsiklis, Itroductio to robabilit, thea cietific, 2008.

2 What is robabilit? robabilit was develoed to describe heomea that caot be redicted with certait Frequec of occurreces ubjective beliefs Everoe accets that the robabilit (of a certai thig to hae) is a umber betwee 0 ad (?) Measures deduced from robabilit aioms ad theories (laws/rules) ca hel us deal with ad quatif iformatio Berli Che 2

3 ets (/2) set is a collectio of objects which are the elemets of the set If is a elemet of set, deoted b Otherwise deoted b set that has o elemets is called emt set is deoted b Ø et secificatio Coutabl fiite:,2,3,4,5,6 Coutabl ifiite: 0,2, 2,4, 4,... With a certai roert: k k 2 is iteger 0 satisfies such that Berli Che 3

4 ets (2/2) If ever elemet of a set is also a elemet of a set T, the is a subset of T Deoted b T or T If T ad T, the the two sets are equal Deoted b T The uiversal set, deoted b, which cotais all objects of iterest i a articular cotet cosider sets that are subsets of fter secifig the cotet i terms of uiversal set, we ol Berli Che 4

5 et Oeratios (/3) Comlemet The comlemet of a set with resect to the uiverse, is the set, amel, the set of all elemets that do c ot belog to, deoted b The comlemet of the uiverse Ø Uio The uio of two sets ad is the set of all elemets that belog to or T, deoted b T T or T Itersectio The itersectio of two sets ad is the set of all elemets that belog to both ad T, deoted b T T ad T T c T Berli Che 5

6 et Oeratios (2/3) The uio or the itersectio of several (or eve ifiite ma) sets Disjoit Two sets are disjoit if their itersectio is emt (e.g., T = Ø) artitio 2 2 collectio of sets is said to be a artitio of a set if the sets i the collectio are disjoit ad their uio is for some for all Berli Che 6

7 et Oeratios (3/3) Visualizatio of set oeratios with Ve diagrams Berli Che 7

8 The lgebra of ets The followig equatios are the elemetar cosequeces of the set defiitios ad oeratios De Morga s law Berli Che 8 c c c c,,,, U T U T T T c c., U T U T U T U T c Ø commutative distributive associative distributive

9 robabilistic Models (/2) robabilistic model is a mathematical descritio of a ucertait situatio It has to be i accordace with a fudametal framework to be discussed shortl Elemets of a robabilistic model The samle sace The set of all ossible outcomes of a eerimet The robabilit law ssig to a set of ossible outcomes (also called a evet) a oegative umber (called the robabilit of ) that ecodes our kowledge or belief about the collective likelihood of the elemets of Berli Che 9

10 robabilit ioms Berli Che 0

11 robabilistic Models (2/2) The mai igrediets of a robabilistic model Berli Che

12 amle aces ad Evets Each robabilistic model ivolves a uderlig rocess, called the eerimet That roduces eactl oe out of several ossible outcomes The set of all ossible outcomes is called the samle sace of the eerimet, deoted b subset of the samle sace (a collectio of ossible outcomes) is called a evet Eamles of the eerimet sigle toss of a coi (fiite outcomes) Three tosses of two dice (fiite outcomes) ifiite sequeces of tosses of a coi (ifiite outcomes) Throwig a dart o a square (ifiite outcomes), etc. Berli Che 2

13 amle aces ad Evets (2/2) roerties of the samle sace Elemets of the samle sace must be mutuall eclusive The samle sace must be collectivel ehaustive The samle sace should be at the right graularit (avoidig irrelevat details) Berli Che 3

14 Discrete robabilit Law robabilit Laws If the samle sace cosists of a fiite umber of ossible outcomes, the the robabilit law is secified b the robabilities of the evets that cosist of a sigle elemet. I articular, the robabilit of a evet s, s2,, s is the sum of the robabilities of its elemets: s, s2,, s s s 2 s s s s Discrete Uiform robabilit Law If the samle sace cosists of ossible outcomes which are equall likel (i.e., all sigle-elemet evets have the same robabilit), the the robabilit of a evet is give b umber of 2 elemet of Berli Che 4

15 Cotiuous Models robabilistic models with cotiuous samle saces It is iaroriate to assig robabilit to each sigle-elemet evet (?) Istead, it makes sese to assig robabilit to a iterval (oedimesioal) or area (two-dimesioal) of the samle sace Eamle: Wheel of Fortue b c 0.3? a b d a ??? Berli Che 5

16 roerties of robabilit Laws robabilit laws have a umber of roerties, which ca be deduced from the aioms. ome of them are summarized below Berli Che 6

17 Coditioal robabilit (/2) Coditioal robabilit rovides us with a wa to reaso about the outcome of a eerimet, based o artial iformatio uose that the outcome is withi some give evet B, we wish to quatif the likelihood that the outcome also belogs some other give evet Usig a ew robabilit law, we have the coditioal robabilit of give B, deoted b B, which is defied as: B B B If B has zero robabilit, B is udefied We ca thik of Bas out of the total robabilit of the elemets of B, the fractio that is assiged to ossible outcomes that also belog to B Berli Che 7

18 Coditioal robabilit (2/2) Whe all outcomes of the eerimet are equall likel, the coditioal robabilit also ca be defied as B umber of elemets of umber of elemets of B B ome eamles havig to do with coditioal robabilit. I a eerimet ivolvig two successive rolls of a die, ou are told that the sum of the two rolls is 9. How likel is it that the first roll was a 6? 2. I a word guessig game, the first letter of the word is a t. What is the likelihood that the secod letter is a h? 3. How likel is it that a erso has a disease give that a medical test was egative? 4. sot shows u o a radar scree. How likel is it that it corresods to a aircraft? Berli Che 8

19 Coditioal robabilities atisf the Three ioms Noegative: B 0 Normalizatio: B B 2 B B B dditivit:if ad are two disjoit evets 2 B 2 B 2 B B B 2 B B B 2 B B B B 2 distributive disjoit sets Berli Che 9

20 Multilicatio (Chai) Rule ssumig that all of the coditioig evets have ositive robabilit, we have The above formula ca be verified b writig For the case of just two evets, the multilicatio rule is siml the defiitio of coditioal robabilit Berli Che i i i i i i i i i i 2 2

21 Total robabilit Theorem Let,, be disjoit evets that form a artitio of the samle sace ad assume that i 0, for all i. The, for a evet B, we have B B B B B Note that each ossible outcome of the eerimet (samle sace) is icluded i oe ad ol oe of the evets,, Berli Che 2

22 Baes Rule Let, 2,, be disjoit evets that form a artitio of the samle sace, ad assume that i 0, for all i. The, for a evet such that we have i B i B B B 0 B i B B i i B i k k B k i B i B B Multilicatio rule Total robabilit theorem Berli Che 22

23 Ideedece (/2) Recall that coditioal robabilit B catures the artial iformatio that evet B rovides about evet secial case arises whe the occurrece of B rovides o such iformatio ad does ot alter the robabilit that has occurred B B B is ideedet of ( also is ideedet of ) B B B B B Berli Che 23

24 Ideedece (2/2) B B ad are ideedet => ad are disjoit (?) No! Wh? ad B are disjoit the However, if 0 ad B 0 B 0 B B B B 0 B 0 Two disjoit evets ad with ad are ever ideedet Berli Che 24

25 Coditioal Ideedece (/2) Give a evet C, the evets ad B are called coditioall ideedet if We also kow that B C C B C B C B C C C B C B C C If B C 0, we have a alterative wa to eress coditioal ideedece 3 B C C multilicatio rule 2 Berli Che 25

26 Coditioal Ideedece (2/2) Notice that ideedece of two evets ad B with resect to the ucoditioall robabilit law does ot iml coditioal ideedece, ad vice versa B B B C C B C If ad B are ideedet, the same holds for c (i) ad B c (ii) ad B (iii) c ad c B Berli Che 26

27 Ideedece of a Collectio of Evets We sa that the evets, 2,, are ideedet if i i i, for ever subset of,2,, i For eamle, the ideedece of three evets amouts to satisfig the four coditios ,, Berli Che 27

Radom Variables Give a eerimet ad the corresodig set of ossible outcomes (the samle sace), a radom variable associates a articular umber with each outcome This

28 Radom Variables Give a eerimet ad the corresodig set of ossible outcomes (the samle sace), a radom variable associates a articular umber with each outcome This umber is referred to as the (umerical) value of the radom variable We ca sa a radom variable is a real-valued fuctio of the eerimetal outcome w : w Berli Che 28

29 Radom Variables: Eamle eerimet cosists of two rolls of a 4-sided die, ad the radom variable is the maimum of the two rolls If the outcome of the eerimet is (4, 2), the value of this radom variable is 4 If the outcome of the eerimet is (3, 3), the value of this radom variable is 3 Ca be oe-to-oe or ma-to-oe maig Berli Che 29

30 Discrete/Cotiuous Radom Variables radom variable is called discrete if its rage (the set of values that it ca take) is fiite or at most coutabl ifiite fiite :, 2, 3, 4, coutabl ifiite :, 2, radom variable is called cotiuous (ot discrete) if its rage (the set of values that it ca take) is ucoutabl ifiite E.g., the eerimet of choosig a oit a from the iterval [, ] radom variable that associates the umerical value to the outcome is ot discrete a 2 a Berli Che 30

31 Cocets Related to Discrete Radom Variables For a robabilistic model of a eerimet discrete radom variable is a real-valued fuctio of the outcome of the eerimet that ca take a fiite or coutabl ifiite umber of values (discrete) radom variable has a associated robabilit mass fuctio (MF), which gives the robabilit of each umerical value that the radom variable ca take fuctio of a radom variable defies aother radom variable, whose MF ca be obtaied from the MF of the origial radom variable Berli Che 3

32 robabilit Mass Fuctio (discrete) radom variable is characterized through the robabilities of the values that it ca take, which is catured b the robabilit mass fuctio (MF) of, deoted or The sum of robabilities of all outcomes that give rise to a value of equal to Uer case characters (e.g., ) deote radom variables, while lower case oes (e.g., ) deote the umerical values of a radom variable The summatio of the oututs of the MF fuctio of a radom variable over all it ossible umerical values is equal to oe ' s are disjoit ad form a artitio of the samle sace Berli Che 32

33 Calculatio of the MF For each ossible value of a radom variable :. Collect all the ossible outcomes that give rise to the evet 2. dd their robabilities to obtai eamle: the MF of the radom variable = maimum roll i two ideedet rolls of a fair 4-sided die Berli Che 33

34 Eectatio The eected value (also called the eectatio or the mea) of a radom variable, with MF, is defied b E Ca be iterreted as the ceter of gravit of the MF (Or a weighted average, i roortio to robabilities, of the ossible values of ) The eectatio is well-defied if That is, coverges to a fiite value c 0 c Berli Che 34

35 Eectatios for Fuctios of Radom Variables Let be a radom variable with MF, ad let g be a fuctio of. The, the eected value of the radom variable g is give b E To verif the above rule g Let, ad therefore g g Y Y Y E g E g g Y? g g g Berli Che 35

36 Variace The variace of a radom variable is the eected value of a radom variable var The variace is alwas oegative (wh?) The variace rovides a measure of disersio of aroud its mea The stadard derivatio is aother measure of disersio, which is defied as (a square root of variace) E 2 2 E E 2 E var Easier to iterret, because it has the same uits as Berli Che 36

37 roerties of Mea ad Variace Let be a radom variable ad let a Y a b b where ad are give scalars a liear fuctio of The, E Y ae var b 2 Y a var g If is a liear fuctio of, the g ge E How to verif it? Berli Che 37

38 Joit MF of Radom Variables Let ad Y be radom variables associated with the same eerimet (also the same samle sace ad robabilit laws), the joit MF of ad Y is defied b, Y, Y, Y if evet is the set of all airs, that have a certai roert, the the robabilit of ca be calculated b, Y, Namel, ca be secified i terms of ad, Y, Y Berli Che 38

39 Margial MFs of Radom Variables The MFs of radom variables ad ca be calculated from their joit MF ad are ofte referred to as the margial MFs The above two equatios ca be verified b Berli Che 39 Y Y Y Y,,,,, Y Y Y,,,

40 Coditioig Recall that coditioal robabilit rovides us with a wa to reaso about the outcome of a eerimet, based o artial iformatio I the same sirit, we ca defie coditioal MFs, give the occurrece of a certai evet or give the value of aother radom variable Berli Che 40

41 The coditioal MF of a radom variable, coditioed o a articular evet with, is defied b (where ad are associated with the same eerimet) Normalizatio roert Note that the evets are disjoit for differet values of, their uio is Berli Che 4 Coditioig a Radom Variable o a Evet (/2) 0 Total robabilit theorem

42 Coditioig a Radom Variable o a Evet (2/2) grahical illustratio is obtaied b addig the robabilities of the outcomes that give rise to ad belog to the coditioig evet Berli Che 42

43 Coditioig a Radom Variable o other (/2) Y Let ad be two radom variables associated with the same eerimet. The coditioal MF Y of give Y is defied as, Y Y Y Y, Normalizatio roert, Y Y Y Y is fied o some value The coditioal MF is ofte coveiet for the calculatio of the joit MF multilicatio (chai) rule, ( ), Y Y Y Y Berli Che 43

44 Coditioig a Radom Variable o other (2/2) The coditioal MF ca also be used to calculate the margial MFs Visualizatio of the coditioal MF Berli Che 44 Y Y Y,, Y Y Y Y Y Y,,,,,,

45 Ideedece of a Radom Variable from a Evet radom variable is ideedet of a evet if ad, for all Require two evets ad be ideedet for all If a radom variable is ideedet of a evet ad 0 ad, for all Berli Che 45

46 Ideedece of Radom Variables (/2) Two radom variables ad Y are ideedet if or If a radom variable is ideedet of a radom variable Y, Y, Y, for all,, Y Y, for all, Y Y for all with, 0 all, Y, Y Y Y, for all with 0 ad all Y Berli Che 46

47 Ideedece of Radom Variables (2/2) Radom variables ad Y are said to be coditioall ideedet, give a ositive robabilit evet, if Or equivaletl,,, for all,, Y Y, for all with 0 ad all Y, Y Note here that, as i the case of evets, coditioal ideedece ma ot iml ucoditioal ideedece ad vice versa Berli Che 47

48 Etro (/2) Three iterretatios for quatit of iformatio. The amout of ucertait before seeig a evet 2. The amout of surrise whe seeig a evet 3. The amout of iformatio after seeig a evet The defiitio of iformatio: I ( i ) log 2 log 2 i i the robabilit of a evet Etro: the average amout of iformatio Have maimum value whe the robabilit (mass) fuctio is a uiform distributio i i I ( ) E log 2 i i log 2 i H ( ) E defie i 0log2 0 0 where, 2,..., i,... Berli Che 48

49 Etro (2/2) For Boolea classificatio (0 or ) 2,, 0 Etro ( ) log 2 2 log 2 2 Etro ca be eressed as the miimum umber of bits of iformatio eeded to ecode the classificatio of a arbitrar umber of eamles If c classes are geerated, the maimum of etro ca be Etro ( ) log 2 c Berli Che 49

Sets and Probabilistic Models

Sets and Probabilistic Models ets ad Probabilistic Models Berli Che Departmet of Computer ciece & Iformatio Egieerig Natioal Taiwa Normal iversity Referece: - D. P. Bertsekas, J. N. Tsitsiklis, Itroductio to Probability, ectios 1.1-1.2