Set Theory and Probability

Transcription

1 Set Theory ad Probablty Dr. Bob Baley Set Theory We beg wth a toc of dscusso. A set ca be cosdered to be ay collecto of zero or more obects or ettes covered by the toc of dscusso. Examles: hadwrtte umerals 0 to 9, cars that ca be foud Tawa. The collecto of all such obects wth the toc of dscusso s called the sace or uversal set Ω. A dvdual obect the set (a secfc umeral, or a secfc car) reresets a elemet of the sace. A elemet s also referred to as a obect or a ot the sace. Each elemet ca aear the set oly oce,.e., reeated elemets are ot allowed, or else they are gored ad are ot couted. Also, the orderg of the elemets s rrelevat to the defto of a set,.e., the elemets may be ut to ay arbtrary orderg wthout chagg the meag of the set. We ofte use catal letters (usually from begg of the alhabet) to refer to a set, ad lower case letters to refer to elemets. If a elemet a s a member of a set A, we wrte a A. If a set cotas o elemets from the sace, the t s called the ull set or emty set,. Oe set may be a subset or suerset of aother set. If all elemets of A are also elemets of B, the A s a subset of B, A B. I ths case B s a suerset of A, B A. A ad B are equvalet sets f A B ad B A. A s a roer subset of B, A B, f A s a subset of B ad B cotas at least oe elemet that s ot A. We defe the uo (sometmes called the sum) of two sets, A B, as the collecto of all elemets that are cotaed ether set A or set B. A B {x: x A or x B}, whch reads all elemets x such that x s a elemet of a ad x s a elemet of B. We defe the tersecto (sometmes called the roduct) of two sets, A B or AB, as the collecto of all elemets that are cotaed strctly both A ad B. A B {x: x A ad x B} Uo ad tersecto ca be exteded to ay umber of sets: ad where s a dex ad I s a dex set, e.g., I {,, }. The comlemet of a set A s all the elemets Ω that are ot A; we deote the comlemet by A c or A. The set dfferece, B A, s the set of elemets of B that are ot cotaed A. Ths s also called the comlemet of A relatve to B. Note that the comlemet of A s A c Ω A. The symmetrc set dfferece s A B (A B) (B A). A Cartesa roduct of two sets A ad B, deoted A B, s the set of all ordered ars of the form (a, b), where (a A, b B). Ordered ar meas that the frst elemet s a elemet of set A, ad the secod elemet s a elemet of set B. Therefore, A B B A. Cartesa roduct ca be exteded to more tha two sets, e.g., A B C. If all the elemets belog to the same set, we ca wrte, for examle, A 3 for A A A. We refer to the sze of a set as ts cardalty. Ay two sets have the same cardalty f all the elemets of both sets ca be ut to a oe-to-oe corresodece wth each other. The elemets of a set may or may ot be coutable. (Coutable meas that the elemets may by some meas be ut a oe-to-oe corresodece wth the atural umbers, N. By some meas may refer to a smle lg u of the rows of elemets of the two sets, or t may refer I A I A 0/4/09 Set Theory ad Probablty of 0

2 to a clever arragemet of the elemets of the coutable set. For examle, the set Q of ratoal umbers may be ut a oe-to-oe corresodece wth the atural umbers by a systematc dagoal selecto of the order f the elemets of the ratoal umbers. Ths meas that Q ad N have the same cardalty!) For a coutable set f there s a maxmum corresodg umber N, the the set s sad to be fte, ad the cardalty of the set s, whch ths case s ust ts umber of elemets. If there s o maxmum umber, the the set s sad to be fte. If the set s ot coutable (caot be ut to a oe-to-oe corresodece wth the atural umbers), the t s called ucoutable, whch case t s also a fte set. However, there are dfferet degrees of fty,.e., oe fte set may cota a larger amout of elemets tha aother. For examle, the (coutable) set of atural umbers ad the (ucoutable) set of real umbers are both fte sets, but the latter obvously (well, t ca be roved) cotas may more elemets. So, the cardalty of the set R of real umbers s greater tha that of the set Z of all tegers. The symbol ℵ 0 (the frst letter of the Hebrew alhabet, aleh) s used to rereset coutable fty, ad c s used to rereset ucoutable fty. Ofte the cardalty of set A s deoted A. For two fte sets A B A B. We ca state a secfc value for the cardalty of oly a fte set, whch s the umber of elemets, as already dcated. However, for ay two sets, whether fte ad/or fte, we ca always state whether they have the same cardalty or whether oe has a larger cardalty tha the other. Note though that, for examle, the cardalty of Z Z s stll ℵ 0, ad that of R R s stll c, sce t s ossble to fd corresodeces wth N ad R, resectvely. Sets are sometmes llustrated vsually by the use of Ve dagrams. For such dagrams, t s ossble to wrte the meags of each rego or sub-rego set otato. Some useful laws or theorems of set theory are: Commutatve laws A B B A, where ca rereset ether or. Assocatve laws A (B C) (A B) C, where ca rereset ether or. Dstrbutve laws A (B C) (A B) (A C), ad A (B C) (A B) (A C). Trastvty If A B ad B C, the A C. Double comlemet (A c ) c A Dualty rcle other dettes DeMorga s laws I a set detty (lhs rhs) f we relace uos by tersectos, tersectos by uos, by Ω, ad Ω by, the the detty stll holds true. (The above laws are dettes). A Ω A; A Ω Ω; A ; A A c c A A ; A A Ω; A B A B c A A A A A A B A B, or geeral, A A I I A B A B, or geeral, A A Two sets, A,B Ω, are defed to be dsot (or mutually exclusve) f A B. Subsets A, A, are defed to be mutually exclusve f A A for every. I I 0/4/09 Set Theory ad Probablty of 0

3 Sometmes we may wat to comare the cotets of two sets whch may have some elemets commo. Oe way s to comute a closeess dex whch s gve by A B c where A meas the cardalty (umber of elemets) of set A. A B Probablty I robablty we refer to a exermet, whch s the coductg (or erformg) of a test or rocedure whose outcome (the result of the exermet) s ukow before erformg the exermet. For a toc of dscusso, we decde what costtutes (.e., defes, forms, makes u, reresets) a exermet, ad what costtutes a outcome. (A secod defto of exermet s gve below.) Performg a exermet oe tme s called a tral. I robablty we are cocered wth erformg a artcular exermet over ad over,.e., coductg may trals of a exermet. For each tme we erform the exermet (coduct a tral) aga, we may get a dfferet outcome. We say that the outcome of our exermet s radom. Because of ths, we sometmes refer to ths defto of exermet as a radom exermet. The samle sace, deoted by Ω, s the set of all ossble outcomes of a gve exermet. A samle sace s a sace the sese of set theory. A samle ot corresods to oe of the ossble outcomes of a exermet. A coutable samle sace (fte or fte, ad cludg Cartesa roducts of such sets) s called a dscrete samle sace. Ad a samle sace made of a terval of a ucoutable set (or Cartesa roducts of tervals) s called a cotuous samle sace. (It s ossble to have more comlcated samle saces.) Evets ad Evet Saces A evet ca be defed varous ways, from casual to very formal. Two casual deftos are () a collecto of outcomes of a exermet, ad () somethg whch may or may ot hae as the result of a exermet. I rollg a de, the samle sace s Ω {,,3,4,5,6}. Examles of a evet are a artcular umber from to 6, or the occurrece of a eve umber, or the occurrece of a umber 3. More formally, a evet s a grou or class of certa subsets of a samle sace. These subsets must be measurable. Measure (actually Lebesgue measure s requred) s a roerty that all subsets of a coutable set have, but that ot all ossble subsets of a cotuous set have. (Ths s related to the sgma algebra roerty below. Measure s lke the legth of a set of ots o the real axs or the area uder a fucto of x.) The result s that oly collectos of outcomes to whch we ca assg robabltes are evets. So, f a samle sace s dscrete, the all subsets of the samle sace are evets, but oly sesble subsets of a cotuous samle sace are evets. I robablty, we usually deote a evet by a catal, talcs letter, e.g., A. If we erform a radom exermet, the we say that evet A has occurred f the outcome belogs to the set A. The set of all ossble evets s called the evet sace. I robablty theory, the evet sace s requred to be a sgma-algebra. A sace S s defed to be a algebra f () Ω S, () f A s a elemet of the sace, the A c s also a elemet, ad () the uo of two elemets of S s also a elemet. To defe sgma algebra, the uo of two the art () s relaced by the uo of a coutably fte umber of elemets. Notce that ths requres that both Ω ad be elemets of the evet sace. It ca be show (usg DeMorga s laws) that () the 0/4/09 Set Theory ad Probablty 3 of 0

4 tersecto of two elemets of S s also a elemet of S, ad () the set dfferece of two elemets of S s also a elemet. Examle: A sace Ω has sx elemets: Ω {,,3,4,5,6}. The grou of subsets, Ω, {,3,5}, {,4,6}, {} s ot a algebra (ad hece, caot be a evet sace) because the comlemet of {} s ot a member of the grou, ad also because the uo of {} ad {,4,6}, whch s {,,4,6} also s ot a member. However, f we remove {} from the grou, the the grou s a algebra. O the real le,.e., Ω R, ay coutable uo or tersecto of oe or closed tervals or ots o the le forms a sgma algebra, ad hece, ca be regarded as a evet sace. Probablty fuctos, dstrbutos ad saces A robablty fucto ] s a fucto wth a evet sace as the doma, ad the real umber terval [0,] as the rage, ad satsfes the roertes: () 0 for every evet A the evet sace () Ω] (See certa evet below.) (3) If A, A, are mutually exclusve evets the evet sace, ad f A A s a elemet of the evet sace, the P A A ] A ] + ] +. [ A The robablty of the occurrece of a evet A s. Ths s the axomatc defto of robablty. (From ths, mathematcas have develoed a rgorous theory of robablty, but ths develomet requres the use of measure theory. However, for use ractcal robablty ad statstcs, ths defto s stll very useful.) Every evet has a robablty assocated wth t. Some roertes of ]: ] 0 If A s a evet a evet sace, the A c ]. If A ad B are evets a evet sace, the A B] + B] A B]. If A ad B are evets a evet sace ad A B, the B]. Boole s equalty If A, A, A are evets a evet sace, the P A A A ] A ] + A ] + + A ]. [ A samle sace (sce t s a sace) may cota a fte or fte umber of samle ots, N N(Ω), the cardalty of Ω. A robablty fucto s also called a robablty dstrbuto, artcularly for dscrete radom varables. The robablty of a evet,, ca alteratvely be defed as the chace (or lkelhood) that a evet wll occur, or as the relatve frequecy of a evet occurrg. (These ca be derved from the axomatc defto.) If A s the umber of occurreces of evet A trals,.e., the frequecy of occurrece of evet A, the the relatve frequecy of evet A s A /. The axomatc robablty ca be show to be gve by A lm. I a exermet, f we have o formato o whch to assg robabltes to outcomes, we ormally assume that all the outcomes are equally lkely,.e., have equal robabltes of 0/4/09 Set Theory ad Probablty 4 of 0

5 occurrg. If there are N ossble outcomes, the we assg a robablty of /N to each outcome. Examles are tossg a co (/) ad rollg a far de (/6). Secod defto: A robablty sace or exermet s the trlet (Ω, E, ]): () A uversal (or samle) sace Ω of all ossble outcomes. Ths s also called the certa evet or sure evet because t wll always hae. The ossble outcomes must be secfed. () A evet sace E whch s a algebra or sgma algebra collecto of evets (certa usually all subsets of Ω). (3) A robablty fucto assged to every evet A E; ths s called the robablty of evet A. The doma of ] s E. I lght of ths defto, ca also be called the mossble evet. Its robablty s ] 0. Mathematcally t reresets the evet that ca ever occur, such as defg a evet for rollg a de to be both eve ad less tha. However, t s ossble to have a evet to have zero robablty of occurrg, such as a artcular teger whe the samle sace s the set of all tegers Therefore, 0 does ot mly that evet A s the mossble evet. Smlarly, f B], t does ot follow that B s equal to the certa evet. I other words, B ad the certa evet are ot ecessarly equvalet sets. However, we ca say that evet A occurs wth robablty 0, ad that evet B occurs wth robablty. Samlg Samlg leads us from the toc of robablty to the toc of statstcs. We are ofte cocered about the roertes of elemets a samle sace. For examle, what s the average sze of the cars o Tawa s streets, or what s the most commo color? What s the rage sze,.e., what are the largest ad smallest? Clearly, ths could be aswered accurately f we sect every car o the streets of Tawa. However, ths s mractcal because t would take too log ad cost too may resources (moey, ma-ower, etc.) to do ths. Ths s where statstcs comes hady. Istead of sectg every car, we make do wth a samle that s much smaller tha samle sace. The we aalyze our samle to make statstcal fereces about the samle sace. I statstcs the samle sace s also referred to as the oulato, ad the oulato sze s the sze of the samle sace. To samle ca be defed two ways: () erform a radom exermet several tmes ad collect the outcomes to a set; () make radom selectos of elemets from a samle sace ad ut the selected elemets to a set. Ths set s called a samle. A samle ot s a elemet of the samle. The umber of elemets the samle s called the samle sze. (It s ossble to have a samle of ust oe elemet.) We also refer to () as to draw samles ots, although we ofte (somewhat accuratel say to draw samles. The samle sze should ot be cofused wth the sze of the samle sace (or oulato sze). There are two ways to make selectos from a samle sace. We ca draw wth relacemet, whch meas that whe we draw aga, the same samle ot s stll avalable to be draw. Theoretcally, we could draw the same samle ot reeatedly. Or, we ca draw wthout relacemet, whch meas that after we draw a samle ot, t s removed from the set from whch we are takg samle ots (.e., the samle sace), so that t wll ot be ossble to draw t a secod tme. We ofte use a samle to estmate roertes of the samle sace. For examle, comutg the mea of a samle s oe way to estmate the mea of the samle sace. 0/4/09 Set Theory ad Probablty 5 of 0

6 It should be oted out that the robablty dstrbuto of the samle wll ot be exactly equal to that of the samle sace. However, f our samle s large eough, ad we use a good method of samlg, the the dstrbuto of the samle should be a reasoable aroxmato of that of the samle sace. The, we should be able to make good estmates of the roertes that we desre from the samle sace. The detals of ths aroxmato s a maor area of statstcs. It ofte s suffcet to make estmates of certa arameters of a robablty dstrbuto, e.g., the mea, varace, rage, etc. Kowledge of oly these arameters ofte allows us to erform the aalyss that we wat to do. Codtoal robablty I may stuatos we are cocered wth the occurrece of more tha oe evet. I the above dscusso we have referred to evets A ad B, whch are members of the same evet sace. The evet (A ad B) s called a ot evet. We ca talk about the robablty of a ot evet, A B], or smly AB], whch s the robablty that the ot evet occurs. A ot evet ca cosst of more tha two evets. It s ofte the case that oe evet s flueced (or affected) by the occurrece of aother evet or stuato. For examle, f evet A s flueced by evet B, the we ca ask, what s the robablty of evet A gve that evet B has occurred? Ths s the codtoal robablty of A gve B, whch we deote A B]. Mathematcally, codtoal robablty s defed by AB] P [ A B] f B] > 0, ad s left udefed f B] 0. B] From ths we ca see that AB] A B] B] B f both ad B] are ozero. We call ad B] ucodtoal robabltes, sce they are the robabltes that A (or B) occur, whether or ot B (or A) occurs;.e., they do ot deed o (are ot codtoed o) aother evet. Defg the codtoal robablty has caused us to descrbe a ew robablty sace. Although t has the same samle sace ad outcomes, ad also the same evet sace as the robablty sace assocated wth the ucodtoal robablty, the robablty fucto for the codtoal robablty s dfferet. That s, the orgal fucto ] s dfferet from the fucto B]. It ca be show that B] satsfes the defto of a robablty fucto. Assume that B] > 0, the roertes of B] clude: B] 0, If A, A, A are mutually exclusve evets a evet sace, the A B] A B] A A c B] A B] A A B] A B] + A B] A A B] If A A, the A B] A B]. These are ust the same roertes as for ], whch s ot surrsg sce B] s art of a robablty sace ust as s ]. 0/4/09 Set Theory ad Probablty 6 of 0

7 Theorem of total robabltes. If B, B, B s a collecto of mutually exclusve evets a evet sace whch satsfy Ω B, ad B] > 0 for,,. (We ca say that the B s artto the samle sace.) The for every A a evet sace, A B ] B ]. s called the total robablty. Ths theorem remas true eve as. Corollary. For A ad B a evet sace, where 0 < B] <, the for every A, A B] B] + A B c ] B c ]. Bayes Theorem. If B, B, B s a collecto of mutually exclusve evets a evet sace whch satsfy the same codtos as B the theorem of total robablty, the for every A the evet sace for whch > 0, PB [ k PA [ Bk] PB [ k]. PA [ B] PB [ ] Ths theorem stll holds eve whe. Ths theorem gves us the a osteror robabltes B k of evets B k terms of ther a ror robabltes B k ] ad the codtoal robabltes A B k ]. The latter are sometmes called the trastoal robabltes [Peebles]. Corollary. For A ad B a evet sace, where > 0 ad 0 < B] <, the A B] B] B c c A B] B] + A B ] B ] Two evets A ad B are defed to be deedet f ad oly f ay oe of the followg codtos s met: () AB] B] () A B] f B] > 0 (3) B B] f > 0 Note that () says that the robablty of a occurrece of A does ot deed o B, ad hece, we call them deedet evets. Sometmes ths s also referred to as statstcal deedece. It s easy to show that two mutually exclusve evets C ad D are deedet f ad oly f C] D] 0. (Note: CD C D, so C D] 0 C] D], whch ca be true oly f ether C] 0 or D] 0.) O the other had, f C ad D are deedet ad C] 0 ad D] 0, the C ad D are ot mutually exclusve, because otherwse () would be volated. Please ote: Ideedece does ot mly mutually exclusve, or vce versa. I fact, oftetmes deedet sets are ot mutually exclusve. Several evets are deedet f they satsfy the followg. Let A, A, A be evets a evet sace. They are defed to be deedet f ad oly f A A ] A ] A ] A A A k ] A ] A ] A k ] for for, k, k 0/4/09 Set Theory ad Probablty 7 of 0

8 Note that ar-wse deedece aloe s ot suffcet for > evets to be deedet. Recall that A B] + B] A B]. Therefore f A ad B are deedet, A B] + B] B]. If two evets are deedet, the ther comlemets are also deedet, ad so are oe of the evets ad the comlemet of the other. Radom varables, cumulatve dstrbutos ad desty fuctos Radom varables are used to descrbe evets, ad a cumulatve dstrbuto fucto s used to assg the robabltes of certa evets defed terms of radom varables. For a robablty sace (Ω, E, ]), a radom varable or ( ) s a fucto wth doma the samle sace Ω ad rage the set of real umbers. The fucto ( ) must have the roerty that the subset of Ω, defed by A r {ω: (ω) r}, belogs to E for every real umber r. (Of course, ω Ω.) Sometmes we may abbrevate radom varable by rv. Note that ( ) s ot a fucto the ormal sese. It does ot have a umber as ts ut, but rather, the outcome of a exermet. The set A r s a evet. The secfed roerty allows us to talk about the robablty that a rv s less tha some umber r. For ay real umber r, the outcomes ω corresodg to (ω) r form (or costtute) a evet whch s E. I other words, gve ay umber r, there s a evet or collecto of evets for whch the corresodg rv s less tha or equal to r. Ths allows us to assg evets to every ossble value of a radom varable over the rage (, +),.e. R. Wth ths, we ca the talk about the robablty that a rv s less tha the umber r. Aother referece [Paouls] also requres ] +] 0 for the defto of a radom varable. (But, mathematcas say that x ad x + are meagless. A mathematca would take these the lmt.) From ths defto of radom varable, we ca ow show that a rv lyg a terval of the real umbers also corresods to a evet: {ω: r < (ω) r } s equvalet to {ω: r < (ω) r } {ω: (ω) r } {ω: (ω) r } The rght-had sde set dfferece s also a elemet of the evet sace E; see the roertes of a algebra gve above. Of course, we ormally dro the ω, ad ust wrte, e.g., r < r. Note that t s ot ecessary that the ossble values of sa all real values. For stace, mght be lmted to ust 0 ad, whch could rereset the absece ad resece, resectvely, of some feature or rmtve. We ca ow defe the followg: The cumulatve dstrbuto fucto (cdf) of a radom varable, deoted F ( ), s defed to be the fucto wth all real umbers as the doma, ad rage o the terval [0, ], ad whch satsfes F ( x] {ω: (ω) x}] for every real umber x from to +. Notce that the cdf s a fucto the usual sese,.e., from oe real umber to aother. It tells us how the radom varables are dstrbuted. Sometmes the term cumulatve s omtted. Whe we deal wth more tha oe rv, we use the subscrt of F to deote the rv, F Y ( for the rv Y. If there s o cofuso or ambguty, we ca omt the subscrt. I ths 0/4/09 Set Theory ad Probablty 8 of 0

9 case, we usually at least use the letter reresetg the rv as the argumet of the fucto F, e.g., F(z) for the cdf of rv Z. The cumulatve dstrbuto fucto has the followg roertes: () F( ) 0 F(+) () F( s a mootoe, odecreasg fucto of x: F(x ) F(x ) for x < x. (3) F( s cotuous from the rght: F(x + ) lm F( x + h) F(. 0< h 0 Note that (3) says that F( s ot ecessarly a cotuous fucto. Alteratve defto: Ay fucto wth all real umbers for the doma ad rage the terval [0, ] ad satsfyg the above three roertes s defed to be a cumulatve dstrbuto fucto. If x < x, the x < x ] F(x ) F(x ). If F( s dscotuous at x x, the sze of the dscotuty (the um) s equal to x ]. A radom varable s dscrete f ts rage s coutable. That s, takes o a coutable (ether fte or fte) umber of values, say x, x, x 3, Its cdf F (.) s also defed to be dscrete. We ca defe a dscrete desty fucto for a dscrete rv: x ] 0 f x x f x x,,, A fucto ( wth doma the real umbers ad rage [0, ] ca be sad to be a dscrete desty fucto f for some coutable set x, x, x 3,, t has the followg three roertes: () (x ) > 0 for x,,, 3, () ( 0 for x x,,, 3, (3) Σ (x ), where,, 3, A radom varable s defed to be cotuous f there exsts a fucto ( ) such that ts cdf x F ( t) dt for every real umber x. The robablty desty fucto (df) of s defed as the fucto ( ) whch satsfes ths tegral. Note that t s ot requred that ( ) be a cotuous fucto, as the case of a uform df. Lkewse, the dervatve of F ( ) eed ot be cotuous. It s oly requred that the umber of ots at whch t s ot dfferetable are coutable. Also, the fucto ( ) s ot ecessarly uque; t may abrutly chage value at a few ots. The requremet s that these ots do ot affect the value of the tegral, that s, they must have zero area uder them, or that the ots have zero measure. However, we usually gore such cases ad requre that ( ) satsfy certa cotuty cosderatos. I ths case, the df ca be cosdered uque. As the above equato, the cdf of a cotuous rv ca be obtaed from ts df by tegrato. Coversely, the df ca be obtaed by dfferetatg the cdf. A fucto ( wth doma the real umbers ad rage [0, ) s a robablty desty fucto f ad oly f t has these two roertes: 0/4/09 Set Theory ad Probablty 9 of 0

10 () ( 0 for all x ad () dx. It s ossble (ad actually commo) that a cdf be artly dscrete ad artly cotuous. For examle, a rv may be cotuous at all but a fte, coutable set of ots. At the coutable set of ots t acts lke a dscrete rv, leadg to dscotutes the cdf. Fally, ote that df ( s ot the same as a robablty fucto. Do ot cofuse them! The value of ( does ot rereset a robablty. It must be tegrated over some rage to be terreted as a robablty. Hece, the ame robablty desty fucto. Codtoal dstrbutos ad destes (Paouls. 98) Above we reseted codtoal robabltes. Here we reset codtoal dstrbutos ad destes whch combe a radom varable wth a evet. The codtoal dstrbuto of rv gve that evet A has occurred s ( x, F A (x A) x, where [ x, s the evet cosstg of all outcomes ω such that (ω) x ad ω A. The subscrt A s used oly to revet cofusg ths F(.) wth some other cdf fucto, e.g., F (. Oe s free to use whatever otato that s clear. See the remarks o. 6 about B]. Also, remember that [ x] s also a evet. Therefore, F A (x A) s ust a cdf, ad all roertes of cdf s also aly to t (. 7-8). We ofte dro the subscrt from F A (x A) ad smly wrte F(x A) whe the meag s clear. If s a cotuous rv, the ts df A (x A) ca be defed as the dervatve of the cdf: df( x A) x x + x A ( x A) lm. dx x 0 x Aga, t has all the roertes of ordary df s, ad we also usually dro the subscrt whe the meag s clear. There s also the roerty of total robablty. If evets A, A, A artto a samle sace,.e., they are mutually exclusve ad ther uo equals the sace, the the total cdf of x s F ( F (x A ) A ] F (x A ) A ] Total robablty ad Bayes theorem for cotuous radom varables Let A be some arbtrary evet ad B be the evet B {x < x }, B] 0, wth x < x. We kow that B] F(x ) F(x ), ad hece, B F(x A) F(x A), so A x < x ] A B] A B] B [ F( x B] B] A) F( x A)] F( x ) F( x ) Ths s true for ay tye of robablty fucto, dscrete, cotuous, or mxed. We are ow terested refg ths for the cotuous case oly. For a cotuous rv we ca defe the evet B { x}. If we lmt our terest to values of such that ( 0, the we ca defe A x] as the lmt [ F( x + x A) F( x A)] A x] lm A x x + x] lm x 0 x 0 F( x + F( x ) 0/4/09 Set Theory ad Probablty 0 of 0

11 where we have used the above equato, substtutg x x ad x x + x. Recall that the df s the dervatve of the cdf ad ca be defed as [ F( x + F( ] ( lm, ad smlarly for (x A). Usg these we fd that x 0 x P [ A x] ( x A) ( Notce that ths s true as log as ( 0, eve though x] 0. Note that Therefore, ( x A) dx, sce the tegrad has all the ormal roertes of a df. P [ A x] ( dx ( x A) dx ( x A) dx Ths gves the total robablty theorem for cotuous rv s: P [ A x] ( dx Now from the above we ca wrte Bayes theorem for cotuous rv s: ( x A) A x] ( A x] ( dx The cdf couterart to ths s A x] F( F( x A) Exected value of a fucto of a radom varable Let be a radom varable ad g( ) be a fucto wth doma ad rage the set of real umbers. The exected value of the fucto g( ) of the radom varable s defed by: () E[g()] g( x ) ( x ) f s a dscrete rv, ad rovded ths seres s absolutely coverget. () E[g()] g( dx f s a cotuous rv, ad rovded that g( dx <. A fucto of a radom varable, lke Y g(), s also a radom varable. However, after takg the exected value (by summg or tegratg over the desty fucto), the result s ot a radom varable. So, E[g()] s ust a real umber, ot a radom varable. 0/4/09 Set Theory ad Probablty of 0

12 We ca smlarly defe codtoal exected value. The cotuous case s E[g( A)] g ( x A dx, where evet A s ay evet the evet sace, ad could, A ) for examle, be some restrcto o, such as a. The ame exected value s somewhat arorate, sce, for examle, the exected value the dscrete case s usually ot a actual value that ca be obtaed,.e., t s ot oe of the dscrete values of the rv. Istead, oe actually gets a average value. The exected value of the radom varable tself s the mea µ, ad that of ( µ) s the varace σ. The varace s also the secod cetral momet. The th (raw) momet of s defed as m E[ ], f the exected value exsts. Some roertes of exected values are: () E[c] c for a costat c () E[c g()] c E[g()] for a costat c (3) E[c g () + c g ()] c E[g ()] + c E[g ()] (4) E[g ()] E[g ()] f g () g () for all Proerty () s called the homogeeous roerty; roerty (3) s called the learty roerty. There are some theorems cocerg exected value. Two are: Markov equalty: If s a rv ad g( ) s a oegatve fucto over real umbers, the E[ g( )] g( ) r] for every r > 0. r Note that we have made o assumtos about the ature of the dstrbuto. It may be dscrete, cotuous, or a mxture of the two. Chebyshev equalty: If s a radom varable wth fte varace, the µ rσ ] ( µ ) r σ ] for every r > 0. r Ths ca be show drectly from the Markov equalty. The Chebyshev equalty s useful for fdg a uer boud for the robablty of a evet from oly the mea ad varace. Aga, o kowledge of the dstrbuto s eeded; artcular we do ot eed to kow the df. The characterstc fucto of a radom varable Φ (u) s the verse Fourer trasform of the u desty fucto: Φ ( u ) E[ e ]. The characterstc fucto s esecally useful for fdg the robablty desty fuctos of fuctos of radom varables. For examle, t ca be used to show that the lear combato of two Gaussa (ormal) dstrbutos s also ormal. The characterstc fucto of a Gaussa dstrbuto s Φ we use to rereset the square root of. ( u) e µ u σ u, where If certa tegrato roertes ad other roertes hold (see Paouls), the the th momet of a arbtrary cotuous rv ca be foud from d Φ( u) du u0 m. Jot ad vector dstrbutos ad desty fuctos (Paouls ch. 6 ad 7) We ow tur to multle radom varables. Let ad Y be two dscrete or cotuous radom varables defed o the same robablty sace (Ω, E, ]). The {ω: (ω) x} ad {ω: Y(ω) y} are both evets. (see defto of rv o. 7) The ot cumulatve dstrbuto 0/4/09 Set Theory ad Probablty of 0

13 fucto of ad Y s defed by F Y (x, x, Y y]. Note that F Y (x, s a scalar valued fucto. Sometmes we dro the Y subscrt whe the meag s clear. Proertes of F Y (, ) are: () F(, 0 for all y, F(x, ) 0 for all x, F(,) () If x < x ad y < y, the x < x, y < Y y ] F(x,y ) F(x,y ) F(x,y ) + F(x,y ) 0 (3) F(x, s cotuous o the rght for each argumet: lm F[ x + x, y] lm F[ x, y + y] F( x, 0< x 0 0< y 0 A fucto that satsfes these three roertes s defed to be a bvarate cumulatve dstrbuto fucto. Ths defto does ot requre a referece to rv s. For a ot cdf F Y (x,, the dvdual cdf s F ( ad F Y ( are called the margal cdf s. They have the roertes: F ( F Y (x, ), ad F Y ( F Y (,. Therefore, whle we ca fd the margal cdf s from the ot cdf, the ooste s ot geerally true,.e., kowledge of F ( ad F Y ( s ot geeral eough formato to determe F Y (x,. The exceto s whe F Y (x, s searable. The ot cdf s bouded as follows: F + F ( F ( x, F F ( for all x ad y. Y Y + Jot dstrbutos are easly exteded to d dmesos. Istead of ad Y, we use a d- dmesoal radom varable,,, d. Ths ca be arraged to a vector: (,, d ) T, whch we call a radom vector. The the ot cumulatve dstrbuto fucto ca be wrtte F ( x,, d x d ]. As before, ths scalar valued fucto has a rage of [0, ], ad the same roertes aly as the -dmesoal case. Examles: F(, x, x 3 ) 0, for all x, ad x 3 ; margal cdf s clude F (x ) F (x,, ) ad F (x ) F (x, x, ), where x (x, x ). Y Jot dscrete ad cotuous robablty desty fuctos are defed smlarly to - dmesoal fuctos. For a ot dscrete desty fucto: ξ ] 0 f x ξ, f x ξ,, Note that ( ). If s ay reduced dmesoal vector of radom vector, the x (x ) s a margal desty ad ca be foud from ( by summg over all dscrete values of the comoets ot cotaed. s a cotuous radom vector (d-dmesoal ot cotuous radom varable) f ad oly f there exsts a (scalar valued) fucto ( 0 such that the ot cdf s x x xd ( t,, td ) dt F ( t) dt dt The ot robablty desty fucto (or vector df) s defed to be the fucto ( ). Whe we dscuss radom vectors, we usually omt the term ot sce t s mled by the term vector. (I other words, the term radom vector meas ot radom varables, ad vector df meas ot robablty desty fucto.) A fucto ( wth roertes smlar to those show o. 9 for the oe dmesoal case, s sad to be a ot robablty desty fucto f ad oly f t satsfes the two roertes: d 0/4/09 Set Theory ad Probablty 3 of 0

14 () ( 0 for all x ad () d x. The ot cotuous cdf ca be foud from the corresodg ot cotuous df by tegratg as the equato above. Therefore, the ot cotuous df ca be comuted from the corresodg cdf by dfferetatg d F x... x d We ca also comute exected values of fuctos (ether scalar or vector valued fuctos) from vector df s. The mea of a radom vector s the vector cotag the meas of each comoet: µ E [ ] x E [ E [ d x E [ d ] ] ] where E x ( x,, xd ) dx [ ] dx. The covarace matrx of s d T T T µ )( µ ) ] E [ µ µ Σ E [( ] Whe the dstrbuto s ot kow, t s commo to estmate the mea ad covarace from observatos. For examle, the mea s frequetly estmated by the average of each comoet, whch s the otmum estmator f the dstrbuto s Gaussa. The exected value for radom vectors has the same roertes, such as learty, as the exected value for -dmesoal radom varables (. 0-). Margal df s ca be comuted from the ot cotuous df by tegratg from to over oe or more comoets of the ot df, or summg for the dscrete df. O ages 6-7 we dscussed deedece of evets. Now we dscuss deedece of cdf s ad df s. A set of radom varables are deedet f the evets { x }, { x },, { d x d }are deedet for ay ad all x,, x d. Therefore, (,, d ) are deedet f ad oly f the ot cdf F (x,, x d ) F (x ) F d (x d ). The same s true for the dscrete or cotuous df: (x,, x d ) (x ) d (x d ). Aga, ar-wse deedece s ot suffcet. (P 37) However, oe subgrou of rv s ca be deedet of aother subgrou. That s, subgrou (,, k ) s deedet of subgrou ( k+,, d ) f ( (x ) (x ). By tegratg over ay oe or more comoets of ether or both grous, the resultg margals are stll deedet,.e., ay subgrou of s deedet of ay subgrou of. Note, however, that comoets wth (or ) are ot deedet. For two radom varables ad Y we have Y (x, ( Y ( f ad Y are deedet. But, we also have from the codtoal robablty Y (x, Y (x Y (. (Ths s show the ext secto.) Therefore, f ad Y are deedet, Y (x (. The, to show 0/4/09 Set Theory ad Probablty 4 of 0

15 that two radom varables are ot deedet, we eed oly show that Y (x deeds o y. Not oly ca we have deedet evets ad robablty dstrbutos, we ca have deedet trals of a exermet. (Ths s qute mortat atter recogto.) If we erform a exermet tmes such a way that each tral s deedet, the the outcomes ω wll be deedet. If a radom varable (ω ) s assocated wth each outcome, the before erformg the exermet tmes, we have a set of radom varables. Sce the exermets are deedet, the radom varables wll also be deedet. Hece, the cdf ad df of the ot radom varable (or the radom vector ) wll be deedet, e.g., (x,, x ) (x ) ( x ). Ths result s very mortat the develomet of the maxmum lkelhood estmator. Codtoal dstrbutos ad destes (Paouls. 0) Above (. 9) we defed the codtoal cdf of a rv gve that evet A has occurred, ad called t F(x A). We ow exted ths to bvarate dstrbuto ad desty fuctos of (-D) radom varables ad Y. If we let A {Y y}, the usg the above we have (assumg F Y ( 0): F x, Y y] ( x Y Y y] Y FY ( x, F ( To comute the corresodg df for cotuous rv s, we dfferetate wth resect to x: Y F ( x Y Y ( x Y F x Y Y ( x, / x F ( Y y y ξ Y Y ( x, ζ ) dζ ( ξ, ζ ) dξ dζ Of course, these equatos, we ca reverse the roles of x ad y. Now let us cosder the evet A {Y y}ad assume Y ( 0. We defe the codtoal cdf of x gve Y y as the lmt F ( x Y lm F ( x y < Y y + Y y 0 Y After some work ths ca be show to be F Y ( x Y x Y ( ξ, dξ ( Dfferetatg wth resect to x gves the codtoal df Y Y Y ( x, ( x Y ( Y Y Y ( x, ( ξ, dξ Usually these wll be wrtte as F Y (x ad Y (x, or eve as F(x ad (x whe the meag s clear. Note that aga we ca terchage x ad y. Y ) From the margal desty ( x, y dy ad the codtoal desty, rewrtte as Y(x, Y (x Y Y (, we have the total robablty desty 0/4/09 Set Theory ad Probablty 5 of 0

16 Y ( x Y ( dy The cotuous radom varable form of Bayes Theorem gves the a osteror robablty desty fucto terms of the a ror, codtoal, ad total robablty desty fuctos: ( y Y Y ( x Y ( For examle, t may be the case that for some exermet we wat to estmate a otmal value of radom varable Y, but we caot observe y drectly. The value of y s hdde from vew. (For examle, y may rereset a arameter of a robablty desty fucto.) However, the value of y s related to aother radom varable x, whch we ca observe. We kow how x deeds o y (the codtoal robablty ofte from theoretcal or assumed cosderatos), ad we kow from log exerece the a ror dstrbuto of y (the a ror robablt. Our goal s to mrove our estmate of the value of y gve observatos of x. To do ths we use ths form of Bayes Theorem. The above also geeralzes to vectors. Let a d-elemet radom vector be searated to two dsot sets of comoets ad. That s, the comoets of are comosed of the comoets of both ad. It does ot matter whch comoets are selected to be whch subvector. The the codtoal ot codtoal robablty desty of gve s (, ), x x ( ) x x ( x ) Ths ales to both dscrete ad cotuous radom vectors, rovded, of course, that the deomator s ot zero. The codtoal dscrete cumulatve dstrbuto s gve by F ( x x ) x x ], where the equalty ales to each comoet. Codtoal Exected Values (Paouls. 3, MGB. 57) The codtoal exected value of a fucto g(x, gve x s (for cotuous rv s): E [ g(, Y ) x] g( x, Y ( y dy g( x, Y ( x, dy Ths ca easly be exteded to the vector case. We smly relace x wth x, ad smlarly for ad y. If we let g(x, g(, the the exectato becomes E[g(Y) x] whch s a fucto of real varable x oly. Note that x s some usecfed, but ot radom, varable. The we ca wrte h( E[g(Y) x]. Now let argumet of fucto h( ) be a radom varable,.e., h(). The the exected value of h() s 0/4/09 Set Theory ad Probablty 6 of 0

17 E [E Y [ g( Y ) ]] E E E Y E[ g( Y ) x] Y g( g( [ g( Y )] g( g( [ g( Y )] [ h( )] Y Y Y, Y x ( dy dx ( y dy ( y h( x ( x, dy dx x dx dx dy dx (from last tegral) Hece, we have rove the theorem: If (,Y) s a -dmesoal radom varable, the E[g(Y)] E[E[g(Y) ]] ad, also as a artcular case, E[Y] E[E[Y ]]. (MGB ) To make the otato more clear, we could wrte E Y [g(y)] E Y [g(y)] E [E Y [g(y) ]]. But, t s customary to omt the subscrts from the exectato, relyg stead o the argumet to dcate the uderlyg df to be used. We ca smlarly show that E Y [g(,y)] E [E Y [g(,y) ]]. (Paouls) Aga, these ca be exteded to radom vectors. More o ot radom varables: As we kow, two radom varables are deedet f Y (x, ( Y ( The covarace of two radom varables s (see. 3): σ cov[ Y, ] E[( µ ) ( Y µ )] E[ Y] E[ EY ] [ ] xy x y Ther correlato coeffcet s ρ σ xy xy, whch has the roerty that ρ xy σ xσ y Note that the correlato coeffcet measures oly lear correlato. Two radom varables are called ucorrelated f ther covarace s 0. The followg are equvalet: σ xy 0 ρ xy 0 E[Y] E[] E[Y] Two radom varables are called orthogoal f E[Y] 0 (Whe are two rv s both ucorrelated ad orthogoal?) Theorem: (MGB. 53) If fucto g(x,, x ) h(x ), the E[g(,, )] E [h( )], ad as a secal case, f g(x,, x ) x, the E[g(x,, x )] E [ ] µ. (Remember that () s the ot df of, ad that (x ) s the margal df of x.) x Theorem: (MGB. 78) For radom varables,, (radom vector ) 0/4/09 Set Theory ad Probablty 7 of 0

18 E E [ ] (rove) ad var var[ ] + < cov[, ] where var[] s the varace, ad cov[] s the covarace. Note that the fal term ( tmes the double summato) s ust the sum of all the off-dagoal terms of the covarace matrx. There s a mortat dfferece betwee the above exected value of the sum ad the learty roerty of exected value (otes,. ). I the learty roerty, there s oly oe radom varable,, whch has a df (. I the reset theorem, we have ot radom varables, as metoed the recedg aragrah. But, as we ca see from ths theorem, the et result s the same. O the other had, ths theorem s ot obvous. Corollary: If,, are ucorrelated radom varables (.e., σ ρ 0, ), the var var[ ] Theorem: (MGB. 79) If R ad Y R m are two radom vectors, ad a ad b are two costat vectors, the cov m T T [ a, b Y ] a b cov[, Y ] Corollary: If R s a radom vector, ad a s a costat vector, the var T [ a ] a m a a cov[, var[ ] + ] a a a T Σ a cov[, ] More o samlg O ages 4 to 5 we dscussed samlg. It was oted that we ofte make estmates of oulato arameters (e.g., mea, varace or covarace). Now we are reared to take a closer look at estmatg the mea ad covarace. It s commo to comute the estmated mea µˆ of a oulato by takg the average (x ) of a set of observatos: ˆµ x x Each observato s the outcome of a exermet, so we ca cosder each observato to be radom varable;.e., a radom varable utl ts value s kow. I the most geeral case, we assume that they have a ot df, sce there wll be correlatos betwee them f they are ot deedet. Now, sce the estmated mea s a fucto of radom varables, t s tself a radom varable. Ad sce t s a radom varable, we ca ask what are ts mea ad varace. We use the above theorem for the exected value of a sum of radom varables: E [ ˆ] µ E E [ ] 0/4/09 Set Theory ad Probablty 8 of 0

19 But, sce the observatos are from the same oulato, they have the same mea, so E ˆ] µ µ, where µ s the oulato mea. [ We ca also comute the varace of the estmated mea, whch gves σ s the oulato varace. var[ ˆ] µ I addto to estmatg the mea of the oulato, we ca estmate ts varace: ˆ σ ( x It ca be show that E[ ˆ σ ] σ. σ where The estmate of oulato mea ad varace ca be exteded to estmatg the mea vector ad covarace matrx of vector samles: ˆµ ad x Σˆ ( x T T T ˆ)( µ x ˆ) µ x x ˆ µµ ˆ Aga, we have that the exected values of these two estmates are equal to the mea ad covarace of the oulato. Not oly ca we comute the exected values of the estmated mea ad varace, we could also comute ther varaces or covaraces, e.g., var[ ˆ σ ]. Ths s gve MGB. Refereces Peebles, Peyto, Probablty, Radom Varables ad Radom Sgal Prcles, 4 th edto, McGraw Hll, 00. Yates, Roy D. ad Davd J. Goodma, Probablty ad Stochastc Processes, d edto, Wley, 005. (a moder ad very readable md-level book) Paouls, Athaasos ad S. Ukrsha Plla, Probablty, Radom Varables, ad Stochastc Processes, 4th ed., McGraw Hll, 00. (the most famous book o these tocs for egeerg, ad very wdely used for seor ad frst year graduate level) [MGB] Mood, Alexader, Frakl Graybll, ad Duae Boes, Itroducto to the Theory of Statstcs, McGraw Hll, 974. (A very famous statstcs book at a somewhat advaced level ad stll avalable) 0/4/09 Set Theory ad Probablty 9 of 0

20 Idex a osteror, 7 a ror, 7 algebra, 3 A B Bayes theorem, 7,, 6 bvarate, 3, 5 C cardalty, Cartesa roduct, characterstc fucto, Chebyshev equalty, comlemet, codtoal cdf, 5 codtoal dstrbutos, 0 codtoal df, 5 codtoal robablty, 6, 7 cotuous samle sace, 3 correlato coeffcet, 7 coutable, covarace, 7, 8 covarace matrx, 4 cumulatve dstrbuto fucto, 8 D dscrete, 9 dscrete samle sace, 3 dsot, draw, 5 E elemet, estmate, 5, 4, 6, 8, 9 evet, 3, 5, 6, 8, 0 evet sace, 3, 7 exected value,, 6 exected values, 4 exermet, 3, 5, 8, 5, 6, 8 fte, frequecy, 4 F dex set, fte, ot cumulatve dstrbuto, ot evet, 6 ot robablty, 3 J M margal, 3, 4, 5 Markov equalty, orthogoal, 7 outcome, 3 O artto, 7 oulato, 5 robablty, 4 robablty desty fucto, 9 robablty fucto, 4, 6, 0 robablty sace, 5 radom, 3 radom varable, 8 radom vector, 3 relacemet, 5 samle, 5 samle sace, 3 samlg, 5, 8 sace, P R S T total robablty, 7,, 5 tral, 3 U ucodtoal robablty, 6 ucorrelated, 7, 8 ucoutable, deedet, 7, 4, 7 I 0/4/09 Set Theory ad Probablty 0 of 0