Discrete Random Variables: Joint PMFs Conditioning and Indeendence Berlin Chen Deartment of Comuter Science & Information ngineering National Taiwan Normal Universit Reference: - D. P. Bertsekas J. N. Tsitsiklis Introduction to Probabilit Sections.5-.7
Motivation Given an eeriment e.g. a medical diagnosis The results of blood test is modeled as numerical values of a random variable The results of magnetic resonance imaging i (MRI 核磁共振攝影 is also modeled as numerical values of a random variable We would like to consider robabilities of events involving simultaneousl the numerical values of these two variables and to investigate their mutual coulings P { } I { } (? Probabilit-Berlin Chen
Joint PMF of Random Variables Let and be random variables associated with the same eeriment (also the same samle sace and robabilit laws the joint PMF of and is defined b ( P ( { } I { } P ( P ( if event is the set of all airs that have a certain roert then the robabilit of can be calculated b P (( ( ( Namel can be secified in terms of and Probabilit-Berlin Chen 3
Marginal PMFs of Random Variables (/ The PMFs of random variables and can be calculated from their joint PMF ( ( ( ( ( ( and are often referred to as the marginal PMFs The above two equations can be verified b ( ( P ( P ( Probabilit-Berlin Chen 4
Marginal PMFs of Random Variables (/ Tabular Method: Given the joint PMF of random variables and is secified in a two-dimensional table the marginal PMF of or at a given value is obtained b adding the table entries along a corresonding column or row resectivel Probabilit-Berlin Chen 5
Functions of Multile Random Variables (/ ( function Z g of the random variables and defines another random variable. Its PMF can be calculated from the joint PMF Z ( z ( { ( g ( zz } The eectation for a function of several random variables [ ] g( [ ] g( ( Z Probabilit-Berlin Chen 6
Functions of Multile Random Variables (/ If the function of several random variables is linear and of the form Z g( a b c [ Z ] a[ ] b[ ] c How can we verif the above equation? Probabilit-Berlin Chen 7
n Illustrative amle Given the random variables and whose joint is given in the following figure and a new random variable Z is defined b Z calculate [ Z ] Method : 3 6 8 3 5 [ ] 3 4 0 0 0 0 0 3 7 7 3 50 [ ] 3 4 0 0 0 0 0 5 50 5 [ Z ] 7. 0 0 0 [ ] [ ] 55 Method : ( z {( z} Z Z Z ( Z [ Z ] 3 4 5 6 ( 3 Z ( 4 Z ( 5 ( 6 0 0 0 Z 0 0 0 0 4 3 3 4 3 3 7 8 9 0 ( 7 ( ( ( 0 0 0 0 Z 8 Z 9 Z 0 0 0 0 7.55 ( Z ( 0 0 0 0 0 0 Probabilit-Berlin Chen 8
More than Two Random Variables (/ The joint PMF of three random variables and is defined in analog with the above as ( z P ( Z z Z P Z The corresonding marginal PMFs and ( ( z Z z ( ( z z Z Probabilit-Berlin Chen 9
More than Two Random Variables (/ The eectation for the function of random variables and Z [ g ( Z ] g ( z ( z Z z If the function is linear and has the form a b cz d [ a b cz d ] a[ ] b[ ] c[ Z ] d generalization to more than three random variables [ a a L an n ] a [ ] a [ ] L a [ ] n n Probabilit-Berlin Chen 0
n Illustrative amle amle.0. Mean of the Binomial. our robabilit class has 300 students and each student has robabilit /3 of getting an indeendentl of an other student. What is the mean of the number of students that get an? Let if the ith student gets an i 0 otherwise K 300 are bernoulli random variables with common mean /3 Their sum K 300 can be interreted as a binomial random variable with arameters n ( n 300 and ( /3. That is is the number of success in n ( n 300 indeendent trials 300 300 i i [ ] [ K ] [ ] 300 /3 00 Probabilit-Berlin Chen
Conditioning Recall that conditional robabilit rovides us with a wa to reason about the outcome of an eeriment based on artial information In the same sirit we can define conditional PMFs given the occurrence of a certain event or given the value of another random variable Probabilit-Berlin Chen
Conditioning a Random Variable on an vent (/ The conditional PMF of a random variable conditioned d on a articular event with P ( > 0 is defined b (where and are associated with the same eeriment ( ( P( { } I P P P( Normalization Proert ({ } Note that the events P I are disjoint for different values of their union is P ( P( { } I P ( P Total robabilit theorem P( { } I ({ } I P( P ( P ( P ( Probabilit-Berlin Chen 3
Conditioning a Random Variable on an vent (/ grahical illustration ( P Is obtained b adding the robabilities biliti of the outcomes that give rise to and be long to the conditioning event Probabilit-Berlin Chen 4
Illustrative amles (/ amle.. Let be the roll of a fair si-sided die and be the event that the roll is an even number P ( P( roll is even ( and is even P( is even P / 3 if 46 0 otherwise Probabilit-Berlin Chen 5
Let Let Illustrative amles (/ amle.4. student will take a certain test reeatedl u to a maimum of n times each time with a robabilit of assing indeendentl of the number of revious attemts. be What is the PMF of the number of attemts given that the student asses the test? be a geometric reresenti fist success ( the w ithin ( ng ( the comes event that n attemts t random number ( n ( m 0 the u ( m variable of student { n } if with attemts ass the arameter until test K n otherwise the P( ( n- n ( n- n Probabilit-Berlin Chen 6
Conditioning a Random Variable on nother (/ Let and be two random variables associated with the same eeriment. The conditional PMF of given is defined as P( ( ( P P ( ( Normalization Proert ( ( is fied on some value The conditional PMF is often convenient for the calculation of the joint PMF multilication (chain rule ( ( ( ( ( ( Probabilit-Berlin Chen 7
Conditioning a Random Variable on nother (/ The conditional PMF can also be used to calculate the marginal PMFs ( ( ( ( Visualization of the conditional PMF ( ( ( ( ( Probabilit-Berlin Chen 8
n Illustrative amle (/ amle.4. Professor Ma B. Right often has her facts wrong and answers each of her students questions incorrectl with robabilit /4 indeendentl of other questions. In each lecture Ma is asked 0 or questions with equal robabilit /3. What is the robabilit that she gives at least one wrong answer? Let be the number of questions asked be the number of questions answered P( P( P( P( P ( P( 3 P( P( 4 P( P( 3 P( 4 P( 3 4 3 4 wrong n k - k n k ( modeled as binomial distributions P( 4 48 Probabilit-Berlin Chen 9
n Illustrative amle (/ ( Calculation of the joint PMF in amle.4. Probabilit-Berlin Chen 0
Conditional ectation Recall that a conditional PMF can be thought of as an ordinar PMF over a new universe determined b the conditioning event In the same sirit a conditional eectation is the same as an ordinar eectation ecet that it refers to the new universe and all robabilities and PMFs are relaced b their conditional counterarts Probabilit-Berlin Chen
Summar of Facts bout Conditional ectations Let and be two random variables associated with the same eeriment The conditional eectation of given an event with P > is defined b ( 0 For a function [ ] ( g ( it is given b [ g ( ] g ( ( Probabilit-Berlin Chen
Total ectation Theorem (/ The conditional eectation of given a value of is defined b We have [ ] ( [ ] ( ( Let L n be disjoint events that form a artition of the samle sace and assume that P ( i > 0 for all i. Then n [ ] P ( [ ] i i i Probabilit-Berlin Chen 3
Total ectation Theorem (/ Let L n be disjoint events that form a artition of an event B and assume that P( i IB > 0 for all i. Then n [ B ] P ( B [ I B ] i Verification of total eectation theorem i [ ] ( ( ( ( ( ( ( [ ] i Probabilit-Berlin Chen 4
n Illustrative amle (/ amle.7. Mean and Variance of the Geometric Random Variable geometric random variable has PMF ( ( K be the event that { } [ ] 0 be the event that { > } [ ] 0 ( Let [ ] P( [ ] P( [ ] where P ( P( (?? ( ( 0 ( otherwise (?? > 0 otherwise Note that (See amle.3 : ( ( n ( m 0 m if K n [ ] [( ] ( ( [ ] ( ( [ ] P( [ ] P( [ ] [ ] P ( ( ( [ ] otherwise Probabilit-Berlin Chen 5
n Illustrative amle (/ [ ] ( [ ] ( [ ] [ ] 0 P P [ ] [ ] ( 0 0 ( ( Q ( ( ( ( ( ( ( ( ( ( ( ( ( ( [ ] ( ( ( set [ ] [ ] [ ] ( [ ] [ ] ( ( [ ] [ ] ( [ ] [ ] [ ] shown that have we Probabilit-Berlin Chen 6 [ ] ( [ ] [ ] ( var
Indeendence of a Random Variable from an vent random variable is indeendent of an event if ( and P( P( for all P { } Require two events and be indeendent for all If a random variable is indeendent of an event and P > ( 0 ( P P P ( and P( ( P ( P( ( ( for all Probabilit-Berlin Chen 7
n Illustrative amle amle.9. Consider two indeendent tosses of a fair coin. Let random variable be the number of heads Let random variable be 0 if the first toss is head and if the first toss is tail Let be the event that the number of head is even Possible outcomes (TT (TH (HT (HH / 4 if 0 / if 0 / if 0 if ( / 4 if / if and are not indeenden ( / if 0 ( ( t! ( / if ( P( and ( / if 0 P / if ( ( ( and are indeenden t! P / Probabilit-Berlin Chen 8
Indeendence of a Random Variables (/ Two random variables and are indeendent if ( ( ( for all ( P( P( for all or P If a random variable is indeendent d of an random variable ( ( for all with ih ( 0 all > ( ( ( ( ( ( ( for all with ( > 0 and all Probabilit-Berlin Chen 9
Indeendence of a Random Variables (/ Random variables and are said to be conditionall indeendent given a ositive robabilit event if Or equivalentl ( ( ( for all ( ( for all with ( 0 and all > Note here that as in the case of events conditional indeendence ma not iml unconditional indeendence and vice versa Probabilit-Berlin Chen 30
n Illustrative amle (/ Figure.5: amle illustrating that conditional indeendence d ma not iml unconditional indeendence d For the PMF shown the random variables and are not indeendent To show and are not indeendent we onl have to find a air of values of and that ( ( ( For eamle and are not indeendent ( 0 ( 3 0 Probabilit-Berlin Chen 3
n Illustrative amle (/ P To show and are not deendent we onl have to find all air of values of and that 9 0 ( ( ( For eamle and are indeendent conditioned on the event ( ( / 0 6 / 0 3 / 0 ( 4 3 / 0 3 4 / 0 3 6 / 0 3 / 0 ( 4 3 / 0 3 { 3} ( I I P( I P ( 3 ( 3 / 0 9 / 0 6 / 0 9 / 0 / 3 ( ( / 3 Probabilit-Berlin Chen 3
Functions of Two Indeendent Random Variables Given and be two indeendent random variables let g ( and h( be two functions of and resectivel. Show that g ( and h( are indeendent. Let ( ( U g and V h then ( u v ( {( g ( u h( v } U V ( g ( u h( { v } ( { g u} U ( u ( v V ( ( ( ( ( { h v} Probabilit-Berlin Chen 33
More Factors about Indeendent Random Variables (/ If and are indeendent random variables then [ ] [ ] [ ] s shown b the following calculation l [ ] ( [ ] [ ] ( ( ( ( b indeendence Similarl if and are indeendent random variables then g h g h [ ( ( ] [ ( ] [ ( ] Probabilit-Berlin Chen 34
More Factors about Indeendent Random Variables (/ If and are indeendent random variables then ( ( ( var var var s shown b the following calculation ( ( [ ] ( [ ] var ( ( [ ] [ ] ( [ ] [ ] ( [ ] ( ( [ ] [ ] ( [ ] [ ] [ ] ( [ ] [ ] ( [ ] [ ] [ ] ( ( ( ( [ ] ( [ ] ( [ ] [ ] Probabilit-Berlin Chen 35 [ ] ( [ ] ( [ ] [ ] [ ] [ ] ( ( [ ] [ ] ( ( ( ( var var
More than Two Random Variables Indeendence of several random variables Z Three random variable and are indeendent if ( z ( ( ( z for all Z Z? Comared to the conditions to be satisfied for three indeendent events and 3 (in P.39 of the tetbook f ( g ( ( n three random variables of the form and are also indeendent Variance of the sum of indeendent random variables If are indeendent random variables then K n ( L var( var( L var( var n n h Probabilit-Berlin Chen 36
Illustrative amles (/3 amle.0. Variance of the Binomial. We consider n indeendent coin tosses with each toss having robabilit of coming u a head. For each i we let be the Bernoulli random variable which h is equal to if the i-th toss comes u a head and is 0 otherwise. Then L is a binomial random variable. n i Q var var ( ( for all i n i ( var( n( (Note that ' s are indeendent! i i i Probabilit-Berlin Chen 37
Illustrative amles (/3 amle.. Mean and Variance of the Samle Mean. We wish to estimate the aroval rating of a resident to be called B. To this end we ask n ersons drawn at random from the voter oulation and we let i be a random variable that encodes the resonse of the i-th erson: i if the i - th erson aroves B' s erformanc e 0 if the i - th erson disarove s B' s erformanc e i ssume that indeendent and are the same random variable (Bernoulli with the common arameter ( for Bernoulli which is unknown to us i are indeendent and identicall distributed (i.i.d. with If the samle mean S n (is a random variable is defined as L n n n Sn n arameter Probabilit-Berlin Chen 38
Illustrative amles (3/3 The eectation of S n will be the true mean of i L n n n [ i ] n i i ( for the Bernoulli [ S ] n [ ] we assumed here The variance of S will aroimate 0 if is large enough n L lim var ( S n var n n n ( var ( ( lim i i n lim lim n n n n n n Which means that Sn will be a good estimate of [ i ] if n is large enough n n 0 Probabilit-Berlin Chen 39
Recitation SCTION.5 Joint PMFs of Multile Random Variables Problems 7 8 30 SCTION.6 Conditioning Problems 33 34 35 37 SCTION.6 Indeendence Problems 4 43 45 46 Probabilit-Berlin Chen 40