Conditioning and Independence

Size: px

Start display at page:

Download "Conditioning and Independence"

Jayson Wright
5 years ago
Views:

1 Discrete Random Variables: Joint PMFs Conditioning and Indeendence Berlin Chen Deartment of Comuter Science & Information ngineering National Taiwan Normal Universit Reference: - D. P. Bertsekas J. N. Tsitsiklis Introduction to Probabilit Sections.5-.7

2 Motivation Given an eeriment e.g. a medical diagnosis The results of blood test is modeled as numerical values of a random variable The results of magnetic resonance imaging i (MRI 核磁共振攝影 is also modeled as numerical values of a random variable We would like to consider robabilities of events involving simultaneousl the numerical values of these two variables and to investigate their mutual coulings P { } I { } (? Probabilit-Berlin Chen

3 Joint PMF of Random Variables Let and be random variables associated with the same eeriment (also the same samle sace and robabilit laws the joint PMF of and is defined b ( P ( { } I { } P ( P ( if event is the set of all airs that have a certain roert then the robabilit of can be calculated b P (( ( ( Namel can be secified in terms of and Probabilit-Berlin Chen 3

4 Marginal PMFs of Random Variables (/ The PMFs of random variables and can be calculated from their joint PMF ( ( ( ( ( ( and are often referred to as the marginal PMFs The above two equations can be verified b ( ( P ( P ( Probabilit-Berlin Chen 4

5 Marginal PMFs of Random Variables (/ Tabular Method: Given the joint PMF of random variables and is secified in a two-dimensional table the marginal PMF of or at a given value is obtained b adding the table entries along a corresonding column or row resectivel Probabilit-Berlin Chen 5

6 Functions of Multile Random Variables (/ ( function Z g of the random variables and defines another random variable. Its PMF can be calculated from the joint PMF Z ( z ( { ( g ( zz } The eectation for a function of several random variables [ ] g( [ ] g( ( Z Probabilit-Berlin Chen 6

7 Functions of Multile Random Variables (/ If the function of several random variables is linear and of the form Z g( a b c [ Z ] a[ ] b[ ] c How can we verif the above equation? Probabilit-Berlin Chen 7

8 n Illustrative amle Given the random variables and whose joint is given in the following figure and a new random variable Z is defined b Z calculate [ Z ] Method : [ ] [ ] [ Z ] [ ] [ ] 55 Method : ( z {( z} Z Z Z ( Z [ Z ] ( 3 Z ( 4 Z ( 5 ( Z ( 7 ( ( ( Z 8 Z 9 Z ( Z ( Probabilit-Berlin Chen 8

9 More than Two Random Variables (/ The joint PMF of three random variables and is defined in analog with the above as ( z P ( Z z Z P Z The corresonding marginal PMFs and ( ( z Z z ( ( z z Z Probabilit-Berlin Chen 9

10 More than Two Random Variables (/ The eectation for the function of random variables and Z [ g ( Z ] g ( z ( z Z z If the function is linear and has the form a b cz d [ a b cz d ] a[ ] b[ ] c[ Z ] d generalization to more than three random variables [ a a L an n ] a [ ] a [ ] L a [ ] n n Probabilit-Berlin Chen 0

11 n Illustrative amle amle.0. Mean of the Binomial. our robabilit class has 300 students and each student has robabilit /3 of getting an indeendentl of an other student. What is the mean of the number of students that get an? Let if the ith student gets an i 0 otherwise K 300 are bernoulli random variables with common mean /3 Their sum K 300 can be interreted as a binomial random variable with arameters n ( n 300 and ( /3. That is is the number of success in n ( n 300 indeendent trials i i [ ] [ K ] [ ] 300 /3 00 Probabilit-Berlin Chen

12 Conditioning Recall that conditional robabilit rovides us with a wa to reason about the outcome of an eeriment based on artial information In the same sirit we can define conditional PMFs given the occurrence of a certain event or given the value of another random variable Probabilit-Berlin Chen

13 Conditioning a Random Variable on an vent (/ The conditional PMF of a random variable conditioned d on a articular event with P ( > 0 is defined b (where and are associated with the same eeriment ( ( P( { } I P P P( Normalization Proert ({ } Note that the events P I are disjoint for different values of their union is P ( P( { } I P ( P Total robabilit theorem P( { } I ({ } I P( P ( P ( P ( Probabilit-Berlin Chen 3

14 Conditioning a Random Variable on an vent (/ grahical illustration ( P Is obtained b adding the robabilities biliti of the outcomes that give rise to and be long to the conditioning event Probabilit-Berlin Chen 4

15 Illustrative amles (/ amle.. Let be the roll of a fair si-sided die and be the event that the roll is an even number P ( P( roll is even ( and is even P( is even P / 3 if 46 0 otherwise Probabilit-Berlin Chen 5

16 Let Let Illustrative amles (/ amle.4. student will take a certain test reeatedl u to a maimum of n times each time with a robabilit of assing indeendentl of the number of revious attemts. be What is the PMF of the number of attemts given that the student asses the test? be a geometric reresenti fist success ( the w ithin ( ng ( the comes event that n attemts t random number ( n ( m 0 the u ( m variable of student { n } if with attemts ass the arameter until test K n otherwise the P( ( n- n ( n- n Probabilit-Berlin Chen 6

17 Conditioning a Random Variable on nother (/ Let and be two random variables associated with the same eeriment. The conditional PMF of given is defined as P( ( ( P P ( ( Normalization Proert ( ( is fied on some value The conditional PMF is often convenient for the calculation of the joint PMF multilication (chain rule ( ( ( ( ( ( Probabilit-Berlin Chen 7

18 Conditioning a Random Variable on nother (/ The conditional PMF can also be used to calculate the marginal PMFs ( ( ( ( Visualization of the conditional PMF ( ( ( ( ( Probabilit-Berlin Chen 8

19 n Illustrative amle (/ amle.4. Professor Ma B. Right often has her facts wrong and answers each of her students questions incorrectl with robabilit /4 indeendentl of other questions. In each lecture Ma is asked 0 or questions with equal robabilit /3. What is the robabilit that she gives at least one wrong answer? Let be the number of questions asked be the number of questions answered P( P( P( P( P ( P( 3 P( P( 4 P( P( 3 P( 4 P( wrong n k - k n k ( modeled as binomial distributions P( 4 48 Probabilit-Berlin Chen 9

20 n Illustrative amle (/ ( Calculation of the joint PMF in amle.4. Probabilit-Berlin Chen 0

21 Conditional ectation Recall that a conditional PMF can be thought of as an ordinar PMF over a new universe determined b the conditioning event In the same sirit a conditional eectation is the same as an ordinar eectation ecet that it refers to the new universe and all robabilities and PMFs are relaced b their conditional counterarts Probabilit-Berlin Chen

22 Summar of Facts bout Conditional ectations Let and be two random variables associated with the same eeriment The conditional eectation of given an event with P > is defined b ( 0 For a function [ ] ( g ( it is given b [ g ( ] g ( ( Probabilit-Berlin Chen

23 Total ectation Theorem (/ The conditional eectation of given a value of is defined b We have [ ] ( [ ] ( ( Let L n be disjoint events that form a artition of the samle sace and assume that P ( i > 0 for all i. Then n [ ] P ( [ ] i i i Probabilit-Berlin Chen 3

24 Total ectation Theorem (/ Let L n be disjoint events that form a artition of an event B and assume that P( i IB > 0 for all i. Then n [ B ] P ( B [ I B ] i Verification of total eectation theorem i [ ] ( ( ( ( ( ( ( [ ] i Probabilit-Berlin Chen 4

25 n Illustrative amle (/ amle.7. Mean and Variance of the Geometric Random Variable geometric random variable has PMF ( ( K be the event that { } [ ] 0 be the event that { > } [ ] 0 ( Let [ ] P( [ ] P( [ ] where P ( P( (?? ( ( 0 ( otherwise (?? > 0 otherwise Note that (See amle.3 : ( ( n ( m 0 m if K n [ ] [( ] ( ( [ ] ( ( [ ] P( [ ] P( [ ] [ ] P ( ( ( [ ] otherwise Probabilit-Berlin Chen 5

26 n Illustrative amle (/ [ ] ( [ ] ( [ ] [ ] 0 P P [ ] [ ] ( 0 0 ( ( Q ( ( ( ( ( ( ( ( ( ( ( ( ( ( [ ] ( ( ( set [ ] [ ] [ ] ( [ ] [ ] ( ( [ ] [ ] ( [ ] [ ] [ ] shown that have we Probabilit-Berlin Chen 6 [ ] ( [ ] [ ] ( var

27 Indeendence of a Random Variable from an vent random variable is indeendent of an event if ( and P( P( for all P { } Require two events and be indeendent for all If a random variable is indeendent of an event and P > ( 0 ( P P P ( and P( ( P ( P( ( ( for all Probabilit-Berlin Chen 7

28 n Illustrative amle amle.9. Consider two indeendent tosses of a fair coin. Let random variable be the number of heads Let random variable be 0 if the first toss is head and if the first toss is tail Let be the event that the number of head is even Possible outcomes (TT (TH (HT (HH / 4 if 0 / if 0 / if 0 if ( / 4 if / if and are not indeenden ( / if 0 ( ( t! ( / if ( P( and ( / if 0 P / if ( ( ( and are indeenden t! P / Probabilit-Berlin Chen 8

29 Indeendence of a Random Variables (/ Two random variables and are indeendent if ( ( ( for all ( P( P( for all or P If a random variable is indeendent d of an random variable ( ( for all with ih ( 0 all > ( ( ( ( ( ( ( for all with ( > 0 and all Probabilit-Berlin Chen 9

30 Indeendence of a Random Variables (/ Random variables and are said to be conditionall indeendent given a ositive robabilit event if Or equivalentl ( ( ( for all ( ( for all with ( 0 and all > Note here that as in the case of events conditional indeendence ma not iml unconditional indeendence and vice versa Probabilit-Berlin Chen 30

indeendence d For the PMF shown the random variables and are not indeendent To show

31 n Illustrative amle (/ Figure.5: amle illustrating that conditional indeendence d ma not iml unconditional indeendence d For the PMF shown the random variables and are not indeendent To show and are not indeendent we onl have to find a air of values of and that ( ( ( For eamle and are not indeendent ( 0 ( 3 0 Probabilit-Berlin Chen 3

32 n Illustrative amle (/ P To show and are not deendent we onl have to find all air of values of and that 9 0 ( ( ( For eamle and are indeendent conditioned on the event ( ( / 0 6 / 0 3 / 0 ( 4 3 / / / 0 3 / 0 ( 4 3 / 0 3 { 3} ( I I P( I P ( 3 ( 3 / 0 9 / 0 6 / 0 9 / 0 / 3 ( ( / 3 Probabilit-Berlin Chen 3

33 Functions of Two Indeendent Random Variables Given and be two indeendent random variables let g ( and h( be two functions of and resectivel. Show that g ( and h( are indeendent. Let ( ( U g and V h then ( u v ( {( g ( u h( v } U V ( g ( u h( { v } ( { g u} U ( u ( v V ( ( ( ( ( { h v} Probabilit-Berlin Chen 33

34 More Factors about Indeendent Random Variables (/ If and are indeendent random variables then [ ] [ ] [ ] s shown b the following calculation l [ ] ( [ ] [ ] ( ( ( ( b indeendence Similarl if and are indeendent random variables then g h g h [ ( ( ] [ ( ] [ ( ] Probabilit-Berlin Chen 34

35 More Factors about Indeendent Random Variables (/ If and are indeendent random variables then ( ( ( var var var s shown b the following calculation ( ( [ ] ( [ ] var ( ( [ ] [ ] ( [ ] [ ] ( [ ] ( ( [ ] [ ] ( [ ] [ ] [ ] ( [ ] [ ] ( [ ] [ ] [ ] ( ( ( ( [ ] ( [ ] ( [ ] [ ] Probabilit-Berlin Chen 35 [ ] ( [ ] ( [ ] [ ] [ ] [ ] ( ( [ ] [ ] ( ( ( ( var var

36 More than Two Random Variables Indeendence of several random variables Z Three random variable and are indeendent if ( z ( ( ( z for all Z Z? Comared to the conditions to be satisfied for three indeendent events and 3 (in P.39 of the tetbook f ( g ( ( n three random variables of the form and are also indeendent Variance of the sum of indeendent random variables If are indeendent random variables then K n ( L var( var( L var( var n n h Probabilit-Berlin Chen 36

37 Illustrative amles (/3 amle.0. Variance of the Binomial. We consider n indeendent coin tosses with each toss having robabilit of coming u a head. For each i we let be the Bernoulli random variable which h is equal to if the i-th toss comes u a head and is 0 otherwise. Then L is a binomial random variable. n i Q var var ( ( for all i n i ( var( n( (Note that ' s are indeendent! i i i Probabilit-Berlin Chen 37

38 Illustrative amles (/3 amle.. Mean and Variance of the Samle Mean. We wish to estimate the aroval rating of a resident to be called B. To this end we ask n ersons drawn at random from the voter oulation and we let i be a random variable that encodes the resonse of the i-th erson: i if the i - th erson aroves B' s erformanc e 0 if the i - th erson disarove s B' s erformanc e i ssume that indeendent and are the same random variable (Bernoulli with the common arameter ( for Bernoulli which is unknown to us i are indeendent and identicall distributed (i.i.d. with If the samle mean S n (is a random variable is defined as L n n n Sn n arameter Probabilit-Berlin Chen 38

39 Illustrative amles (3/3 The eectation of S n will be the true mean of i L n n n [ i ] n i i ( for the Bernoulli [ S ] n [ ] we assumed here The variance of S will aroimate 0 if is large enough n L lim var ( S n var n n n ( var ( ( lim i i n lim lim n n n n n n Which means that Sn will be a good estimate of [ i ] if n is large enough n n 0 Probabilit-Berlin Chen 39

40 Recitation SCTION.5 Joint PMFs of Multile Random Variables Problems SCTION.6 Conditioning Problems SCTION.6 Indeendence Problems Probabilit-Berlin Chen 40

Discrete Random Variable

Discrete Random Variable Outcome of a random experiment need not to be a number. We are generally interested in some measurement or numerical attribute of the outcome, rather than the outcome itself. n