Lecture Notes 3 Multiple Random Variables. Joint, Marginal, and Conditional pmfs. Bayes Rule and Independence for pmfs

Lecture Notes 3 Multiple Random Variables Joint, Marginal, and Conditional pmfs Bayes Rule and Independence for pmfs Joint, Marginal, and Conditional pdfs Bayes Rule and Independence for pdfs Functions of Two RVs One Discrete and One Continuous RVs More Than Two Random Variables Corresponding pages from B&T textbook: 110-111, 158-159, 164-170, 173-178, 186-190, 221-225. EE 178/278A: Multiple Random Variables Page 3 1 Two Discrete Random Variables Joint PMFs As we have seen, one can define several r.v.s on the sample space of a random experiment. How do we jointly specify multiple r.v.s, i.e., be able to determine the probability of any event involving multiple r.v.s? We first consider two discrete r.v.s Let X and Y be two discrete random variables defined on the same experiment. They are completely specified by their joint pmf p X,Y (x,y) P{X x,y y} for all x X, y Y Clearly, p X,Y (x,y) 0, and x X y Y p X,Y(x,y) 1 Notation: We use (X,Y) p X,Y (x,y) to mean that the two discrete r.v.s have the specified joint pmf EE 178/278A: Multiple Random Variables Page 3 2

The joint pmf can be described by a table Example: Consider X,Y with the following joint pmf p X,Y (x,y) X 1 2 3 4 1 1/16 0 1/8 1/16 Y 2 1/32 1/32 1/4 0 3 0 1/8 1/16 1/16 4 1/16 1/32 1/16 1/32 EE 178/278A: Multiple Random Variables Page 3 3 Marginal PMFs Consider two discrete r.v.s X and Y. They are described by their joint pmf p X,Y (x,y). We can also define their marginal pmfs p X (x) and p Y (y). How are these related? To find the marginal pmf of X, we use the law of total probability p X (x) y Yp(x,y) for x X Similarly to find the marginal pmf of Y, we sum over x X Example: Find the marginal pmfs for the previous example X 1 2 3 4 1 1/16 0 1/8 1/16 Y 2 1/32 1/32 1/4 0 3 0 1/8 1/16 1/16 4 1/16 1/32 1/16 1/32 p X (x) p Y (y) EE 178/278A: Multiple Random Variables Page 3 4

Conditional PMFs The conditional pmf of X given Y y is defined as p X Y (x y) p X,Y(x,y) p Y (y) for p Y (y) 0 and x X Also, the conditional pmf of Y given X x is p Y X (y x) p X,Y(x,y) p X (x) for p X (x) 0 and y Y For fixed x, p Y X (y x) is a pmf for Y Example: Find p Y X (y 2) for the previous example EE 178/278A: Multiple Random Variables Page 3 5 Chain rule: Can write p X,Y (x,y) p X (x)p Y X (y x) Bayes rule for pmfs: Given p X (x) and p Y X (y x) for all (x,y) X Y, we can find p X Y (x y) p X,Y(x,y) p Y (y) p Y X(y x) p X (x) p Y (y) p Y X(y x) p X,Y (x,y) p X(x), by total probability x X Using the chain rule, we obtain another version of Bayes rule p X Y (x y) x X p Y X (y x) p X (x )p Y X (y x ) p X(x) EE 178/278A: Multiple Random Variables Page 3 6

Independence The random variables X and Y are said to be independent if for any events A X and B Y P{X A,Y B} P{X A}P{Y B} Can show that the above definition is equivalent to saying that the r.v.s X and Y are independent if p X,Y (x,y) p X (x)p Y (y) for all (x,y) X Y Independence implies that p X Y (x y) p X (x) for all (x,y) X Y EE 178/278A: Multiple Random Variables Page 3 7 Example: Binary Symmetric Channel Consider the following binary communication channel Z {0,1} X {0,1} Y {0,1} The bit sent X Bern(p), the noise Z Bern(ǫ), the bit received Y (X +Z) mod 2 X Z, and X and Z are independent. Find 1. p X Y (x y), 2. p Y (y), and 3. the probability of error P{X Y} EE 178/278A: Multiple Random Variables Page 3 8

1. To find p X Y (x y) we use Bayes rule p Y X (y x) p X Y (x y) p Y X (y x )p X (x ) p X(x) x X We know p X (x). To find p Y X, note that p Y X (y x) P{Y y X x} P{X Z y X x} P{Z y X X x} P{Z y x X x} P{Z y x}, since Z and X are independent p Z (y x) So we have p Y X (0 0) p Z (0 0) p Z (0) 1 ǫ, p Y X (0 1) p Z (0 1) p Z (1) ǫ p Y X (1 0) p Z (1 0) p Z (1) ǫ, p Y X (1 1) p Z (1 1) p Z (0) 1 ǫ EE 178/278A: Multiple Random Variables Page 3 9 Plugging in the Bayes rule equation, we obtain p X Y (0 0) p Y X (0 0) p Y X (0 0)p X (0)+p Y X (0 1)p X (1) p X(0) (1 ǫ)(1 p) (1 ǫ)(1 p)+ǫp p Y X (0 1) p X Y (1 0) p Y X (0 0)p X (0)+p Y X (0 1)p X (1) p X(1) 1 p X Y (0 0) ǫp (1 ǫ)(1 p)+ǫp p X Y (0 1) p X Y (1 1) p Y X (1 0) p Y X (1 0)p X (0)+p Y X (1 1)p X (1) p X(0) ǫ(1 p) ǫ(1 p)+(1 ǫ)p p Y X (1 1) p Y X (1 0)p X (0)+p Y X (1 1)p X (1) p X(1) 1 p X Y (0 1) (1 ǫ)p ǫ(1 p)+(1 ǫ)p EE 178/278A: Multiple Random Variables Page 3 10

2. We already found p Y (y): p Y (1) ǫ(1 p)+(1 ǫ)p 3. Now to find the probability of error P{X Y}, consider P{X Y} p X,Y (0,1)+p X,Y (1,0) p Y X (1 0)p X (0)+p Y X (0 1)p X (1) ǫ(1 p)+ǫp ǫ An interesting special case is when ǫ 1/2 Here, P{X Y} 1/2, which is the worst possible (no information is sent), and p Y (0) 1 2 p+ 1 2 (1 p) 1 2 p Y(1), i.e., Y Bern(1/2), independent of the value of p! Also in this case, the bit sent X and the bit received Y are independent (check this) EE 178/278A: Multiple Random Variables Page 3 11 Two Continuous Random variables Joint PDFs Two continuous r.v.s defined over the same experiment are jointly continuous if they take on a continuum of values each with probability 0. They are completely specified by a joint pdf f X,Y such that for any event A (, ) 2, P{(X,Y) A} f X,Y (x,y)dxdy For example, for a rectangular area (x,y) A P{a < X b, c < Y d} Properties of a joint pdf f X,Y : 1. f X,Y (x,y) 0 2. f X,Y(x,y)dxdy 1 d b c a f X,Y (x,y)dxdy Again joint pdf is not a probability. Can relate it to probability as P{x < X x+ x, y < Y y + y} f X,Y (x,y) x y EE 178/278A: Multiple Random Variables Page 3 12

Marginal PDF The Marginal pdf of X can be obtained from the joint pdf by integrating the joint over the other variable y f X (x) f X,Y (x,y)dy This follows by the law of total probability. To see this, consider y x x+ x x EE 178/278A: Multiple Random Variables Page 3 13 P{x < X x+ x} f X (x) lim x 0 x lim x 0 lim x 0 1 x lim 1 x y 0 n P{x < X x+ x, n y < Y (n+1) y} f X,Y (x,y)dy x f X,Y (x,y)dy EE 178/278A: Multiple Random Variables Page 3 14

Example Let (X,Y) f(x,y), where 1. Find c 2. Find f Y (y) 3. Find P{X 1 2 Y} Solution: 1. To find c, note that thus 1 2c 1, or c 2 { c if x,y 0, and x+y 1 f(x,y) 0 otherwise f(x,y)dxdy 1, EE 178/278A: Multiple Random Variables Page 3 15 2. To find f Y (y), we use the law of total probability f Y (y) f(x,y)dx { (1 y) 2dx 0 y 1 0 0 otherwise { 2(1 y) 0 y 1 0 otherwise 2 f Y (y) 0 1 y EE 178/278A: Multiple Random Variables Page 3 16

3. To find the probability of the set {X 1 2Y} we first sketch it y 1 2 3 1 x 1 2 y x from the figure we find that P {X 12 } Y {(x,y):x 2 1y}f X,Y(x,y)dxdy 2 3 (1 y) 0 y 2 2dxdy 2 3 EE 178/278A: Multiple Random Variables Page 3 17 Example: Buffon s Needle Problem The plane is ruled with equidistant parallel lines at distance d apart. Throw needle of length l < d at random. What is the probability that it will intersect one of the lines? d Θ X l Solution: Let X be the distance from the midpoint of the needle to the nearest of the parallel lines, and Θ be the acute angle determined by a line through the needle and the nearest parallel line EE 178/278A: Multiple Random Variables Page 3 18

(X, Θ) are uniformly distributed within the rectangle [0, d/2] [0, π/2], thus f X,Θ (x,θ) 4, x [0,d/2], θ [0,π/2] πd The needle intersects a line iff X < l 2 sinθ The probability of intersection is: P {X < l2 } sinθ f X,Θ (x,θ)dxdθ {(x,θ): x< 2 l sinθ} 4 πd 4 πd 2l πd π/2 l 2 sinθ 0 π/2 0 0 l 2 sinθdθ dxdθ EE 178/278A: Multiple Random Variables Page 3 19 Example: Darts Throw a dart on a disk of radius r. Probability on the coordinates (X,Y) is described by a uniform pdf on the disk: Find the marginal pdfs f X,Y (x,y) { 1 πr 2, if x 2 +y 2 r 2 0, otherwise Solution: To find the pdf of Y (same as X), consider f Y (y) f X,Y (x,y)dx 1 dx πr 2 {x: x 2 +y 2 r 2 } 1 r 2 y 2 πr 2 dx r 2 y 2 { 2 πr 2 r2 y 2, if y r 0, otherwise EE 178/278A: Multiple Random Variables Page 3 20

Conditional pdf Let X and Y be continuous random variables with joint pdf f X,Y (x,y), we define the conditional pdf of Y given X as f Y X (y x) f X,Y(x,y) f X (x) for f X (x) 0 Note that, for a fixed X x, f Y X (y x) is a legitimate pdf on Y it is nonnegative and integrates to 1 We want the conditional pdf to be interpreted as: f Y X (y x) y P{y < Y y + y X x} The RHS can be interpreted as a limit P{y < Y y + y X x} lim x 0 P{y < Y y + y, x < X x+ x} P{x < X x+ x} f X,Y (x,y) x y lim x 0 f X (x) x f X,Y(x,y) y f X (x) EE 178/278A: Multiple Random Variables Page 3 21 Example: Let Find f X Y (x y). { 2, x,y 0,x+y 1 f(x,y) 0, otherwise Solution: We already know that f Y (y) Therefore { 2(1 y), 0 y 1 0, otherwise f X Y (x y) f X,Y(x,y) f Y (y) { 1 1 y, x,y 0, x+y 1, y < 1 0, otherwise f X Y (x y) 1 (1 y) (1 y) x EE 178/278A: Multiple Random Variables Page 3 22

Chain rule: p X,Y (x,y) p X (x)p Y X (y x) EE 178/278A: Multiple Random Variables Page 3 23 Independence and Bayes Rule Independence: Two continuous r.v.s are said to be independent if f X,Y (x,y) f X (x)f Y (y) for all x,y It can be shown that this definition is equivalent to saying that X and Y are independent if for any two events A,B (, ) P{X A,Y B} P{X A}P{Y B} Example: Are X and Y in the previous example independent? Bayes rule for densities: Given f X (x) and f Y X (y x), we can find f X Y (x y) f Y X(y x) f X (x) f Y (y) f Y X (y x) f X,Y(u,y)du f X(x) f Y X (y x) f X(u)f Y X (y u)du f X(x) EE 178/278A: Multiple Random Variables Page 3 24

Example: Let Λ U[0,1], and the conditional pdf of X given Λ λ f X Λ (x λ) λe λx, 0 λ 1, i.e., X {Λ λ} Exp(λ). Now, given that X 3, find f Λ X (λ 3) Solution: We use Bayes rule f X Λ (3 λ)f Λ (λ) f Λ X (λ 3) 1 0 f Λ(u)f X Λ (3 u)du { λe 3λ 1,0 λ 1 9 (1 4e 3 ) 0, otherwise f Λ (λ) 0 1.378 1 f Λ X (λ 3) λ 1 3 1 0.56 λ EE 178/278A: Multiple Random Variables Page 3 25 Joint cdf If X and Y are two r.v.s over the same experiment, they are completely specified by their joint cdf F X,Y (x,y) P{X x,y y} for x,y (, ) y (x,y) x EE 178/278A: Multiple Random Variables Page 3 26

Properties of the joint cdf: 1. F X,Y (x,y) 0 2. F X,Y (x 1,y 1 ) F X,Y (x 2,y 2 ) whenever x 1 x 2 and y 1 y 2 3. lim x,y F X,Y (x,y) 1 4. lim y F X,Y (x,y) F X (x) and lim x F X,Y (x,y) F Y (y) 5. lim y F X,Y (x,y) 0 and lim x F(x,y) 0 6. The probability of any set can be determined from the joint cdf, for example, y d c x a b P{a < X b,c < Y d} F(b,d) F(a,d) F(b,c)+F(a,c) EE 178/278A: Multiple Random Variables Page 3 27 If X and Y are continuous random variables having a joint pdf f X,Y (x,y), then F X,Y (x,y) x y f X,Y (u,v)dudv for x,y (, ) Moreover, if F X,Y (x,y) is differentiable in both x and y, then f X,Y (x,y) 2 F(x,y) x y P{x < X x+ x,y < Y y + y} lim x, y 0 x y Two random variables are independent if F X,Y (x,y) F X (x)f Y (y) EE 178/278A: Multiple Random Variables Page 3 28

Functions of Two Random Variables Let X and Y be two r.v.s with known pdf f X,Y (x,y) and Z g(x,y) be a function of X and Y. We wish to find f Z (z) We use the same procedure as before: First calculate the cdf of Z, then differentiate it to find f Z (z) Example: Max and Min of Independent Random Variables Let X f X (x) and Y f Y (y) be independent, and define Find the pdfs of U and V U max{x,y}, and V min{x,y} Solution: To find the pdf of U, we first find its cdf F U (u) P{U u} P{X u,y u} F X (u)f Y (u), by independence EE 178/278A: Multiple Random Variables Page 3 29 Now, to find the pdf, we take the derivative w.r.t. u f U (u) f X (u)f Y (u)+f Y (u)f X (u) For example, if X and Y are uniformly distributed between 0 and 1, f U (u) 2u for 0 u 1 Next, to find the pdf of V, consider F V (v) P{V v} 1 P{V > v} 1 P{X > v,y > v} 1 (1 F X (v))(1 F Y (v)) F X (v)+f Y (v) F X (v)f Y (v), thus f V (v) f X (v)+f Y (v) f X (v)f Y (v) f Y (v)f X (v) For example, if X Exp(λ 1 ) and Y Exp(λ 2 ), then V Exp(λ 1 +λ 2 ) EE 178/278A: Multiple Random Variables Page 3 30

Sum of Independent Random Variables Let X and Y be independent r.v.s with known distributions. We wish to find the distribution of their sum W X +Y First assume X p X (x) and Y p Y (y) are independent integer-valued r.v.s, then for any integer w, the pmf of their sum p W (w) P{X +Y w} P{X x, Y y} {(x,y): x+yw} x x x P{X x, Y w x} P{X x}p{y w x}, by independence p X (x)p Y (w x) This is the discrete convolution of the two pmfs EE 178/278A: Multiple Random Variables Page 3 31 For example, let X Poisson(λ 1 ) and Y Poisson(λ 2 ) be independent, then the pmf of their sum p W (w) p X (x)p Y (w x) x w p X (x)p Y (w x) for w 0,1,... x0 w λ x 1 x! e λ 1 x0 (λ 1+λ 2 ) w w! λ w x 2 (w x)! e λ 2 e (λ 1+λ 2 ) w x0 ( )( ) x ( ) w x w λ1 λ2 x λ 1 +λ 2 λ 1 +λ 2 (λ 1+λ 2 ) w e (λ 1+λ 2 ) w! Thus W X +Y Poisson(λ 1 +λ 2 ) In general a Poisson r.v. with parameter λ can be written as the sum of any number of independent Poisson(λ i ) r.v.s, so long as λ i λ. This property of a distribution is called infinite divisibility EE 178/278A: Multiple Random Variables Page 3 32

Now, let s assume that X f X (x), and Y f Y (y) are independent continuous r.v.s. We wish to find the pdf of their sum W X +Y. To do so, first note that P{W w X x} P{X +Y w X x} P{x+Y w X x} P{x+Y w}, by independence F Y (w x) Thus f W X (w x) f Y (w x), a very useful result Now, to find the pdf of W, consider f W (w) f W,X (w,x) dx f X (x)f W X (w x) dx This is the convolution of f X (x) and f Y (y) f X (x)f Y (w x) dx EE 178/278A: Multiple Random Variables Page 3 33 Example: Assume that X U[0,1] and Y U[0,1] are independent r.v.s. Find the pdf of their sum W X +Y Solution: To find the pdf of the sum, we convolve the two pdfs f W (w) w 0 1 w 1 f X (x)f Y (w x) dx dx, if 0 w 1 dx, if 1 < w 2 0, otherwise w, if 0 w 1 2 w, if 1 < w 2 0, otherwise Example: If X N(µ 1,σ 2 1) and Y N(µ 2,σ 2 2) are indepedent, then their sum W N(µ 1 +µ 2,σ 2 1 +σ 2 2), i.e., Gaussian is also an infinitely divisible distribution any Gaussian r.v. can be written as the sum of any number of independent Gaussians as long as their means sum to its mean and their variances sum to its variance (will prove this using transforms later) EE 178/278A: Multiple Random Variables Page 3 34

One Discrete and One Continuous RVs Let Θ be a discrete random variable with pmf p Θ (θ) For each Θ θ, such that p Θ (θ) 0, let Y be a continuous random variable with conditional pdf f Y Θ (y θ) The conditional pmf of Θ given Y can be defined as a limit P{Θ θ,y < Y y + y} p Θ Y (θ y) lim y 0 P{y < Y y + y} p Θ (θ)f Y Θ (y θ) y lim y 0 f Y (y) y f Y Θ(y θ) p Θ (θ) f Y (y) So we obtain yet another version of Bayes rule p Θ Y (θ y) f Y Θ (y θ) θ p Θ (θ )f Y Θ (y θ ) p Θ(θ) EE 178/278A: Multiple Random Variables Page 3 35 Example: Additive Gaussian Noise Channel Consider the following communication channel model Z N(0,N) Θ Y where the signal sent Θ { +1, with probability p 1, with probability 1 p, the signal received (also called observation) Y Θ+Z, and Θ and Z are independent Given Y y is received (observed), find the a posteriori pmf of Θ, p Θ Y (θ y) EE 178/278A: Multiple Random Variables Page 3 36

Solution: We use Bayes rule p Θ Y (θ y) f Y Θ(y θ)p Θ (θ) θ p Θ (θ )f Y Θ (y θ ) We know p Θ (θ). To find f Y Θ (y θ), consider P{Y y Θ 1} P{Θ+Z y Θ 1} P{Z y Θ Θ 1} P{Z y 1 Θ 1} P{Z y 1}, by independence of Θ and Z Therefore, Y {Θ +1} N(+1,N). Also, Y {Θ 1} N( 1,N) Thus p Θ Y (1 y) p 2πN e (y 1)2 2N p 2πN e (y 1)2 2N + (1 p) 2πN e (y+1)2 2N pe y pe y for < y < +(1 p)e y EE 178/278A: Multiple Random Variables Page 3 37 Now, let p 1/2. Suppose the receiver decides that the signal transmitted is 1 if Y > 0, otherwise he decides that it is a 1. What is the probability of decision error? Solution: First we plot the conditional pdfs f Y Θ (y 1) f Y Θ (y 1) 1 1 y This decision rule make sense, since you decide that the signal transmitted is 1 if f Y Θ (y 1) > f Y Θ (y 1) An error occurs if Θ 1 is transmitted and Y 0, or Θ 1 is transmitted and Y > 0 EE 178/278A: Multiple Random Variables Page 3 38

But if Θ 1, then Y 0 iff Z < 1, and if Θ 1, then Y > 0 iff Z > 1 Thus the probability of error is P{ error } P{Θ 1,Y 0 or Θ 1,Y > 0} P{Θ 1,Y 0}+P{Θ 1,Y > 0} P{Θ 1}P{Y 0 Θ 1}+P{Θ 1}P{Y > 0 Θ 1} 1 2 P{Z < 1}+ 1 P{Z > 1} 2 ( ) 1 Q N EE 178/278A: Multiple Random Variables Page 3 39 Summary: Total Probability and Bayes Rule Law of total probability: events: P(B) i P(A i B), A i s partition Ω pmf: p X (x) y p(x,y) pdf: f X (x) f X,Y (x,y)dy mixed: f Y (y) θ p Θ(θ)f Y Θ (y θ), p Θ (θ) f Y (y)p Θ Y (θ y)dy Bayes rule: events: P(A j B) pmf: p X Y (x y) pdf: f X Y (x y) mixed: p Θ Y (θ y) f Y Θ (y θ) P(B A j ) i P(B A i)p(a i ) P(A j), A i s partition Ω p Y X (y x) x p Y X (y x )p X (x ) p X(x) f Y X (y x) fx (x )f Y X (y x )dx f X (x) f Y Θ (y θ) θ p Θ (θ )f Y Θ (y θ ) p Θ(θ), p Θ Y (θ y) fy (y )p Θ Y (θ y )dy f Y (y) EE 178/278A: Multiple Random Variables Page 3 40

More Than Two RVs Let X 1,X 2,...,X n be random variables (defined over the same experiment) If the r.v.s are discrete then they can be jointly specified by their joint pmf p X1,X 2,...,X n (x 1,x 2,...,x n ), for all (x 1,x 2,...,x n ) X 1 X 2,... X n If the r.v.s are jointly continuous, then they can be specified by the joint pdf f X1,X 2,...,X n (x 1,x 2,...,x n ), for all (x 1,x 2,...,x n ) Marginal pdf (or pmf) is the joint pdf (or pmf) for a subset of {X 1,...,X n }; e.g. for three r.v.s X 1,X 2,X 3, the marginals are f Xi (x i ) and f Xi,X j (x i,x j ) for i j The marginals can be obtained from the joint in the usual way, e.g. for the n 3 example f X1,X 2 (x 1,x 2 ) f X1,X 2,X 3 (x 1,x 2,x 3 )dx 3 EE 178/278A: Multiple Random Variables Page 3 41 Conditional pmf or pdf can be defined in the usual way, e.g. the conditional pdf of (X k+1,x k+2,...,x n ) given (X 1,X 2,...,X k ) is f Xk+1,...,X n X 1,...,X k (x k+1,...,x n x 1,...,x k ) f X 1,...,X n (x 1,...,x n ) f X1,...,X k (x 1,...,x k ) Chain rule: We can write f X1,...,X n (x 1,...,x n ) f X1 (x 1 )f X2 X 1 (x 2 x 1 )f X3 X 1,X 2 (x 3 x 1,x 2 )......f Xn X 1,...,X n 1 (x n x 1,...,x n 1 ) In general X 1,X 2,...,X n are completely specified by their joint cdf F X1,...,X n (x 1,...,x n ) P{X 1 x 1,...,X n x n }, for all (x 1,...,x n ) EE 178/278A: Multiple Random Variables Page 3 42

Independence and Conditional Independence Independence is defined in the usual way: X 1,X 2,...,X n are said to be independent iff n f X1,X 2,...,X n (x 1,x 2,...,x n ) f Xi (x i ) for all (x 1,x 2,...,x n ) Important special case, i.i.d. r.v.s: X 1,X 2,...,X n are said to be independent and identically distributed (i.i.d.) if they are independent and have the same marginal, e.g. if we flip a coin n times independently we can generate X 1,X 2,...,X n i.i.d. Bern(p) r.v.s The r.v.s X 1 and X 3 are said to be conditionally independent given X 2 iff i1 f X1,X 3 X 2 (x 1,x 3 x 2 ) f X1 X 2 (x 1 x 2 )f X3 X 2 (x 3 x 2 ) for all (x 1,x 2,x 3 ) Conditional independence does not necessarily imply or is implied by independence, i.e., X 1 and X 3 independent given X 2 does not necessarily mean that X 1 and X 3 are independent (or vice versa) EE 178/278A: Multiple Random Variables Page 3 43 Example: Series Binary Symmetric Channels: Z 1 Z 2 X 1 X 2 X3 Here X 1 Bern(p), Z 1 Bern(ǫ 1 ), and Z 2 Bern(ǫ 2 ) are independent, and X 3 X 1 +Z 1 +Z 2 mod 2 In general X 1 and X 3 are not independent X 1 and X 3 are conditionally independent given X 2 Also X 1 and Z 1 are independent, but not conditionally independent given X 2 Example Coin with Random Bias: Consider a coin with random bias P f P (p). Flip it n times independently to generate the r.v.s X 1,X 2,...,X n (X i 1 if i-th flip is heads, 0 otherwise) The r.v.s X 1,X 2,...,X n are not independent However, X 1,X 2,...,X n are conditionally independent given P in fact, for any P p, they are i.i.d. Bern(p) EE 178/278A: Multiple Random Variables Page 3 44