Chapter 5,6 Multiple RandomVariables

Transcription

1 Chapter 5,6 Multiple RandomVariables ENCS66 - Probabilityand Stochastic Processes Concordia University

2 Vector RandomVariables A vector r.v. is a function where is the sample space of a random experiment. Example: randomly pick up a student name from a list. all student names on the list. Let be a given outcome, e.g. Tom height of student weight of student age of student are r.v.s. Let then is a vector r.v. ENCS66 p./47

3 Events Each event involving has a corresponding region in Example: is a two-dimensional r.v. ENCS66 p./47

4 Pairs ofrandomvariables Pairs of discrete random variables Joint probability mass function P X;Y (x j ;y k )= P X = x j Y=y k =P X = x j ;Y = y k Obviously Marginal Probability Mass Function P X (x j )= P X = x j =P X = x j ;Y = anything = k= P X;Y (x j ;y k ) Similarly = ENCS66 p.3/47

5 Pairs ofrandomvariables The joint CDF of and (for both discrete and continuous r.v.s) y 6 (x,y) - x ENCS66 p.4/47

6 Pairs ofrandomvariables Properties of the joint CDF:.,if anything Marginal cdf ENCS66 p.5/47

7 Pairs ofrandomvariables The joint pdf of two jointly continuous r.v.s. Obviously, and ENCS66 p.6/47

8 Pairs ofrandomvariables The probability In general, Example: y 6 A x ENCS66 p.7/47

9 Pairs ofrandomvariables Marginal pdf: ENCS66 p.8/47

10 Pairs ofrandomvariables Example: if 0 x ; 0 y f X;Y (x;y) = 0 otherwise. Find F X;Y (x;y) ) x 0 or y 0; F X;Y (x;y) = 0 ) 0 x ; and 0 y x y F X;Y (x;y) = dy 0 dx 0 =xy 0 0 3) 0 x ; and y> F X;Y (x;y) = dy 0 dx 0 =x 4) x>and 0 y< 0 0 F X;Y (x;y) = y 5) x>and y> F X;Y (x;y) = x ENCS66 p.9/47

11 Independence for all and (discrete r.v.s) or for all and or for all and Example: a) 6 b) otherwise - 6 otherwise - ENCS66 p.0/47

12 Conditional Probability If X is discrete, P Y y; X = x F Y (y x) = for P X = x >0 P X = x d f Y (y x) = dy F Y (y x) If X is continuous, P X = x =0 P Y y; x < X x+ h F Y (y x) = lim F Y (y x<x x + h) = lim h! 0 h! 0 P x<x x+h f Y (y x) = = lim h! 0 = lim h! 0 y y x+ h x f X;Y (x 0 ;y 0 )dx 0 dy 0 x+ h x f X (x 0 )dx 0 f X;Y(x;y 0 )dy 0 h f X (x) h d dy F Y (y x) = f X;Y(x;y) f X (x) = y f X;Y(x;y 0 )dy 0 f X (x) ENCS66 p./47

13 Conditional Probability If, independent, Similarly, So, Bayes Rule: ENCS66 p./47

14 Conditional Probability Example: A r.v. is uniformly selected in and then is selected uniformly in Find Solution: for and is elsewhere. y x for and elsewhere. ENCS66 p.3/47

15 Conditional Probability Example: x 0; Decide Decide - Y- R f Y (y x) Detector - ^X Channel (volts), weassume 0 (volts), 0 if if This is called the Maximum a posterior probability (MAP) detection. ENCS66 p.4/47

16 Conditional Probability Binary communication over Additive White Gaussian Noise (AWGN) channel (y+ A) (y A) Apply the MAP detection, we need to find and.notehere is discrete, is continuous. ENCS66 p.5/47

17 Conditional Probability Use thesimilar approach (considering x< X x+h, and let h 0), wehave P X = 0y = P X = 0 f Y (y 0) f Y (y) P X = y = P X = f Y (y 0) f Y (y) Decide ^X = 0,if Decide ^X =,if P X = 0y P X = y y A ln p 0 p P X = 0y <P X = y y> A ln p 0 p When p 0 =p = : Decide ^X = 0,ify 0 Decide ^X =,ify> 0 ENCS66 p.6/47

18 Conditional Probability Prob of error: considering the special case 0 0 Similarly, 0 0 (y+ A) ) ENCS66 p.7/47

19 Conditional Expectation The conditional expectation E[Y x] = yf Y (y x)dy In discrete case, E[Y x] = y i P Y (y i x) y i An important fact: E[Y ] = E[E[Y X]] Proof: E[E[Y X]] = = In general: E[Y x]f X (x)dx = yf X;Y (x;y)dydx = E[Y] E[h(Y)] = E[E[h(Y) X]] yf Y (y x)f X (x)dydx ENCS66 p.8/47

20 Multiple RandomVariables Joint cdf Joint pdf n n If discrete, joint pmf Marginal pdf i n n all n except i ENCS66 p.9/47

21 Independence are independent iff n n for all If we use pdf, n n for all ENCS66 p.0/47

22 Functions of Several r.v.s One function of several r.v.s Let s.t. then z n ENCS66 p./47

23 Functions of Several r.v.s Example:, find and in terms of y y=z-x x If and are independent ENCS66 p./47

24 Functions of Several r.v.s Example: let. Find the pdf of if and are independent and both exponentially distributed with mean one. Can use the similar approach as previous example, but complicated. Fix,then and. So ENCS66 p.3/47

25 Functions of Several r.v.s Since are indep. exponentially distributed 0 0 for ENCS66 p.4/47

26 Transformation of RandomVectors Transformation of Random Vectors The joint CDF of is n n : k ( ) k ENCS66 p.5/47

27 pdf of Linear Transformations If,where is a invertible matrix. n n = is the absolute value of the determinant of. e.g if, then ENCS66 p.6/47

28 pdf of General Transformations where We assume that the set of equations: has a unique solution given by The joint pdf of n is given by n n where is called the Jacobian of the transformation. ENCS66 p.7/47

29 pdf of General Transformations The Jacobian of the transformation and Linear transformation is a special case of n n n n n n n n = ( ) ENCS66 p.8/47

30 pdf of General Transformations Example: let and be zero-mean unit-variance independent Gaussian r.vs. Find the joint pdf of and defined by: \ This is a transformation from Cartesian to Polar coordinates. The inverse transformation is: y y v w x x ENCS66 p.9/47

31 pdf of General Transformations The Jacobian Since and are zero-mean unit-variance independent Gaussian r.v.s, x y x +y The joint pdf of is then (v cos w+ v si n w) v for and ENCS66 p.30/47

32 pdf of General Transformations The marginal pdf of for for Since and 0 v v. This is called the Rayleigh Distribution. 0 are independent. v ENCS66 p.3/47

33 Expected Value of Functions of r.v.s Let then For discrete case, n all possible n Example: n ENCS66 p.3/47

34 Expected Value of Functions of r.v.s Example: If are indep. n The -th moment of two r.v.s is If,itiscalled the correlation. If,wecall are orthogonal. ENCS66 p.33/47

35 Expected Value of Functions of r.v.s The -th central moment of is when When,itiscalled the covariance of The correlation coefficient of and is defined as where and. ENCS66 p.34/47

36 Expected Value of Functions of r.v.s The correlation coefficient Proof: If, are said to be uncorrelated. If Hence, are independent, are uncorrelated. The converse is not always true. It is true in the case of Gaussian r.v.s (will be discussed later) ENCS66 p.35/47

37 Expected Value of Functions of r.v.s Example: is uniform in Let and and are not independent, since. However 0 We can also show. So are uncorrelated but not independent. ENCS66 p.36/47

38 JointCharacteristic Function Joint Characteristic Function n ( + + n n ) For two variables ( + ) Marginal characteristic function If are independent + ENCS66 p.37/47

39 JointCharacteristic Function If ( + ) If, and are independent ENCS66 p.38/47

40 Jointly Gaussian RandomVariables Consider a vector of random variables. Each withmean for and the covariance matrix. Let be the mean vector and where denotes transpose. Then are said to be the jointly Gaussian if their joint pdf is: n where is the determinant of... ENCS66 p.39/47

41 Jointly Gaussian RandomVariables For,let (x m), then p For, denote the r.v.s by and.let m = m X m Y K = X XY X Y Then K = X Y ( XY ) and K = f X;Y (x;y) = XY X Y Y Y XY X Y X Y ( XY ) XY X Y X X Y XY XY ( x exp ( XY ) ( x m x x ) m x x )( y m y y )+ ( y m y y ) ENCS66 p.40/47

42 Jointly Gaussian RandomVariables The marginal pdf of X : The marginal pdf of Y : If XY =0 The conditional pdf. f XjY (x y) = f X (x) = f Y (y) = X;Y are independent. f X;Y (x;y) f Y (y) = N( XY X Y (y e (xm x ) x x e (ym y ) y y exp ( X Y ) X + E[XjY] m y )+ m x x XY X Y (y m y ) m x X ( XY ) ;X ( XY) ) + Var(XjY) ENCS66 p.4/47

43 Linear Transformation of Gaussian r.v.s Let X N(m;K), Y =AXthen Y N(^m;C),where ^m = Amand C= AKA T proof: f Y (y) = f X (A y) A = () n A K exp (A y m) T K (A y m) Note that A y m =A (y Am)= A (y ^m), so (A y m) T K (A y m) = (y ^m) T (A ) T K A (y ^m) = (y ^m) T C (y ^m) and A K = A K = AK A T = C f Y (y) = () n C exp (y ^m)t C (y ^m) N(^m;C) ENCS66 p.4/47

44 Linear Transformation of Gaussian r.v.s Since K is symmetric, itis always possible to find a matrix A s.t. 0 = AKA T is diagonal. =... so 0 n f Y (y) = = () n e (y^m ) T (y^m ) e (y ^m ) e (yn ^m n ) n n That is, we can transform X into n independent Gaussian r.v.s Y Y n with means ^m i and variance i. ENCS66 p.43/47

45 Mean Square Estimation Weuse to estimate, write as. The costassociated with the estimation error is e.g. Y Case :if X The mean square error The mean square error: ENCS66 p.44/47

46 Mean Square Estimation Case :if,then This is called the MMSE linear estimation. The mean square error: If,,error, reduces to case. Y If,, and error X ENCS66 p.45/47

47 Mean Square Estimation Case 3: is a general function of. For any, choose to minimize E[(Y g(x)) X = x] This is called the MMSE estimation. Example:, joint Gaussian. The MMSE estimation is linear for Gaussian. ENCS66 p.46/47

48 Mean Square Estimation Example: uniform,. We have So, the MMSE linear estimation: 3 and the error is. The MMSE estimation: So and the error is. ENCS66 p.47/47