Multivariate Random Variable

Transcription

1 Multivariate Random Variable Author: Author: Andrés Hincapié and Linyi Cao This Version: August 7, 2016

2 Multivariate Random Variable 3 Now we consider models with more than one r.v. These are called multivariate models For instance: height and weight An n-dimensional random vector is a function from a sample space Ω into R n

3 Multivariate Random Variable 4 1 Joint and Marginal Distributions 1.1 Discrete Case A random vector is called discrete if it has only a countable number of possible values. For instance if we rolled two dice and X =sum and Y = difference. Then (X, Y ) has finite support. Let (X 1,... X n ) be a discrete random vector. The function is called the joint pmf of (X 1,... X n ) f (x 1,..., x n ) = P r (X 1 = x 1,... X n = x n ) Notation is sometimes f X1,...X n (x 1,..., x n )

4 Multivariate Random Variable 5 For any event A R n, P r ((X 1,... X n ) A) = (X 1,...X n ) A f (x 1,..., x n ) And similarly Eg (X 1,... X n ) = (X 1,...X n ) R n g (x 1,..., x n ) f (x 1,..., x n )

5 Multivariate Random Variable 6 Multivariate pmf s must satisfy same properties as univariate pmf s so (X 1,...X n ) R n f (x 1,..., x n ) = 1 and f (x 1,..., x n ) 0 for any (X 1,... X n ) R n Even when working with a vector, we might be interested in probabilities of only one of the variables in the vector For instance we may want to obtain f X1 (x 1 ). This is called the marginal pmf of X 1 For a random vector with joint pmf f X1,...X n (x 1,..., x n ), the marginal pmf of X i, f Xi (x i ) = P r (X i = x i ) is given by f Xi (x i ) = f X1,...X n (x 1,..., x n ) x 1 x i 1 x i+1 x n

6 Multivariate Random Variable 7 EXAMPLE: f (0, 0) = f (0, 1) = 1/6 f (1, 0) = f (1, 1) = 1/3 f (x, y) = 0 for any other combination Find P r (X = Y ) = Find f X (x), f Y (x)

7 Multivariate Random Variable 8 Marginal distributions do not completely describe the joint distribution of the vector Many different joint distributions can have the same marginals EXAMPLE: f (0, 0) = 1/12, f (1, 0) = 5/12, f (0, 1) = f (1, 1) = 3/12, f (x, y) = 0, for all other values Find f X (x), f Y (x) Same as previous example

8 Multivariate Random Variable Continuous Case A function f (x 1,..., x n ) from R n into R is called the joint probability density function or joint pdf of the continuous random vector (X 1,..., X n ) if, for every Borel set A R n P r ((X 1,..., X n ) A) = f (x 1,..., x n ) dx 1... dx n A Similar to the discrete case Eg (X 1,... X n ) =... g (x 1,..., x n ) f (x 1,..., x n ) dx 1... dx n And the marginal pdf of X i f Xi (x i ) = f X1,...X n (x 1,..., x n ) dx 1... dx i 1 dx i+1... dx n x 1,...x i 1,x i+1,...x n

9 Multivariate Random Variable 10 EXAMPLE: Let the joint pdf of X and Y be given by f (x, y) = 6xy 2, 0 < x < 1 0 < y < 1 Find the marginal pdf of X and Y What s P r (X + Y 1)?

10 Multivariate Random Variable 11 EXAMPLE: Let the joint pdf of X and Y be given by f (x, y) = e y, 0 < x < y < What s P r (X + Y 1)? Use intersection of support and complement of event.

11 Multivariate Random Variable 12 In the bivariate case, the joint cdf is given by F (x 1, x 2 ) = P r((x 1, X 2 ) (, x 1 ] (, x 2 ]) And in the discrete r.v. case we have: F (x 1, x 2 ) = s x 1 t x 2 f(s, t) x 1 x 2 in the continuous r.v. case we have: F (x 1, x 2 ) = f (s, t) dsdt Generalization to higher dimensions is straight forward Can we define joint cdf first, then joint pdf/pmf as we did in the univariate case? Yes.

12 Multivariate Random Variable Multinomial Moment Generating Function If a random vector X R n, its mgf is defined as M X (t) = E(exp{t X}), t R n provided existence in some neighborhood of 0 R n Case when n = 1 seen before If n 2, i.e. vector is (X 1,..., X n ) E [ X i ] i i n Xi n n = t i t i M X (t) n t=0 Example: if X R 2 E [ X 1 X2 3 ] 1+3 = t 1 t 3 M X (t) 2 t=0

13 Multivariate Random Variable 14 2 Conditional Distributions and Expectations If we have information on one r.v. in a random vector, we may be able to sharpen our knowledge about the other P r (wage > occupation = P hd Student) = 0!!! Let (x 1, x 2 ) be an 2-dimensional random vector. The conditional distribution of X 1 given X 2 = x 2 is defined by f (x 1 x 2 ) = f (x 1, x 2 ) f X2 (x 2 ) For the discrete case the conditional pmf obtains from computing the correspondent conditional probabilities.

14 Multivariate Random Variable 15 Conditional expectation of a function of r.v. Y w.r.t. X = x is given by E (g (Y ) x) = g (y) f (y x) dy The conditional expectation E (g (Y ) x), sometimes written as E(g(Y ) X = x), is a function of x. And it maps from the support of X to R. So when we write E(g (Y ) X), it is a composed function: Ω R, so it is a r.v.! Similarly the conditional variance of Y w.r.t. X = x is defined as Again, V ar(y X) is a r.v. V ar (Y x) = E ( Y 2 x ) (E (Y x)) 2

15 Multivariate Random Variable 16 EXAMPLE: Let the joint pdf of X and Y be given by f (x, y) = e y, 0 < x < y < What s E (Y X = x)?

16 Multivariate Random Variable 17 If X and Y are r.v. then Law of Iterated Expectations EX = E [E (X Y )] Conditional Variance Identity V arx = E [V ar (X Y )] + V ar [E (X Y )]

17 Multivariate Random Variable 18 3 Independence In some situations, knowledge of X does not provide extra information about Y Such relationship is called independence (between X and Y ) Definition 1 Let (X 1,... X n ) be an n-dimensional random vector with joint pdf or pmf f (x 1,..., x n ). Then x 1,..., x n are independent if f (x 1,..., x n ) = f X1 (x 1 )... f Xn (x n )

18 Multivariate Random Variable 19 But we do not necessarily need to know each individual marginal pdf or pmf to check independence. Theorem: Let (X 1,... X n ) be an n-dimensional random vector with joint pdf or pmf f (x 1,..., x n ). Then X 1,... X n are independent iff there exist functions g 1 (x 1 ),..., g n (x n ) such that f (x 1,..., x n ) = g 1 (x 1 )... g n (x n ) i.e.: as long as they can be separated in the joint pdf

19 Multivariate Random Variable 20 EXAMPLE: Are X and Y independent? f (x, y) = x2 y 4 e y x/2, x, y > 0 f (x, y) = e y, 0 < x < y < Are X and Y independent?

20 Multivariate Random Variable 21 Let X and Y be independent random variables. Then a) For any Borel sets A R and B R, P r (X A, Y B) = P r (X A) P r (Y B) i.e.: events {X A} and {Y B} are independent. This is actually an alternative (and better) way to define independence! b) If g (x) is a function of X, and h (y) is a function of Y, then E [g (X) h (Y )] = E [g (X)] E [h (Y )] c) If Z = X + Y, then the mgf of Z is given by M Z (t) = M X (t) M Y (t)

21 Multivariate Random Variable 22 EXAMPLE: Let X N ( µ x, σ 2 x) and Y N ( µy, σ 2 y) be independent r.v. s Find the pdf of Z = X + Y

22 Multivariate Random Variable 23 4 Covariance and Correlation Numerical measures of the strength of the relation between random variables Let X and Y be two random variables, and E(X) = µ X, V ar(x) = σ 2 X, similar for Y. The covariance of X and Y is the number defined by σ XY = Cov (X, Y ) = E ((X µ X ) (Y µ Y )) We call X and Y uncorrelated if Cov(X, Y ) = 0. The correlation coefficient of X and Y is the number defined by ρ XY = σ XY σ X σ Y

23 Multivariate Random Variable Properties of Covariance and Correlation Let X and Y be any random variables a) Cov (X, Y ) = E (XY ) µ X µ Y b) If X and Y are independent then Cov (X, Y ) = 0, namely independence implies being uncorrelated, but not the other way around. Independence is much stronger! c) If a, b, and c are any three constants, then V ar (ax + by + c) = a 2 V arx + b 2 V ary + 2abCov (X, Y ) d) 1 ρ XY 1

24 Multivariate Random Variable 25 EXAMPLE: X U ( 1, 1), Z U(0, 1/10). X and Z independent, and Y = X 2 + Z Are X and Y independent? Are they uncorrelated? Orthogonality: We call X and Y orthogonal if E(XY ) = 0 Generally, orthogonality and uncorrelation don t imply each other. But if we have either EX = 0 or EY = 0, then they do. Where to apply: In a linear regression Y = α + β X + ε, we want X and ε to be orthogonal. Check orthogonal projection. A sufficient condition would be: E(ε X = x) = 0 for all possible x, or equivalently E(ε X) 0.

25 Multivariate Random Variable 26 5 Multivariate Normal (Gaussian) The random vector X = (X 1,... X n ) has a multivariate Gaussian distribution with mean µ [n 1] and variance-covariance matrix Σ [n n] if f (x 1,..., x n ) = 1 (2π) n/2 det (Σ) 1/2 exp { 1 } 2 (X µ) Σ 1 (X µ) where µ {i} = µ i and Σ {i,j} = Cov (X i, X j )

26 Multivariate Random Variable Properties of the Bivariate Normal Consider a bivariate normal vector (X, Y ). Then a) The marginal of W is N ( µ W, σ 2 W ), for W = X, Y b) The conditional distribution of Y given X is N ( µ Y x, σ Y x ) where µ Y x = µ Y + ρ (σ x /σ y ) (x µ x ) σ Y x = σ 2 ( y 1 ρ 2 )

27 Multivariate Random Variable 28 Important difference between a vector (X 1, X 2 ) that is distributed jointly normal, and a vector (X 1, X 2 ) whose elements are distributed marginally normal: If (X 1, X 2 ) is distributed jointly normal, then ρ = 0 X 1 and X 2 are independent. If (X 1, X 2 ) is a random vector whose elements are distributed marginally normal, then ρ = 0 X 1 and X 2 are independent but ρ = 0 X 1 and X 2 are independent

28 Multivariate Random Variable 29 Even if the elements X 1, X 2 are distributed marginally normal, their joint distribution may not be normal. EXAMPLE s.p. joint pdf of a vector (X, Y ) is f(x, y) = 2zφ(x)φ(y) where z = 1 if xy > 0, z = 0 otherwise where φ(c) is the pdf of a N(0, 1)

29 Multivariate Random Variable 30 Marginal pdf of X is f X (x) = = 2zφ(x)φ(y)dy = 2φ(x) 2φ(x)(1 Φ(0)) if x > 0 2φ(x)Φ(0) if x < 0 zφ(y)dy = = φ(x) x R 2φ(x) 0 0 2φ(x) φ(y)dy if x > 0 φ(y)dy if x < 0 Hence, even though joint distribution is not Normal, marginals are Normal. Normal marginal does not imply that the joint pdf of the vector is normal.