III - MULTIVARIATE RANDOM VARIABLES

Transcription

1 Computational Methods and advanced Statistics Tools III - MULTIVARIATE RANDOM VARIABLES A random vector, or multivariate random variable, is a vector of n scalar random variables. The random vector is written X 1 X X = 2 (1)... X n A random vector is also said an n dimensional random variable. 1 Joint ad marginal densities Every random variable has a distribution function. DEFINITION III.1A - (DISTRIBUTION) - The n dimensional random variable X has the distribution function F(x 1,...,x n ) = P{X 1 x 1,...,X n x n }. (2) A random vector, X, is called absolutely continuous if there exists the joint probability density f such that x1 xn F(x 1,...,x n ) =... f(t 1,...,t n )dt 1,...dt n. (3) A random vector, X, is called discrete if the set of its values is discrete, that is either finite or countable. It means that X : Ω R n, where Ω is discrete. In this case the joint probability function (or, less properly, the joint discrete density) is defined as f(x 1,...,x n ) = P{X 1 = x 1,...,X n = x n }. (4) The joint distribution and the joint probability function are related by F(x 1,...,x n ) =... f(t 1,...,t n ). (5) t 1 x 1 t n x n 1

2 Ex. III.1B (n = 2) The joint distribution function of the random vector (X,Y) is defined by the relation F(x,y) = P(X x, Y y) The function F(x,y) is defined on R 2 and, as a function each variable, is monotone non-decreasing and such that lim F(x,y) = 1, lim F(x,y) = lim F(x,y) = 0 x,y + x y To know F(x,y) allows to know the probability that (X,Y) falls in any given rectangle (a,b] (c,d] : P[(X,Y) (a,b] (c,d] ] = F(b,d) F(a,d) F(b,c)+F(c,d). Sincerectangles approximate anymeasurableset ofr 2, ifwe knowf then we know P[(X,Y) A] for any measurable set A R 2 ; that is, the distribution function F(x, y) thoroughly describes the distribution measure of the couple (X,Y). If the random vector (X,Y) is absolutely continuous, i.e. if there is an integrable function f(x, y) such that f(x,y) 0, R 2 f(x,y)dxdy = 1, F(x,y) = x y f(ξ, η)dξdη, then A f(x,y)dxdy = P[(X,Y) A], measurable set A R 2, which illustrates the meaning of the joint probability density f(x, y). An example of an absolutely continuous random vector: (X,Y) with probability density and distribution function f(x,y) = 1 2π exp[ 1 2 (x2 +y 2 )]. For example (X,Y) falls in the unit disk D = {(x,y) : x 2 + y 2 1} with probability D f(x,y)dxdy = 1 2π /2 2π e r2 r drdθ = 2π [ ] e r 2 /2 1 2π = 0 1 e 1/2. Anexampleofadiscreterandom vector is(x,y)with X =scoregot bytossing a die, Y = score got by tossing another die. Then the joint probability 2

3 function has the value 1/36 for any (x i,y j ) {1,...,6} 2 and zero elsewhere. The distribution function F(x,y) = P(X = x i,y = y j ) x i x y j y is computed by counting and dividing by 36. For example F(2,4) = 8/36, F(3,3) = 9/36,..., F(6,6) = 1,. For a sub-vector S = (X 1,...,X k ) T (k < n) of the random vector X, the marginal density function is f S (x 1,...,x k ) = in the continuous case, and f S (x 1,...,x k ) = x k+1 + f(x 1,...,x n ) dx k+1...dx n (6)... x n f(x 1,...,x n ) dx k+1...dx n (7) if X is discrete. In both cases the marginal distribution function is F S (x 1,...,x k ) = F(x 1,...,x k,,..., ). (8) For the sake of notation, we will write f X (x) and F X (x) instead of f(x) and F(x), respectively, whenever it is necessary to emphasize to which random variable the function belongs. EX. III.1C f X,Y (x,y) = { 1 8, 0 y x 4 0, else Then the marginal density function of X is { x 0 1 f X (x) = f X,Y (x,y) dy = dy = 1 x, x [0,4] x / [0,4] R DEF. III.1D (INDEPENDENCE) The random vectors X and Y are independent 1 if f X,Y (x,y) = f X (x) f Y (y), x,y (9) 1 In most textbooks the n dimensional and m dimensional r.v. X,Y are said to be independent if P(X A,Y B) = P(X A) P(Y B), measurable A R n, B R m. A suitable choice of A, B leads directly to the equivalent definition (10) 3

4 which corresponds to F X,Y (x,y) = F X (x) F Y (y), x,y (10) 2 Moments of scalar random variables Evolution presupposes the existence of something that can develop. The theory says nothing about where this something came from. Furthermore, questions about the being, essence, dignity, mission, meaning and wherefore of the world and man cannot be answered in biological terms. (YouCat 42) An average of the possible values of X, each value being weighted by its probability, E[X] = x P{X = x}, x is ad litteram the notion of expectation for a discrete variable X (provided that such series is absolutely convergent). For continuous variables, the sums are replaced by integrals. DEFINITION III.2A (EXPECTATION) The expectation (or mean value) of an absolutely continuous variable X with density function f X is E[X] = x f X (x) dx (11) provided that the integral is absolutely convergent, i.e. E[ X ] <. If X is a random variable and g is a function, then Y = g(x) is also a random variable. To calculate the expectation of Y, we could find f Y and use (11). However the process of finding f Y can be complicated; instead, we can simply use E[Y] = E[g(X)] = g(x)f X (x) dx (12) This provides a method for calculating the moments of a distribution. R DEF. III.2B (MOMENTS) 4

5 The n-th moment of X is E[X n ] = and the n-th central moment is E[(X E[X]) n ] = R R x n f X (x) dx (13) (x E[X]) n f X (x) dx (14) The second central moment is also called the variance, and the the variance of X is given by Var[X] = E[(X E[X]) 2 ] = E[X 2 ] (E[X]) 2. (15) Now let X be an n dimensional random variable and put Y = g(x), where g is a function. As in (12) we have E[Y] = E[g(X)] = g(x 1...,x n )f X (x 1,...,x n )dx 1...dx n R n This gives sense to the mean of a linear combination of variables, and to any type of moments. Since E[aX 1 +bx 2 ] = ae[x 1 ]+be[x 2 ] the expectation operator is a linear operator. Of special interest is the covariance of X i and X j : DEF: III.2C - Covariance - The covariance of two scalar r.v. s X 1, 2 is Cov[X 1,X 2 ] = E[(X 1 E[X 1 ])(X 2 E[X 2 ])] (16) The covariance gives information about the simultaneous variation of two variables and is useful to find whether X 1,X 2 are dependent on each other. Since the expectation is linear, the covariance is a bilinear operator, i.e. linear in each of the two arguments, whence its calculation rule: Cov[aX 1 +bx 2,cX 3 +dx 4 ] = = ac Cov[X 1,X 3 ]+ad Cov[X 1,X 4 ]+bc Cov[X 2,X 3 ]+bd Cov[X 2,X 4 ](17) for constants a,b,c,d and r.v. X 1,...,X 4. 5

6 PROP. III.2D - For two scalar random variables X,Y we have the formulae: Cov(X,Y) = E(XY) µ x µ y (18) Var(X +Y) = Var(X)+2 Cov(X,Y)+Var(Y) (19) Proof E[(X µ x ) (Y µ y )] = E[XY] µ x E[Y] µ y E[X]+µ x µ y = = E[XY] 2µ x µ y +µ x µ y = E[XY] µ x µ y In particular, as we already know, Var(X) Cov(X,X) = E[X 2 ] µ 2 x. Besides, E[{X+Y (µ x +µ y )} 2 ] = E[(X µ x ) 2 +2(X µ x )(Y µ y )+(Y µ y ) 2 ] = = Var(X)+2 Cov(X,Y)+Var(Y) DEF. III.2D - Correlation coefficient - The correlation coefficient, or correlation, of two scalar r.v. s X 1,X 2 is ρ ρ(x 1,X 2 ) := E[ X 1 µ σ 1 X2 µ 2 σ 2 ] THM. III.2D - The covariance and the correlation of two scalar r.v. s X,Y satisfy Cov(X,Y) 2 Var(X)Var(Y) (Schwarz s inequality), 1 ρ(x,y) 1 respectively. Proof Since the expectation of a nonnegative variable is always nonnegative, 0 E[(θ X + Y ) 2 ] = θ 2 E[X 2 ]+2θE[ XY ]+E[Y 2 ], θ. 6

7 Thus a polynomial with degree 2 (in theta) does not take negative values, so that the discriminant must be 0: E[ XY ] 2 E[X 2 ]E[Y 2 ] 0, or E[ XY ] 2 E[X 2 ]E[Y 2 ]. Replacing X and Y by X µ x and Y µ y, and using f(t)g(t)dt f(t)g(t) dt, this implies E[(X mu x )(Y µ y )] 2 E[ (X mu x )(Y µ y ) ] 2 E[(X µ x ) 2 ]E[(Y µ y ) 2 ] = Var(X)Var(Y) Since ρ = Cov(X,Y)/ Var(X)Var(Y), this also means [ρ(x,y)] 2 1, i.e. 1 ρ +1. DEFINITION III.2E - Uncorrelation - Two scalar random variables, Y are said uncorrelated, or orthogonal, when Cov(X,Y) = 0. Since Cov(X,Y) = E(XY) µ x µ y, However X,Y independent X,Y uncorrelated X, Y uncorrelated does not imply X, Y independent as we see in the following example. EX. III.2F - Let U be uniformly distributed in [0,1] and consider the r.v. s X = sin2πu, Y = cos2πu. X and Y are not independent since Y has only two possible values if X is known. However they are orthogonal or uncorrelated: so that E(X) = 1 Cov(X,Y) = E(X Y) = 0 sin2πt dt = 0, E(Y) = sin2πt cos2πt dt = cos2πt dt = 0, 1 0 sin4πt dt = 0. In this case the covariance is zero, but the variables are not independent. 7

8 3 Moments of multivariate random variables We are interested in defining a (vector) first moment ad a (matrix) second moment of a multivariate random variable. DEF. III.3A (EXPECTATION OF A RANDOM VECTOR) The expectation (or the mean value) of the random vector X is µ = E[X] = E[X 1 ] E[X 2 ]. E[X n ] (20) DEFINITION III.3B (COVARIANCE matrix) The covariance matrix of the random vector X is Σ X = Var[X] = E[(X µ)(x µ) T ] = = Var[X 1 ] Cov[X 1,X 2 ]... Cov[X 1,X n ] Cov[X 2,X 1 ] Var[X 2 ]... Cov[X 2,X n ].. Cov[X n,x 1 ] Cov[X n,x 2 ]... Var[X n ] (21) Sometimes we shall use the notation σ1 2 σ σ 1n σ 21 σ σ 2n.. σ n1 σ n2... σn 2 Both σ ii and σ 2 i are possible notations for the variance of X i. By the calculation rule (17), the variance of a linear combination of X 1,X 2 is known from the variance matrix of (X 1,X 2 ): Var[z 1 X 1 +z 2 X 2 ] = z 2 1Var(X 1 )+2z 1 z 2 Cov(X 1,X 2 )+z 2 2Var(X 2 ). and in general, for any n 2, it is the the quadratic form of Σ calculated in the coefficients z 1,...,z n : Var[z 1 X z n X n ] = z T Σz. 8

9 According to III.2C, the correlation of two random variables X i, X j is: ρ ij = Equations (21, 23) lead to the definition Cov[X i,x j ] Var[Xi ]Var[X j ] = σ ij σ i σ j (22) DEFINITION III.3C The correlation matrix for X is 1 ρ ρ 1n ρ ρ 2n.. ρ n1 ρ n (23) THEOREM III.3D The covariance matrix Σ and the correlation matrix R are a) symmetric and b) nonnegative. Proof. (a)thesymmetryisobvious. (b)bydefinitionofvariance,var(z 1 X z n X n ) 0 for any z R n. Hence the quadratic form z T Σz Var[z T X] 0, z, i.e. the matrix Σ is nonnegative semidefinite.. REMARK III.3E - Very often Σ is positive definite, i.e. z T Σz 0 and z T Σz = 0 z = (0,...,0) When Σ is positive definite, we write Σ > 0. For our purposes, we always assume Σ > 0. COROLLARY III.3F - Linear transformations of random vectors Let X = (X 1,...,X n ) T have mean µ and covariance Σ X. Now we introduce the new random variable Y = (Y 1,...,Y n ) T by the transformation Y = a+bx where a is (n 1) vector and B is a (n n) matrix. Then the expectation and variance matrix of Y are: E[Y] = E[a+BX] = a+be[x] (24) Σ Y = Var[a+BX] = Var[BX] = BVar[X]B T (25) 9

10 Proof The first formula is obvious since the expectation operator is linear. As for the variance of BX we have: (Var[a+BX]) i,j = Cov[(BX) i, (BX) j ] = = Cov[ l b i,l X l, m b j,m X m ] = b i,l b j,m Cov(X l,x m ) = ( BΣ X B T). i,j l,m 4 The multivariate normal distribution We assume that X 1,...,X n are independent random variables with means µ 1,...,µ n, and variances σ 2 1,...,σ 2 n. We write X i N(µ i,σ 2 i). Now, define the random vector X = (X 1,...,X n ) T. Because the random variables are independent, the joint density is the product of the marginal densities: f X (x 1,...,x n ) = f X1 (x 1 ) f Xn (x n ) = n 1 σ i 2π exp 1 ( n exp σ i )(2π) n/2 [ (xi µ i ) 2 ] 2σi 2 [ 1 n [ ] ] xi µ 2 i. 2 σ i By introducing the mean µ = (µ 1,...,µ n ) T, and the covariance Σ X = Var[X] = diag(σ 2 1,...,σ 2 n), this is written f X (x) = [ 1 (2π) n/2 exp 1 ] detσ X 2 (x µ)t Σ 1 X (x µ) (26) A generalization to the case where the covariance matrix is a full matrix leads to the following DEF. III.4A - The multivariate normal distribution - The joint density function for the n dimensional random variable X with mean µ and covariance matrix Σ is 10

11 f X (x) = [ 1 (2π) n/2 exp 1 ] detσ X 2 (x µ)t Σ 1 X (x µ) (27) where Σ > 0. We write X N(µ,Σ). If X N(0,I) we say that X is standardized normally distributed. THM. III.4B - Within the class of normal variables, X 1,..,X n are uncorrelated if ad only if they are independent. Proof Cov(X i,x j ) = 0, i j implies that the variance matrix is diagonal. Thus the quadratic form in the exponent of f is a diagonal form. So the density is the product of the marginal densities, hence the X i s are independent. THM. III.4C - Principal components of N(µ;Σ). - Let X N(µ;Σ), with values in R n. Each level surface (or contour ) of the joint density f(x 1,...,x n ) is an ellipsoid {x : (x µ) T Σ 1 (x µ) = c 2 } (28) in R n, c being a constant. The axes of the ellipsoid are ±c λ i e i where (λ i,e i ) are the eigenvalue-eigenvector pairs of the covariance matrix Σ (i = 1,..,n). If the matrix G is defined with the normalized eigenvectors e i as columns, then the equation Y = G T (X µ), (29) determines independent and normally distributed variables Y i with mean zero and variance λ i. Such Y i s are said principal components of the normal vector X. Proof The covariance matrix Σ is symmetric and definite positive. Thus by the spectral decomposition theorem, G T ΣG = Λ, (or G T Σ 1 G = Λ 1 ) 11

12 where Λ = Diag(λ 1,...,λ n ) is the diagonal matrix of (positive) eigenvalues of Σ, while G is an orthogonal matrix whose columns are the corresponding normalized eigenvectors. By the principal component trasformation the ellipsoid becomes y = G T (x µ), i.e. x µ = Gy (Gy) T Σ 1 Gy = c 2, i.e. y T G T Σ 1 Gy = c 2, i.e. y T Λ 1 y = c 2 i.e n 1 λ i y 2 i = c 2. This is the equation of an ellipsoid with semiaxes of length c λ i. y 1,...,y n represent the proper axes of the ellipsoid. The direction of the axes are given by the eigenvectors of Σ since y = G T (x µ) is a rotation and G is orthogonal with eigenvectors of Σ as columns. EX. III.4D Principal components and contours of a two-dimensional normal vector. To obtain contours of a bivariate normal density we have to find the eigenvalues and eigenvectors of Σ. Suppose ( ) 12 9 µ 1 = 5, µ 2 = 8, Σ = Here the eiegnvalue equation becomes ( ) 12 λ 9 0 = Σ λi = det = (12 λ)(25 λ) 81 = λ The equation gives λ 1 = , λ 2 = The corresponding normalized eigenvectors are ( ) ( ) e 1 = e =..455 The ellipse of constant probability density (??) has center at µ = (5, 8) T and axes ±5.44 c e 1 and ±2.72 c e 2. The eigenvector e 1 lies along the line which makes an angle of cos 1 (.455) = with the X 1 axis and passes through the point (5, 8) in the (X 1,X 2 ) plane. By principal component transformation y = G T (x µ) ) where G = (e 1,e 2 ), the ellipse (??) has the equation n 1 {y : yi 2 = c 2 }. λ i 12

13 The ellipse as center at (y 1 = 0,y 2 = 0) and axes ±c λ i f i where f i, i = 1,2, are the eigenvectors of L = Diag(λ 1,λ 2 ). In terms of the new co-ordinate axes Y 1,Y 2, the eigenvectors are f 1 = (1,0) T, f 2 = (0,1) T. The lengths of the axes are c λ i, i = 1,2. The axes Y 1,Y 2 are the Principal Components of Σ, > Sigma<-matrix(c(12,9,9,25),nrow=2,ncol=2) > eigen(sigma)$values [1] > eigen(sigma)$vectors [,1] [,2] [1,] [2,] > THM. III.4E (n-dimensional standardization) Let X N(µ,Σ), with dimension n. Then there exists a square matrix C such that U := C 1 (X µ ) N(0;I). i.e. the components (U 1,...,U n ) are independent N(0;1) variables. Proof Due to symmetry of the positive matrix Σ, there always exists a non singular real matrix C such that Σ = CC T (a matrix square root of Σ). Setting U = C 1 (X µ), by means of Corollary III.3G we find: E[U] = C 1 E[X µ] = 0 Var[U] = C 1 Var[X](C 1 ) T = C 1 CC T (C T ) 1 = I. 5 Distributions derived from the normal distribution Most of the test quantities used in Statistics are based on distributions derived from the normal one. 13

14 Any linear combination of normally distributed random variables is normal. If, for instance, X N(µ; Σ), then the linear transformation Y = a+bx defines a normally distributed random variable Y N(a+Bµ, BΣB T ) (30) Compare with Corollary III.3E, where the transformation rule Σ BΣB T of the variance of X, for any affine map X a+bx, is established. Let U = (U 1,...,U n ) T be a vector of independent N(0;1) random variables. The (central) χ 2 distribution with n degrees of freedom is obtained as the squared sum of n independent N(0;1) random variables, i.e. n X 2 = Ui 2 = U T U χ 2 (n) (31) From this it is clear that if Y 1,...,Y n are independent N(µ i ;σ 2 i) r.v. s, then X 2 = n [ ] Yi µ 2 i χ 2 (n) since U i = (Y i µ i )/σ i is N(0;1) distributed. σ i THM. III.5A If X i, for i = 1,...,n, are independent normal variables from a population N(µ;σ 2 ), then the sample mean and the sample variance follow a normal and a chi squared distribution, respectively: X = 1 n n S 2 = 1 (Xi n 1 X) 2 X i X N(µ, σ2 n ) (n 1)S2 σ 2 χ 2 (n 1). Proof. X is normal as a linear transformation of normal variables. By linearity E[ X] = 1 n ie[x i ] = 1 nµ = µ and by independence n Var[ 1 n X i ] = 1 Var[X n n 2 i ] = nσ2 = σ2 i n 2 n. whence X N(µ,σ 2 /n). Now using this fact and the definition of χ 2, i ( X i µ σ ) 2 χ 2 (n), ( X µ σ/ n )2 = ( X µ) 2 σ 2 /n 14 χ 2 (1)

15 Now writing X i µ = X i X + X µ, the following decomposition can be verified: n ( X i µ n ) 2 (X i = X) 2 +n( X µ ) 2 = σ σ 2 σ = (n 1)S2 +n( X µ ) 2. σ 2 σ Therefore the random variable (n 1)S 2 σ 2 is the difference between a χ 2 (n) r.v. and a χ 2 (1) r.v., that is a r. v. with distribution χ 2 (n 1).. DEF. III.5B - The Student s t distribution with n degrees of freedom is obtained as T = Z (K/n) 1/2 t(n) (32) where Z N(0;1), K χ 2 (n), and Z and K are independent. The Fisher-Snedecor F distribution with (n 1,n 2 ) degrees of freedom appears as the following ratio F = K 1/n 1 K 2 /n 2 F(n 1,n 2 ) (33) where K 1 χ 2 (n 1 ), and K 2 χ 2 (n 2 ), and K 1,K 2 are indepedent. It is clearly seen from (32) that T 2 F(1,n 1 ). PROP. III.5C - For i = 1,...,n, let the X i be i.i.d. variables from a normal population N(µ;σ 2 ). If S 2 = 1 n (X n 1 i X) 2 then the estimated standardization of the mean T = X µ S/ n obeys a Student t distribution with n 1 degrees of freedom. 15

16 Figure 1: Chi-squared density with 5 degrees of freedom Figure 2: Student s t density with 8 degrees of freedom 16

17 Figure 3: Fisher s F density with 5 and 7 degrees of freedom If two independent samples {X i } i and {Y j } j (i = 1,...,n 1 ; j = 1,...n 2 ) are taken from normal populations with the same variance, the ratio F = S 2 1/S 2 2 obeys a Fisher s distribution with n 1 1,n 2 1 degrees of freedom. Proof Since X N(µ;σ 2 /n), we have: where T = X µ S/ n = X µ σ/ n = U S/σ = U K/(n 1) σ/ n S/ n = U = X µ σ/ n, K = S2 (n 1) σ 2 are independent with U N(0;1) and K χ 2 (n 1). Finally, under the assumptions of equal variance σ 2, S1 2 S2 2 = 1 n 1 1 S2 1 (n 1 1)/σ 2 1 n 2 1 S2 2 (n 2 1)/σ = K 1/(n 1 1) 2 K 2 /(n 2 1) where K 1 and K 2 are independent, K 1 χ 2 (n 1 1), K 2 χ 2 (n 2 1). 17

18 In the language and enviroment R, each distribution above described has four functions, as described in below (cited from: R reference card). Distributions rnorm(n, mean=0, sd=1) Gaussian (normal) rt(n, df) Student (t) rf(n, df1, df2) FisherSnedecor (F) (c2) rchisq(n, df) Pearson rbinom(n, size, prob) binomial runif(n, min=0, max=1) uniform All these functions can be used by replacing the letter r with d, p or q to get, respectively, the probability density (dfunc(x,...)), the cumulative probability density (pfunc(x,...)), and the value of quantile (qfunc(p,...), with 0 < p < 1). Some examples: > pnorm(5.8, mean=2, sd=3) funz. distribuzione N(2;9) in 5.8 [1] > dnorm(2.7,mean=0,sd=5) funz.densita N(0;25) in 2.7 [1] > qt(0.95,df=13) quantile 0.95 di "t" di Student [1] con 13 gradi di liberta > qchisq(0.99,df=10) quantile 0.99 di "chi quadro" [1] con 10 gradi di liberta > z <- seq( 0,40,length=1000) > plot(z, dchisq(z,df=10),ty="l") densita di chi quadro con 10 gradi di liberta 18