3. Probability and Statistics

Transcription

1 FE661 - Statistical Methods for Financial Engineering 3. Probability and Statistics Jitkomut Songsiri definitions, probability measures conditional expectations correlation and covariance some important random variables multivariate random variables 3-1

2 Definition a random variable X is a function mapping an outcome to a real number the sample space, S, is the domain of the random variable S X is the range of the random variable example: toss a coin three times and note the sequence of heads and tails S = {HHH,HHT,HTH,THH,HTT,THT,TTH,TTT} Let X be the number of heads in the three tosses S X = {0, 1, 2, 3} Probability and Statistics 3-2

3 Probability measures Cumulative distribution function (CDF) F (a) = P (X a) Probability mass function (PMF) for discrete RVs p(k) = P (X = k) Probability density function (PDF) for continuous RVs f(x) = df (x) dx Probability and Statistics 3-3

4 Probability Density Function Probability Density Function (PDF) f(x) 0 P (a X b) = b a f(x)dx x F (x) = f(u)du Probability Mass Function (PMF) p(k) 0 for all k p(k) = 1 k S Probability and Statistics 3-4

5 Expected values let g(x) be a function of random variable X g(x)p(x) x S E[g(X)] = g(x)f(x)dx X is discrete X is continuous Mean Variance n th Moment xp(x) X is discrete x S µ = E[X] = xf(x)dx X is continuous σ 2 = var[x] = E[(X µ) 2 ] E[X n ] Probability and Statistics 3-5

6 Facts Let Y = g(x) = ax + b, a, b are constants E[Y ] = ae[x] + b var[y ] = a 2 var[x] var[x] = E[X 2 ] (E[X]) 2 Probability and Statistics 3-6

7 Example of Random Variables Discrete RVs Bernoulli Binomial Geometric Negative binomial Poisson Uniform Continuous RVs Uniform Exponential Gaussian (Normal) Gamma, Chi-squared, Student s t, F Logistics Probability and Statistics 3-7

8 Joint cumulative distribution function F XY (a, b) = P (X a, Y b) a joint CDF is a nondecreasing function of x and y: F XY (x 1, y 1 ) F XY (x 2, y 2 ), if x 1 x 2 and y 1 y 2 F XY (x 1, ) = 0, F XY (, y 1 ) = 0, F XY (, ) = 1 P (x 1 < X x 2, y 1 < Y y 2 ) = F XY (x 2, y 2 ) F XY (x 2, y 1 ) F XY (x 1, y 2 ) + F XY (x 1, y 1 ) Probability and Statistics 3-8

9 Joint PMF for discrete RVs p XY (x, y) = P (X = x, Y = y), (x, y) S Joint PDF for continuous RVs Marginal PMF f XY (x, y) = 2 F XY (x, y) x y p X (x) = y S p XY (x, y), p Y (y) = x S p XY (x, y) Marginal PDF f X (x) = f XY (x, z)dz, f Y (y) = f XY (z, y)dz Probability and Statistics 3-9

10 Conditional Probability Discrete RVs the conditional PMF of Y given X = x is defined by p Y X (y x) = P (Y = y X = x) = = p XY (x, y) p X (x) P (X = x, Y = y) P (X = x) Continuous RVs the conditional PDF of Y given X = x is defined by f Y X (y x) = f XY (x, y) f X (x) Probability and Statistics 3-10

11 Conditional Expectation the conditional expectation of Y given X = x is defined by Continuous RVs E[Y X] = y f Y X (y x)dy Discrete RVs E[Y X] = y y p Y X (y x) E[Y X] is the center of mass associated with the conditional pdf or pmf E[Y X] can be viewed as a function of random variable X E[E[Y X]] = E[Y ] Probability and Statistics 3-11

12 in fact, we can show that E[h(Y )] = E[E[h(Y ) X]] for any function h( ) that E[ h(y ) ] < proof. E[E[h(Y ) X]] = = = = E[h(Y ) x]f X (x)dx h(y)f Y X (y x)dy f X (x)dx h(y) = E[h(Y )] h(y)f Y (y) dy f XY (x, y) dxdy Probability and Statistics 3-12

13 Independence of two random variables X and Y are independent if and only if F XY (x, y) = F X (x)f Y (y), x, y this is equivalent to Discrete Random Variables p XY (x, y) = p X (x)p Y (y) p Y X (y x) = p Y (y) Continuous Random Variables f XY (x, y) = f X (x)f Y (y) f Y X (y x) = f Y X (y) If X and Y are independent, so are any pair of functions g(x) and h(y ) Probability and Statistics 3-13

14 Expected Values and Covariance the expected value of Z = g(x, Y ) is defined as E[Z] = E[Z] = x g(x, y) f XY (x, y) dx dy g(x, y) p XY (x, y) y X, Y continuous X, Y discrete E[X + Y ] = E[X] + E[Y ] E[XY ] = E[X]E[Y ] if X and Y are independent Covariance of X and Y cov(x, Y ) = E[(X E[X])(Y E[Y ])] = E[XY ] E[X]E[Y ] cov(x, Y ) = 0 if X and Y are independent (the converse is NOT true) Probability and Statistics 3-14

15 Correlation Coefficient denote σ X = var(x), σ Y = var(y ) the standard deviations of X and Y the correlation coefficient of X and Y is defined by 1 ρ XY 1 ρ XY = cov(x, Y ) σ X σ Y ρ XY gives the linear dependence between X and Y : for Y = ax + b, ρ XY = 1 if a > 0 and ρ XY = 1 if a < 0 X and Y are said to be uncorrelated if ρ XY = 0 Probability and Statistics 3-15

16 if X and Y are independent then X and Y are uncorrelated but the converse is NOT true Probability and Statistics 3-16

17 Law of Total Variance suppose that X and Y are random variables var(y ) = E[var(Y X)] + var(e[y X]) aka Eve s Law; we say the unconditional variance equals EV plus VE Proof. using E[E[Y X]] = E[Y ] var(y ) = E[Y 2 ] (E[Y ]) 2 = E[E[Y 2 X]] (E[E[Y X]]) 2 = E[var(Y X)] + E[(E[Y X]) 2 ] (E[E[Y X]]) 2 = E[var(Y X)] + var[e[y X]] Probability and Statistics 3-17

18 Moment Generating Functions the moment generating function (MGF) Φ(t) is defined for all t by Φ(t) = E[e tx ] = { etx f(x)dx, X is continuous x etx p(x), X is discrete except for a sign change, Φ(t) is the 2-sided Laplace transform of pdf knowing Φ(t) is equivalent to knowing f(x) E[X n ] = dn Φ(t) dt n t=0 MGF of the sum of independent RVs is the product of the individual MGF Φ X+Y (t) = E[e t(x+y ) ] = Φ X (t)φ Y (t) Probability and Statistics 3-18

19 Gaussian (Normal) random variables arise as the outcome of the central limit theorem the sum of a large number of RVs is distributed approximately normally many results involving Gaussian RVs can be derived in analytical form let X be a Gaussian RV with parameters mean µ and variance σ 2 Notation X N (µ, σ 2 ) PDF f(x) = 1 2πσ 2 exp (x µ)2 2σ 2, < x < Mean E[X] = µ Variance var[x] = σ 2 MGF Φ(t) = e µt+σ2 t 2 /2 Probability and Statistics 3-19

20 let Z N (0, 1) be the normalized Gaussian variable CDF of Z is F Z (z) = 1 2π z e t2 /2 dt Φ(z) then CDF of X N (µ, σ 2 ) can be obtained by ( ) x µ F X (x) = Φ σ in MATLAB, the error function is defined as erf(x) = 2 π x 0 e t2 dt hence, Φ(z) can be computed via the erf command as Φ(z) = 1 [ ( )] z 1 + erf 2 2 Probability and Statistics 3-20

21 Gamma random variables PDF f(x) = λ(λx)α 1 e λx, x 0; α, λ > 0 Γ(α) where Γ(z) is the gamma function, defined by Γ(z) = 0 x z 1 e x dx, z > 0 Mean E[X] = α λ Variance var[x] = α ( ) λ 2 α 1 MGF Φ(t) = λ λ t if X 1 and X 2 are independent gamma RVs with parameters (α 1, λ) and (α 2, λ) then X 1 + X 2 is a gamma RV with parameters (α 1 + α 2, λ) Probability and Statistics 3-21

22 Properties of the gamma function Γ(1/2) = π Γ(z + 1) = zγ(z) for z > 0 Γ(m + 1) = m!, for m a nonnegative integer Special cases a Gamma RV becomes exponential RV when α = 1 m-erlang RV when α = m, a positive integer chi-square RV with n DF when α = n/2, λ = 1/2 Probability and Statistics 3-22

23 Chi-square random variables if Z 1, Z 2,..., Z n are independent normal RVs, then X defined by X = Z Z Z 2 n χ 2 n is said to have a chi-square distribution with n degrees of freedom Φ(t) = E[ n i=1 etz2 i ] = n i=1 E[etZ2 i ] = (1 2t) n/2 we recognize that X is a gamma RV with parameters (n/2, 1/2) sum of independent chi-square RVs with n 1 and n 2 DF is the chi-square with n 1 + n 2 DF PDF f(x) = (1/2)e x/2 (x/2) n/2 1, x > 0 Γ(n/2) Mean E[X] = n Variance var(x) = 2n Probability and Statistics 3-23

24 t random variables if Z N (0, 1) and χ 2 n are independent then T n = Z χ 2 n /n is said to have a t-distribution with n degree of freedom t density is symmetric about zero t has greater variability than the normal T n Z for n large for 0 < α < 1 such that P (T n t α,n ) = α), P (T n t α,n ) = 1 α t α,n = t 1 α,n Probability and Statistics 3-24

25 F random variables if χ 2 n and χ 2 m are independent chi-square RVs then the RV F n,m defined by F n,m = χ2 n/n χ 2 m/m is said to have an F -distribution with n and m degree of freedoms for any α (0, 1), let F α,n,m be such that P (F n,m > F α,n,m ) = α then P ( χ 2 m /m χ 2 n/n 1 ) F α,n,m = 1 α since χ2 m /m χ 2 n /n is another F m,n RV, it follows that 1 α = P ( ) χ 2 m /m χ 2 n/n F 1 α,n,m 1 F α,n,m = F 1 α,m,n Probability and Statistics 3-25

26 Logistics random variables CDF PDF F (x) = Mean E[X] = µ f(x) = e(x µ)/ν, < x <, µ, ν > e (x µ)/ν e (x µ)/ν ν(1 + e (x µ)/ν ) 2, < x < if µ = 0, ν = 1 then X is a standard logistic µ is the mean of the logistic ν is called the dispersion parameter Probability and Statistics 3-26

27 Multivariate Random Variables probabilities cross correlation, cross covariance Gaussian random vectors Probability and Statistics 3-27

28 Random vectors we denote X a random vector X is a function that maps each outcome ζ to a vector of real numbers an n-dimensional random variable has n components: X 1 X n X = X 2. also called a multivariate or multiple random variable Probability and Statistics 3-28

29 Probabilities Joint CDF F (X) F X (x 1, x 2,..., x n ) = P (X 1 x 1, X 2 x 2,..., X n x n ) Joint PMF p(x) p X (x 1, x 2,..., x n ) = P (X 1 = x 1, X 2 = x 2,..., X n = x n ) Joint PDF f(x) f X (x 1, x 2,..., x n ) = n x 1... x n F (X) Probability and Statistics 3-29

30 Marginal PMF p Xj (x j ) = P (X j = x j ) = x p X (x 1, x 2,..., x n ) x j 1 x j+1 x n Marginal PDF f Xj (x j ) =... f X (x 1, x 2,..., x n ) dx 1... dx j 1 dx j+1... dx n Conditional PDF: the PDF of X n given X 1,..., X n 1 is f(x n x 1,..., x n 1 ) = f X (x 1,..., x n ) f X1,...,X n 1 (x 1,..., x n 1 ) Probability and Statistics 3-30

31 Characteristic Function the characteristic function of an n-dimensional RV is defined by Φ(ω) = Φ(ω 1,..., ω n ) = E[e i(ω 1X 1 + +ω n X n ) ] = e iωt X f(x)dx X where ω = ω 1 ω 2., X = x 1 x 2. ω n x n Φ(ω) is the n-dimensional Fourier transform of f(x) Probability and Statistics 3-31

32 Independence the random variables X 1,..., X n are independent if the joint pdf (or pmf) is equal to the product of their marginal s Discrete Continuous p X (x 1,..., x n ) = p X1 (x 1 ) p Xn (x n ) f X (x 1,..., x n ) = f X1 (x 1 ) f Xn (x n ) we can specify an RV by the characteristic function in place of the pdf, X 1,..., X n are independent if Φ(ω) = Φ 1 (ω 1 ) Φ n (ω n ) Probability and Statistics 3-32

33 the expected value of a function Expected Values g(x) = g(x 1,..., X n ) of a vector random variable X is defined by E[g(X)] = g(x)f(x)dx E[g(X)] = g(x)p(x) x x Continuous Discrete Mean vector X 1 X n µ = E[X] = E X 2. E[X 1 ] E[X 2 ]. E[X n ] Probability and Statistics 3-33

34 Correlation and Covariance matrices Correlation matrix has the second moments of X as its entries: R E[XX T ] = E[X 1 X 1 ] E[X 1 X 2 ] E[X 1 X n ] E[X 2 X 1 ] E[X 2 X 2 ] E[X 2 X n ] E[X n X 1 ] E[X n X 2 ] E[X n X n ] with R ij = E[X i X j ] Covariance matrix has the second-order central moments as its entries: C E[(X µ)(x µ) T ] with C ij = cov(x i, X j ) = E[(X i µ i )(X j µ j )] Probability and Statistics 3-34

35 Properties of correlation and covariance matrices let X be a (real) n-dimensional random vector with mean µ Facts: R and C are n n symmetric matrices R and C are positive semidefinite If X 1,..., X n are independent, then C is diagonal the diagonals of C are given by the variances of X k if X has zero mean, then R = C C = R µµ T Probability and Statistics 3-35

36 Cross Correlation and Cross Covariance let X, Y be vector random variables with means µ X, µ Y respectively Cross Correlation cor(x, Y ) = E[XY T ] if cor(x, Y ) = 0 then X and Y are said to be orthogonal Cross Covariance cov(x, Y ) = E[(X µ X )(Y µ Y ) T ] = cor(x, Y ) µ X µ T Y if cov(x, Y ) = 0 then X and Y are said to be uncorrelated Probability and Statistics 3-36

37 Affine transformation let Y be an affine transformation of X: Y = AX + b where A and b are deterministic matrices µ Y = Aµ X + b µ Y = E[AX + b] = AE[X] + E[b] = Aµ X + b C Y = AC X A T C Y = E[(Y µ Y )(Y µ Y ) T ] = E[(A(X µ X ))(A(X µ X )) T ] = AE[(X µ X )(X µ X ) T ]A T = AC X A T Probability and Statistics 3-37

38 Gaussian random vector X 1,..., X n are said to be jointly Gaussian if their joint pdf is given by f(x) f X (x 1, x 2,..., x n ) = 1 (2π) n/2 det(σ) 1/2 exp 1 2 (X µ)t Σ 1 (X µ) µ is the mean (n 1) and Σ 0 is the covariance matrix (n n): µ 1 µ n µ = µ 2., Σ = Σ 11 Σ 12 Σ 1n Σ 21 Σ 22 Σ 2n Σ n1 Σ n2 Σ nn and µ k = E[X k ], Σ ij = E[(X i µ i )(X j µ j )] Probability and Statistics 3-38

39 example: the joint density function of X (not normalized) is given by f(x 1, x 2, x 3 ) = exp x x (x 3 1) 2 + 2x 1 (x 3 1) 2 f is an exponential of negative quadratic in x so X must be a Gaussian f(x 1, x 2, x 3 ) = exp 1 2 x 1 x 2 x x 1 x x 3 1 T the mean vector is (0, 0, 1) the covariance matrix is C = = / Probability and Statistics 3-39

40 the variance of x 1 is highest while x 2 is smallest x 1 and x 2 are uncorrelated, so are x 2 and x 3 Probability and Statistics 3-40

41 examples of Gaussian density contour (the exponent of exponential) [ x1 ] T [ ] 1 [ ] Σ11 Σ 12 x1 x 2 Σ 12 Σ 22 x 2 = 1 uncorrelated different variances correlated Σ = [ ] Σ = [ ] 1/ Σ = [ 1 ] Probability and Statistics 3-41

42 Properties of Gaussian variables many results on Gaussian RVs can be obtained analytically: marginal s of X is also Gaussian conditional pdf of X k given the other variables is a Gaussian distribution uncorrelated Gaussian random variables are independent any affine transformation of a Gaussian is also a Gaussian these are well-known facts and more can be found in the areas of estimation, statistical learning, etc. Probability and Statistics 3-42

43 Characteristic function of Gaussian Φ(ω) = Φ(ω 1, ω 2,..., ω n ) = e iµt ω e ωt Σω 2 Proof. By definition and arranging the quadratic term in the power of exp 1 Φ(ω) = e ixt ω e (X µ)t Σ 1 (X µ) (2π) n/2 Σ 1/2 2 dx = eiµt ω e ω T Σω 2 (2π) n/2 Σ 1/2 X X = exp (iµ T ω) exp ( 1 2 ωt Σω) e (X µ iσω)t Σ 1 (X µ iσω) 2 dx (the integral equals 1 since it is a form of Gaussian distribution) for one-dimensional Gaussian with zero mean and variance Σ = σ 2, Φ(ω) = e σ2 ω 2 2 Probability and Statistics 3-43

44 Affine Transformation of a Gaussian is Gaussian let X be an n-dimensional Gaussian, X N (µ, Σ) and define Y = AX + b where A is m n and b is m 1 (so Y is m 1) Φ Y (ω) = E[e iωt Y ] = E[e iωt (AX+b) ] = E[e iωt AX e iωt b ] = e iωt b Φ X (A T ω) = e iωt b e iµt A T ω e ωt AΣA T ω/2 = e iωt (Aµ+b) e ωt AΣA T ω/2 we read off that Y is Gaussian with mean Aµ + b and covariance AΣA T Probability and Statistics 3-44

45 Marginal of Gaussian is Gaussian the k th component of X is obtained by X k = [ ] X e T k X (e k is a standard unit column vector; all entries are zero except the k th position) hence, X k is simply a linear transformation (in fact, a projection) of X X k is then a Gaussian with mean e T k µ = µ k and covariance e T k Σ e k = Σ kk Probability and Statistics 3-45

46 Uncorrelated Gaussians are independent suppose (X, Y ) is a jointly Gaussian vector with [ ] µx mean µ = and covariance µ y [ ] CX 0 0 C Y in otherwords, X and Y are uncorrelated Gaussians: the joint density can be written as f XY (x, y) = = cov(x, Y ) = E[XY T ] E[X]E[Y ] T = 0 1 (2π) n C X 1/2 C Y 1/2 exp (2π) n/2 C X 1/2e 1 2 (x µ T x) C 1 X (x µ x) proving the independence [ ] T [ x µx C 1 y µ y X 0 0 C 1 Y ] [ ] x µx y µ y 1 (2π) n/2 C Y 1/2e 1 2 (y µ T y) C 1 Y (y µ y) Probability and Statistics 3-46

47 Conditional of Gaussian is Gaussian let Z be an n-dimensional Gaussian which can be decomposed as Z = [ X Y ] N ([ ] [ ]) µx Σxx Σ, xy µ y Σ T xy Σ yy the conditional pdf of X given Y is also Gaussian with conditional mean and conditional covariance µ X Y = µ x + Σ xy Σ 1 yy (Y µ y ) Σ X Y = Σ x Σ xy Σ 1 yy Σ T xy Probability and Statistics 3-47

48 Proof: from the matrix inversion lemma, Σ 1 can be written as Σ 1 = S 1 Σ 1 yy Σ T xys 1 S 1 Σ xy Σ 1 yy Σ 1 yy + Σ 1 yy Σ T xys 1 Σ xy Σ 1 yy where S is called the Schur complement of Σ xx in Σ and S = Σ xx Σ xy Σ 1 yy Σ T xy det Σ = det S det Σ yy we can show that Σ 0 if any only if S 0 and Σ yy 0 Probability and Statistics 3-48

49 from f X Y (x y) = f X (x, y)/f Y (y), we calculate the exponent terms [ ] T [ ] x µx Σ 1 x µx y µ y y µ y (y µ y ) T Σ 1 yy (y µ y ) = (x µ x ) T S 1 (x µ x ) (x µ x ) T S 1 Σ xy Σ 1 yy (y µ y ) (y µ y ) T Σ 1 yy Σ T xys 1 (x µ x ) +(y µ y ) T (Σ 1 yy Σ T xys 1 Σ xy Σ 1 yy )(y µ y ) = [x µ x Σ xy Σ 1 yy (y µ y )] T S 1 [x µ x Σ xy Σ 1 yy (y µ y )] (x µ X Y ) T Σ 1 X Y (x µ X Y ) f X Y (x y) is an exponential of quadratic function in x so it has a form of Gaussian Probability and Statistics 3-49

50 Standard Gaussian vectors for an n-dimensional Gaussian vector X N (µ, C) with C 0 let A be an n n invertible matrix such that AA T = C (A is called a factor of C) then the random vector Z = A 1 (X µ) is a standard Gaussian vector, i.e., Z N (0, I) (obtain A via eigenvalue decomposition or Cholesky factorization) Probability and Statistics 3-50

51 Quadratic Form Theorems let X = (X 1,..., X n ) be a standard n-dimensional Gaussian vector: then the following results hold X N (0, I) X T X χ 2 (n) let A be a symmetric and idempotent matrix and m = tr(a) then X T AX χ 2 (m) Probability and Statistics 3-51

52 Proof: the eigenvalue decomposition of A: A = UDU T where λ(a) = 0, 1 U T U = UU T = I it follows that X T AX = X T UDU T X = Y T DY = n i=1 d ii Y 2 i since U is orthogonal, Y is also a standard Gaussian vector since A is idempotent, d ii is either 0 or 1 and tr(d) = m therefore X T AX is the m-sum of standard normal RVs Probability and Statistics 3-52

53 References Chapter 1-3 in J.M. Wooldridge, Econometric Analysis of Cross Section and Panel Data, 2nd edition, the MIT press, 2010 Chapter 1-5 in S.M. Ross, Introduction to Probability and Statistics for Engineers and Scientists, 4th edition, Academic press, 2009 Appendix A in A.C. Cameron and P.K. Trevedi, Microeconometrics: Method and Applications, Cambridge University Press, 2005 Probability and Statistics 3-53