In this course we: do distribution theory when ǫ i N(0, σ 2 ) discuss what if the errors, ǫ i are not normal? omit proofs.

Transcription

1 Distribution Theory Question: What is distribution theory? Answer: How to compute the distribution of an estimator, test or other statistic, T : Find P(T t), the Cumulative Distribution Function (CDF) of T. Find f T (t) the density of T. Say something like T is N(12, 25) or T is Binomial(100,0.75) or other named distribution. (Possible distributions include Normal, Binomial, t, Bernoulli, F, χ 2, Gamma, Geometric, Negative Binomial, Poisson, Weibull, Logistic...) Find E(T ), Var(T ) or other moments of T

2 In this course we: do distribution theory when ǫ i N(0, σ 2 ) discuss what if the errors, ǫ i are not normal? omit proofs.

3 Standard normal distribution Standard normal density is f (x) = 1 2π e x 2 /2 < x < We say that Z N(0, 1) if the density of Z is standard normal: Reminder: if X has density f (x) then Moments of Z are E(g(X )) = g(x)f (x)dx E(Z ) = Var(Z ) = E(Z 2 ) = 1. z 2π e z 2 /2 dz = 0

4 General univariate normal distribution Definition: If Z N(0, 1) then X = µ + σz N(µ, σ 2 ). Moments: E(X ) = µ + σe(z ) = µ Var(X ) = σ 2 Var(Z ) = σ 2

5 Multivariate normal distribution Definition: If Z 1,..., Z n are independent N(0, 1) then Z = [ Z 1 Z n ] T MVNn (0, I ) We say that the vector Z has a standard n-dimensional multivariate normal distribution. We can define E(Z ) and Var(Z ) for vectors like Z as follows: If X is a random vector of length n, say X T = [X 1 X n ] then E(X 1 ) µ X E(X ) =. E(X n ) and Var(X ) is an n n matrix E [(X ] µ X )(X µ X ) T

6 Variance Covariance Matrices Note that the ijth entry of (X µ X )(X µ X ) T is Definition: (X i E(X i ))(X j E(X j )) Cov(X i, X j ) = E [(X i E(X i ))(X j E(X j ))] = E(X i X j ) E(X i )E(X j ) Definition: If M is a matrix then E(M) is a matrix whose ijth entry is E(M ij ). So Var(X ) has ijth entry Cov(X i, X j ) and diagonal entries Cov(X i, X i ) = Var(X i ).

7 Standard Multivariate Normal Moments Now suppose Z MVN n (o, I ). Then E(Z 1 ) E(Z ) =. E(Z n ) = 0 n The 0 n matches the 0 in MVN n (0, I ). Next we compute the variance of Z. Note that E [ (Z 0)(Z 0) T ] has ijth entry E(Z i Z j ) = { 0 i j (independence) E(Z 2 i ) = 1 i = j So Var(Z ) = I n n the n n identity matrix.

8 General Multivariate Normal Now suppose that where X = AZ + µ A is an m n matrix of constants. µ is a vector in R m Z MVN n (0, I ) Then we say that X has a MVN m (µ, AA T ) distribution. Now E(X ) = E(AX + µ) has ith component E[(AZ ) i ] + µ i = E( j A ij Z j ) + µ i = j A ij E(Z j ) + µ i = µ i

9 Variance Covariance of MVN Moreover, Var(X ) = E( [(X ] µ)(x µ) T = E[(AZ )(AZ ) T ] = E[AZZ T A T ] = AE[ZZ T ]A T = AIA T = AA T Thus X MVN(µ, Σ) means that E(X ) = µ Var(X ) = Σ X is normal.

10 Things to notice along the way 1. E(AX + b) = AE(X ) + b when A m n, X n 1 b m 1 and A and b are constant. 2. E(AMB) = AE(M)B whenever A, B and M are matrices whose dimensions make the multiplication possible and A and B are non-random constant matrices while M is a random matrix. 3. Var(AX + b) = AVar(X )A T where A and X are as in 1). The notation Cov(X ) is sometimes used for Var(X ). This matrix is called the variance-covariance matrix of X.

11 Application to Least Squares The following do not use the normal assumption: ˆβ = (X T X ) 1 X T Y = (X T X ) 1 X T (X β + ǫ) = (X T X ) 1 (X T X )β + (X T X ) 1 X T ǫ = β + (X T X ) 1 X T ǫ E( ˆβ) = β + (X T X ) 1 X T E(ǫ) = β So ˆβ is unbiased.

12 Variance of Least Squares Estimator Var( ˆβ) = E[( ˆβ β)( ˆβ β) T ] = (X T X ) 1 X T E(ǫǫ T ) ((X ) T X ) 1 X T T = (X T X ) 1 X T σ 2 I ((X T X ) 1 X T ) T = σ 2 (X T X ) 1

13 Distribution Theory for Least Squares Assume: E(ǫ i ) = 0 and Y = X β + ǫ Then 1. E( ˆβ) = β 2. E(ˆµ) = X E( ˆβ) = X β = µ 3. ˆǫ = (I X (X T X ) 1 X T )ǫ Mǫ where M = I X (X T X ) 1 X T. 4. E(ˆǫ) = ME(ǫ) = 0

14 Homoscedastic errors Define: H = X (X T X ) 1 X T, the hat matrix so that M = I H. If also Var(ǫ) = σ 2 I (as will be the case for instance if the ǫ i are iid) then 1. Var( ˆβ) = σ 2 (X T X ) 1 2. Var(ˆµ) = σ 2 X (X T X ) 1 X T = σ 2 H 3. Var(ˆǫ) = MVar(ǫ)M T = σ 2 MM T

15 Normal errors If also ǫ i N(0, σ 2 ) are independent then ˆβ MVN(β, σ 2 (X T X ) 1 ) ˆµ MVN(µ, σ 2 H) Var(ˆǫ) = σ 2 MM T = σ 2 M Definition: A matrix Q is idempotent if QQ Q 2 = Q

16 So What? 1. Distribution theory of Sums of Squares in ANOVA tables uses this. 2. F tests for hypotheses about parameters are justified using these matrix ideas. 3. t-tests, and confidence intervals for c T β (where c is a vector of length p can be derived using these ideas.

17 Estimation of σ Based on the error sum of squares defined by ESS = ˆǫ 2 = ˆǫ 2 i = (Mǫ) T (Mǫ) = ǫ T M T Mǫ = ǫ T Mǫ Fact E[ESS] = E(ǫ T Mǫ) = σ 2 i M ii = σ 2 trace(m)

18 Counting Degrees of Freedom Definition: The trace of a square matrix Q is defined by trace(q) = i Q ii Fact: trace(m) = trace(i H) = trace(i ) trace(h) = n trace( }{{} X (X T X ) 1 X T ) }{{} A B = n trace((x T X ) 1 X T X) }{{} I p p = n trace(i p p ) = n p Notice that p is the number of columns of X including the column of 1 s if present.

19 Summary of result E [ ] ESS n p = σ 2 So the Mean Squared Error, ESS/(n p) is an unbiased estimate of σ 2.

20 Inference for linear combinations of entries in β Examples of linear combinations: Confidence intervals and tests for µ x, the mean of Y corresponding to a particular value x of the covariate. (See Polynomial Regression page for example.) ˆµ x = x 1 ˆβ1 + + x p ˆβp = x t ˆβ Confidence intervals and tests for β k = [0,..., 0, 1, 0,..., 0]β where the 1 is in position k.

21 Basic Ingredients 1. E(x t ˆβ) = x T β = µ x. 2. Var(x t ˆβ) = x T Var( ˆβ)x = σ 2 x T (X T X ) 1 x 3. ˆσ 2 = ESS/(n p) is consistent (that is, converges to σ 2 as n ). 4. If ǫ i N(0, σ 2 ) or n is large then 5. If as in 4 then ˆβ MVN(β, σ 2 (X T X ) 1 ) ˆµ x MVN(µ x, σ 2 x T (X T X ) 1 x) 6. If ǫ i N(0, σ 2 ) then ESS/σ 2 χ 2 n p 7. If ǫ i N(0, σ 2 ) then ˆβ is independent of ESS.

22 More points 8. If ǫ i N(0, σ 2 ) then ˆµ x µ x ˆσ x T (X T X ) 1 x t n p 9. A confidence interval for x T β is x t ESS ˆβ ± tα/2,n p x n p T (X T X ) 1 x 10. The confidence interval in 9 is justified either by Normal errors OR large n