Outline for today. Maximum likelihood estimation. Computation with multivariate normal distributions. Multivariate normal distribution

Transcription

1 Outline for today Maximum likelihood estimation Rasmus Waageetersen Deartment of Mathematics Aalborg University Denmark October 30, 2007 the multivariate normal distribution linear and linear mixed models the likelihood function maximum likelihood estimation restricted maximum likelihood estimation likelihood ratio tests 1/ 22 2/ 22 Multivariate normal distribution Comutation with multivariate normal distributions Stochastic vector Y = (Y 1,...,Y n ) T is multivariate normal N n (µ,σ) with mean vector µ and ositive definite covariance matrix Σ if it has density function 1 f (y;µ,σ) = ( (2π) n Σ )1/2 ex( 1 2 (y µ)t Σ 1 (y µ)) EY i = µ i, VarY i = Σ ii and Cov(Y i,y j ) = Σ ij. Examle: Y 1,...,Y n indeendent N(µ,σ 2 ). µ = (µ,...,µ) T, Σ = σ 2 I where I is identity matrix. Suose the n-dimensional random vector Y is N n (µ,σ) and A is a m n. Then AY N(Aµ,AΣA T ) NB: AΣA T only ositive definite if rank of A is m. 3/ 22 4/ 22

2 Model with random intercets as multivariate normal Linear models Y ij = β 1 + U i + β 2 x ij + ǫ ij U i N(0,τ 2 ) and ǫ ij N(0,σ 2 ) indeendent. EY ij = β 1 + β 2 x ij VarY ij = τ 2 + σ 2 0 i l Cov(Y ij,y lk ) = τ 2 i = l,j k τ 2 + σ 2 i = l,j = k 1 x 11 [ ] µ = 1 x 12 β β 2 1 x nm General formulation of mean vector for linear models: µ = Xβ 5/ 22 6/ 22 Linear mixed models Linear mixed model and multivariate normal distribution Consider mixed model: Y ij = β 1 + U i + β 2 x ij + ǫ ij May be written in matrix vector form as Y = Xβ + ZU + ǫ where β = (β 1,β 2 ) T, U = (U 1,...,U m ) T and ǫ = (ǫ 11,ǫ 12,...,ǫ nm ) T. Consider stochastic vector Y = Xβ + ZU + ǫ where U N(0,Ψ) and ǫ N(0,σ 2 D) are indeendent. Then Y has multivariate normal distribution Y N(Xβ,ZΨZ T + σ 2 D) Shall need multivariate normal density of Y for maximum likelihood estimation. 7/ 22 8/ 22

3 Linear mixed models using lmer Likelihood function General lmer model formulation y~ fixed formula +( rand formula1 Grou1)+...+( rand. formulan Groun) translates into linear mixed model with indeendent sets of random effects for each grouing variable and e.g. (z Groui) corresonds to U i + V i z i.e. model with random intercet and random sloe for covariate z within each level of grouing factor Groui. NB indeendence between levels but intercet and sloe deendent within level. Simle examle: binomial data. Consider binomial random vector Y b(n,) and a secific realization y {0,1,...,n}. Probability of observing x: P(Y = y;) = ( ) n y (1 ) n y y Likelihood function is P(Y = y;) but regarded as function of arameter : ( ) n L() = y (1 ) n y y 9/ 22 10/ 22 Examle: Y b(10,) y = 3 Maximum likelihood estimation Maximum likelihood estimate ˆ is the value of that makes it most likely to observe the data y. I.e. L() ˆ = arg max L() = arg max y (1 ) n y = arg max y log +(n y)log(1 ) (binomial coeffiecient does not deend on and can be ignored) Makes sense intuitively and MLE is otimal in the sense of having small variance (best recision). By differentiating and equating to zero we obtain ˆ = y n Hence, MLE in this case coincides with moment estimate based on equation EY /n = 11/ 22 12/ 22

4 Likelihood function for simle normal samle Likelihood for linear mixed model Suose Y 1,...,Y n are indeendent N(µ,σ 2 ). Density function for Y i is f (y;µ,σ 2 ) = 1 2πσ 2 ex( (y µ)2 /(2σ 2 )) and likelihood function based on observations y 1,...,y n is L(µ,σ 2 1 ) = ( σ 2 )n ex( n (y i µ) 2 /(2σ 2 )) i=1 log likelihood for linear mixed model: 1 2 log( Σ(ψ) ) 1 2 (y Xβ)T Σ(ψ) 1 (y Xβ)) ψ: arameters for Σ(ψ) (e.g. variance comonents) MLE are ˆµ = ȳ and ˆσ 2 = n i=1 (y i ȳ) 2 /n. NB: ˆσ 2 s 2! 13/ 22 14/ 22 MLE and weighted least squares MLE for balanced one-way anova Assume ψ known. MLE for β is weighted least squares estimate ˆβ(ψ) = arg min β (y Xβ) T Σ(ψ) 1 (y Xβ) Differentiate and equate to zero: X T Σ(ψ) 1 (y Xβ) = 0 ˆβ(ψ) = (X T Σ 1 X) 1 X T Σ(ψ) 1 y (rovided relevant inverses exist) Covariance arameters for ψ: often numerical otimization is needed to maximize rofile likelihood 1 2 log( Σ(ψ) ) 1 2 (y X ˆβ(ψ)) T Σ(ψ) 1 (y X ˆβ(ψ)) MLE: MLE for σ 2 is biased: Y ij = β i + ǫ ij ˆβ i = ȳ i ˆσ 2 = 1 nm (y ij ȳ i ) 2 i,j Eˆσ 2 = nm m nm σ2 and differs from ANOVA estimate s 2 = 1 nm m i,j (y ij ȳ i ) 2. 15/ 22 16/ 22

5 REML (restricted/residual maximum likelihood I REML (restricted/residual maximum likelihood II Bias of MLE for variance arameters is due to loss of degrees of freedom when estimating mean arameters. Idea: consider n (n ) matrix A with columns orthogonal to columns in X: A T X = 0 and filtered data vector Ỹ = AT Y Then Ỹ N(0,A T Σ(ψ)A) and we may comute MLE for covariance arameter ψ based on likelihood for Ỹ (called REML estimates). Given REML estimate ˆψ we use weighted least squares estimate of β: ˆβ = ˆβ = (X T Σ( ˆψ) 1 X) 1 X T Σ 1 ( ˆψ)y Secific choice of A does not matter, i.e. same REML estimate obtained with B = AC where C is invertible (n ) (n ). 17/ 22 18/ 22 Likelihood ratio test for binomial data Return to the binomial examle: Y b(n,). Suose we wish to test hyothesis H 0 : = 0.4 versus H 1 : 0.4. Likelihood ratio tests Consider two models M 1 and M 2 where M 2 is a submodel of M 1 obtained by setting one or more arameters in M 1 equal to zero. Let L 1 and L 2 denote the maximal likelihoods under the two models. Then 2 times the log likelihood ratio 2log(L 2 /L 1 ) Intuitive idea: comare likelihood L(0.4) with the maximal ossible likelihood under the alternative L(ˆ). is aroximately χ 2 (d 1 d 2 ) where d 1 and d 2 are the numbers of free arameters for the two models. Then small valus of likelihood ratio critical. L(0.4) L(ˆ) Equivalently: large values of 2log(L(0.4)/L(ˆ)) critical. 19/ 22 Aroximation requires that number of observations is large (tends to infinity). In simle situations log likelihood ratio test is equivalent to usual ANOVA F-test. χ 2 aroximation not accurate for testing variance arameters equal to zero (-values too large, Faraway (2006) recommends arametric bootstra). 20/ 22

6 Likelihood ratio tests in R For variance comonents ψ we may use REML likelihoods but not for mean arameters β since REML deends on model for the mean. Likelihood ratio test for (systematic) cardboard effect: > out3=lmer(reflektans~factor(pa.nr.)+(1 StedPa.nr.),method="ML") > loglik(out3) log Lik (df=6) > out4=lmer(reflektans~(1 StedPa.nr.),method="ML") > loglik(out4) log Lik (df=2) > 1-chisq(2*( ),4) [1] e-12 > #using anova > anova(out3,out4)... Df AIC BIC loglik Chisq Chi Df Pr(>Chisq) out out e-12 *** Exercises 1. formulate random intercet and sloe model for Orthodont data (day 1) as multivariate normal N(µ, Σ). What are the design matrices X and Z? 2. Find the formula for the REML estimate of the variance in a simle oneway anova Y ij = µ + α i + ǫ ij with two grous i = 1, 2 each with two observations j = 1, 2 (hint: find aroriate matrix A). 3. Use lmer to comare ML and REML estimates for the Orthodont data (consider mixed model with age, Sex and random intercets). 4. show that MLE for indeendent N(µ, σ 2 ) normal samle x 1,..., x n are x and (n 1)s 2 /n. 5. Comute variance of MLE ˆσ 2 and s 2 given that n i=1 (x i x) 2 is σ 2 χ 2 (n 1) (hint: Varχ 2 (f ) = 2f ). What haens with the difference between the two estimates when n tends to infinity? 6. Comute the -value for the hyothesis H 0 : = 0.4 when data y = 3 is observed from Y b(10, ) (dbinom()). 7. Comute likelihood ratio test for the Sex effect and the random intercet variance comonent in the mixed model from 3. 21/ 22 22/ 22