Covariance Models (*) X i : (n i p) design matrix for fixed effects β : (p 1) regression coefficient for fixed effects

Transcription

1 Covariance Models (*) Mixed Models Laird & Ware (1982) Y i = X i β + Z i b i + e i Y i : (n i 1) response vector X i : (n i p) design matrix for fixed effects β : (p 1) regression coefficient for fixed effects Note: see pg 60 for specific examples Note: FLW Appendix A = Gentle Intro to Matrices 139 Heagerty, 2006

2 Covariance Models (*) Mixed Models Laird & Ware (1982) Z i : (n i q) design matrix for random effects b i : (q 1) vector of random effects e i : (n i 1) vector of errors For the random components of the model we typically assume: b i N (0, D) e i N (0, R i ) 140 Heagerty, 2006

3 140-1 Heagerty, 2006 Laird & Ware Chair, Dept Biostatistics HSPH Associate Dean HSPH

4 LMM and components of variation (*) This yields a covariance structure: cov(y i ) = Z i DZ T i }{{} + R i }{{} between-cluster var + within-cluster var We assume that observations on different subjects are independent Note: This is a matrix (compact) way of writing the covariance for any possibe pair Y ij, Y ik, and represents the variance and covariance details that we presented on pp 60-1 and Heagerty, 2006

5 LMM and components of variation Within-Subject: Independence Model : R i = σ 2 I or general diagonal matrix Then, assuming normal errors we have that Y i = (Y i1, Y i2,, Y i,ni ) are conditionally independent given b i This model assumes that the within-subject errors do not have any serial correlation 142 Heagerty, 2006

6 More on Covariance Models Within-Subject: Serial Models Linear mixed models assume that each subject follows his/her own line In some situations the dependence is more local meaning that observations close in time are more similar than those far apart in time One model that we introduced is called the autoregressive model where: cov(e ij, e ik ) = σ 2 ρ j k 143 Heagerty, 2006

7 ID=1, rho=09 ID=2, rho=09 ID=3, rho=09 ID=4, rho=09 y y y y months months months months ID=5, rho=09 ID=6, rho=09 ID=7, rho=09 ID=8, rho=09 y y y y months months months months ID=9, rho=09 ID=10, rho=09 ID=11, rho=09 ID=12, rho=09 y y y y months months months months ID=13, rho=09 ID=14, rho=09 ID=15, rho=09 ID=16, rho=09 y y y y months months months months ID=17, rho=09 ID=18, rho=09 ID=19, rho=09 ID=20, rho=09 y y y y months months months months Heagerty, 2006

8 More on Covariance Models Autoregressive Correlation Assume t ij = j, n i 4: 1 ρ ρ 2 ρ 3 ρ 1 ρ ρ 2 corr(e i ) = ρ 2 ρ 1 ρ ρ 3 ρ 2 ρ Heagerty, 2006

9 More on Covariance Models Autoregressive Correlation Assume t ij = j: corr(e i ) = 1 ρ ρ 2 ρ (n 1) ρ 1 ρ ρ (n 2) ρ 2 ρ 1 ρ (n 3) ρ (n 1) ρ (n 2) ρ (n 3) Heagerty, 2006

10 More on Covariance Models Autoregressive Correlation corr(e i ) = Assume t ij unique: 1 ρ t i1 t i2 ρ t i1 t i3 ρ t i1 t in ρ t i2 t i1 1 ρ t i2 t i3 ρ t i2 t in ρ t i3 t i1 ρ t i3 t i2 1 ρ t i3 t in ρ t in t i1 ρ t in t i2 ρ t in t i Heagerty, 2006

11 More on Covariance Models Mixed + Serial Diggle (1988) proposed the following model Y ij = X ij β + b i,0 + W i (t ij ) + ɛ ij 147 Heagerty, 2006

12 Covariance Models Mixed + Serial The most general type of covariance model will combine some random effects with some additional aspects that characterize within-subject serial correlation One such model contains three sources of random variation: random intercept b i,0 serial process W i (t ij ) measurement error ɛ ij 148 Heagerty, 2006

13 We assume: var(b i,0 ) = ν 2 cov[w (s), W (t)] = σ 2 ρ s t var(ɛ ij ) = τ 2 Then: Total Variance = ν 2 + σ 2 + τ 2 Covariance(Y ij, Y ik ) = ν 2 + σ 2 ρ t ij t ik 149 Heagerty, 2006

14 Covariance Models Mixed + Serial of variation? Q: How to biologically interpret these three sources random intercept: This represents a trait of the subject FEV1 child size not captured by age and height CD4 subject s normal steady-state level serial variation: This represents a state for the subject FEV1 child current health status (infected with PseudoA) CD4 subject s current immune status (diet? treatment?) measurement error: This represents the instrumentation or process used to generate the final quantitative measurement FEV1 result of only one trial with expiration CD4 blood sample, lab processing 150 Heagerty, 2006

15 150-1 Heagerty, 2006

16 EDA for Covariance Structure Numerical Summaries Empirical covariance & correlation Variogram Define: R ij = Y ij X ij β = b i,0 + W (t ij ) + ɛ ij 151 Heagerty, 2006

17 Note: var(r ij ) = ν 2 + σ 2 + τ 2 E [ ] 1 2 (R ij R ik ) 2 = σ 2 (1 ρ t ij t ik ) + τ 2 R ij R ik = (b i,0 + W i (t ij ) + ɛ ij ) (b i,0 + W i (t ik ) + ɛ ik ) = [W i (t ij ) W i (t ik )] + [ɛ ij ɛ ik ] Plot: 1 2 ( R ij R ik ) 2 versus t ij t ik 152 Heagerty, 2006

18 152-1 Heagerty, 2006 Variogram E( dr^2 ) Total Variance deltatime

19 152-2 Heagerty, 2006 Variogram: Key Features When t ij = t ik : E [ ] 1 2 (R ij R ik ) 2 = σ 2 (1 ρ t ij t ik ) + τ 2 = σ 2 (1 ρ 0 ) + τ 2 = τ 2 = measurement error variance

20 152-3 Heagerty, 2006 Variogram: Key Features When t ij >> t ik : E [ ] 1 2 (R ij R ik ) 2 (large time separation) = σ 2 (1 ρ t ij t ik ) + τ 2 = σ 2 (1 ρ ) + τ 2 = σ 2 + τ 2 = serial and measurement error variances

21 dt dr FEV1 residual variogram Heagerty, 2006

22 Recall Simple Linear Regression (**) In simple linear regression we fit the model E(Y i X i ) = β 0 + β 1 X i We can write the estimate of the slope, β 1 as follows: β 1 = 1 i (X (X i X) 2 i X) (Y i Y ) This method is sometimes called ordinary least squares, or OLS i 153 Heagerty, 2006

23 Recall Simple Linear Regression (**) In some applications we still want to fit the regression model: E(Y i X i ) = β 0 + β 1 X i But now we want to assign weights, w i, to each observation Using the weights leads to weighted least squares (WLS) We can write the estimate of the slope, β 1 as follows: β 1 (w) = 1 i w (X i (X i X) 2 i X) w i (Y i Y ) With longitudinal data we have a method of estimation that generalizes this to allow covariance weights i 154 Heagerty, 2006

24 Estimation of β with known Σ i (**) Weighted least squares: In univariate regression, WLS yields estimates of β that minimize the objective function Q(β) = N w i (Y i X i β) 2 where W i is an (n i n i ) positive definite symmetric matrix i=1 Analogously, the multivariate version of WLS finds the value of the parameter β(w ) that minimizes Q W (β) = N (Y i X i β) T W i (Y i X i β) i=1 155 Heagerty, 2006

25 Estimation of β with known Σ i (**) It s straight forward to see that U(β) = β Q W (β) = 2 N X T i W i (Y i X i β) i=1 This is a general way to statistically define the regression estimator a solution to equations In general W i is chosen as the inverse of Σ i 156 Heagerty, 2006

26 The solution to the minimization solves U(β) = 0 and yields ( N ) 1 ( N ) β(w ) = X T i W i X i X T i W i Y i i=1 i=1 157 Heagerty, 2006

27 Properties of β(w ) (**) Given X 1, X 2, X N and W 1, W 2, W N ( ] N ) 1 ( N ) E [ β(w ) = X T i W i X i X T i W i E[Y i ] i=1 i=1 ( N ) 1 ( N ) = X T i W i X i X T i W i X i β = β i=1 i=1 Notice that the estimate β(w ) is unbiased no matter what weighting scheme is used 158 Heagerty, 2006

28 Properties of β(w ) (**) 1 When W i is correctly specified as the inverse of the variance of Y i then: W i = Σ 1 i ] var [ β(σ 1 ) = ( i ) 1 X T i Σ 1 i X i When we use gllamm, SAS PROC MIXED, or S+ lme this is what is returned to provide standard errors for the estimated regression coefficients 159 Heagerty, 2006

29 Properties of β(w ) (**) 2 When W i is not the inverse of the variance of Y i then: W i Σ 1 i ] var [ β(w ) = bread ( ) {}}{ A 1 X T i W i Σ i W i X i i }{{} cheese bread {}}{ A 1 A = i X T i W i X i More on this sandwich later 160 Heagerty, 2006

30 Likelihood Estimation for Linear Mixed Models (**) Parameters: β : regression parameter, fixed effects coefficient α : variance components α D(α) and R(α) where cov(y i ) = Z i DZ T i + R i 161 Heagerty, 2006

31 Normality: E(Y i ) = X i β cov(y i ) = Σ(α) f(y i ; β, α) = Σ 1/2 (2π) ni/2 [ exp 1 ] 2 (Y i X i β) T Σ 1 (Y i X i β) Maximum Likelihood: Find the values for the regression coefficients, β, and the variance components that maximizes the likelihood eg put the highest available probability on the observed data 162 Heagerty, 2006

32 RA Fisher Heagerty, 2006

33 ML versus REML There is a variant of ML estimation known as REML Residual ML Restricted ML REML is used to provide slightly less biased estimates of variance components However, be careful using REML when you change the covariates in your model since one can not use changes in REML log likelihoods to test for fixed effects Useful for a single fitted model, or to compare covariance models with a fixed regression model 163 Heagerty, 2006

34 Inference in the Linear Mixed Model In practice: (1) Saturated mean model & explore the covariance (2) Fix the covariance & explore the mean 164 Heagerty, 2006

35 Likelihood Ratio Tests Fixed Effects Standard likelihood theory can be applied to test H 0 : β 2 = 0 where E[Y ] = [X 1, X 2 ] β 1 β 2 = X 1 β 1 + X 2 β 2 [1]Full Model: E[Y ] = X 1 β 1 + X 2 β 2 [0]Reduced Model: E[Y ] = X 1 β Heagerty, 2006

36 Likelihood Ratio Tests Fixed Effects In this case we have (when null hypothesis is true): Likelihood Ratio = L ML( β 1, β 2, α; ML using model 1) L ML ( β 1, 0, α; ML using model 0) LRstatistic = 2 log Likelihood Ratio = 2 log L ML,1 2 log L ML,0 χ 2 (q) Where q is the number of coefficients that are set to zero in the reduced model 166 Heagerty, 2006

37 Other Tests Fixed Effects (*) We also have for a general linear contrast A and a hypothesis H 0 : Aβ = 0 Wald Test: ( ) (A β) T Avar( β)a T 1 (A β) χ 2 (q) F Test: F = (A β) T (A var( β)a T ) 1 (A β) rank(a) F (ndf = rank(a), ddf) 167 Heagerty, 2006

38 LMM: Selection of the Covariance Matrix A Model that fits the data Compare the fitted covariance to the empirical assessment of it: Σ i = Z i DZ T i + R i ( α) versus cov(y i µ i ) γ( ) = τ 2 + σ 2 [1 ρ( )] versus empirical variogram var(y ij ) = τ 2 + σ 2 + ν 2 versus empirical variance 168 Heagerty, 2006

39 Look at the maximized likelihood: Compare 2 log L AIC, BIC Don t lose sight of the goals of analysis If covariance selection is to obtain valid model based standard errors then we can assess the impact on β and se s We can also calculate an empirical (sandwich) variance estimate 169 Heagerty, 2006

40 Inference in the Linear Mixed Model (*) Likelihood Ratio Tests Variance Components We may want to test whether we have random intercepts and slopes, or just random intercepts H 0 : D = D versus H 1 : D = D 11 D 12 D 21 D 22 Q: What is the distribution of the likelihood ratio statistic LRstat = 2 log L ML( θ ML,model 1 ) L ML ( θ ML,model 0 ) 170 Heagerty, 2006

41 LR Testing for Variance Components (*) D 22 = 0 is on the boundary of the parameter space!!! This violates the standard assumption that we use to justify the χ 2 (p 1 p 0 ) distribution of the LR statistic We appeal to results in Stram and Lee (1994) that build upon results in Self & Liang (1987) showing that LR stat is a mixture of χ 2 Note: For a fixed mean strucure we can use the LR based on either ML or REML (Why?) See: Verbeke and Molenberghs (1997) pages Heagerty, 2006

42 171-1 Heagerty, 2006

43 171-2 Heagerty, 2006

44 171-3 Heagerty, 2006 S+ LMM Program: # # cfkids-cda-newlmmq # # # # PURPOSE: Use linear mixed models to characterize longitudinal # change by gender and genotype # # AUTHOR: P Heagerty # # DATE: 00/07/10 Revised 14Feb2002 # # # # ##### ##### Read data ##### # source("cfkids-readq") # # ##### ##### Trellis plots of individuals and groups #####

45 # # Create Grouped Data Set # ntotal <- cumsum( unlist( lapply( split( cfkids$id, cfkids$id), length ) ) ) cfsubset <- groupeddata( fev1 ~ age id, outer = ~ factor(f508)*female, data = cfkids[ 1:ntotal[(8*4*1)], ] ) # cfkids <- groupeddata( fev1 ~ age id, outer = ~ factor(f508)*female, data = cfkids ) # # trellis plot, by id, first 1 pages, 8x4 # postscript( file="cfkids-trellis1ps", horiz=f ) plot( cfsubset, layout = c(4,8) ) graphicsoff() postscript( file="cfkids-trellis2ps", horiz=t ) par( pch="" ) plot( cfkids, outer = ~ factor(f508)*factor(female), layout=c(3,2), aspect=1 ) graphicsoff() # ##### ##### Linear Mixed Models ##### # options( contrasts=c("contrtreatment","contrhelmert") ) Heagerty, 2006

46 171-5 Heagerty, 2006 # ### Intercept only fit0 <- lme( fev1 ~ age0 + agel + female*agel + factor(f508)*agel, method = "ML", random = restruct( ~ 1 id, pdclass="pdsymm", REML=F), data = cfkids ) summary( fit0 ) ### Intercept plus Slope fit1 <- lme( fev1 ~ age0 + agel + female*agel + factor(f508)*agel, method = "ML", random = restruct( ~ 1 + agel id, pdclass="pdsymm", REML=F), data = cfkids ) summary( fit1 ) ### EDA for serial correlation postscript( file="cfkids-variogramps", horiz=t ) plot( Variogram( fit0, form = ~ age id, restype="response" ) ) graphicsoff() ### Intercept plus AR(1) fit2a <- lme( fev1 ~ age0 + agel + female*agel + factor(f508)*agel,

47 171-6 Heagerty, 2006 method = "ML", random = restruct( ~ 1 id, pdclass="pdsymm", REML=F), correlation = corar1( form = ~ 1 id ), data = cfkids ) summary( fit2a ) ### another way fit2b <- lme( fev1 ~ age0 + agel + female*agel + factor(f508)*agel, method = "ML", random = restruct( ~ 1 id, pdclass="pdsymm", REML=F), correlation = corexp( form = ~ agel id, nugget=f), data = cfkids ) summary( fit2b ) ### another way fit2c <- lme( fev1 ~ age0 + agel + female*agel + factor(f508)*agel, method = "ML", random = restruct( ~ 1 id, pdclass="pdsymm", REML=F), correlation = corcar1( form = ~ agel id ), data = cfkids ) summary( fit2c ) fit2 <- fit2b ### Intercept plus AR(1) plus measurement error

48 171-7 Heagerty, 2006 fit3 <- lme( fev1 ~ age0 + agel + female*agel + factor(f508)*agel, method = "ML", random = restruct( ~ 1 id, pdclass="pdsymm", REML=F), correlation = corexp( form = ~ agel id, nugget=t), data = cfkids ) summary( fit3 ) # ##### compare these models # anova( fit0, fit1, fit2, fit3 ) # ##### ##### Residual Analysis -- using fit3 ##### # popres <- resid( fit3, level=0 ) clusterres <- resid( fit3, level=1 ) print( var( popres ) ) print( var( clusterres ) ) # postscript( file="cfkids-newresidualsps", horiz=f ) par( mfrow=c(2,1) ) plot( cfkids$age0, popres, pch="" ) lines( smoothspline( cfkids$age0, popres, df=5 ) ) title("residuals (pop) vs Age0") abline( h=0, lty=2 )

49 171-8 Heagerty, 2006 plot( cfkids$agel, popres, pch="" ) lines( smoothspline( cfkids$agel, popres, df=5 ) ) title("residuals (pop) vs AgeL") abline( h=0, lty=2 ) graphicsoff() # postscript( file="cfkids-newresiduals2ps", horiz=f ) par( mfrow=c(2,1) ) plot( cfkids$age0, clusterres, pch="" ) lines( smoothspline( cfkids$age0, clusterres, df=5 ) ) abline( h=0, lty=2 ) title("residuals (cluster) vs Age0") b0 <- unlist( fit2$coefficients$random ) age0 <- unlist( lapply( split( cfkids$age0, cfkids$id ), min ) ) plot( age0, b0 ) lines( smoothspline( age0, b0, df=5 ) ) abline( h=0, lty=2 ) title("eb b0 versus Age0") graphicsoff() # ##### Do we need a quadratic age0??? # fit4 <- lme( fev1 ~ age0 + age0^2 + agel + female*agel + factor(f508)*agel, method = "ML", random = restruct( ~ 1 id, pdclass="pdsymm", REML=F), correlation = corexp( form = ~ agel id, nugget=t), data = cfkids )

50 summary( fit4 ) # anova( fit3, fit4 ) # # end-of-file Heagerty, 2006

51 Heagerty, fev age

52 fev age Heagerty, 2006

53 Heagerty, 2006 Fit 0 Random Intercepts Linear mixed-effects model fit by maximum likelihood Data: cfkids AIC BIC loglik Random effects: Formula: ~ 1 id (Intercept) Residual StdDev: Fixed effects: fev1 ~ age0 + agel + female * agel + factor(f508) * agel Value StdError DF t-value p-value (Intercept) <0001 age <0001 agel female factor(f508) factor(f508) female:agel agelfactor(f508) agelfactor(f508) Number of Observations: 1513 Number of Groups: 200

54 Heagerty, Semivariogram Distance

55 Heagerty, 2006 Fit 1 Random Intercepts and Slopes Linear mixed-effects model fit by maximum likelihood Data: cfkids AIC BIC loglik Random effects: StdDev Corr Formula: ~ 1 + agel id (Intercept) (Inter Structure: General positive-definite agel Residual Fixed effects: fev1 ~ age0 + agel + female * agel + factor(f508) * agel Value StdError DF t-value p-value (Intercept) <0001 age <0001 agel female factor(f508) factor(f508) female:agel agelfactor(f508) agelfactor(f508) Number of Observations: 1513 Number of Groups: 200

56 Heagerty, 2006 Fit 2a Random Intercepts + AR(1) errors Linear mixed-effects model fit by maximum likelihood Data: cfkids AIC BIC loglik Random effects: Correlation Structure: AR(1) Formula: ~ 1 id Formula: ~ 1 id (Intercept) Residual Parameter estimate(s): StdDev: Phi Fixed effects: fev1 ~ age0 + agel + female * agel + factor(f508) * agel Value StdError DF t-value p-value (Intercept) <0001 age <0001 agel female factor(f508) factor(f508) female:agel agelfactor(f508) agelfactor(f508) Number of Observations: 1513 Number of Groups: 200

57 Heagerty, 2006 Fit 2b Random Intercepts + corexp errors Linear mixed-effects model fit by maximum likelihood Data: cfkids AIC BIC loglik Random effects: Correlation Structure: Exponential spatial corr Formula: ~ 1 id Formula: ~ agel id (Intercept) Residual Parameter estimate(s): StdDev: range Fixed effects: fev1 ~ age0 + agel + female * agel + factor(f508) * agel Value StdError DF t-value p-value (Intercept) <0001 age <0001 agel female factor(f508) factor(f508) female:agel agelfactor(f508) agelfactor(f508) Number of Observations: 1513 Number of Groups: 200

58 Heagerty, 2006 Fit 2c Random Intercepts + CAR(1) errors Linear mixed-effects model fit by maximum likelihood Data: cfkids AIC BIC loglik Random effects: Correlation Structure: Continuous AR(1) Formula: ~ 1 id Formula: ~ agel id (Intercept) Residual Parameter estimate(s): StdDev: Phi Fixed effects: fev1 ~ age0 + agel + female * agel + factor(f508) * agel Value StdError DF t-value p-value (Intercept) <0001 age <0001 agel female factor(f508) factor(f508) female:agel agelfactor(f508) agelfactor(f508) Number of Observations: 1513 Number of Groups: 200

59 Heagerty, 2006 Fit 3 Random Intercepts + corexp + meas error Linear mixed-effects model fit by maximum likelihood Data: cfkids AIC BIC loglik Random effects: Correlation Structure: Exponential spatial corr Formula: ~ 1 id Formula: ~ agel id (Intercept) Residual Parameter estimate(s): StdDev: range nugget Fixed effects: fev1 ~ age0 + agel + female * agel + factor(f508) * agel Value StdError DF t-value p-value (Intercept) <0001 age <0001 agel female factor(f508) factor(f508) female:agel agelfactor(f508) agelfactor(f508) Number of Observations: 1513 Number of Groups: 200

60 Heagerty, 2006 ANOVA Model df AIC BIC loglik Test LRatio p-value fit fit vs <0001 fit vs e-04 fit vs <0001

61 LMM Summary Observe Y i, i = 1, 2,, m independent clusters Model: (Laird & Ware, 1982) Y i = X i β + Z i b i + e i β is the coefficient that is common to all clusters (fixed across clusters) b i is the deviation of the coefficient that varies from cluster to cluster (random across clusters) (β j + b j,i ) is the coefficient of X i,j for cluster i 172 Heagerty, 2006

62 b i N (0, D) between-cluster e i N (0, R i ) within-cluster Estimation/Inference: WLS, ML Covariance model choice leads to WLS but estimated regression coefficient is unbiased for any choice of weight (covariance) Covariance model choice determines the standard error estimates for the regression coefficients correct covariance model is needed for correct standard errors 173 Heagerty, 2006