Impact of serial correlation structures on random effect misspecification with the linear mixed model.

Transcription

1 Impact of serial correlation structures on random effect misspecification with the linear mixed model. Brandon LeBeau University of Iowa file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 1/24

2 Introduction The linear mixed model (LMM) is a common model used with nested data, including both cross sectional and longitudinal data structures. Y j = X j β + Z j b j + e j Common model assumptions include: e ij iid N(0, σ 2 ) b j iid N(0,G) and V ar( Y j ) = Σ j = Z j G Z T j + σ 2 e I n1j However, with longitudinal data and measurement occasions close in time (i.e. hourly, daily, weekly, etc.) the random effects alone may not be enough to account for the dependency due to repeated measurements. In these situations, it may be needed to model the within cluster covariance structure to adequately account for the dependency due to repeated measurements. V ar( Y j ) = Σ j = Z j G Z T j + σ 2 e I n1j + τ 2 H j file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 2/24

3 Motivation When a model doesn't converge what is commonly done? 100% 90% 80% 70% Fitted SC AR(1) ARMA(1,1) Ind MA(1) MA(2) Convergence Rate 60% 50% 40% 30% 20% 10% 0% Ind AR(1) MA(1) MA(2) ARMA(1,1) Generated Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 3/24

4 Research Questions 1. To what extent can a more complicated covariance structure overcome a missing random effect for time? 2. Does the relationship in question 1 generalize to real world data conditions such as non-normal random effect distributions. file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 4/24

5 Methodology A monte carlo simulation was employed to answer the research questions. Using a factorial research design, the following four conditions were manipulated: Covariance structure (Ind, AR(1), MA(1), ARMA(1,1)) Random effect distribution (Normal, Laplace, Chi(1)) Number of measurement occasions (6, 12) Number of clusters (25, 50, 100) The random effect for time ( b1j ) was deliberately not modeled to explore if serial correlation structures can overcome this random effect misspecification. For each data condition, 500 replications were done. All of the covariance structures were fitted to every dataset. All simulation, model fitting, and analyses were done with the statistical programming language R. file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 5/24

6 Sample Population parameters were generated from data collected by the Minnesota Mathematics Achievement Project (MNMAP). The MNMAP project explored the relationship between high school mathematics curriculum and subsequent college mathematics performance. The database for the MNMAP project consisted of about 20,000 students from over 300 high schools, and 35 post-secondary schools. For the sake of the simulation, the dependent variable was college mathematics GPA, the independent variables were time (semester), difficulty of the college mathematics course, the student's ACT mathematics score, and the interaction between time and ACT. The following model was fitted to the data: GP A ij = β0 + β1tim e ij + β2dif f ij + β3ac T j + β4time ij AC T j + b0 file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 6/24

7 Population Parameters The population parameters estimated from the MNMAP data were: Parameter Value β0 β1 β2 β3 β Var b0j Var b1j Var e ij AR(1) MA(1) MA(2) file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 7/24

8 Analysis Model convergence, relative bias, and type I error rates were analyzed descriptively. The relative bias and type I error rates were also analyzed inferentially. Relative bias was calculated with the following formula: RB = ( θ^ θ) θ Analysis of variance (ANOVA) was used to explore variation in the relative bias. Instead of p-values, effect sizes were calculated. The effect size used took the form of: η 2 = SS trt SS total To compute the empirical type I error rates, a Wald statistic was set up: Z = β SE Logistic regression was used to explore the likelihood of making a type I error. β^ file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 8/24

9 Convergence The convergence in general was very high. The few exceptions occurred when an ARMA(1,1) structure was fitted to less complicated structures such as an independent structure. The higher convergence rate is not surprising as a random effect close to zero was dropped from the model. The random slope term was likely the culprit for the poor convergence rates from study 1. n p RE Dist Gen SC Fit SC Convergence % 25 6 norm Ind ARMA(1,1) norm Ind ARMA(1,1) norm Ind ARMA(1,1) lap Ind ARMA(1,1) lap Ind ARMA(1,1) lap Ind ARMA(1,1) chi Ind ARMA(1,1) chi Ind ARMA(1,1) chi Ind ARMA(1,1) chi Ind ARMA(1,1) file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 9/24

10 Relative Bias Fixed Effects Below are summary statistics for the relative bias of the fixed effects: Term Mean RB Var RB Med RB Min RB Max RB Intercept Time Level Level Time:Level None of the simulation conditions explained variation in the relative bias of the fixed effects. file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 10/24

11 Relative Bias Random Components The relative bias statistics were on average above zero. Term Mean RB Var RB Med RB Min RB Max RB Var Int Var Res file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 11/24

12 Relative Bias Random Components Inference Simulation conditions with effect sizes greater than.001 are shown in the table below: Variable EtaSq p Gen SC Fit SC Term Gen SC:Term Fit SC:Term p:gen SC:Term p:fit SC:Term Gen SC:Fit SC:Term file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 12/24

13 Relative Bias Random Intercept Fitted SC AR(1) ARMA(1,1) Ind MA(1) 0.5 Relative Bias Ind AR(1) MA(1) ARMA(1,1) Generated Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 13/24

14 Relative Bias Within Cluster Residuals Fitted SC AR(1) ARMA(1,1) Ind MA(1) 2 Relative Bias Ind AR(1) MA(1) ARMA(1,1) Generated Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 14/24

15 Empirical Type I Error Rates Empirical Type I Error Rate Intercept Time Level 1 Level 2 Time:Level 2 Parameter t1e outlier file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 15/24

16 Empirical Type I Error Rates Inferential Significant predictors from the logistic regression included: Term Number of measurement occasions Generated Serial Correlation Structure Fitted Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 16/24

17 Empirical Type I Error Rate Int (p = 6) 0.2 Fitted SC Average Type I Error Rate AR(1) ARMA(1,1) Ind MA(1) 0 Ind AR(1) MA(1) ARMA(1,1) Generated Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 17/24

18 Empirical Type I Error Rate Int (p = 12) 0.2 Fitted SC Average Type I Error Rate AR(1) ARMA(1,1) Ind MA(1) Ind AR(1) MA(1) ARMA(1,1) Generated Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 18/24

19 Empirical Type I Error Rate Time (p = 6) Fitted SC AR(1) ARMA(1,1) Ind MA(1) Average Type I Error Rate Ind AR(1) MA(1) ARMA(1,1) Generated Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 19/24

20 Empirical Type I Error Rate Time (p = 12) 0.4 Fitted SC Average Type I Error Rate AR(1) ARMA(1,1) Ind MA(1) Ind AR(1) MA(1) ARMA(1,1) Generated Serial Correlation Structure file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 20/24

21 Conclusions The fixed effects showed very little evidence of bias. In addition, no simulation condition explained variation in the relative bias of the fixed effects. This supports previous research. The random components showed signs of bias. More specifically the random components tended to be overestimated compensating for the missing random slope. The generated and fitted serial correlation structures explained variation in the relative bias of the random components. Including some measure of serial correlation when present helped to limit bias in the random effects, but tended to inflate the bias in the residuals. Unfortunately, bias was still present when the generated serial correlation structure matched the fitted serial correlation structure. This also matches prior research file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 21/24

22 Conclusions (2) There was significant inflation of the type I error rate when a random effect was misspecified. Specifying a serial correlation structure did help to limit the amount of inflation for the slope terms for time, but did not completely overcome the missing random effect. Deviations from the nominal rate was also affected by the number of measurement occasions. file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 22/24

23 Recommendations Recommendations come in three levels. First, if researchers are only interested in estimates of the fixed effects, little to no attention needs to be made with regard to the serial correlation structure or random effect distribution. Secondly, if researchers are interested in estimates of the random effects more care needs to be taken in identifying if serial correlation is present. If the serial correlation is unmodeled, the bias in the random effects is larger. Lastly, if researchers are interested in conducting inference for the fixed effects, then care needs to be taken, especially with more measurement occasions. Including a serial correlation structure in this case can help limit the severe inflation. file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 23/24

24 Future Work Guidelines on how to identify situations where serial correlation is needed and ways to select an appropriate serial correlation structure. Currently there are no guidelines and model selection procedures such as the AIC, BIC, or the likelihood ratio test have not shown promising results. Removing the assumption that random effects be uncorrelated may allow the model to account for a third level of nesting in a two level model. This would be most useful when there are very few level three units sampled, but there is significant variation expected in that third level (i.e. the intraclass correlation is large). file:///c:/users/bleb/onedrive%20 %20University%20of%20Iowa%201/JournalArticlesInProgress/Diss/Study2/Pres/pres.html#(2) 24/24