Serial Correlation Edps/Psych/Stat 587 Carolyn J. Anderson Department of Educational Psychology c Board of Trustees, University of Illinois Fall 017
Model for Level 1 Residuals There are three sources of possible variance Between individuals, modeled by Z i U i where U i N(0,T). Within individuals, R it = e (1)it +e ()it Random (measurement error, variables not included, etc.), e (1)it N(0,σ ). Autocorrelated errors, e()it...need model for this. C.J. Anderson (Illinois) Serial Correlation Fall 017.1/ 97
Autocorrelated Errors: Mini-Outline Possible models for R it Back to Riesby data. Models for U i and R it. Time varying explanatory variables: Riesby see text by Hedeker & Gibbons (006). Detecting serial correlation. C.J. Anderson (Illinois) Serial Correlation Fall 017 3.1/ 97
Possible models for R it In an HLM/linear mixed model, Y i (r i 1) = X i Γ + (r i p) (p 1) Z i U i + R i (r i q) (q 1) (r i 1) where i = 1,...,N and r i = number of time points for individual i. U i N(0,T). R i N(0,σ Ω i ) var(y i ) = V i = Z i TZ i +σ Ω i. C.J. Anderson (Illinois) Serial Correlation Fall 017 4.1/ 97
Possible Models for Serial Correlation Autoregressive (AR). Moving average (MA). Autoregressive with a moving average (ARMA). Toeplitz. Others. C.J. Anderson (Illinois) Serial Correlation Fall 017 5.1/ 97
Autoregressive Errors First order autoregressive process, AR1: Time 1: Time : R i1 = ǫ i1 R i = ρr i1 +ǫ i = ρǫ i1 +ǫ i Time 3: R i3 = ρr i +ǫ i3 = ρ(ρǫ i1 +ǫ i )+ǫ i3 Time t:.. R it = ρr i(t 1) +ǫ it C.J. Anderson (Illinois) Serial Correlation Fall 017 6.1/ 97
Autoregressive Errors First order autoregressive process, AR1: where ǫ it N(0,σ ǫ) i.i.d. R it = ρr i(t 1) +ǫ it ρ is autocorrelation coefficient, 0 ρ < 1. Stationarity: variance of R it and covariance between R it and R it are independent of t. Resulting error variance structure... C.J. Anderson (Illinois) Serial Correlation Fall 017 7.1/ 97
Autoregressive Errors σ Ω = σ ǫ (1 ρ ) 1 ρ ρ... ρ ri 1 ρ 1 ρ... ρ ri ρ ρ 1... ρ ri 3............ ρ ri 1 ρ ri ρ ri 3... 1 In SAS, AR1 is parameterized as 1 ρ ρ... ρ ri 1 ρ 1 ρ... ρ ri σ Ω = σǫ ρ ρ 1... ρ ri 3,......... ρ ri 1 ρ ri ρ ri 3... 1 where σ ǫ = σ ǫ/(1 ρ ).. C.J. Anderson (Illinois) Serial Correlation Fall 017 8.1/ 97
Notes Regarding AR1 AR1 process is a regression equation in which R it depends on it s past values. Since R it only depends on it s past values, this is a Markov process. Ω is defined by ρ, the autocorrelation coefficient. Non-stationarity: If the variance of R it and the covariance between R it and R it increases over time, then you have non-stationarity. Not available in SAS. Available in Hedekker s MIXREG program. Not sure what s in R. C.J. Anderson (Illinois) Serial Correlation Fall 017 9.1/ 97
Simulated Data: No Serial Correlation Random Intercept and Random Slope for Time: where Y it = 10+(time) it +U 0j +U 1j (time) it +R it (time)= t = 1...,0, and N = 50 individuals. U i N(0,T) with R it N(0,4). T = ( 4 4 ) C.J. Anderson (Illinois) Serial Correlation Fall 017 10.1/ 97
No Serial Correlation: Data R it N(0,σ ) iid. C.J. Anderson (Illinois) Serial Correlation Fall 017 11.1/ 97
No Serial Correlation: R it R it N(0,σ ) iid. C.J. Anderson (Illinois) Serial Correlation Fall 017 1.1/ 97
No Serial Correlation: R it R it N(0,σ ) iid. C.J. Anderson (Illinois) Serial Correlation Fall 017 13.1/ 97
Simulated AR1 Data Random intercept and random slope: where Y it = 10+(time) it +U 0j +U 1j (time) it +.75R i(t 1) +ǫ it (time)= t = 1...,0, and N = 50 individuals. U i N(0,T) with ǫ it N(0,4). R it =.75R i(t 1) +ǫ it. T = ( 4 4 ) C.J. Anderson (Illinois) Serial Correlation Fall 017 14.1/ 97
Eg of AR1: The Data C.J. Anderson (Illinois) Serial Correlation Fall 017 15.1/ 97
Eg of AR1: The R it C.J. Anderson (Illinois) Serial Correlation Fall 017 16.1/ 97
Eg of AR1: Some R it C.J. Anderson (Illinois) Serial Correlation Fall 017 17.1/ 97
Eg of AR1: OLS ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 18.1/ 97
Eg of AR1: OLS ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 19.1/ 97
Eg of AR1: OLS mean ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 0.1/ 97
Example of AR1 (continued) C.J. Anderson (Illinois) Serial Correlation Fall 017 1.1/ 97
Example of AR1 (continued) C.J. Anderson (Illinois) Serial Correlation Fall 017.1/ 97
Example of AR1 (continued) C.J. Anderson (Illinois) Serial Correlation Fall 017 3.1/ 97
SAS/MIXED and AR1 (continued) PROC MIXED data=new1 method=ml; CLASS i occasion; MODEL y= time / solution ; RANDOM intercept time /type=un subject=i solution G; REPEATED occasion / subject=i type=ar(1) R; REPEATED works much the same way the RANDOM does. Need two time variables (one continuous/numerical & one classification). C.J. Anderson (Illinois) Serial Correlation Fall 017 4.1/ 97
SAS/MIXED and AR1 (continued) Covariance Parameter Estimates Cov Parm Subject Actual Estimate τ 00 UN(1,1) i 4.00 0.94 τ 10 UN(,1) i.00 1.01 τ 11 UN(,) i 4.00 3.04 ρ AR(1) i 0.75 0.7 σǫ Residual 9.14 8.15 Note: ˆσ = ˆσ (1 ˆρ ) = 8.15(1.75 ) = 3.93. The value used to simulate data was 4. C.J. Anderson (Illinois) Serial Correlation Fall 017 5.1/ 97
SAS/MIXED and AR1 (continued) Estimated ˆσ ˆΩ r Row Col1 Col Col3 Col4 Col5 Col6 Col7 Co 1 8.1509 5.8668 8.1509 3 4.8 5.8668 8.1509 4 3.0395 4.8 5.8668 8.1509 5.1878 3.0395 4.8 5.8668 8.1509 6 1.5747.1878 3.0395 4.8 5.8668 8.1509 7 1.1334 1.5747.1878 3.0395 4.8 5.8668 8.1509 r 8 0.8158 1.1334 1.5747.1878 3.0395 4.8 5.8668 8.15 9 0.587 0.8158 1.1334 1.5747.1878 3.0395 4.8 5.86 10 0.47 0.587 0.8158 1.1334 1.5747.1878 3.0395 4. 11 0.304 0.47 0.587 0.8158 1.1334 1.5747.1878 3.03 1 0.190 0.304 0.47 0.587 0.8158 1.1334 1.5747.18 13 0.1576 0.190 0.304 0.47 0.587 0.8158 1.1334 1.57 14 0.1134 0.1576 0.190 0.304 0.47 0.587 0.8158 1.13 15 0.08166 0.1134 0.1576 0.190 0.304 0.47 0.587 0.81 16 0.05877 0.0817 0.1134 0.1576 0.190 0.304 0.47 0.58 C.J. Anderson (Illinois) Serial Correlation Fall 017 6.1/ 97
First Order Moving Average Time 1: Time : Time 3: Time t: R i1 = ǫ i1 R i = ǫ i θǫ i1 R i3 = ǫ i3 θǫ i.. R it = ǫ it θǫ i,(t 1) where ǫ it N(0,σ ǫ) i.i.d. θ is the autocorrelation coefficient. C.J. Anderson (Illinois) Serial Correlation Fall 017 7.1/ 97
First Order Moving Average The covariance matrix for R it is 1+θ θ 0... 0 θ 1+θ θ... 0 σǫω = σǫ 0 θ 1+θ... 0......... 0 0 0... 1+θ. C.J. Anderson (Illinois) Serial Correlation Fall 017 8.1/ 97
MA1 and SAS/MIXED SAS/MIXED doesn t estimate 1+θ θ 0... 0 θ 1+θ θ... 0 σǫω = σǫ 0 θ 1+θ... 0......... 0 0 0... 1+θ The closest you can come to this is in SAS is TYPE=TOEP(), σ σ 1 0... 0 σ 1 σ σ 1... 0 cov(r i ) = 0 σ 1 σ... 0......... 0 0 0... σ. C.J. Anderson (Illinois) Serial Correlation Fall 017 9.1/ 97
Simulated MA1 Data where Y it = 10+(time) it +U 0j +U 1j (time) it +ǫ it.75ǫ i,(t 1) (time)= t = 1...,0, and N = 50 individuals. U i N(0,T) with ǫ it N(0,4). R it = ǫ it.75ǫ i,(t 1). T = ( 4 4 ) C.J. Anderson (Illinois) Serial Correlation Fall 017 30.1/ 97
Eg of MA1 Data C.J. Anderson (Illinois) Serial Correlation Fall 017 31.1/ 97
Eg of MA1 R it C.J. Anderson (Illinois) Serial Correlation Fall 017 3.1/ 97
Eg of MA1 R it C.J. Anderson (Illinois) Serial Correlation Fall 017 33.1/ 97
Eg of MA1 OLS ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 34.1/ 97
Eg of MA1 OLS ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 35.1/ 97
Eg of MA1 Mean Sq Residuals C.J. Anderson (Illinois) Serial Correlation Fall 017 36.1/ 97
Example of MA1 (continued) C.J. Anderson (Illinois) Serial Correlation Fall 017 37.1/ 97
Example of MA1 (continued) C.J. Anderson (Illinois) Serial Correlation Fall 017 38.1/ 97
SAS/MIXED and TOEP() (continued) Covariance Parameter Estimates Cov Parm Subject Estimate UN(1,1) i 4.0533 UN(,1) i -1.311 UN(,) i 3.064 TOEP() i -.995 Residual 6.66 C.J. Anderson (Illinois) Serial Correlation Fall 017 39.1/ 97
SAS/MIXED and TOEP(): ˆσ ˆΩ Row Col1 Col Col3 Col4 Col5 Col6 Col7 1 6.66 -.995 -.995 6.66 -.995 3 -.995 6.66 -.995 4 -.995 6.66 -.995 5 -.995 6.66 -.995 6 -.995 6.66 -.995 7 -.995 6.66-8 -.995 6 9-10 11 1 C.J. Anderson (Illinois) Serial Correlation Fall 017 40.1/ 97
Autoregressive-Moving Average ARMA(1,1) where ǫ it N(0,σ ǫ) iid. R it = ρr i,t 1 +ǫ it θǫ i,(t 1) ρ is autocorrelation coefficient, ρ < 1. Stationarity: variance of R it and covariance between R it and R it are independent of t. θ is the autocorrelation coefficient. C.J. Anderson (Illinois) Serial Correlation Fall 017 41.1/ 97
ARMA(1,1) Error Covariance Matrix cov(r i ) = σ ǫ (1 ρ ) ξ 0 ξ 1 ρξ 1... ρ r ξ 1 ξ 1 ξ 0 ξ 1... ρ r 3 ξ ρξ 1 ξ 1 ξ 0... ρ r 4 ξ 1 ρ ξ 1 ρξ 1 ξ 1... ρ (r 5) ξ 1......... ρ (r ) ξ 1 ρ (r 3) ξ 1 ρ (r 4)ξ 1... ξ 0 where ξ 0 = 1+θ ρθ and ξ 1 = (1 ρθ)(ρ θ). The MA1 term, θ, changes the lag-1 autocorrelation and then autocorrelations deceases as in AR1. C.J. Anderson (Illinois) Serial Correlation Fall 017 4.1/ 97
ARMA(1,1) SAS Parameterization cov(r i ) = σ 1 ξ ρξ... ρ r ξ ξ 1 ξ... ρ r 3 ξ ρξ ξ 1... ρ r 4 ξ ρ ξ ρξ ξ... ρ (r 5) ξ......... ρ (r ) ξ ρ (r 3) ξ ρ (r 4)ξ... 1 where σ = (σ ǫ/(1 ρ ))ξ 0 = (σ ǫ/(1 ρ ))(1+θ ρθ) ξ = (σ ǫ/(1 ρ ))(1 ρθ)(ρ θ). C.J. Anderson (Illinois) Serial Correlation Fall 017 43.1/ 97
Simulated ARMA(1,1) Data Y it = 10+(time) it +U 0j +U 1j (time) it +R it where R it = ǫ it +.75R i,t 1.5ǫ it (time)= t = 1...,0, and N = 500 individuals. U i N(0,T) with T = ( 4 4 ) ǫ it N(0,4). C.J. Anderson (Illinois) Serial Correlation Fall 017 44.1/ 97
Eg of ARMA(1,1): The data C.J. Anderson (Illinois) Serial Correlation Fall 017 45.1/ 97
Eg of ARMA(1,1): The R it C.J. Anderson (Illinois) Serial Correlation Fall 017 46.1/ 97
Eg of ARMA(1,1): The R it C.J. Anderson (Illinois) Serial Correlation Fall 017 47.1/ 97
Eg of ARMA(1,1): The OLS ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 48.1/ 97
Eg of ARMA(1,1): The OLS ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 49.1/ 97
Eg of ARMA(1,1): The OLS ˆR it C.J. Anderson (Illinois) Serial Correlation Fall 017 50.1/ 97
Example of ARMA(1,1) (continued) C.J. Anderson (Illinois) Serial Correlation Fall 017 51.1/ 97
Example of ARMA(1,1) (continued) C.J. Anderson (Illinois) Serial Correlation Fall 017 5.1/ 97
SAS/MIXED and ARMA(1,1) Covariance Parameter Estimates Cov Parm Subject Estimate τ 00 UN(1,1) i.5978 τ 10 UN(,1) i -1.1575 τ 11 UN(,) i 3.051 ρ Rho i.690 ξ Gamma i.5358 Residual 5.5804 C.J. Anderson (Illinois) Serial Correlation Fall 017 53.1/ 97
SAS/MIXED and ARMA(1,1) (continued) Covariance Matrix for R it : 1 5.5804.9901 5.5804 3.0637.9901 5.5804 4 1.444.0637.9901 5.5804 5 0.9831 1.444.0637.9901 5.5804 6 0.6786 0.9831 1.444.0637.9901 5.5804 7 0.4683 0.6786 0.9831 1.444.0637.9901 5.5804 8 0.333 0.4683 0.6786 0.9831 1.444.0637.9901 9 0.31 0.333 0.4683 0.6786 0.9831 1.444.0637.9901 5.5804 10 0.1540 0.31 0.333 0.4683 0.6786 0.9831 1.444.0637.9901 11 0.1063 0.1540 0.31 0.333 0.4683 0.6786 0.9831 1.444.0637 1 0.07336 0.1063 0.1540 0.31 0.333 0.4683 0.6786 0.9831 1.444 13 0.05063 0.07336 0.1063 0.1540 0.31 0.333 0.4683 0.6786 0.9831 14 0.03495 0.05063 0.07336 0.1063 0.1540 0.31 0.333 0.4683 0.6786 15 0.041 0.03495 0.05063 0.07336 0.1063 0.1540 0.31 0.333 0.4683 16 0.01665 0.041 0.03495 0.05063 0.07336 0.1063 0.1540 0.31 0.333 17 0.01149 0.01665 0.041 0.03495 0.05063 0.07336 0.1063 0.1540 0.31 C.J. Anderson (Illinois) Serial Correlation Fall 017 54.1/ 97
Other Error Structures Autocorrelations of each lag are functionally related: AR(1), MA(1), ARMA(1,1) Gaussian Fractional Polynomials Autocorrelations of each lag are not functionally related: Toeplitz Errors C.J. Anderson (Illinois) Serial Correlation Fall 017 55.1/ 97
Toeplitz Errors General Toeplitz matrix: 1 ρ 1 ρ ρ 3... ρ r 1 ρ 1 1 ρ 1 ρ... ρ r Σ R = σ Ω = σ ρ ρ 1 1 ρ 1... ρ r 3......... ρ r 1 ρ r ρ r 3 ρ r 4... 1 Higher-order lags possible, but usually assumed to be 0. MA(1), which has lag of 1, is Toeplitz(). In SAS TYPE=TOEP(# lags +1) C.J. Anderson (Illinois) Serial Correlation Fall 017 56.1/ 97
Back to Riesby Data Recall that we decided on the following model: Level 1 model: HamD it = β 0i +β 1i (time) it +β i (time) it +R it where R it N(0,σ ) i.i.d & cov(r i ) = σ I. Level model: β 0i = γ 00 +γ 01 (Endog) i +U 0i β 1i = γ 10 +U 1i β i = γ 0 +U i where U i N(0,T). C.J. Anderson (Illinois) Serial Correlation Fall 017 57.1/ 97
New Models for Riesby Data Linear Mixed Model: HamD it = γ 00 +γ 10 (time) it +γ 0 (time) it +γ 01(Endog) i Model for cov(r i ) AR(1) MA(1) ARMA(1,1) +U 0i +U 1i (time) it +U i (time) it +R it C.J. Anderson (Illinois) Serial Correlation Fall 017 58.1/ 97
Global Fit Statistics Model LnLike AIC BIC Empty/Null 501.1 507.1 513.7 Preliminary HLM 04.0 8.0 54.3 No endog*week 04.0 6.0 50.1 No endog*week and no endog 07.6 7.6 49.5 AR(1) 03.1 7.1 53.4 TOEP() 03.3 7.3 53.6 ARMA(1,1) Estimated G matrix is not positive definite C.J. Anderson (Illinois) Serial Correlation Fall 017 59.1/ 97
Cov. Parameter Estimates: AR(1) Standard Z Cov Parm Estimate Error Value Pr Z UN(1,1) 7.45 4.80 1.55 0.06 UN(,1) 0.10.90 0.04 0.97 UN(,) 4.85 3.54 1.37 0.09 UN(3,1) 0.18 0.45-0.41 0.68 UN(3,) 0.65 0.60-1.08 0.8 UN(3,3) 0.14 0.11 1.3 0.11 AR(1) 0.15 0.17 0.88 0.39 Residual 1.41.99 4.14 <.00 C.J. Anderson (Illinois) Serial Correlation Fall 017 60.1/ 97
Estimated cov(r i ): AR(1) Row Col1 Col Col3 Col4 Col5 Col6 1 1.417 1.878 0.84 0.04 0.006 0.000 1.878 1.417 1.878 0.84 0.04 0.006 3 0.84 1.878 1.417 1.878 0.84 0.04 4 0.04 0.84 1.878 1.417 1.878 0.8 5 0.006 0.04 0.84 1.878 1.417 1.87 6 0.000 0.006 0.04 0.84 1.878 1.41 Note: (, 1) covariance equals (.15)(1.417) = 1.878. (3,1) covariance equals (.15) (1.417) =.84. C.J. Anderson (Illinois) Serial Correlation Fall 017 61.1/ 97
Cov. Parameter Estimates: TOEP() Standard Z Cov Parm Estimate Error Value Pr Z UN(1,1) 8.36 3.98.10 0.0 UN(,1) 0.3.65 0.1 0.90 UN(,) 5.44 3.13 1.74 0.04 UN(3,1) 0.15 0.44 0.33 0.73 UN(3,) 0.75 0.55 1.37 0.17 UN(3,3) 0.16 0.10 1.5 0.06 TOEP() 1.18 1.41 0.83 0.41 Residual 11.66 1.89 6.16 <.00 C.J. Anderson (Illinois) Serial Correlation Fall 017 6.1/ 97
Estimated cov(r i ): TOEP(1) week week 1 week week 3 week 4 week 5 week 6 1 11.66 1.18 0 0 0 0 1.18 11.66 1.18 0 0 0 3 0 1.18 11.66 1.18 0 0 4 0 0 1.18 11.66 1.18 0 5 0 0 0 1.18 11.66 1.18 6 0 0 0 0 1.18 11.66 C.J. Anderson (Illinois) Serial Correlation Fall 017 63.1/ 97
Testing Random Quadratic Term, U i Model LnLike AIC BIC Empty/Null 501.1 507.1 513.7 Preliminary HLM 04.0 8.0 54.3 No endog*week 04.0 6.0 50.1 AR(1) 03.1 7.1 53.4 TOEP() 03.3 7.3 53.6 Only U 0i and U 1i (no U i ) AR(1) 06.5 4.5 44. TOEP() 09.0 7.0 46.7 ARMA(1,1) Estimated G matrix is not positive definite C.J. Anderson (Illinois) Serial Correlation Fall 017 64.1/ 97
Dropping Random Quadratic, U i Hypothesis Test: H o : τ = τ 1 = τ 0 = 0 versus H a : Not H o. Test statistic = (06.5 03.1) = 3.4. p-value equals a mixture of χ 3 and χ : p-value = 1 (.334+.183) =.58 Retain H o ; drop quadratic random effect. C.J. Anderson (Illinois) Serial Correlation Fall 017 65.1/ 97
Next Steps? Remove fixed effect for the quadratic term: H o : γ 0 = 0 vs H a : γ 0 0. t = 0.55, Satter. df = 87. p = 0.58. Global fits Model LnLike AIC BIC Empty/Null 501.1 507.1 513.7 Preliminary HLM 04.0 8.0 54.3 No endog*week 04.0 6.0 50.1 AR(1) 03.1 7.1 53.4 AR(1) w/o U i 06.5 4.5 44. AR(1) w/o quadratic 06.8.8 40.4 C.J. Anderson (Illinois) Serial Correlation Fall 017 66.1/ 97
Model Refinement (continued) Random linear trend with AR(1): Covariance Parameter Estimates Standard Z Cov Parm Estimate Error Value Pr Z UN(1,1) 3.90 6.09 0.64 0.6 UN(,1) 0.34 1.40 0.4 0.81 UN(,) 1.8 0.66 1.93 0.03 AR(1) 0.37 0.15.50 0.01 Residual 17.97 4.56 3.94 <.01 Do we need a random intercept? C.J. Anderson (Illinois) Serial Correlation Fall 017 67.1/ 97
Dropping Random Intercept Global fits # Model LnLike AIC BIC Param Empty/Null 501.1 507.1 513.7 3 Preliminary HLM 04.0 8.0 54.3 1 No endog*week 04.0 6.0 50.1 11 AR(1) 03.1 7.1 53.4 1 AR(1) w/o U i 06.5 4.5 44. 9 AR(1) w/o quadratic 06.8.8 40.4 8 U 1i & AR(1) w/o U 0i 09.1 1.1 34. 6 C.J. Anderson (Illinois) Serial Correlation Fall 017 68.1/ 97
Dropping Random Intercept Hypothesis: H o : τ0 = τ 01 = 0 versus H a : Not H o Difference between lnlike 09.1 06.8 =.3 p-value is mixture of χ and χ 1, p value = 1 (.417+.69) =.34 Retain H o ; don t need a random intercept. But we keep fixed effects model for the intercept β 0i = γ 00 +γ 01 (Endog) i C.J. Anderson (Illinois) Serial Correlation Fall 017 69.1/ 97
Parameter Estimates Covariance Parameter Estimates Standard Z Cov Parm Estimate Error Value Pr Z UN(1,1) 1.45 0.43 3.40 0.0003 AR(1) 0.47 0.07 6.95 <.0001 Residual 1.51.74 7.84 <.0001 Solution for Fixed Effects Std. t Effect Estimate Error DF Value Pr t Intercept 4.905 0.666 83.7 36.66 <.0001 time.359 0.01 16 10.57 <.0001 Endog= 0 1.8071 0.8865 9.1.04 0.0444 Endog= 1.... C.J. Anderson (Illinois) Serial Correlation Fall 017 70.1/ 97
Final Model Subjective decision. I like simple ones, so let s look at HamD it = γ 00 +γ 10 (week) it +γ 01 (endog) i +U 1i (week) it +ρr i,(t 1) +ǫ it for (week) it = 0,...,5. Fixed effect structure (overall regression): ĤamD it = 4.9.33(week) it 1.81(endog) i Covariance for Y i? C.J. Anderson (Illinois) Serial Correlation Fall 017 71.1/ 97
Estimated Covariance Matrix for Y i : Σy i = Z i TZ i +σ ǫω where T = τ 1 and Z i = (0,1,,3,4,5), so Z i TZ i = τ1 0 0 0 0 0 0 0 1 3 4 5 0 4 6 8 10 0 3 6 9 1 15 0 4 8 1 16 0 0 5 10 15 0 5, and C.J. Anderson (Illinois) Serial Correlation Fall 017 7.1/ 97
Covariance Matrix for Y i Part due to autoregressive model, 1 ρ ρ ρ 3 ρ 4 ρ 5 ρ 1 ρ ρ ρ 3 ρ 4 σǫ Ω = ρ σ ǫ ρ 1 ρ ρ ρ 3 ρ 3 ρ ρ 1 ρ ρ ρ 4 ρ 3 ρ ρ 1 ρ ρ 5 ρ 4 ρ 3 ρ ρ 1 C.J. Anderson (Illinois) Serial Correlation Fall 017 7./ 97
Covariance Matrix for Y i Σy i = Σy i = σ ρσǫ ρ σǫ ρσǫ 3 ρ 4 σǫ ρσǫ 5 ρσǫ τ1 + σy i τ1 + ρσ ǫ 3τ1 + ρ σǫ 4τ1 + ρ3 σǫ 5τ1 + ρ6 σǫ ρ σǫ τ1 + ρσ ǫ 4τ1 + σ ǫ 6τ1 + ρσ ǫ 8τ1 + ρ σǫ 10τ1 + ρ3 σǫ ρ 3 σǫ 3τ1 + ρ σǫ 6τ1 + ρσ ǫ 9τ1 + σ ǫ 1τ1 + ρσ ǫ 15τ1 + ρ σǫ ρ 4 σǫ 4τ1 + ρ3 σǫ 8τ1 + ρ σǫ 1τ1 + ρσ ǫ 16τ1 + σ ǫ 0τ1 + ρσ ǫ ρ 5 σǫ 5τ1 + ρ4 σǫ 10τ1 + ρ3 σǫ 15τ1 + ρ σǫ 0τ1 + ρσ ǫ 5τ1 + σ ǫ 1.51 10.10.96 4.75 13.01 7.31.3 9.10 18.81 34.56 1.05 8.03 16.35 7.51 44.71 0.49 8.30 16.73 6.50 39.11 57.76 C.J. Anderson (Illinois) Serial Correlation Fall 017 73.1/ 97
Correlation Matrix for Y i ĉorry = 1.45 1.0.5 1.08.3.61 1.03.5.47.70 1.01.3.4.59.77 1 C.J. Anderson (Illinois) Serial Correlation Fall 017 74.1/ 97
Correlation Matrix for Y i Observed = dots, Estimated = lines C.J. Anderson (Illinois) Serial Correlation Fall 017 75.1/ 97
Quadratic Trend or AR(1)? Observed = dots, Estimated = lines C.J. Anderson (Illinois) Serial Correlation Fall 017 76.1/ 97
TOEP(6) Observed = dots, Estimated = lines C.J. Anderson (Illinois) Serial Correlation Fall 017 77.1/ 97
Quadratic Trend or AR(1)? Observed = dots, Estimated = lines C.J. Anderson (Illinois) Serial Correlation Fall 017 78.1/ 97
Quadratic Trend or TOEP(6) Observed = dots, Estimated = lines C.J. Anderson (Illinois) Serial Correlation Fall 017 79.1/ 97
Let the data decide! C.J. Anderson (Illinois) Serial Correlation Fall 017 80.1/ 97
Random Effects vs Serial Correlation Snijders & Bosker: Riesby data: found a reasonable model with complex random effects structure and one with simple random effects with more complex serial correlation structure. Does theory suggest one or the other? There is a wide range of possible combinations of random effects and serial covariance structures... C.J. Anderson (Illinois) Serial Correlation Fall 017 80./ 97
Possible Covariance Structures for Y i Some examples that we ll look at Only random effects (i.e., U i ) We ve looked at these most of the semester. Only serial correlation (i.e., R i ). Both random and serial. Most of this is/was from Hedeker web-site but translated into our notation (any mistakes are mine). C.J. Anderson (Illinois) Serial Correlation Fall 017 81.1/ 97
Only Random Effects Random-intercepts model: Z i = (1,1,...,1) T = τ 0 R i = σ I i Gives τ0 +σ τ0... τ0 Σy i = Z i TZ τ i +σ 0 I i = τ0 +σ... τ0......... τ0 τ0... τ0 +σ C.J. Anderson (Illinois) Serial Correlation Fall 017 8.1/ 97
Random Intercept & Random Slope Random linear trend: Z i = ( 1 1 1 0 1 ) ( τ T = 0 τ 01 τ 01 τ1 ) R i = σ I i Gives Σy i = Z i TZ i +σ I i ( τ 0 +σ τ0 +τ 01 τ0 +τ 01 τ0 +τ 01 τ0 +τ 01 +τ1 +σ τ0 +3τ 01 +τ1 τ0 +τ 01 τ0 +3τ 01 +τ1 τ0 +4τ 01 +4τ1 +σ ) C.J. Anderson (Illinois) Serial Correlation Fall 017 83.1/ 97
Random Intercept & Random Slopes Random quadratic trend: 1 1 1 R i = σ I i Z i = 0 1 T = 0 1 4 τ0 τ 01 τ 0 τ 01 τ1 τ 1 τ 0 τ 1 τ Gives Σy i ( τ0 + σ τ0 + τ 01 + τ 0 τ0 + τ 01 + 4τ 0 τ0 + τ 01 + τ 0 τ0 + τ 01 + τ1 + +τ 1τ + σ τ0 + 3τ 01 + τ1 + 5τ 0 + 6τ 1 + 4τ τ0 + τ 01 + 4τ 0 τ0 + 3τ 01 + τ1 + 5τ 0 + 6τ 1 + 4τ τ0 + 4τ 01 + 4τ1 + 8τ 0 + 16τ 1 + σ ) C.J. Anderson (Illinois) Serial Correlation Fall 017 84.1/ 97
Only Model for R i Compound symmetry: CS σ +σ1 σ1 σ 1 Σy i = σ1 σ +σ1 σ1. σ1 σ1 σ +σ1 How else can you get this structure? Random intercept model, i.e., σ 1 = τ 0. C.J. Anderson (Illinois) Serial Correlation Fall 017 85.1/ 97
Only Model for R i Frist-Order autoregressive: AR(1) 1 ρ ρ Σy i = σ ρ 1 ρ ρ ρ 1. C.J. Anderson (Illinois) Serial Correlation Fall 017 86.1/ 97
Only Model for R i (continued) Second-Order Toeplitz: TOEP() σ1 σ 0 Σy i = σ σ1 σ. 0 σ σ1 General Toeplitz: Σy i = σ 1 σ σ 3 σ σ 1 σ σ 3 σ σ 1. C.J. Anderson (Illinois) Serial Correlation Fall 017 87.1/ 97
Only Model for R i (continued) Unstructured: (UN) Σy i = σ 1 σ 1 σ 13 σ 1 σ σ 3 σ 13 σ 3 σ 3 When do we assume this for the covariance matrix of a response variable? Standard multivariate methods (e.g.,manova) Implication: A way to deal with missing data.. C.J. Anderson (Illinois) Serial Correlation Fall 017 88.1/ 97
Random Intercept with Compound Symmetry Z = (1,1,1) T = τ 0 cov(r) = CS Σy i = Z i TZ i + σ I i +σ11 i 1 i ( ) ( τ0 τ0 τ0 = τ0 τ0 τ0 + τ0 τ0 τ0 ( ) =...Just compound symmetry? σ + σ1 σ1 σ1 σ1 σ + σ1 σ1 σ1 σ1 σ + σ1 σ + σ1 + τ 0 σ1 + τ 0 σ1 + τ 0 σ1 + τ 0 σ + σ1 + τ 0 σ1 + τ 0 σ1 + τ 0 σ1 + τ 0 σ + σ1 + τ 0 ) C.J. Anderson (Illinois) Serial Correlation Fall 017 89.1/ 97
Random Intercept with... First Order Autoregressive Z = (1,1,1) T = τ 0 cov(r) = AR(1) Σy i = = ( τ0 τ0 τ0 τ0 τ0 τ0 τ0 τ0 τ0 )+σ ( 1 ρ ρ ρ 1 ρ ρ ρ 1 ( σ +τ 0 ρσ +τ 0 ρ σ +τ 0 ρσ +τ0 σ +τ0 ρσ +τ0 ρ σ +τ0 ρσ +τ0 σ +τ0 ) ) Constant variance, constant (but differing) bands, decreasing covariances. C.J. Anderson (Illinois) Serial Correlation Fall 017 90.1/ 97
Random Intercept with... Toeplitz Errors Z = (1,1,1) T = τ 0 cov(r) = TOEP() Σy i = = ( τ0 τ0 τ0 τ0 τ0 τ0 τ0 τ0 τ0 ) ( + σ1 σ σ3 σ σ1 σ σ3 σ σ1 ( σ 1 +τ 0 σ +τ 0 σ 3 +τ 0 σ +τ 0 σ1 +τ 0 σ +τ 0 σ3 +τ 0 σ +τ 0 σ1 +τ 0 ) ) Constant variance, constant (by differing) bands, decreasing covariances. C.J. Anderson (Illinois) Serial Correlation Fall 017 91.1/ 97
Random Linear Trend with... Compound symmetry: cov(r i ) = CS ( ) ( Z 1 1 1 τ i = T = 0 τ 01 0 1 τ 01 τ1 ) Σy i = ( ( + τ0 τ0 + τ 01 τ0 + τ 01 τ0 + τ 01 τ0 + τ 01 + τ1 τ0 + 3τ 01 + τ1 τ0 + τ 01 τ0 + 3τ 01 + τ0 + 3τ 01 + 4τ1 σ + σ1 σ1 σ1 σ1 σ + σ1 σ1 σ1 σ1 σ + σ1 ) ) C.J. Anderson (Illinois) Serial Correlation Fall 017 9.1/ 97
Random Linear Trend with... Compound symmetry: cov(r i ) = CS ( ) ( Z 1 1 1 τ i = T = 0 τ 01 0 1 τ 01 τ1 ) ( Σy i = σ + σ1 + τ 0 σ1 + τ 0 + τ 01 σ1 + τ 0 + τ 01 σ1 + τ 0 + τ 01 σ + σ1 + τ 0 + τ 01 + τ1 σ1 + τ 0 + 3τ 01 + τ1 σ1 + τ 0 + τ 01 σ1 + τ 0 + 3τ 01 + σ + σ1 + τ 0 + 3τ 01 + 4τ1 ) Increasing variances, non-constant covariances. C.J. Anderson (Illinois) Serial Correlation Fall 017 93.1/ 97
Random Linear Trend with... First Order Autoregressive: R i = AR(1) ( ) ( Z 1 1 1 τ i = T = 0 τ 01 0 1 τ 01 τ1 ) Σ y i = ( τ0 τ0 + τ 01 τ0 + τ 01 τ0 + τ 01 τ0 + τ 01 + τ1 τ0 + 3τ 01 + τ1 τ0 + τ 01 τ0 + 3τ 01 + τ0 + 3τ 01 + 4τ1 +σ ( 1 ρ ρ ρ 1 ρ ρ ρ 1 ) ) C.J. Anderson (Illinois) Serial Correlation Fall 017 94.1/ 97
Random Linear Trend with... First Order Autoregressive: R i = AR(1) ( ) ( ) Z 1 1 1 τ i = T = 0 τ 01 0 1 τ 01 τ1 ( Σy i = σ τ0 ρσ (τ0 + τ 01) ρ σ (τ0 + τ 01) ρσ (τ0 + τ 01) σ (τ0 + τ 01 + τ1 ) ρσ (τ0 + 3τ 01 + τ1 ) ρ σ (τ0 + τ 01) ρσ (τ0 + 3τ 01 + ) σ (τ0 + 3τ 01 + 4τ1 ) ) C.J. Anderson (Illinois) Serial Correlation Fall 017 95.1/ 97
Random Linear Trend with... Toeplitz Errors: R i = TOEP ( ) Z 1 1 1 i = 0 1 ( τ T = 0 τ 01 τ 01 τ1 ) Σy i = = ( + τ0 τ0 + τ 01 τ0 + τ 01 τ0 + τ 01 τ0 + τ 01 + τ1 τ0 + 3τ 01 + τ1 τ0 + τ 01 τ0 + 3τ 01 + τ0 + 3τ 01 + 4τ1 ( σ 1 σ σ 3 σ σ1 σ σ3 σ σ1 ) σ1 +τ 0 σ +τ 0 +τ 01 σ3 +τ 0 +τ 01 σ +τ 0 +τ 01 σ1 +τ 0 +τ 01 +τ1 σ +τ 0 +3τ 01 +τ1 σ3 +τ 0 +τ 01 σ +τ 0 +3τ 01 + σ1 +τ 0 +3τ 01 +4τ1 ) C.J. Anderson (Illinois) Serial Correlation Fall 017 96.1/ 97
How to Decide? Lots of possibilities. Tools for selection: Look at variances and covariances: Constant or non-constant variance? Constant Bands? Decreasing covariances? LR tests for nested models. Information criteria. What you know about the data & processes. Look for a set of good ones. C.J. Anderson (Illinois) Serial Correlation Fall 017 97.1/ 97