Models for Clustered Data

Models for Clustered Data Edps/Psych/Stat 587 Carolyn J Anderson Department of Educational Psychology c Board of Trustees, University of Illinois Fall 2017

Outline Notation NELS88 data Fixed Effects ANOVA Random Effects ANOVA Multiple Regression HLM and the general linear mixed model Reading: Snijders & Bosker, chapter 3 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 21/ 63

Notation Snijders & Bosker s notation j index for groups (macro-level units) N = number of groups j = 1,,N i index for individuals (micro-level units) n j = number of individuals in group j i = 1,,n j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 31/ 63

Notation (continued) Y ij response or dependent variable y ij is a value on the response/dependent variable x ij explanatory variable n + = total number of level 1 observations, n + = N j=1 n j This is different from Snijder & Bosker (they use M) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 41/ 63

NELS88 National Education Longitudinal Study: Conducted by National Center for Education Statistics of the US department of Education These data constitute the first in a series of longitudinal measurements of students starting in 8th grade These data were collected Spring 1988 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 51/ 63

NELS88: The Data for 10 Schools CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 61/ 63

Descriptive Statistics Math scores Total sample: N j n j = n + = 260, Ȳ = 5130, s = 1114 By School: sch id n j Mean Std Dev Min Max 7472 23 4573 753 3300 6400 7829 20 4215 831 3100 6500 7930 24 5325 1152 3300 7000 24725 22 4354 1000 3100 6500 25456 22 4986 844 3200 6200 25642 20 4640 432 3900 5700 62821 67 6282 567 4300 7100 68448 21 4966 1033 3400 6900 68493 21 4633 955 3400 7100 72292 20 4785 1130 3400 6800 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 71/ 63

Another Look at the Data Mean math scores ±1 standard deviation for the N = 10 schools CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 81/ 63

Review of Fixed Effects ANOVA The model: Y ij = µ+α j +R ij where µ is the overall mean (constant) α j is group j effect (constant) R ij is the residual or error (random) R ij N(0,σ 2 ) and independent Accounts for all group differences Only interested in the N groups CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 91/ 63

Example: NELS88 using 10 schools Math scores as response variable ANOVA Summary Table From SAS/GLM and MIXED (method=reml) Sum of Mean Source df Squares Square F p Model (school) 9 1403054 155895 2155 < 0001 Error (residual) 250 1808606 7234 Corrected total 259 3211660 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 101/ 63

Example (continued) Only from SAS/MIXED: Maximum Likelihood Estimation Effect Num df Den df F value p school 9 250 2241 < 0001 Expected mean squares Sum of Mean Expected Error Source df squares square mean square term sch id 9 14031 155895 Var(Res) + Q(sch id ) MS(Res Residual (Res) 250 18086 7234 Var(Res) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 111/ 63

Linear Regression Formulation of fixed effects ANOVA Y ij = µ j +R ij = µ+α j +R ij = β 0 +β 1 x 1ij +β 2 x 2ij ++β (N 1) x (N 1),ij +R ij where x kij = 1 if j = k, and 0 otherwise (ie, dummy codes) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 121/ 63

Linear Regression in Matrix Notation Y 11 Y 21 Y 23,1 Y 1,2 Y 2,2 Y 20,2 Y 1,10 Y 20,10 = 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 β o β 1 β 2 β 3 β 4 β 5 β 6 β 7 β 8 β 9 + R 11 R 21 R 23,1 R 1,2 R 2,2 R 20,2 R 1,10 R 20,10 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 131/ 63

Linear Regression in Matrix Notation Y 1 Y 2 = X 1 X 2 β + R 1 R 2 Y 10 X 10 R 10 Y = Xβ + R CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 141/ 63

Fixed Effects Model (continued) Y ij = β 0 +β 1 x 1ij +β 2 x 2ij ++β (N 1) x (N 1)ij +R ij The parameters β 0,β 1,,β N 1 are considered fixed The R ij s are random: R ij N(0,σ 2 ) and independent Therefore, Y ij N(β 0 +β 1 x 1ij ++β (N 1) x (N 1)ij,σ 2 ) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 151/ 63

In Matrix Terms The model : Y = Xβ +R where β is an (N 1) vector of fixed parameters R is an (n + 1) vectors of random variables: Y N n+ (Xβ,σ 2 I) R N n+ (0,σ 2 I) Covariance matrix, σ 2 I = σ 2 0 0 0 σ 2 0 0 0 σ 2 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 161/ 63

Estimated model parameters, σ and β From the ANOVA table: ˆσ 2 = 7234 Solution for Fixed Effects β Standard Effect sch id Estimate Error df t Pr > t Intercept 4785 190 250 2516 < 0001 sch id 7472 211 260 250 081 42 sch id 7829 570 269 250 212 04 sch id 7930 540 258 250 210 04 sch id 24725 431 263 250 164 10 sch id 25456 201 263 250 077 44 sch id 25642 145 269 250 054 59 sch id 62821 1497 217 250 691 < 0001 sch id 68448 182 266 250 068 049 sch id 68493 152 266 250 057 057 sch id 72292 0 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 171/ 63

Random Effects ANOVA Situation: When groups are a random sample from some larger population of groups and you want to make inferences about this population Data are hierarchically structured data with no explanatory variables y CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 181/ 63

Random Effects ANOVA: The Model where Y ij = µ + U }{{} j +R ij }{{} fixed stochastic µ is the overall mean (constant) U j is group j effect (random) U j N(0,τ 2 ) and independent R ij is the w/in groups residual or error (random) R ij N(0,σ 2 ) and independent U j and R ij are independent, ie, cov(u j,r ij ) = 0 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 191/ 63

Notes Regarding Random Effects ANOVA If τ 2 > 0, group differences exist Instead of estimating N parameters for groups, we estimate a single parameter, τ 2 The variance of Y ij = (τ 2 +σ 2 ) Random effects ANOVA can be represented as a linear regression model with a random intercept CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 201/ 63

Random Effects ANOVA As Linear Regression Model: No explanatory variables Y ij = β 0j +R ij The intercept: β 0j = γ +U j where γ is Constant (overall mean) and U j is Random, N(0,τ 2 ) β 0j is random with β 0j N(γ,τ 2 ) U j are Level 2 residuals, R ij are Level 1 residuals, and they re independent The empty or null HLM CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 211/ 63

Random Effects ANOVA: NELS88 Using SAS/MIXED with Maximum likelihood estimation: Parameter Estimate std error Fixed effect(s) γ 4887 183 Variance parameters intercept τ 2 3052 1448 residual σ 2 7223 1120 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 221/ 63

A Closer Look (Intuitive) Y ij = β 0j +R ij = γ +U j +R ij Y ij is random due to U j and R ij Since U j and R ij are independent and normally distributed, (i) Y ij N(γ,(τ 2 +σ 2 )) (ii) cov(y ij,y kj ) = 0 (between groups) eg, Y ij = γ +U j +R ij Y i j = γ +U j +R i j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 231/ 63

A Closer Look (continued) Y ij = β 0j +R ij = γ +U j +R ij Within groups, observations are dependent Y ij = γ +U j +R ij Y i j = γ +U j +R i j γ is fixed U j is random but the same value for both i & i R ij and R i j are random and different values for i & i CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 241/ 63

Within Group Dependency (formal) cov(y ij,y i j) E[(Y ij E(Y ij ))(Y i j E(Y i j))] = E[((γ +U j +R ij ) γ)((γ +U j +R i j) γ)] = E[(U j +R ij )(U j +R i j)] = E[U 2 j +U j R i j +R ij U j +R ij R i j] = E[U 2 j ]+E[U j R i j]+e[r ij U j ]+E[R ij R i j] = E[U 2 j ] = E[(U j 0) 2 ] var(u j ) = τ 2 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 251/ 63

Intra-class Correlation Measure of within group dependency ρ I corr(y ij,y i j) = = = cov(y ij,y i j) var(yij ) var(y i j) τ 2 τ 2 +σ 2 τ 2 +σ 2 τ 2 τ 2 +σ 2 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 261/ 63

Intra-class Correlation: ρ I ρ I = τ2 τ 2 +σ 2 Assuming within group dependency is the same in all groups If τ 2 = 0, then don t need HLM Could be used as a measure of how much variance accounted for by the model, eg, Our NELS88 example, ˆρ I = 3052 3052+7224 = 30 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 271/ 63

Matrix representation of Y ij = γ +U j +R ij Y 11 Y 21 Y 23,1 Y 1,2 Y 2,2 Y 20,2 Y 1,10 Y 20,10 = 1 1 1 1 1 1 1 1 ( γ ) + U 1 U 1 U 1 U 2 U 2 U 2 U 10 U 10 + R 11 R 21 R 23,1 R 1,2 R 2,2 R 20,2 R 1,10 R 20,10 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 281/ 63

Matrix representation (continued) Y 1 Y 2 = X 1 X 2 β + U 1 U 2 + R 1 R 2 Y 10 X 10 U 10 R 10 Y = Xβ + U + R CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 291/ 63

Matrix Representation (continued) For group j, the distribution of Y j is Y j N nj (X j β,σ j ) where X j = 1 j is the an (n j 1) column vector with elements equal to one β = γ The covariance matrix for the j th group is an (n j n j ) symmetric matrix with elements Σ j = σ 2 I j +τ 2 1 j 1 j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 301/ 63

Covariance Matrix Σ j (n j n j ) = σ 2 I j + τ 2 1 j 1 j 1 0 0 1 1 1 = σ 2 0 1 0 1 1 1 +τ2 0 0 1 1 1 1 (τ 2 +σ 2 ) τ 2 τ 2 τ 2 (τ 2 +σ 2 ) τ 2 = τ 2 τ 2 (τ 2 +σ 2 ) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 311/ 63

Covariance Matrix (continued) Notes: Σ j = σ 2 I j +τ 2 1 j 1 j Compound symmetry Only difference between groups is the size of this matrix (ie, (n j n j )) For simplicity, drop the sub-script from Σ CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 321/ 63

Covariance Matrix: Y For the whole data vector Y, the covariance matrix is of size (n + n + ) and equals Σ 0 0 (n 1 n 1 ) 0 Σ 0 (n 2 n 2 ) Σ 0 0 0 (n N n N ) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 331/ 63

Summary: Random Effects ANOVA Model isn t too interesting No explanatory variables If we add variables to the regression, we only have a shift intercept in the intercept CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 341/ 63

Multiple Regression Intercepts and Slopes as outcomes Two stages: Stage 1: Fit multiple regression model to each group (level 1 analysis) ie, Summarize groups by their regression parameters Stage 2: Use the estimated intercept and slope coefficients as outcomes (level 2 analysis) ie, Analysis of the summary statistics CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 351/ 63

NELS88: Overall Regression Before we do the two stage analysis, CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 361/ 63

Individual Schools Data CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 371/ 63

Individual Schools Regressions CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 381/ 63

Two Stage Example: NELS88 Stage 1: Within groups regressions Y ij = student s math score x ij = time spent doing homework Proposition: More time spent doing homework, the higher the math score Appropriate model: where R ij N(0,σ 2 ) (math) ij = β 0j +β 1j (homew) ij +R ij Y ij = β 0j +β 1j x ij +R ij CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 391/ 63

Stage One Model fit to each of the 10 NELS88 schools: School ID ˆβ0j ˆβ1j R 2 7272 5068354-355380 2780 7829 4901229-292012 2111 7930 3875000 790909 6007 24725 3439382 559266 7002 25456 5393863-471841 1873 25642 4925896-248606 2186 62821 5921022 109464 1105 68448 3605535 649631 5098 68493 3852000 586000 3137 72292 3771392 63350 6417 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 401/ 63

Stage One (continued) Parameter n mean std dev se variance ˆβ 0j 10 4475 866 274 7493 ˆβ 1j 10 196 498 158 2476 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 411/ 63

Stage Two: NELS88 We use the estimated regression parameters as response variables and try to model the parameters We ll start with the intercepts Example simple model for the ˆβ 0j s: ˆβ 0j = γ 0 +U 0j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 421/ 63

Stage Two: Intercepts The REG Procedure Model: MODEL1 Dependent Variable: alpha Analysis of Variance Sum of Mean Source DF Squares Square F Pr > F Model 0 0 Error 9 67440021 7493336 Corrected Total 9 67440021 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 431/ 63

Stage Two: Intercepts Root MSE 865641 R-Square 00000 Dependent Mean 4475367 Adj R-Sq 00000 Coeff Var 1934234 Parameter Standard Variable DF Estimate Error t Pr > t Intercept 1 4475367 273740 1635 < 0001 Descriptives statistics CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 441/ 63

A little more complex model Model for the level 2 analysis: ˆβ 0j = γ 00 +γ 01 (mean SES j )+U 0j = γ 00 +γ 01 Z j +U 0j where U 0j N(0,τ 2 0 ) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 451/ 63

SAS PROC REG CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 461/ 63

SAS PROC REG (intercepts) The REG Procedure Model: MODEL2 Dependent Variable: alpha Analysis of Variance Sum of Mean Source DF Squares Square F Pr > F Model 1 21910398 21910398 385 0854 Error 8 45529623 5691203 Corrected Total 9 67440021 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 471/ 63

SAS PROC REG (intercepts) Root MSE 754401 R-Square 03249 Dependent Mean 4475367 Adj R 2 02405 Coeff Var 1685673 Parameter Estimates Parameter Standard Variable DF Estimate Error t Pr > t Intercept 1 4242466 266461 1592 < 0001 meanses 1-828170 422082-196 00854 Conclusions? Note regarding p-value decimal places CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 481/ 63

Stage Two: Slopes If the estimated slopes differ over groups, the effect of time spent doing homework on math scores changes depending on what school a student is in Simplest case: no explanatory variable The model: ˆβ 1j = γ 10 +U 1j where U 1j N(0,τ 2 1 ) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 491/ 63

Slopes: No Explanatory Variables The REG Procedure Model: MODEL1 Dependent Variable: beta Analysis of Variance Sum of Mean Source DF Squares Square F Pr > F Model 0 0 Error 9 22285163 2476129 Corrected Total 9 22285163 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 501/ 63

Slopes: No Explanatory Variables Root MSE 497607 R-Square 00000 Dependent Mean 196093 Adj R-Sq 00000 Coeff Var 25376069 Parameter Estimates Parameter Standard Variable F Estimate Error t Pr > t Intercept 1 196093 157357 125 02442 Descriptives statistics: see page 41 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 511/ 63

Slopes: More Complex Model Proposition: The higher the SES, the less of an effect time spent doing homework has on math achievement Reasoning for this: Schools with higher mean SES, more wealthy area, more money for schools, better school Diagram for this: Z x y CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 521/ 63

SAS PROC REG CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 531/ 63

Slopes: SAS PROC REG Output The model: ˆβ 1j = γ 10 +γ 11 (mean SES) j +U 1j where U 1j N(0,τ1 2) Dependent Variable: beta Analysis of Variance Sum of Mean Source DF Squares Square F Pr>F Model 1 1438612 1438612 055 48 Error 8 20846552 2605819 Corrected Total 9 22285163 CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 541/ 63

Slopes: SAS PROC REG Output Root MSE 510472 Dependent Mean 196093 R-Square 00646 Adj R 2 00524 Parameter Standard Variable DF Estimate Error t Pr> t Intercept 1 255772 180304 142 1938 meanses 1 212210 285605 074 4787 Conclusions? Problems with this analysis? CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 551/ 63

Possible Improvements To the two stage analysis of NELS88 Center SES Use multivariate (multiple) regression so that the intercepts and slopes are modeled simultaneously Would get an estimate of τ 01, covariance between U 0j and U 1j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 561/ 63

Serious problems remain Lost information because regression coefficients are used to summarize groups The number of observations per group is not taken into account when analyzing the estimated regression coefficients (in the 2nd stage) Random variability is introduced by using estimated regression coefficients How can model fit to data be assessed? Incorrect estimation of standard errors (level 2) CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 571/ 63

A Simple Hierarchical Linear Models Level 1: Y ij = β 0j +β 1j x ij +R ij where R ij N(0,σ 2 ) and independent and more compactly, where Y j N(X j β j,σ 2 I)iid Y j = X j β j +R j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 581/ 63

HLM: Level 2 Level 2: β 0j = γ 00 +U 0j β 1j = γ 10 +U 1j where ( U0j U 1j ) = N (( 0 0 ) ( τ 2, 0 τ 01 τ 01 τ1 2 )) iid or more compactly, where U j N(0,T)iid β j = Z j Γ+U j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 591/ 63

Linear Mixed Model Put models for β 0j and β 1j into the level 1 model: Y ij = β 0j +β 1j x ij +R ij = (γ 0o +U 0j )+(γ 1o +U 1j )x ij +R ij = γ 00 +γ 10 x ij }{{} +U 0j +U 1j x ij +R ij }{{} fixed part random part CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 601/ 63

Linear Mixed Model (continued) Given all the assumptions of the hierarchical model, Y ij N((γ 00 +γ 10 x ij ),var(y ij )) iid where var(y ij ) = (τ 2 0 +2τ 01x ij +τ 2 1 x2 ij )+σ2 This is the Marginal Model for Y ij CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 611/ 63

In Matrix Form The Model: Y j = X j Γ+Z j U j +R j where Y j N nj (X j Γ,V j )iid and the covariance matrix equals, V j = Z j TZ j +σ 2 I and later for longitudinal data V j = Z j TZ j +Σ j CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 621/ 63

The Marginal Model The distinction between the hierarchical formulation (and linear mixed model) and the marginal model are important with respect to Interpretation Estimation Inference but more on this later CJ Anderson (Illinois) Hierarchical Linear Models/Multilevel Analysis Fall 2017 631/ 63