Random Effects. Edps/Psych/Stat 587. Carolyn J. Anderson. Fall Department of Educational Psychology. university of illinois at urbana-champaign

Transcription

1 Random Effects Edps/Psych/Stat 587 Carolyn J. Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign Fall 2012

2 Outline Introduction Empirical Bayes inference Henderson s mixed-model equations BLUP: Best Linear Unbiased Prediction Shrinkage The normality assumption for random effects SAS/MIXED Reading: Snijders & Bosker: Further reference: Verbeke & Molenberghs (Chapter 7) and therein. C.J. Anderson (Illinois) Random Effects Fall / 68

3 Introduction Why get estimate of the random effects, Ûj? See how much groups (macro units) deviate from the average regression. Detect outlying groups. Predict group specific outcomes. In education might be tempted to use these to Selecting specific school for your child to attend (see Snijders & Bosker). Hold schools/teachers/students accountable. C.J. Anderson (Illinois) Random Effects Fall / 68

4 Empirical Bayes Inference Bayes Theorem Empirical Bayes Estimates Example C.J. Anderson (Illinois) Random Effects Fall / 68

5 Bayes Theorem Based on probability theory, for two discrete variables, A and B, : joint probability = (conditional prob)(marginal prob) Bayes Theroem: P(A and B) = P(A B)P(B) = P(B A)P(A) P(A B) = P(B A)P(A) P(B) or P(B A) = P(A B)P(B) P(A) C.J. Anderson (Illinois) Random Effects Fall / 68

6 Bayes Theorem: Continuous Variables Replace probabilities (i.e., P(.)) by probability density functions, i.e., f (.). For two continuous variables, x and y, and Bayes Theroem: f (x, y) = f (x y)f (y) = f (y x)f (x) or f (x y) = f (y x) = f (y x)f (x) f (y) f (x y)f (y) f (x) C.J. Anderson (Illinois) Random Effects Fall / 68

7 Bayes Theorem for Vectors We have vectors of continuous random variables: y j, observations/measures on the response variable from individuals in group j. U j, our random effects for group j. The relationships that hold for two single variables also hold for sets of random variables and f (y j, U j ) = f (y j U j )f (U j ) = f (U j y j )f (y j ) f (U j y j ) = f (y j U j )f (U j ) f (y j ) C.J. Anderson (Illinois) Random Effects Fall / 68

8 Empirical Bayes Estimates where f (U j y j ) = f (y j U j )f (U j ) f (y j ) f (U j ) is the prior distribution of the random effects. This distribution is N (0, T). f (y j ) is the marginal distribution of the response variable, Y j N (X j Γ, (Z j TZ j + σ 2 I)) C.J. Anderson (Illinois) Random Effects Fall / 68

9 Empirical Bayes Estimates (continued) f (y j U j ) is the conditional distribution of the response variable given the random effects, f (y j U j ) is N ((X j Γ + Z j U j ), σ 2 I) f (U j y j ) is the posterior distribution of the random effects; it s the distribution of the random effects conditional on the data (our observations on our response variable and our estimated parameters, ˆΓ and ˆT). C.J. Anderson (Illinois) Random Effects Fall / 68

10 Empirical Bayes Estimates (continued) f (U j y j ) = f (y j U j )f (U j ) f (y j ) Replacing everything on the right hand side of this equation gives the distribution for f (U j y j ), which is multivariate normal. This is what we need to get estimates/predictions of U j. C.J. Anderson (Illinois) Random Effects Fall / 68

11 Estimating U j One way, estimate the mean of the distribution of the posterior distribution, f (U j y j ). By the definition of the mean, algebra and some calculus thrown in, U j(γ,t) = E(U j Y j = y j ) = U j f (U j y j )du j = TZ jv 1 j (y j X j Γ) C.J. Anderson (Illinois) Random Effects Fall / 68

12 Estimating U j (continued) Û j(γ,t) = U j f (U j y j )du j = TZ jv 1 j (y j X j Γ) The integration is over all possible values of U j (like a sum). V j = (Z j TZ j + σ2 I). (y j X j Γ) = (y j ŷ j ). The estimate of U j depends on Γ and T, as well as on y j. C.J. Anderson (Illinois) Random Effects Fall / 68

13 Estimating Covariance Matrix for U j The covariance matrix for is Ûj(Γ,T) [ var(ûj) = TZ j V 1 j V 1 j ( X j Xj V 1 j X j ) 1 ] X j V 1 j Z j T. This covariance matrix isn t good for statistical inference about (Ûj(Γ,T) U j ), because it underestimates the variability of (Ûj(Γ,T) U j )... it ignores the variability of U j. For statistical inference, the variance of (Ûj(Γ,T) U j ) is used, var(ûj U j ) = T var(ûj). When Γ and T are unknown and we use estimates of them, the estimate of U j is known as the Empirical Bayes (EB) estimate, Ûj. C.J. Anderson (Illinois) Random Effects Fall / 68

14 Example: HSB A simple model. (math) ij = γ 00 + γ 10 (cses) ij + U 0j + R ij SAS/MIXED input: PROC MIXED data=hsbcent noclprint covtest method=ml ic; CLASS id; MODEL mathach = cses /solution; RANDOM intercept / subject=id type=un solution cl alpha=.05; ODS output SolutionR=RanUs; C.J. Anderson (Illinois) Random Effects Fall / 68

15 SAS/MIXED Notes solution option in the RANDOM statement tells SAS to compute Ûj s. The ODS command ( Output delivery system ) replaces the MAKE command in earlier versions of SAS (MAKE can still be used). SolutionR is an ODS table name that have random effect solution vectors. RanUs is the name of the SAS data set output that contains the contents of the random effect solution vectors. For other ODS table names, see SAS 9.1 PROC MIXED online documentation. C.J. Anderson (Illinois) Random Effects Fall / 68

16 SAS/MIXED: Output Convergence criteria met. Covariance Parameter Estimates Standard Z Cov Parm Subject Estimate Error Value Pr Z UN(1,1) id <.0001 Residual <.0001 Solution for Fixed Effects Standard t Effect Estimate Error DF Value Pr> t Intercept <.0001 cses <.0001 C.J. Anderson (Illinois) Random Effects Fall / 68

17 Solution for Random Effects Solution for Random Effects Std Err t Effect school Estimate Pred DF Value Pr> t Intercept Intercept Solution for Random Effects Effect school Alpha Lower Upper Intercept Intercept C.J. Anderson (Illinois) Random Effects Fall /

18 Solution for Random Effects Estimate are the empirical Bayes estimates. The Std Err Pred are the standard errors of (Û j(γ,t) U j ), which are useful for statistical inference. The t Value is for testing H 0 : U j = 0 versus H a : U j 0. To understand what s in the SAS/MIXED mannual, you need to know about Henderson s Mixed Model and BLUP.... But first let s look atthe Ûoj s graphically... C.J. Anderson (Illinois) Random Effects Fall / 68

19 Empirical Distribution of Û0j s C.J. Anderson (Illinois) Random Effects Fall / 68

20 Û 0j s versus Schools with 95% CL Schools ordered according to Û0j. C.J. Anderson (Illinois) Random Effects Fall / 68

21 Effect of Micro Sample Size on Ûj: N = 160, n j = 2, 5, 10, 100 C.J. Anderson (Illinois) Random Effects Fall / 68

22 Effect of Micro Sample Size on Ûj: N = 160, n j = 2, 5, 10, 100 C.J. Anderson (Illinois) Random Effects Fall / 68

23 Effect of Macro Sample Size on Ûj: n j = 10, N = 20, 50, 100, 500 C.J. Anderson (Illinois) Random Effects Fall / 68

24 Effect of Macro Sample Size on Ûj: n j = 10, N = 20, 50, 100, 500 C.J. Anderson (Illinois) Random Effects Fall / 68

25 Effect of macro Sample Size on ˆγ: n j = 10, N = 20, 50, 100, 500 Marco sample Standard size Effect Estimate Error N=20 Intercept x N=50 Intercept x N=100 Intercept x N=500 Intercept x C.J. Anderson (Illinois) Random Effects Fall / 68

26 Effect of micro Sample Size on ˆγ: N = 160, n j = 2, 5, 10, 100 Micro sample Standard size Effect Estimate Error n j = 2 Intercept x n j = 5 Intercept x n j = 10 Intercept x n j = 100 Intercept x C.J. Anderson (Illinois) Random Effects Fall / 68

27 Summary: Effect of N and n j on Ûj & ˆγ Effect on shape of distribution of Û j : Increasing nj (micro) doesn t have much of an effect. Increasing N (macro) leads to more normal looking distribution. Effect on confidence limits for Ûj (i.e. standard errors) Increasing n j (micro) leads to smaller standard errors of Û j vspace.05in Increasing N (macro) doesn t change the standard errors of Û j. Effect on parameter estimates (i.e.,γ s): none Effect on standard errors of parameters: Increase N or n j, standard errors get smaller. Why? Is there a differential effect?

28 Effect of N and n j on s.e s of γ s N n j n + s.e Intercept x Intercept x Intercept x Intercept x , 000 Intercept x , 600 Intercept x , 000 Intercept x , 000 Intercept x

29 Henderson s Mixed-Model Equations The equation Û j(γ,t) = TZ jv 1 j (y j X j Γ) was also derived by Henderson who used a system of linear equations rather than Bayes Theorem. He basically stacks the problem. C.J. Anderson (Illinois) Random Effects Fall / 68

30 Henderson s Mixed-Model Equations Staring with the linear mixed model, y 1 X 1 Z y 2 X 2 0 Z =. Γ. }{{} Γ y N X N Z N }{{}}{{}}{{} y X Z i.e., y = XΓ + ZU + R. U 1 U 2. U N } {{ } U + Given estimates of variance componentst, estimates of U can be obtained by solving the mixed-model equations for Γ and U, ( X Σ 1 X X ) ( ) ( Σ 1 Z Γ X ) Σ 1 y Z Σ 1 X ZΣ 1 Z + T 1 = U Z 1 Σ 1 y R 1 R 2. R N } {{ } R C.J. Anderson (Illinois) Random Effects Fall / 68

31 Henderson s Mixed-Model Equations Note that the within and between covariance matrices are block diagonal σ 2 I T σ 2 I Σ = and T = 0 T σ 2 I N T N. C.J. Anderson (Illinois) Random Effects Fall / 68

32 and Solution to Henderson s Equations ˆΓ = which we ve seen before. ( X ˆV 1 X ) 1 X ˆV 1 y Û = TZ ˆV 1 (y XˆΓ) Using Henderson s approach is fine so long as you don t have large data sets (otherwise it s computationally difficult). In the SAS/MIXED documentation it is reported that Henderson s estimates are used...these are the same as the EB ones. In practice, the more direct approach is more efficient. (i.e., equation for Û j given at the beginning of this section and under EB and the equations previously given for. ˆΓ and ˆT in the notes on estimation). C.J. Anderson (Illinois) Random Effects Fall / 68

33 BLUP: Best Linear Unbiased Prediction Best means a predictor or estimator is Unbiased. Has the smallest variance among all possible unbiased estimators (of a particular form). If the variance components are known, Then the Bayes predictions of U j are the Best Linear nbaised Predictors or BLUP. But the variance components are not known... C.J. Anderson (Illinois) Random Effects Fall / 68

34 BLUP: Best Linear Unbiased Prediction Since estimates of the variance components ˆT are used to estimate or predict Y j, i.e., Ŷ j = XˆΓ + ZÛ j, The EB estimates of U j are Empricial Best Linear Unbaised Predictors or EBLUP. This is related to what follows and has or interpretation of the EB estimates of U j. C.J. Anderson (Illinois) Random Effects Fall / 68

35 Shrinkage 1. Simple Model: Null/Empty 2. Example: HSB random intercept models with (cses) i j. 3. Complex/General Model. 4. Example 2: More complex model. C.J. Anderson (Illinois) Random Effects Fall / 68

36 Shrinkage: Null/Empty Model Y ij = β 0j + R ij = γ 00 + U 0j + R ij Using information from group j, the OLS estimates of β 0j is The group mean. n j ˆβ 0j = (1/n j ) Y ij = Ȳ.j i=1 C.J. Anderson (Illinois) Random Effects Fall / 68

37 Shrinkage: Null Model (continued) If we used information from all the groups, we could estimate β 0j as the mean over populations (i.e., γ 00 ); that is, ( ) M n 1 j M ˆγ 00 = j n Y ij = j So we can estimate β 0j using Group information. j=1 i=1 j=1 Information from all groups (population). A combination of information. n j M Ȳ.j = Ȳ.. C.J. Anderson (Illinois) Random Effects Fall / 68

38 Optimal Estimator of β 0j The optimal (linear) combination (BLUP) is the empirical Bayes estimator of β 0j. It is a weighted average. ˆβ EB 0j = λ j ˆβ0j + (1 λ j )ˆγ 00 ( τ 2 ) o = τ0 2 + ˆβ 0j + σ2 /n j ( τo 2 = τ0 2 + σ2 /n j ( = 1 σ2 /n j τ0 2 + σ2 /n j ( 1 ) ( Ȳ.j + 1 ) Ȳ.j + τ 2 o τ σ2 /n j τ 2 o τ0 2 + σ2 /n j ( σ 2 /n j τ σ2 /n j ) ˆγ 00 ) Ȳ.. ) Ȳ.. C.J. Anderson (Illinois) Random Effects Fall / 68

39 Optimal Estimator of β 0j (continued) ˆβ EB ( ) ( 0j = λ j ˆβ 0j + (1 λ j )ˆγ 00 = 1 σ2 /n j σ 2 ) /n j τ0 2 + Ȳ.j + σ2 /n j τ0 2 + Ȳ.. σ2 /n j The weights are both less than 1, so the EB estimate will be closer to the overall mean than the OLS estimator of β 0j. Consider the extreme cases: τ 2 0 = 0 and σ2 = 0. This phenomenon is known as Shrinkage. The observed data are shrunken toward the prior average (i.e., X j Γ) since the prior mean of the random effects is 0. C.J. Anderson (Illinois) Random Effects Fall / 68

40 HSB Example of Shrinkage Random intercept model with (cses) ij. Consider school 8367 where n j = 14. The weight for the group data is ˆτ o 2 /(ˆτ o 2 + ˆσ 2 /n j ) = /( /n j ) =.76 The weight for overall average regression is 1 ˆτ 2 o /(ˆτ 2 o + ˆσ 2 /n j ) =.23. C.J. Anderson (Illinois) Random Effects Fall / 68

41 HSB Example of Shrinkage (continued) C.J. Anderson (Illinois) Random Effects Fall / 68

42 HSB Example of Shrinkage (continued) The observed data are shrunken toward the prior average (i.e., X j Γ) where the prior mean of the random effects is 0. The variance of the estimates is less than (or equal) to the data. See SAS/INSIGHT output.... next slide... C.J. Anderson (Illinois) Random Effects Fall / 68

43 HSB: Estimate = Û0j where ˆτ 2 0 = 8.61 C.J. Anderson (Illinois) Random Effects Fall / 68

44 Shrinkage for More General/Complex Shrinkage also occurs in more complex models. Instead of developing this in terms of terms of predicted values of Y ij... Ŷ ij = ˆβ 0j. ˆβ EB 0j we can do it in and Because I had to show it to myself... C.J. Anderson (Illinois) Random Effects Fall / 68

45 Shrinkage for More General/Complex Ŷ j = X j ˆΓ + Z j Û j = X j ˆΓ + Z j (TZ jv 1 j )(y j X j ˆΓ) = X j ˆΓ Z j TZ jv 1 j X j ˆΓ + Z j TZ jv 1 j y j = (I nj Z j TZ jv 1 j )X j ˆΓ + Z j TZ jv 1 j y j = (I nj (V j σ 2 I nj )V 1 j )X j ˆΓ + (V j σ 2 I nj )V 1 j = (I nj I nj + σ 2 V 1 j = (σ 2 V 1 j )X j ˆΓ + (I nj σ 2 I nj V 1 j )y j )X j ˆΓ + (I nj σ 2 V 1 )y j j y j C.J. Anderson (Illinois) Random Effects Fall / 68

46 English translation Ŷ j = (σ 2 V 1 j )X j ˆΓ + (I nj σ 2 V 1 )y j Predictions of Y ij are weighted combinations of j The overall/average population regression (i.e., X j ˆΓ), and The data from group j (i.e., yj ). Recall that that covariance matrix for Y j is V j = Z j TZ j + σ 2 I nj C.J. Anderson (Illinois) Random Effects Fall / 68

47 Weights for Extreme Case: T = 0 V j = σ 2 I nj and V 1 j = (1/σ 2 )I nj Weight for the overall average regression is (σ 2 V 1 j ) = (σ 2 (1/σ 2 )I nj ) = I nj, Weight for group j data is (I nj σ 2 V 1 j ) = (I nj I nj ) = 0 Predicted value of the response variable is Ŷ j = (σ 2 V 1 j )X j ˆΓ + (I nj σ 2 V 1 )y j = X j ˆΓ j C.J. Anderson (Illinois) Random Effects Fall / 68

48 Weights for Extreme Case: σ 2 = 0 Then V j = Z j TZ j + σ2 I nj = Z j TZ j Weight for the overall average regression is Weight for the group j data is (σ 2 V 1 j ) = 0, (I nj σ 2 V 1 j ) = I nj Predicted value of the response variable is Ŷ j = (σ 2 V 1 j )X j ˆΓ + (I nj σ 2 V 1 )y j = y j C.J. Anderson (Illinois) Random Effects Fall / 68 j

49 Comments on Complex Case Generally, the covariance matrix for the data is in between the two extremes and the predictions are shrunken toward toward the priori average regression (X j Γ). This also implies that for any linear combination (say vector L), var(l Û j ) var(l U j ) C.J. Anderson (Illinois) Random Effects Fall / 68

50 Shrinkage Example w/ Complex Model HSB: The linear mixed model is (math) ij = γ 00 + γ 10 (cses) ij + γ 20 (female) ij + γ 30 (minority) ij +γ 01 (sector) j + γ 02 (size) j + γ 03 (SES) j +γ 11 (sector) j (cses) ij + γ 22 (size) j (female) ij +γ 23 (SES) j (female) ij + γ 31 (sector) j (minority) ij +U 0j + U 1j (female) ij + U 2j (minority) ij + R ij C.J. Anderson (Illinois) Random Effects Fall / 68

51 Shrinkage Example w/ Complex Model The Mixed Procedure Model Information Data Set WORK.HSBCENT Dependent Variable mathach Covariance Structure Unstructured Subject Effect id Estimation Method ML Residual Variance Method Profile Fixed Effects SE Method Model-Based Degrees of Freedom Method Containment Convergence criteria met. C.J. Anderson (Illinois) Random Effects Fall / 68

52 Covariance Parameter Estimates Standard Z Cov Parm Subject Estimate Error Value Pr Z UN(1,1) id <.0001 UN(2,1) id UN(2,2) id UN(3,1) id UN(3,2) id UN(3,3) id Residual <.0001 C.J. Anderson (Illinois) Random Effects Fall / 68

53 The covariances from PROC CORR are smaller (indicates shrinkage). C.J. Anderson (Illinois) Random Effects Fall / 68 Empirical Bayes Inference Henderson s Mixed-Model Equations BLUP Shrinkage Normality Assumption SAS/MIXED Covariance Parameter Estimates Estimated from MIXED and computed variances and covariances (PROC CORR) of the EB Ûj.: PROX PROC Parameter MIXED CORR intercept, intercept τ female, intercept τ female, female τ minority, intercept τ minority, female τ minority,minority τ Residual σ

54 Weight Matrices: School 8367 Lower triangle ˆσ 2 ˆV 1 j and upper I ˆσ 2 ˆV 1 j.95 * * * * * * * * * * * * * * Overall being given more weight? C.J. Anderson (Illinois) Random Effects Fall / 68

55 Normality Assumption: Random Effects Using Ûj is to examine the assumption of normality for the random effects. Problem Even when the linear mixed model is correctly specified, the distribution of the U j s are all different unless all groups have the same X j and Z j. Solution: Standardize the Ûj s, Û j = Ûj s.e. j And then examine for normality.... C.J. Anderson (Illinois) Random Effects Fall / 68

56 HSB: stdu = Û 0j /Ŝ.E. j w/ (cses ij ) C.J. Anderson (Illinois) Random Effects Fall / 68

57 Normality: Problem of Shrinkage Problem: Due to shrinkage, Ûj s show less variability than is actually present in the population of random effects. Plots of Ûj and Û j do not necessarily reflect the actual distribution of the random effects. The EB estimates of U j are very dependent on their assumed prior distribution. C.J. Anderson (Illinois) Random Effects Fall / 68

58 Impact of Non-normality How sensitive are Ûj to the normality assumption? Study by Verbeke & Molenberghs, which I ran (out of curiosity and to show you too): 1. Simulate samples from a population with 1000 marco units where each macro unit had 5 observations (i.e., n j = 5) where the random effects followed a mixture of two normal distributions, U 0j [ 1 2 N ( 2, 1) + 1 ] N (2, 1). 2 C.J. Anderson (Illinois) Random Effects Fall / 68

59 Simulated Distribution for U 0j C.J. Anderson (Illinois) Random Effects Fall / 68

60 Impact of Non-normality 2 Using the simulated U 0j s, simulate y ij s according to y ij = U 0j + R ij with R ij N (0, σ 2 ) where I used σ 2 = 1, 9, 25 and Fit random intercept model to the simulated data. 4 Examine the resulting distribution of the Û0j s. C.J. Anderson (Illinois) Random Effects Fall / 68

61 Distributions for Û0j (Note: ˆτ 2 0 5) C.J. Anderson (Illinois) Random Effects Fall / 68

62 Impact of Non-Normality Both σ 2 and τ 2 0 influence the shape of the distribution of Ûoj: If σ 2 is large relative to τ 2 0, ρ I = τ 2 0 σ 2 + τ 2 0 is small and it s difficult to detect sub-group in the random effects. That is sub-groups in the population are difficult to recognize. If a model includes random slopes, it will be easier to detect subgroups in the random effects. C.J. Anderson (Illinois) Random Effects Fall / 68

63 Non-Normality & Marginal Model Based on a number of simulation studies (by others), the wrong distributional assumption for U j, Has little effect on the ˆΓ and ˆT. (yeah) Effects the estimated standard errors of ˆΓ and ˆT. (boo) Estimated standard errors for ˆΓ are generally pretty close to the robust ones. (yeah) Estimated standard errors for ˆT can be really bad. The uncorrected s.e. s could be 5 times too large or 3 times too small. But we generally don t use s.e. s to test whether τk 2 = 0, so this result is not too critical for valid tests of such hypotheses... How would/could you test this hypothesis?

64 Checking the Normality Assumption The EB s estimates of U j depend heavily on the distribution assumption for them. Best way to check for non-normality? Compare Ûj obtained assuming normality to model with those obtained from relaxing the normality assumption. Alternative distributional assumption for Uj s is a mixture of a number of (multivariate) normal s, where k p r = 1. U j g p r N (µ r, T) r=1 Maybe a Gamma distribution (e.g., skewed distribution reaction times)... But this would be for Y ij. C.J. Anderson (Illinois) Random Effects Fall / 68

65 Alternative Distribution for U j s A mixture of (multivariate) normals would be found if There is unobserved heterogeneity in the population.... p r represents a cluster of the total population. You have not included a categorical variable that s important. Such a mixture of normals implies a range of possible non-normal distributions for U j s. For examples, see Verbeke & Molenberghs (page 90). C.J. Anderson (Illinois) Random Effects Fall / 68

66 SAS/MIXED Options Options to produce predictions and estimates of the random effects, as well as a little data manipulation: PROC MIXED data=hsbcent noclprint covtest method=ml ic; CLASS id; MODEL mathach = cses /solution outpred=hsbpred outpredm=hsbpm; RANDOM intercept / subject=id type=un solution cl; ODS output SolutionR=RanUs; C.J. Anderson (Illinois) Random Effects Fall / 68

67 Data Steps DATA tmp99; SET hsbpred; predij=pred; residij=residual; lowij=lower; upij= upper; stdij=stderrpred; KEEP predij residij lowij upij stdij id; PROC SORT data=tmp99; by id; PROC SORT data=hsbpm; by id; C.J. Anderson (Illinois) Random Effects Fall / 68

68 More SAS Data & Tricks DATA predict; merge tmp99 hsbpm; by id; SAS/INSIGHT (WORD file & demonstration) SAS/ASSIST (Word file & demonstration) C.J. Anderson (Illinois) Random Effects Fall / 68