Analysis of variance and regression. December 4, 2007

Transcription

1 Analysis of variance and regression December 4, 2007

2 Variance component models Variance components One-way anova with random variation estimation interpretations Two-way anova with random variation Crossed random effects Ecological analyses

3 Lene Theil Skovgaard, Dept. of Biostatistics, Institute of Public Health, University of Copenhagen

4 Variance Component Models, December Terminology for correlated measurements: Multivariate outcome: Several outcomes (responses) for each individual, e.g. a number of hormone measurements that we want to study simultaneously. Cluster design: Same outcome (response) measured on all individuals in a number of families/villages/school classes Repeated measurements: Same outcome (response) measured in different situations (or at different spots) for the same individual. Longitudinal measurements: Same outcome (response) measured consecutively over time for each individual.

5 Variance Component Models, December Variance component models Generalisations of ANOVA-type models or regression models, involving several sources of random variation (variance components) environmental variation between regions, hospitals or countries biological variation variation between individuals, families or animals within-individual variation variation between arms, teeth, injection sites, days variation due to uncontrollable circumstances time of day, temperature, observer measurement error

6 Variance Component Models, December Typical studies involve data from: a number of family members from a sample of households pupils from a sample of school classes measurements on several spots of each individual Alternative name (for some of them): Multilevel models variation on each level (variance component) possibly systematic effects (covariates) on each level

7 Variance Component Models, December Examples of hierarchies: individual context/cluster level 1 level 2 level 3 subjects twin pairs countries subjects families regions students classes schools visits subjects centres

8 Variance Component Models, December Merits Certain effects may be estimated more precisely, since some sources of variation are eliminated, e.g. by making comparisons within a family. This is analogous to the paired comparison situation. When planning subsequent investigations, the knowledge of the relative sizes of the variance components will be of help in deciding the number of repetitions needed at each level (if possible).

9 Variance Component Models, December Drawbacks When making inference (estimation and testing), it is important to take all sources of variation into account, and effects have to be evaluated using the relevant variation! Bias may result, if one or more sources of variation are disregarded

10 Variance Component Models, December Measurements belonging together in the same cluster look alike (are correlated) If we fail to take this correlation into account, we will experience: possible bias in the mean value structure low efficiency (type 2 error) for evaluation of level 1 covariates (within-cluster effects) too small standard errors (type 1 error) for estimates of level 2 effects (between-cluster effects)

11 Variance Component Models, December Concepts of the day: advantage/necessity of random effects generalisations of ANOVA-type models Examples with small data sets some of them too small to allow for trustworthy interpretations illustrative precisely because of their limited size Illustrated with SAS PROC MIXED

12 Variance Component Models, December One-way analysis of variance with random variation: Comparison of k groups/clusters, satisfying The groups are not of individual interest and it is of no interest to test whether they have identical means The groups may be thought of as representatives from a population, that we want to describe. Example: 10 consecutive measurements of blood pressure on a sample of 50 women: We know that the women differ and we do not care! We only want to learn something about blood pressure in the female population in general

13 Variance Component Models, December Example of one-way anova structure: 6 rabbits are vaccinated, each in 6 spots on the back Response Y : swelling in cm 2 Model: swelling = grand mean + rabbit deviation y rs = µ + α r + ε rs, r = 1,, R = 6 denotes the rabbit, + variation ε rs N(0,σ 2 ), where s = 1,, S = 6 denotes the spot The variation can be regarded either as within-rabbit variation or measurement error (probably a combination of the two).

14 Variance Component Models, December Rabbit means: µ r = µ + α r

15 Variance Component Models, December anova-table: SS df MS=SS/df F Between R 1 = Within R(S 1) = Total RS 1 = Test for identical rabbits means: F = 4.39 F(5, 30), P = 0.004, But: We are not interested in these particular 6 rabbits, only in rabbits in general, as a species! We assume these 6 rabbits to have been randomly selected from the species.

16 Variance Component Models, December We choose to model rabbit variation instead of rabbit levels: swelling = grand mean + between-rabbit variation + within-rabbit variation y rs = µ + a r + ε rs, where the a r s and the ε rs s are assumed independent, normally distributed with Var(a r )=ω 2 B, Var(ε rs)=σ 2 W The variation between rabbits has been made random ω 2 B and σ2 W are variance components, and the model is also called a two-level model

17 Variance Component Models, December Fixed vs. random effects? Fixed: all values of the factor present (typically only a few, e.g. treatment) allows inference for these particular factor values only must include a reasonable number of observations for each factor value Random: a representative sample of values of the factor is present allows inference to be extended beyond the values in the experiment and to the population of possible factor values (e.g. geographical areas, classes, rabbits) is necessary when we have a covariate for this level

18 Variance Component Models, December Interpretation: All observations have common mean and variance: y rs N(µ, ω 2 B + σ 2 W) but: Measurements made on the same rabbit are correlated with the intra-class correlation Corr(y r1, y r2 ) = ρ = ω 2 B ω 2 B + σ2 W Measurements made on the same rabbit tend to look more alike than measurements made on different rabbits. All measurements on the same rabbit look equally much alike. This correlation structure is called compound symmetry (CS) or exchangeability.

19 Variance Component Models, December Estimation of variance components First step is to determine the mean values of the mean squares (in balanced situations): E(MS B ) = Rω 2 B + σ2 W E(MS W ) = σ 2 W and from this we get the estimates σ 2 W = MS W σ 2 B = MS B MS W R

20 Variance Component Models, December Note: It may happen that σ 2 B becomes negative! by a coincidence as a result of competition between units belonging together, e.g. when measuring yield for plants grown in the same pot In this case, it will be reported as a zero

21 Variance Component Models, December Reading in data in SAS: data rabbit_orig; input spot $ y1-y6; cards; a b c d e f ; run; data rabbit; set rabbit_orig; rabbit=1; swelling=y1; output; rabbit=2; swelling=y2; output; rabbit=3; swelling=y3; output; rabbit=4; swelling=y4; output; rabbit=5; swelling=y5; output; rabbit=6; swelling=y6; output; run;

22 Variance Component Models, December In SAS, the estimation can be performed as: proc mixed data=rabbit; class rabbit; model swelling = / s; random rabbit; run; Covariance Parameter Estimates Cov Parm Estimate rabbit Residual Solution for Fixed Effects Standard Effect Estimate Error DF t Value Pr > t Intercept <.0001

23 Variance Component Models, December Interpretation of variance components: Proportion of Variation Variance component Estimate variation Between ωb % Within σw % Total ωb 2 + σw % Typical differences (95% Prediction Intervals): for spots on the same rabbit ± = ±2.16 cm 2 for spots on different rabbits ± = ±2.70 cm 2

24 Variance Component Models, December Interpretation of the size of the variance components: Approx. 2 3 of the variation in the measurements comes from the variation within rabbits. Maybe there is a systematic difference between the injection spots? Two-way anova: Source DF Type III SS Mean Square F Value Pr > F rabbit spot It does not look as if there is any systematic difference (P=0.26).

25 Variance Component Models, December Design considerations Imaginary experiment with measurements on R rabbits, and S spots for each rabbit. Var(ȳ) = ω2 B R + σ2 W RS For S=#spots, varying from 1 to 10: standard error rabbits

26 Variance Component Models, December Effective sample size If we had only one observation for each of k rabbits, how many would we need to obtain the same precision? k = R S 1 + ρ(s 1) We have here ρ = ω2 B ω 2 B +σ2 W = = k = 12.8 Effectively, we have only approximately two independent observations from each rabbit!

27 Variance Component Models, December Quantification of overall swelling method estimate (s.e.) 1: forget rabbit (0.155) 2: fixed rabbit (0.127) 3: rabbit averages (0.267) 4: random rabbit (0.267) 1. We pool all 36 measurements, mixing up the two variance components, and assuming independence 2. We estimate the mean swelling of exactly these 6 rabbits (using only within-rabbit variation) 3. We only look at averages for each rabbit (ecological analysis) 4. We estimate the mean swelling of rabbits as a species (the correct approach)

28 Variance Component Models, December Estimation of individual rabbit means: simple averages rely on individual measurements only, ȳ r. BLUP s (or EBLUP s, expected best unbiased linear predictor) rely on the assumption that individuals come from the same population, and become weighted averages: ω 2 B ω 2 B + σ2 W S ȳ r. + σ 2 W S ω 2 B + σ2 W S ȳ.. which have been shrinked towards the overall mean, ȳ..

29 Variance Component Models, December

30 Variance Component Models, December When the 3 smallest measurements from rabbit 2 (largest level) are omitted, the results become: method estimate (s.e.) 1: forget rabbit (0.163) 2: fixed rabbit (0.136) 3a: rabbit averages (0.265) (weighted) 3b: rabbit averages (0.333) (unweighted) 4: random rabbit (0.298) reference (0.267) 1 we have omitted some of the largest observations 2+3a rabbit 2 has a lower weight in the average due to only 3 observations 3b average for rabbit 2 has increased 4 rabbit 2 has a lower weight in the average due to a larger standard error

31 Variance Component Models, December EBLUPS for the reduced data set: 2 2 Larger shrinkage than before, for rabbit no

32 Variance Component Models, December Confidence limits for the variance components: Intra-individual variation σ 2 W : < σ 2 W < Inter-individual variation ω 2 B : < ω 2 B < 2.48 So, we should take care not to over-interpret...

33 Variance Component Models, December We imagine, that rabbits are grouped in two (grp=1,2) proc mixed data=rabbit; class grp rabbit; model swelling = grp / s; random rabbit(grp); run; Cov Parm Estimate rabbit(grp) < this changes Residual < this stays the same Solution for Fixed Effects Standard Effect grp Estimate Error DF t Value Pr > t Intercept <.0001 grp grp

34 Variance Component Models, December Such a comparison can not be performed in the usual way (ignoring the rabbits), since we then perform the comparison/test against a wrong variation. Type I error will occur! proc glm data=rabbit; class grp; model swelling=grp / solution; run; T for H0: Pr > T Std Error of Parameter Estimate Parameter=0 Estimate INTERCEPT B GRP B B...

35 Variance Component Models, December Two-level model: level unit variation covariates 1 rabbit*spot within rabbit spot 2 rabbit between rabbits group overall mean When the random rabbit variation is ignored: low efficiency (type 2 error) for evaluation of level 1 covariates (spot) too small standard errors (type 1 error) for estimates of level 2 effects (group, overall mean)

36 Variance Component Models, December Factor diagrams: In the traditional one-way anova: [I] = [R*S] [R] 0 In case of grouping: [I] = [R*S] [R] G 0 We have here used the notation arrows indicating simplifications / groupings [ ] for the random effects, corresponding to variance components on the various levels.

37 Variance Component Models, December Example: Number of nuclei per cell in the rat pancreas, used for the evaluation of cytostatica Henrik Winther Nielsen, Inst. Med. Anat. 4 rats (R) 3 sections for each rat (S) 5 randomly chosen fields from each section (F) 3-level model fields sections rats σ 2 τ 2 ω 2 Factor diagram: [I] = [R*S*F] [R*S] [R] 0

38 Variance Component Models, December Covariance Parameter Estimates Cov Parm Estimate rat section(rat) Residual Solution for Fixed Effects Standard Effect Estimate Error DF t Value Pr > t Intercept

39 Variance Component Models, December Estimation of variance components Proportion of Variation Variance component Estimate variation Rats ω % Sections τ % Fields σ % Total ω 2 + τ 2 + σ %

40 Variance Component Models, December Typical differences: for sections on different rats ±2 2 ( ) = ±1.319 for different sections on the same rat ±2 2 ( ) = ±1.264 for different fields on the same section ± = ±1.255

41 Variance Component Models, December The correlation between two measurements on the same rat becomes: if they are measured on the same section: Corr(y rs1, y rs2 ) = ω 2 + τ 2 ω 2 + τ 2 + σ 2 = if they are measured on different sections: Corr(y r11, y r22 ) = ω 2 ω 2 + τ 2 + σ 2 = 0.082

42 Variance Component Models, December Examples of hierarchies: individual cluster level 1 level 2 level 3 spots rabbits fields sections rats subjects twin pairs countries subjects families regions students classes schools visits subjects centres On all levels, we may have random variation (random effects or variance components), as well as covariates

43 Variance Component Models, December Average profiles: Example: 2 groups of dogs (5 resp. 6 dogs). Outcome: Osmolality, measured at 4 different times (with treatments along the way)

44 Variance Component Models, December Do we have repetitions?

45 Variance Component Models, December Residual plot (after suitable analysis): We see a clear trumpet shape, corresponding to the fact, that Dogs with a high level also vary more than dogs with a low level. Solution: Make a logarithmic transformation!

46 Variance Component Models, December Profiles on logarithmic scale with corresponding residual plot:

47 Variance Component Models, December Two-level model: level unit variation covariates 1 dog*time within dogs grp*time time 2 dog between dogs group overall mean proc mixed data=dogs; class grp time dog; model losmol=grp time grp*time / outpm=fit1 ddfm=satterth; random dog(grp); run;

48 Variance Component Models, December Class Level Information Class Levels Values grp time dog Covariance Parameter Estimates Cov Parm Estimate dog(grp) Residual P=0.08 for test of interaction, i.e. no convincing indication of this. Type 3 Tests of Fixed Effects Num Den Effect DF DF F Value Pr > F grp time <.0001 grp*time

49 Variance Component Models, December When there is no interaction, we simply omit the term from the model (but we could also just use averages, since the design is balanced) proc mixed covtest data=dogs; class grp time dog; model losmol=grp time / outpm=fit2 ddfm=satterth s; random dog(grp); run; Covariance Parameter Estimates Standard Z Cov Parm Estimate Error Value Pr Z dog(grp) Residual <.0001

50 Variance Component Models, December Solution for Fixed Effects Standard Effect grp time Estimate Error DF t Value Pr > t Intercept grp grp time time time <.0001 time Type 3 Tests of Fixed Effects Num Den Effect DF DF F Value Pr > F grp time <.0001

51 Variance Component Models, December In contrast, if we forget the random dog-effect, and perform a traditional two-way anova: proc glm data=dogs; class grp time; model losmol=grp time / solution; run; The GLM Procedure Dependent Variable: losmol Source DF Type III SS Mean Square F Value Pr > F grp time

52 Variance Component Models, December Standard Parameter Estimate Error t Value Pr > t Intercept B <.0001 grp B grp B... time B time B time B time B... Type 2 error for effect of time (level 1 covariate) time is evaluated in an unpaired fashion Type 1 error for effect of grp (level 2 covariate) we think we have more information than we actually have (we disregard the correlation)

53 Variance Component Models, December Factor diagram: [Dog] Grp [I] [Dog Time] Grp Time Time We note the following: The effect of GRP*TIME is evaluated against DOG*TIME If GRP*TIME is not considered significant, we thereafter evaluate TIME against DOG*TIME GRP against DOG, also called DOG(GRP)

54 Variance Component Models, December Interpretation of group effect: The estimated group difference is (0.166), corresponding to a 95% confidence interval of (-0.053,0.611). But this is on a logarithmic scale! We perform a back transformation with the exponential function and may conclude, that group 1 lies exp(0.279)=1.321 times higher than group 2, i.e. 32.1% higher. The 95% confidence interval is (exp(-0.053),exp(0.611))=(0.948,1.842)

55 Variance Component Models, December Example of a non-hierarchical model: Visual acuity: time in msec. from a stimulus (light flash) to the electrical response at the back of the cortex, measured for 7 individuals (patient), 2 eyes for each individual (eye) 4 lens magnifications (power) for each eye Crowder & Hand (1990)

56 Variance Component Models, December Predictors of visual acuity: Main effects: Systematic (mean value): eye, power Random: patient Interactions: Systematic (mean value): eye*power Random: patient*eye, patient*power Example of crossed random factors

57 Variance Component Models, December proc mixed data=visual; class patient eye power; model acuity=eye power eye*power / ddfm=satterth; random patient patient*eye patient*power; run; Cov Parm Estimate patient patient*eye patient*power Residual Type 3 Tests of Fixed Effects Effect Num DF Den DF F Value Pr > F eye power eye*power

58 Variance Component Models, December Factor diagram: [I] = [Pa Ey Po] [Pa Ey] Ey Po [Pa Po] Ey [Pa] Po 0

59 Variance Component Models, December level unit covariates 1 single measurements Ey*Po 2 interactions 2A [Pa*Ey] Ey 2B [Pa*Po] Po 2 individuals, [Pa] overall level

60 Variance Component Models, December

61 Variance Component Models, December Blood pressure and social inequity women in 17 regions of Malmö Covariates: Individual (level 1): low educational achievement (x 1 ) (less than 9 years of school) age group (x 2 ) Regional (level 2): rate of people with low educational achievement (z 1 ) from the Skåne Council Statistics Office

62 Variance Component Models, December Ecological analysis level 2 analysis (analysis of regional averages): Y: average blood pressure in residential area Z 1 : rate of people with low educational achievement Estimate of regression coefficient: 4.655(1.420) It seems to be an important explanatory variable!?

63 Variance Component Models, December Size of circle indicates size of investigation Obvious effect of aggregated covariate

64 Variance Component Models, December

65 Variance Component Models, December Estimates from variance component model: Covariates Individual variation Rate of variation in low education between low-education between model individuals regions x 1 σw 2 z 1 ωb 2 none age x 1, age (0.170) z 1, age (1.345) x 1, z 1, age (0.167) (1.250) 0.087

66 Variance Component Models, December We note the following: Region as a random effect could only account for 0.4% of the variation in blood pressure! Ecological variable ( Rate of low-income ) will have very little impact! Ecological analysis sums up the two effects, but is not able to distinguish between the two effects It overestimates the level 2 effect It cannot be interpreted as a level 1 effect

67 Variance Component Models, December

68 Variance Component Models, December Covariate effects on level 1 and level 2 can be very different Example: Reading ability, as a function of age and cohort:

69 Variance Component Models, December Misspecification missing random effect Result - type 2 error for x (unpaired) - type 1 error for z (too many df s, wrong variation) missing z - estimate of ω 2 B - estimate of σ 2 W too big perhaps too big (in unbalanced designs) missing x - estimate of ω 2 B or too small - estimate of σ 2 W too big too big

70 Variance Component Models, December Simulated data: Random effect of individual y p

71 Variance Component Models, December Estimates: Level Variation standard deviation 1 within individuals ˆσ W = between individuals ˆω B = 1.23

72 Variance Component Models, December Individual (level 1) covariate x, e.g. time/age: y x

73 Variance Component Models, December Estimates: standard regression Level Variation deviation coefficient 1 within individuals ˆσ W = 0.41 β x = 1.028(0.046) 2 between individuals ˆω B =

74 Variance Component Models, December Addition of a level 2 covariate: z, e.g. age: y x

75 Variance Component Models, December Estimates: standard regression Level Variation deviation coefficient 1 within individuals ˆσ W = 0.41 ˆβx = 1.033(0.046) 2 between individuals ˆω B = 1.14 ˆβz = 1.316(0.206)

76 Variance Component Models, December Comparison of estimates: within individual between individual Model ˆβx sd, ˆσ W ˆβz sd, ˆω B x (0.046) z (0.201) 1.03 x, z (0.046) (0.206) 1.14

77 Variance Component Models, December Example: suicide and religion Ecological analysis of regions: % suicides increases with % protestants, i.e. Protestants are more likely to commit suicide Or??

78 Variance Component Models, December level unit variation covariates 1 individuals within region, σ 2 W religion, x 2 regions between regions, ω 2 B % protestants, z True explanation: Interaction between individual effect (x) and region covariate (z) More suicides among catholics, in regions with many protestants