Unit 7: Random Effects, Subsampling, Nested and Crossed Factor Designs

Transcription

1 Unit 7: Random Effects, Subsampling, Nested and Crossed Factor Designs STA 643: Advanced Experimental Design Derek S. Young 1

2 Learning Objectives Understand how to interpret a random effect Know the different types of variance components Be familiar with how to obtain point estimates of variance components Know how to calculate and interpret the intraclass correlation coefficient Understand the use of subsampling as a strategy to procure data Know how to model and analyze data from factorial treatment designs with random effects Know how to model and analyze data from nested factor designs Know how to model and analyze data from crossed factors 2

3 Outline of Topics 1 Random Effects 2 Subsampling and Satterthwaite s Approximation 3 Multifactor Random and Mixed Effects Models 4 Nested and Crossed Factor Designs 3

4 Outline of Topics 1 Random Effects 2 Subsampling and Satterthwaite s Approximation 3 Multifactor Random and Mixed Effects Models 4 Nested and Crossed Factor Designs 4

5 Motivation for Random Effects With quantitative factors, we may want to make inference to levels not measured in the experiment by interpolation on the measurement scale. For example, suppose we have an experiment where temperature is a factor with two levels: 80 F and 100 F. Clearly, any temperature could be of interest (e.g., 90 F). With categorical factors, we may only be able to use a subset of all possible levels, but we would still like to be able to make inference to other levels. Consider some of the examples we have already seen: cereal packaging type, meat storage type, or car type. All of these can be considered to be a subset of a larger population of all levels. The factors we have seen thus far were presented as being the only levels of interest and, thus, they are considered fixed effects. When the factor levels are considered to be a subset of a broader set of factor levels of potential interest, then the effects are considered random effects. 5

6 Random Effects Model Imagine that we randomly select t of the possible levels of the factor of interest; i.e., a random effect. As before, the usual single-factor ANOVA applies, which is Y ij = µ + a i + ɛ ij, (1) but now where both the error term and treatment effects are random variables. Since both the error and treatment effects are random, we assume they are independent of each other such that ɛ ij N (0, σ 2 e) and a i N (0, σ 2 a). The variances σ 2 e and σ 2 a are called components of variance and the model above is often called the variance components model or random effects model. We call them components of variance because the variance of an observation, say σy 2, is simply the sum of the two components: σy 2 = σ2 a + σe. 2 6

7 Single-Factor Random Effects Model (Balanced Design) Y ij = µ + a i + ɛ ij (2) i = 1,..., t is the index over the factor level j = 1,..., n is the index over the EUs from treatment i Y ij is the measured response of observation j in treatment i µ is the grand (or overall) mean a i N (0, σ 2 a) is a random variable for the treatment distribution and ɛ ij N (0, σ 2 e) is the experimental error, where a i and ɛ ij are independent 7

8 Variance Components ANOVA Table Below is the ANOVA table for the variance components model. Note that in this model, we use the terminology between groups (also referred to as among groups) in place of treatments and within groups in place of error. Source df SS MS Expected MS Between t 1 SSB MSB = SSB t 1 σe 2 + nσa 2 Within N t SSW MSW = SSW N t σe 2 Total N 1 SSTot 8

9 Testing and Point Estimates In the fixed effect models, we test the equality of the treatment means; however, this is no longer appropriate in the variance components set-up because treatments are randomly selected and we are interested in the population of treatments rather than any individual treatment. The appropriate hypothesis test for a random effect is: H 0 : σ 2 a = 0 H A : σ 2 a > 0 The standard ANOVA partition of the total sum of squares results in the ANOVA table on the previous slide; however, as before, the form of the appropriate test statistic depends on the expected mean squares. In this case, the appropriate test statistic would be F = MSB/MSW, which follows an F -distribution with t 1 and N t df. To estimate the variance components, equate the expected mean squares to their observed values in the ANOVA table: MSW = ˆσ 2 e MSB = ˆσ 2 e + nˆσ2 a ˆσ 2 Y = ˆσ2 a + ˆσ2 e. = ˆσ 2 e = MSW = ˆσ2 a = (MSB MSW)/n 9

10 Negative Variance Estimates By definition, a variance measure is positive; however, estimates of σ 2 a may be negative. Various strategies can be taken if σ 2 a < 0 : 1 Accept the estimate as evidence of a true value of 0 and use 0 as the estimate, but under the caveat that the estimator will no longer be unbiased. 2 Retain the negative estimate, but understand that subsequent calculations using the results may not make sense. 3 Interpret the negative component estimate as an indication of an incorrect statistical model and consider revising the model. 4 Utilize a different estimation method instead of ANOVA. 5 Collect more data and analyze them separately or in conjunction with existing data and hope that the increased information will result in a positive estimate. 10

11 CI Estimates for Variance Components CI estimates can be computed for both variance components. For a balanced design, where N = nt, the exact 100(1 α)% CI estimator for σ 2 e is SSW χ 2 α/2,(n t) < σ 2 e < SSW χ 2 1 α/2,(n t), (3) where χ 2 γ,ν is the upper γ th percentile of a χ 2 distribution with ν df. An approximate 100(1 α)% CI estimator for σ 2 a is SSB(1 F u/f 0) nχ 2 α/2,(t 1) < σa 2 < SSB(1 F l/f 0), (4) nχ 2 1 α/2,(t 1) where F 0 = MSB/MSW is the observed F -statistic and the quantities F l = F 1 α/2,(t 1),(N t) and F u = F α/2,(t 1),(N t) are values of the upper (1 α/2) th and (α/2) th percentile, respectively, of an F -distribution with (t 1) and (N t) df. 11

12 Intraclass Correlation Coefficient (ICC) The intraclass correlation coefficient (or ICC) is a measure of the similarity of observations within groups relative to that between groups. When the similarity of the observations within groups is very high, σ 2 e will be very small. Consequently, σ 2 a will be a larger proportion of the total variation σ 2 y. The ICC is defined as the following ratio: ρ I = σ2 a σa 2 + σe 2. (5) 12

13 Point Estimates and CI Rewriting the expected mean squares from the ANOVA table gives E(MSB) = σ 2 Y {1 + (n 1)ρ I} E(MSW) = σ 2 Y (1 ρ I), which can be rearranged to find estimators for σ 2 Y and ρ I: ˆσ 2 Y = MSB + (n 1)MSW n ˆρ I = (MSB MSW) {MSB + (n 1)MSW} Note that the estimate of the ICC can have a minimum value of 1/(n 1) and a maximum value of +1. The 100(1 α)% CI estimator for ρ I is F 0 F u F 0 + (n 1)F u < ρ I < F 0 F l F 0 + (n 1)F l The above interval may be used for testing H 0 : ρ I = 0, where the hypothesis is not rejected if the interval includes 0. 13

14 Example: Tensile Strength of Castings We consider data on three different castings of a particular alloy. The castings were randomly selected fabrications conducted in the same facility. Each casting was broken down into bars (the EUs). Since the castings used in the study represent a sample of the potential population of castings that could be produced, the casting type is treated as a random effect. The investigator is interested in the variation in tensile strength among the castings produced in the facility and not just the three castings used in this study. Data are collected on n = 10 bars from each casting. We proceed to fit a random effects model to estimate the variance components. Casting Strength , 88.0, 94.8, 90.0, 93.0, 89.0, 86.0, 92.9, 89.0, , 88.6, 90.0, 87.1, 85.6, 86.0, 91.0, 89.6, 93.0, , 91.5, 92.0, 96.5, 95.6, 93.8, 92.5, 93.2, 96.2,

15 Example: Tensile Strength of Castings Analysis of Variance Table Response: strength Df Sum Sq Mean Sq F value Pr(>F) casting *** Residuals Signif. codes: 0 *** ** 0.01 * Above is the ANOVA table for the single factor random effects model. The computation is the same as when we considered the fixed effects model. Note that the lines pertaining to casting and Residuals are for the between and within sources of variation, respectively. Under the random effects model, this ANOVA table provides the test for H 0 : σ 2 a = 0 H A : σ 2 a > 0. The p-value is , thus indicating that the between group variation is significantly larger than 0. 15

16 Example: Tensile Strength of Castings Random effects: Groups Name Variance Std.Dev. casting (Intercept) Residual Number of obs: 30, groups: casting, 3 Above are the estimates of the variance components, which can also be calculated using the mean squares quantities on the previous slide: ˆσ 2 e = MSW = ˆσ a 2 MSB MSW = = = n 10 ˆσ Y 2 = ˆσ2 a + ˆσ e 2 = = We can also use those mean squares to calculate the estimate of the ICC: ˆρ I = (MSB MSW) {MSB + (n 1)MSW} = (10 1)5.819 =

17 Example: Tensile Strength of Castings 0.05% 0.95% sig2.a sig2.e rho.i Above are the 90% confidence intervals for the two variance components and the ICC. We can confirm these calculations as follows: ( σ 2 e : SSW SSW χ 2, χ α/2,(n t) 2 = , ) = (3.916, 9.727) α/2,(n t) σ 2 a :, SSB(1 F l/f 0 ) nχ 2 1 α/2,(t 1) SSB(1 Fu/F 0) nχ 2 α/2,(t 1) ( ( /12.708) = 10(5.991) = (1.817, ) ( ) F 0 F u F 0 F l ρ I :, = F 0 + (n 1)F u F 0 + (n 1)F l = (0.218, 0.961) ) ( /12.708), 10(0.103) ( ) (9)3.354, (9)

18 RCBD Model (Random Treatment Effect) Blocking can also be incorporated where the treatment is considered a random effect: Y ij = µ + a i + ρ j + ɛ ij (6) µ is the overall mean (a constant) i = 1,..., t and j = 1,..., n a i is the random treatment effect at level i and are iid normal with mean 0 and variance σa 2 ρj is the block effect at block j and is subject to the constraint n j=1 ρ j = 0 ɛ ij is the experimental error and are iid normal with mean 0 and variance σ 2 a i and ɛ ij are independent 18

19 RCBD Model (Random Treatment Effect) The primary differences the model in (6) and the model where treatment is a fixed effect is that a test of the treatment effect is H 0 : σ 2 a = 0 H A : σ 2 a > 0 and the expected mean square for the treatment effect is E(MSTr) = σ 2 + nσ 2 a. 19

20 Outline of Topics 1 Random Effects 2 Subsampling and Satterthwaite s Approximation 3 Multifactor Random and Mixed Effects Models 4 Nested and Crossed Factor Designs 20

21 Subsampling Up to this point, our discussion of experimental designs have focused on CRDs in which one observation of the response variable is made on the EUs. Sometimes, more than one observation is desired or simply convenient. Subsampling is when the experimenter takes repeated observations on the same EU. Thus, the MU is a subsample taken from a larger EU. 21

22 Statistical Model with Subsamples When there are m subsamples from each of n EUs for t treatments, the statistical model is Y ijk = µ + τ i + ɛ ij + δ ijk, (7) where i = 1,..., t, j = 1,..., n, and k = 1,..., m; µ is the grand mean; τ i are fixed effects subject to the constraint i τ i = 0; ɛ ij are the random experimental error effects for the j th EU of the i th treatment; δ ijk is the random effect for the k th subsample of the j th EU of the i th treatment; ɛ ij and δ ijk are independent random effects and each are normally distributed with mean 0 and variances σe 2 and σd 2, respectively. The above can be written for a random effect, which would require replacing τ i by a i, and the a i would be normally distributed with mean 0 and variance σa 2 and would be independent of the ɛ ij and δ ijk. The ANOVA table is calculated similarly under each case, except that the expected mean square for the treatment is computed differently. 22

23 SS Partitioning The observations, expressed as deviations from the grand mean, can be written as a sum of three separate deviations that represent the sources of variation in the experiment: where (Y ijk Ȳ ) = (Ȳi Ȳ ) + (Ȳij Ȳi ) + (Y ijk Ȳij ), (Ȳi Ȳ ) is the treatment deviation; (Ȳij Ȳi ) is the experimental error; and (Yijk Ȳij ) is the sampling error. Squaring and summing both sides results in the SSTot on the left-hand side expressed as the SS for treatments, error, and sampling, respectively: SSTot = SSTr + SSE + SSS 23

24 SS Formulas Source df SS MS E(MS) Treatments t 1 SSTr MSTr σ 2 d + mσ2 e + nm i τ2 i t 1 Error t(n 1) SSE MSE σ 2 d + mσ2 e Sampling tn(m 1) SSS MSS σ 2 d Total tnm 1 SSTot The SS quantities are given by: t SSTr = nm (Ȳi Ȳ )2 i=1 t n SSE = m (Ȳij Ȳi ) 2 i=1 j=1 t n m SSS = (Y ijk Ȳij ) 2 i=1 j=1 k=1 t n m SSTot = (Y ijk Ȳ )2 i=1 j=1 k=1 If the treatment effect is random, then the expected mean square for the treatment is E(MSTr) = σ 2 d + mσ2 e + nmσ2 a 24

25 Example: Bread Crustiness An experimenter wishes to test the effect of oven temperature on the crustiness of bread. Three temperatures (treated as a fixed effect) were utilized (high, medium, and low) and two EUs (batches of flour) were randomly assigned to each treatment. It was not economical to use the entire batch to bake breads. Hence, three subsamples were selected from each batch to make three loaves, which were baked at a given temperature. Here, three observations (subsamples) were made on each EU (batch). The response are scores of crustiness on a scale from 1 to 20. The data are given in the table below: Temperature MU Low Medium High Batch 1 Batch 2 Batch 3 Batch 4 Batch 5 Batch

26 Example: Bread Crustiness Analysis of Variance Table Response: crustiness Df Sum Sq Mean Sq F value Pr(>F) temperature e-06 *** batch ** Residuals Signif. codes: 0 *** ** 0.01 * Above is the ANOVA table for the single factor fixed effects model with subsampling. Note that the lines are temperature, batch, and Residuals, which correspond to the treatment, error, and sampling sources of variation, respectively. The test for the temperature effect is H 0 : τ 1 = τ 2 = τ 3 = 0 H A : not all τ i equal 0 This test has a p-value of 2.625e-06, which means we reject H 0 and conclude that at least one of the treatments is not 0. The test for the batch differences is H 0 : σ 2 e = 0 H A : σ 2 e > 0 This test has a p-value of , thus indicating that there are batch effects on the crustiness of bread. 26

27 Example: Bread Crustiness Random effects: Groups Name Variance Std.Dev. batch (Intercept) Residual Number of obs: 18, groups: batch, 6 Above are the estimates of the variance components, which can also be calculated using the mean squares quantities on the previous slide: ˆσ 2 d = MSS = ˆσ e 2 MSE MSS = = = m 3 ˆσ Y 2 = ˆσ d 2 + ˆσ e 2 = =

28 Subsampling with Unbalanced Data? Sometimes, the subsampling process results in unbalanced data. This requires a modification to the model in (7). We consider the setting where there are n i EUs for the i th treatment group and m ij subsamples for the j th EU of the i th treatment. 28

29 Model with Subsampling and Unbalanced Data When there are m i EUs for the i th treatment group and m ij subsamples for the j th EU of the i th treatment, the statistical model is Y ijk = µ + τ i + ɛ ij + δ ijk, (8) where i = 1,..., t, j = 1,..., n i, and k = 1,..., m ij ; N = t ni i=1 j=1 m ij and m i = n i j=1 m ij; µ is the grand mean; τ i are fixed effects subject to the constraint i τ i = 0; ɛ ij are the random experimental error effects for the j th EU of the i th treatment; δ ijk is the random effect for the k th subsample of the j th EU of the i th treatment; and ɛ ij and δ ijk are independent random effects and each are normally distributed with mean 0 and variances σe 2 and σ2 d, respectively. 29

30 SS Formulas Source df SS MS E(MS) Treatments t 1 SSTr MSTr σ i d 2 + c 1σe 2 + mi τ2 i t 1 Error i n i t SSE MSE σd 2 + c 2σe 2 Sampling N i n i SSS MSS σd 2 Total N 1 SSTot The SS quantities are given by: t SSTr = m i (Ȳi Ȳ )2 i=1 t n i SSE = m ij (Ȳij Ȳi ) 2 i=1 j=1 n m t i ij SSS = (Y ijk Ȳij ) 2 i=1 j=1 k=1 n m t i ij SSTot = (Y ijk Ȳ )2, where i=1 j=1 k=1 c 1 = 1 t n i m 2 ti=1 ni ( ij j=1 m2 t ) 1 ij t n i m 2 ij c 2 = n i t N. t 1 i=1 j=1 m i N i=1 i=1 j=1 m i 30

31 Test of Hypotheses Require Approximations When the subsample numbers are not equal, c 1 and c 2 on the previous slide can have different values. Thus, there is no exact test of the null hypothesis for treatment effects because no two mean squares have the same expected mean squares under the null hypothesis if c 1 and c 2 have different values. We can construct an approximate F -test to test the null hypothesis of no treatment effects. 31

32 Satterthwaite Approximation for DF In unbalanced cases, the required mean square quantities are often constructed as linear functions of other mean squares like the following: M = a 1 MS 1 + a 2 MS a k MS k, where MS h is a mean square with df ν h, h = 1,..., k. The Satterthwaite approximation for the df of M is given by ν = k h=1 M 2 (a h MS h ) 2 ν h. (9) Note that this approximation will almost never be an integer value, which is acceptable since it is only an approximation. 32

33 Derivation of Satterthwaite s DF Suppose we have non-negative parameters γ i with estimators ˆγ i such that the d iˆγ i γ i χ 2 d i are independent for all i, which implies that E[ˆγ i ] = and Var(ˆγ i ) = so that the ˆγ i are unbiased estimators of γ i. Let ω = i a i γ i and ˆω = i a iˆγ i. We would like an approximation for the distribution of ˆω of the form dˆω ω U χ 2 d. 33

34 Derivation of Satterthwaite s DF Note that Var [ dˆω E ω ( dˆω ω ] = E[U] = ) = Var(U) = This suggest that we approximate d by ω 2 d i a2 i γ2 i /d, i but we don t know the γ i or ω. The Satterthwaite approximation for this situation chooses ˆω ˆd 2 i a2 i ˆγ2 i /d, (10) i which matches ν defined in (9). Note that the Satterthwaite approximation will almost never be an integer value, which is acceptable since it is only an approximation. 34

35 Outline of Topics 1 Random Effects 2 Subsampling and Satterthwaite s Approximation 3 Multifactor Random and Mixed Effects Models 4 Nested and Crossed Factor Designs 35

36 Multiple Factors Earlier we introduced random effects in our linear models to estimate variance components, where the effects of the treatment factor were random samples from a population of treatments. The objective with using random effects is to decompose the total variance into identifiable components. The variability caused by one source or factor can depend on the conditions under which it is evaluated. Thus, some of the total variance is associated with the interaction between two (or more) factors. We will develop random effects models involving two and three factors. 36

37 Two-Factor Factorial Design with Random Effects Variability due to the interaction of random factors can play an important role in the inferential process. Suppose that we have two factors, A and B, that both have a large number of levels that are of interest, where we choose a random levels of factor A and b random levels of factor B, and n observations at each treatment combination. A random effects model for the two-factor experiment in a CRD is Y ijk = µ + a i + b j + (ab) ij + ɛ ijk, (11) where µ is the overall mean (a constant); i = 1,..., a, j = 1,..., b, and k = 1,..., n; a i are the random effects of factor A and are iid normal with mean 0 and variance σa; 2 b j are the random effects of factor B and are iid normal with mean 0 and variance σb 2; (ab) ij are the random interaction effects and are iid normal with mean 0 and variance σab 2 ; a i, b j, and (ab) ij are all assumed independent of one another; and ɛ ijk is the experimental error and are iid normal with mean 0 and variance σ 2. 37

38 Variance Components The observations Y ijk in the two-factor random effects model have a normal distribution with mean µ and variance: σ 2 Y = σ 2 + σ 2 a + σ 2 b + σ2 ab The relevant hypotheses that we are interested in testing are: H 0 : σ 2 a = 0 H 0 : σ 2 b = 0 H 0 : σ 2 ab = 0 H A : σ 2 a > 0 H A : σ 2 b > 0 H A : σ 2 ab > 0 We would, of course, start with testing the interaction effect (i.e., the third set of hypotheses) and, if not statistically significant, proceed to test the two main effects (i.e., the first two sets of hypotheses). 38

39 Expected Mean Squares and Testing The numerical calculations for the ANOVA are exactly like in the fixed effect case; however, again, the expected mean squares need to be taken into account when forming the test statistics for the tests on the previous slide. We state the expected mean squares below and, assuming the hypothesis is true, we form the F -test statistics, so that under that assumption, both the numerator and denominator of the F statistic have the same expectation. Note that the test for the main effects are no longer what they were in the fixed factor situation as they are now compared to the interaction effect. E(MSA) = σ 2 + nσ 2 ab + nbσ2 a = F = MSA MSAB F (a 1),(a 1)(b 1) E(MSB) = σ 2 + nσ 2 ab + naσ2 b = F = MSB MSAB F (b 1),(a 1)(b 1) E(MSAB) = σ 2 + nσab 2 = F = MSAB MSE F (a 1)(b 1),ab(n 1) E(MSE) = σ 2 39

40 Estimating Variance Components We can, again, estimate the variance components by equating the expected mean squares to their observed values: ˆσ 2 a = ˆσ 2 b MSA MSAB nb = MSB MSAB na ˆσ ab 2 MSAB MSE = n ˆσ 2 = MSE Then, the estimate of total variation for a single observation is ˆσ 2 Y = ˆσ 2 + ˆσ 2 a + ˆσ 2 b + ˆσ2 ab 40

41 Example: Machine Performance A manufacturer is developing a new spectrophotometer for the use in medical clinical laboratories. A critical component of instrument performance is the consistency of day-to-day measurements among machines. The researcher wants to know if the variability of measurements among machines operated over several days was within acceptable standards for clinical applications. The researcher sets up a factorial treatment design with two factors: machines (factor A) and days (factor B). a = 4 machines were to be tested over b = 4 days. Eight replicate serum samples were prepared each day and two of the serum samples were randomly assigned to each of the four machines on each of the four days. Therefore, we have n = 2 replicates for a total of N = 32 EUs. The response is the amount of triglyceride levels (mg/dl) in the serum samples. The data is given in the table below. Day Machine (j) (i) , , , , , , , , , , , , , , , ,

42 Example: Machine Performance Source df SS MS Var. Comp. E(MS) A σ 2 + nσ 2 ab + nbσ2 a B σ 2 + nσ 2 ab + naσ2 b AB σ 2 + nσ 2 ab Error σ 2 Total Above is the output for the ANOVA table for the two-factor factorial design with random effects. We can calculate the estimate of total variation for a single observation by ˆσ Y 2 = ˆσ2 + ˆσ a 2 + ˆσ2 b + ˆσ2 ab = = The proportion of variation in the preparation of the serum samples (i.e., the error component) is about 100 (ˆσ 2 /ˆσ Y 2 ) 11.5%. The variability in the machines contribute about 100 (ˆσ a 2/ˆσ2 Y ) 37.2% of the total variation. The variability in the day-to-day operational set-up contributes about 100 (ˆσ b 2/ˆσ2 Y ) 28.8% of the total variation. The interaction component contributes about 100 (ˆσ ab 2 /ˆσ2 Y ) 22.4% of the total variation. 42

43 Example: Machine Performance For the relevant hypotheses of interest: H 0 : σa 2 = 0 H 0 : σb 2 = 0 H 0 : σab 2 = 0 H A : σa 2 > 0 H A : σb 2 > 0 H A : σab 2 > 0 we get the following F -statistics: F = MSA MSAB = = 6.29 F = MSB MSAB = = 5.09 F = MSAB MSE = = 4.88 The F -statistics follow the respective F -distributions: F 3,9, F 3,9, and F 9,16. The p-values for each test are, respectively, 0.014, 0.025, and We would, of course, start with the test of the interaction term, which is significantly different from 0 with a p-value of Since the interaction term is significant, we would usually not proceed to test the significance of the other random effects. However, for illustrative purposes, we calculated the corresponding p-values. We see that the p-values for the machine and day effect are each significant. 43

44 Three-Factor Factorial Design with Random Effects Suppose that we have three factors, A, B, and C, where we choose a random levels of factor A, b random levels of factor B, c random levels of factor C, and n observations at each treatment combination. A random effects model for the three-factor experiment in a CRD is Y ijkl = µ + a i + b j + c k + (ab) ij + (ac) ik + (bc) jk + (abc) ijk + ɛ ijkl, (12) where µ is the overall mean (a constant); i = 1,..., a, j = 1,..., b, k = 1,..., c, and l = 1,..., n; a i, b j, c k, (ab) ij, (ac) ik, (bc) jk, and (abc) ijk are all the random effects and random interaction terms such that they are all independent of one another and iid normal with mean 0 and respective variances σa 2, σ2 b, σ2 c, σab 2, σ2 ac, σbc 2, and σ2 abc ; and ɛ ijkl is the experimental error and are iid normal with mean 0 and variance σ 2. SS quantities are done similar to previous ANOVA models; however, approximate F -tests are used when testing the null hypotheses of H 0 : σ 2 a = 0, H 0 : σ 2 b = 0, or H 0 : σ 2 c = 0. We will not develop these F -tests, but we will present the expected mean square formulas. 44

45 Expected Mean Squares Source df Expected MS A a 1 σ 2 + nσabc 2 + ncσab 2 + nbσac 2 + nbcσa 2 B b 1 σ 2 + nσabc 2 + ncσab 2 + naσbc 2 + nacσb 2 C c 1 σ 2 + nσabc 2 + nbσac 2 + naσbc 2 + nabσc 2 AB (a 1)(b 1) σ 2 + nσabc 2 + ncσab 2 AC (a 1)(c 1) σ 2 + nσabc 2 + nbσac 2 BC (b 1)(c 1) σ 2 + nσabc 2 + naσbc 2 ABC (a 1)(b 1)(c 1) σ 2 + nσabc 2 Error abc(n 1) σ 2 45

46 Two-Factor Mixed Model (Unrestricted Case) When models have both fixed and random effects, they are called mixed effects models. Suppose that we have fixed factor A and random factor B, where A has a levels and B has b random levels, and n observations at each treatment combination. A mixed effects model where one factor is fixed and one factor is random in a CRD is Y ijk = µ + α i + b j + (αb) ij + ɛ ijk, (13) where µ is the overall mean (a constant); i = 1,..., a, j = 1,..., b, and k = 1,..., n; α i are the fixed effects of factor A and a i=1 α i = 0; b j are the random effects of factor B and are iid normal with mean 0 and variance σb 2; (αb) ij are the random interaction effects and are iid normal with mean 0 and variance σαb 2 ; and ɛ ijk is the experimental error and are iid normal with mean 0 and variance σ 2. 46

47 Testing in the Two-Factor Mixed Model (Unrestricted) In the two-factor mixed model we presented, all of the random terms are assumed to be iid normally distributed and there is no restriction on the interaction term hence, we refer to this particular type of model as an unrestricted mixed model. The relevant hypotheses that we are interested in testing are: H 0 : α i = 0 for all i H 0 : σ 2 b = 0 H 0 : σ 2 αb = 0 H A : α i 0 for some i H A : σ 2 b > 0 H A : σ 2 αb > 0 We would, of course, start with testing the interaction effect (i.e., the third set of hypotheses) and, if not statistically significant, proceed to test the two main effects (i.e., the first two sets of hypotheses). 47

48 Expected Mean Squares and Testing (Unrestricted) The numerical calculations for the ANOVA are exactly like in previous settings; however, again, the expected mean squares need to be taken into account when forming the test statistics for the tests on the previous slide. We state the expected mean squares below and, assuming the hypothesis is true, we form the F -test statistics, so that under that assumption, both the numerator and denominator of the F statistic have the same expectation. E(MSA) = σ 2 + nσ 2 αb + nb i=1 α2 i a 1 E(MSB) = σ 2 + nσ 2 αb + naσ2 b = F = MSB E(MSAB) = σ 2 + nσ 2 αb = F = MSA MSAB F (a 1),(a 1)(b 1) = F = MSAB MSE E(MSE) = σ 2 MSAB F (b 1),(a 1)(b 1) F (a 1)(b 1),ab(n 1) We can, again, estimate the variance components by equating the expected mean squares to their observed values: ˆσ 2 b = MSB MSAB na ˆσ 2 αb ˆσ 2 = MSE = MSAB MSE n 48

49 Two-Factor Mixed Model (Restricted Case) There are several versions of the mixed model based on the definition used for the interaction effects. One alternative is that we place a constraint on the interaction effects, which yields what we call a restricted mixed model. A mixed effects model where one factor is fixed and one factor is random in a CRD is Y ijk = µ + α i + b j + (αb) ij + ɛ ijk, (14) where µ is the overall mean (a constant); i = 1,..., a, j = 1,..., b, and k = 1,..., n; α i are the fixed effects of factor A and a i=1 α i = 0; b j are the random effects of factor B and are iid normal with mean 0 and variance σb 2; (αb) ij is a random interaction effect that is normally distributed with mean 0 and variance [(a 1)/a]σαb 2, but is not independent; the restriction a i=1 (αb) ij = 0 is imposed on the interaction; and ɛ ijk is the experimental error and are iid normal with mean 0 and variance σ 2. 49

50 Testing in the Two-Factor Mixed Model (Restricted) Note that while a i=1 (αb)ij = 0, it is not the case that b j=1 (αb)ij = 0 since the levels of B are only a random sample. There is, then, covariance between two interaction effects at the same level of the random effect and different levels of the fixed effect, which is equal to σ 2 αb/a. The relevant hypotheses that we are interested in testing are: H 0 : α i = 0 for all i H 0 : σ 2 b = 0 H 0 : σ 2 αb = 0 H A : α i 0 for some i H A : σ 2 b > 0 H A : σ 2 αb > 0 We would, of course, start with testing the interaction effect (i.e., the third set of hypotheses) and, if not statistically significant, proceed to test the two main effects (i.e., the first two sets of hypotheses). In the mixed model, we can estimate the factor effects as before: ˆµ = ȳ ˆα i = ȳ i ȳ 50

51 Expected Mean Squares and Testing (Restricted) The numerical calculations for the ANOVA are exactly like in the unrestricted setting. Below are the expected mean squares: E(MSA) = σ 2 + nσ 2 αb + nb i=1 α2 i a 1 E(MSB) = σ 2 + naσ 2 b E(MSAB) = σ 2 + nσ 2 αb = F = MSA MSAB F (a 1),(a 1)(b 1) = F = MSB MSE F (b 1),ab(n 1) = F = MSAB MSE E(MSE) = σ 2 F (a 1)(b 1),ab(n 1) We can, again, estimate the variance components by equating the expected mean squares to their observed values: ˆσ 2 b = MSB MSE na ˆσ 2 αb ˆσ 2 = MSE = MSAB MSE n 51

52 Example: Chemistry Methods We consider an additional prong of the research study discussed earlier where a manufacturer is developing a new spectrophotometer for the use in medical clinical laboratories. Now, a researcher considers two chemistry methods (factor A) administered across four days (factor B). The chemistry method is considered fixed and the day is considered random. Two replicate samples of serum were prepared with each of the two chemistry methods. The experiment is set-up using a CRD. The response is, again, the amount of triglyceride levels (mg/dl) in the serum samples. The data is given in the table below. Method Day (j) (i) , , , , , , , ,

53 Example: Chemistry Methods Source df SS MS Var. Comp. E(MS) A σ 2 + nσ 2 αb + nb i=1 α2 i a 1 B σ 2 + nσ 2 αb + naσ2 b AB σ 2 + nσ 2 αb Error σ 2 Above is the output for the ANOVA table for the two-factor unrestricted mixed effects factorial design. For the relevant hypotheses H 0 : α i = 0 for all i H 0 : σ 2 b = 0 H 0 : σ 2 αb = 0 H A : α i 0 for some i H A : σ 2 b > 0 H A : σ 2 αb > 0 we have the following test statistics: F = MSA MSAB = = 5.35 F = MSB MSAB = = 2.33 F = MSAB MSE = = 4.27 The F -statistics follow the respective F -distributions: F 1,3, F 3,3, and F 3,8. The p-values for each test are, respectively, 0.103, 0.253, and The presence of interaction with days suggests a possibility of differences among the chemistry methods that vary with days. While the main effects are each not significant, they would be retained due to the significant interaction. 53

54 Example: Chemistry Methods Source df SS MS Var. Comp. E(MS) A σ 2 + nσ 2 αb + nb i=1 α2 i a 1 B σ 2 + naσ 2 b AB σ 2 + nσ 2 αb Error σ 2 Above is the output for the ANOVA table for the two-factor restricted mixed effects factorial design. We have the same hypotheses as in the restricted case H 0 : α i = 0 for all i H 0 : σ 2 b = 0 H 0 : σ 2 αb = 0 H A : α i 0 for some i H A : σ 2 b > 0 H A : σ 2 αb > 0 but we now have the following test statistics: F = MSA MSAB = = 5.35 F = MSB MSE = = 9.97 F = MSAB MSE = = 4.27 The F -statistics follow the respective F -distributions: F 1,3, F 3,8, and F 3,8. The p-values for each test are, respectively, 0.103, 0.004, and The results for method and the interaction are the same, but the result for the random effect (days) is now significant. Regardless, both effects would be retained due to the significant interaction. 54

55 Some Comments on Restricted vs. Unrestricted It is difficult to provide guidelines for when the restricted or unrestricted mixed model should be used, because there is not even consensus among statisticians. Inference for the fixed effects does not differ for the 2 factor mixed model which is most often seen, and is usually the same in more complicated models as well. If faced with using these models, the researcher should seek to make a reasonable choice and justify which model they think is most appropriate for their experimental situation. One major advantage is that the expected mean squares for unbalanced data are consistent with results from the unrestricted model; however, the restricted model is not considered in the unbalanced case. 55

56 Some Comments on Restricted vs. Unrestricted If there is a possibility of correlation between effects of a fixed factor for a given level of the random effect, and the data are balanced, then the restricted model may be more appropriate. Confidence intervals for contrasts of fixed effects are calculated as in fixed effects models, except that the denominator used in the corresponding hypothesis test must be used instead of the MSE. Confidence intervals for variance components of random effects or interactions involving random effects are calculated just as for random effects models, but using the random part of the model. 56

57 Outline of Topics 1 Random Effects 2 Subsampling and Satterthwaite s Approximation 3 Multifactor Random and Mixed Effects Models 4 Nested and Crossed Factor Designs 57

58 Nested and Crossed Factors In the factorial studies we have considered thus far, every level of one factor appears with each level of every other factor. For this setting, we say that the factors are crossed. Sometimes, the levels of one factor (say, factor B) are not identical to each other at different levels of another factor (say, factor A), although they might have the same labels. For this setting, we say that the factors are nested. For example, suppose that factor A is school and factor B is teacher. If Teacher 1 is different between schools (which is likely the case), then the design is nested. If the design were crossed, then Teacher 1 would have to be the same at both schools. Note that, formally speaking, subsampling of EUs is completely analogous to having nested factors therefore you are already familiar with the basics for working with nested designs! 58

59 Example: Training School A manufacturing company operates three regional training schools for mechanics: one in Atlanta, one in Chicago, and one in San Francisco. The schools have two instructors each, who teach the classes. The company was concerned about the effect of school (factor A) and instructor (factor B) on the learning achieved. To investigate these effects, classes in each district were formed in the usual way and then randomly assigned to one of the two instructors in the school. This was done for two sessions (n = 2 replicates) and at the end of each session, a suitable summary measure of learning for the class was obtained. The data are in the table below. Note that the instructor labeling is relative to the school. Factor A Factor B (Instructor) (School) 1 2 Atlanta 25, 29 ȳ 11 = , 11 ȳ 12 = 12.5 Chicago ȳ 21 = 8.5 ȳ 22 = , 6 22, 18 San Francisco 17, 20 5, 2 ȳ 31 = 18.5 ȳ 32 = 3.5 Average ȳ 1 = ȳ 2 = ȳ 3 =

60 Example: Training School Below is how the design would be treated if the factors were crossed. Namely, each instructor and school factor levels would be crossed to form treatments. This would results in a total of 18 cells (treatments), where their inclusion in the design is denoted by the shaded cell. School3(Factor3A) Instructor3(Factor3B) Atlanta Chicago San3Francisco Figure: Crossed Factors 60

61 Example: Training School Below is this study s design, where the factors are nested. Two instructors are specific to each school. There are a total of 6 cells, where the inclusion of a treatment in the design is denoted by the shaded cell and exclusion is denoted by an X in the cell. School3(Factor3A) Instructor3(Factor3B) Atlanta Chicago San3Francisco Figure: Nested Factors 61

62 Example: Training School Below is a better representation of this study s design, which shows the hierarchical structure of the design as well as reflects the replications in the study. Atlanta& (i=1) && Chicago& (i=2)& San&Francisco& (i=3)& Instructor&1& (j=1)& Instructor&2& (j=2)& Instructor&3& (j=1)& Instructor&4& (j=2)& Instructor&5& (j=1)& Instructor&6& (j=2)& Class&1& (k=1)& Class&2& (k=2)& Class&3& (k=1)& Class&4& (k=2)& Class&5& (k=1)& Class&6& (k=2)& Class&7& (k=1)& Class&8& (k=2)& Class&9& (k=1)& Class&10& (k=2)& Class&11& (k=1)& Class&12& (k=2)& Figure: Alternative Figure 62

63 Nested Design Model Let Y ijk denote the response for the k th replicate obtained under level i of the fixed factor A and level j of the fixed factor B, which is nested within A (written B(A)). A (balanced) nested effects model in a CRD is Y ijk = µ + α i + β j(i) + ɛ ijk, (15) where µ is the overall mean (a constant); i = 1,..., a, j = 1,..., b, and k = 1,..., n; α i are the fixed effects of factor A and a i=1 α i = 0; β j(i) are the fixed effects of factor B nested in A subject to the restriction a i=1 β j(i) = 0; and ɛ ijk is the experimental error and are iid normal with mean 0 and variance σ 2. There is no need for an interaction term in the nested design model above, since factor B is nested within factor A, not crossed with it. 63

64 Remark on Nested Design Model The nested design model is also called a balanced two-stage nested design because we can think about the design construction in two stages. The first stage consists of the main effects and the second stage consists of the effect nested within the main effect. We can further constructed m-stage nested designs, which would involve m such stages. For example, a three-stage nested design (i.e., m = 3) means we have a main effect (A), a second effect nested within the main effect (B(A)), and then a third effect nested within the combination of the first two effects (C(AB)). 64

65 Fitting the Model The least squares and maximum likelihood estimators of the parameters in the nested design model are obtained as usual: µ : ˆµ = ȳ α i : ˆα i = ȳ i ȳ β j(i) : ˆβj(i) = ȳ ij ȳ i The fitted values are, therefore, ŷ ijk = ȳ + (ȳ i ȳ ) + (ȳ ij ȳ i ) = ȳ ij 65

66 Partitioning Total Deviation and SSTot We decompose the total deviation by: (y ijk ȳ ) = (ȳ i ȳ ) + (ȳ ij ȳ i ) + (y ijk ȳ ij ) (y ijk ȳ ) is the total deviation. (ȳi ȳ ) is the deviation due to the factor effect A. (ȳij ȳ i ) is the deviation due to the specific effect B when at level i of factor A. (yijk ȳ ij ) is the residual deviation. When we square the above and sum over all cases, the cross-product terms drop out and we obtain: SSTot = SSA + SSB(A) + SSE, where SSTot = (y ijk ȳ ) 2 i j k SSA = nb (ȳ i ȳ ) 2 i SSB(A) = n (ȳ ij ȳ i ) 2 i j SSE = (y ijk ȳ ij ) 2 i j k 66

67 ANOVA Table Remember we are considering both A and B are fixed effects in the nested effects model. The df, mean squares, and expected mean squares can be summarized in the following ANOVA table: Source df SS MS Expected MS A a 1 SSA MSA σ 2 + nb i B(A) a(b 1) SSB(A) MSB(A) σ 2 + n Error ab(n 1) SSE MSE σ 2 Total abn 1 SSTot i α2 i a 1 j β2 j(i) a(b 1) 67

68 Testing The hypotheses of interest are H 0 : α i = 0 for all i H 0 : β j(i) = 0 H A : α i 0 for some i The respective test statistics are F = MSA MSE F a 1,ab(n 1) H A : β j(i) 0 for some j(i) F = MSB(A) MSE F a(b 1),ab(n 1) 68

69 Example: Training School Source df SS MS A B(A) Error Above is the output for the ANOVA table for the nested effects model, where both factors are treated as fixed. For the relevant hypotheses H 0 : α i = 0 for all i H 0 : β j(i) = 0 H A : α i 0 for some i we have the following test statistics: H A : β j(i) 0 for some j(i) F = MSA MSE = = 11.2 F = MSB(A) MSE = = 27.0 The F -statistics follow the respective F -distributions: F 2,6 and F 3,6. The p-values for each test are, respectively, and Since both results are statistically significant, we conclude that the three schools differ in mean learning effects and we conclude that the instructors within at least one school differ in terms of mean learning effects. 69

70 Random Effects in the Nested Design Factor B can also be treated as random, while factor A can also be treated as fixed or random. For such settings, the major differences are with how the expected mean squares and test statistics are calculated see below. Mean Expected Mean Square Square A Fixed, B Fixed A Fixed, B Random A Random, B Random MSA σ 2 i + nb α2 i σ a 1 2 i + nb α2 i a 1 + nσ2 b σ 2 + nbσa 2 + nσ2 b i j β2 j(i) MSB(A) σ 2 + n a(b 1) σ 2 + nσb 2 σ 2 + nσb 2 MSE σ 2 σ 2 σ 2 Test Appropriate Test Statistic A Fixed, B Fixed A Fixed, B Random A Random, B Random Factor A MSA/MSE MSA/MSB(A) MSA/MSB(A) Factor B(A) MSB(A)/MSE MSB(A)/MSE MSB(A)/MSE 70

71 This is the end of Unit 7. 71