Three-Level Modeling for Factorial Experiments With Experimentally Induced Clustering

Transcription

1 Three-Level Modeling for Factorial Experiments With Experimentally Induced Clustering John J. Dziak The Pennsylvania State University Inbal Nahum-Shani The University of Michigan Copyright 016, Penn State. All rights reserved. Technical Report Number Please send questions and comments to John Dziak, The suggested citation for this technical report is Dziak, J. J., & Nahum-Shani, I. (016). Three-level modeling for factorial experiments with experimentally induced clustering (Technical Report No ). University Park, PA: The Methodology Center, Penn State. This work was supported by Awards P50 DA010075, P50 DA039838, P01 CA180945, R01 DK097364, R01 AA0931, R01 DA039901, and K05 DA01806 from the National Institutes of Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

2 Methodology Center Tech Report Three-Level Modeling With EIC Introduction In some experiments in the social sciences, participants enter the study as independent, but are then assigned into clusters (such as teams or therapy groups) by the investigators during the course of the study. This can be described as experimentally induced clustering (EIC). A recent manuscript by Nahum-Shani, Dziak, and Collins (016) provides specially constructed multilevel models for analyzing data from factorial experiments with EIC, and also provides power and sample size resources for planning them. They consider two kinds of EIC: full EIC, in which each participants are assigned to a cluster, and partial EIC, in which only some participants are so assigned, and one of the factors in the design tells whether a participant is assigned to belong to a cluster or not. They also provide analysis and power planning resources for EIC studies with or without a pretest measurement. Nahum-Shani, Dziak, and Collins (016) focused on two-level models, with Level 1 representing the participant and Level representing the cluster. These models assume that if the participant s pretest response (e.g., the participant s weight or body mass index at baseline) is available, it will be used by the investigator as a covariate in an ANCOVA-like model for predicting the posttest (e.g., the participant s weight or body mass index at a 6-month follow-up). However, another possible approach is to include the pretest as a repeated measure in a threelevel model in which time (pretest versus posttest) is used as an additional level. The covariateadjusted approach, as in Nahum-Shani, Dziak, and Collins (016), and the repeated measures approach, as in this technical report, require different assumptions. There are advantages and disadvantages to each approach, and neither is necessarily superior (see, e.g., Allison, 1990; Murray, 1998; Janega et al., 004).

3 Methodology Center Tech Report Three-Level Modeling With EIC 3 In Nahum-Shani, Dziak, and Collins (016) we focused on covariate-adjusted models for several reasons. First, they require fewer levels of nesting and hence are computationally much simpler and less costly than the repeated measures models. Second, the covariate-adjusted approach also offers more statistical power than the repeated measures approach under some scenarios (Vickers, 001; Vickers & Altman, 001; Dimitrov & Rumrill 003) although not all (Oakes & Feldman, 001). Third, the covariate-adjusted approach does not require that the pretest and posttest be measured on the same scale. In fact, the covariate-adjusted approach requires practically no assumptions about the distribution of the pretest, except those required of any regression predictor (i.e., that it have some variability in the sample, and that its relationship with posttest, if any, be linear). The three-level approach, on the other hand, requires that the variance structure of the pretest be jointly specified together with that of the posttest-- a much stronger requirement. However, it is possible that the three-level approach may still be of substantive or theoretical interest in specific situations. Therefore, as a supplement to Nahum-Shani, Dziak, and Collins (016), this technical report extends the models and power formulas provided in the manuscript to include another level for observation time, so as to model the response between pretest and posttest. Thus, this report provides models and power planning resources for the three-level approach. As in the manuscript, we begin for simplicity with a model for a single factor experiment with full EIC, and a similar model for partial EIC. We then generalize both to a model which allows multiple factors and either full or partial EIC. Last, we provide power formulas for factorial experiments with full and partial EIC using the repeated measures approach.

4 Methodology Center Tech Report Three-Level Modeling With EIC 4 Models for Repeated Measures Experiments with EIC Full EIC with a Single Factor Let us begin by considering the case of a randomized controlled trial (RCT), involving a single dichotomous treatment factor X, in which there is full EIC (i.e., clusters are generated for all participants). Nahum-Shani, Dziak, and Collins (016) point out that the pretest in an experiment with EIC must be included in the analysis model in a special way. Specifically, such a model must specify that the cluster-level random effect exists only at posttest and not at pretest, because clusters are generated after treatment assignment. Let T denote the observation time, coded as T = 0 for pretest and T = 1 for posttest. Let j be the index representing cluster membership, and i be the index of individuals within a cluster. For example, the dataset might appear as follows: Cluster (j) Individual (i) Time (t) Within Cluster (0=Pre, 1=Post) X Y

5 Methodology Center Tech Report Three-Level Modeling With EIC 5 With Levels 1, and 3 representing observations time, participant and cluster respectively, the response Y tij at observation time t, for participant i, in cluster j can be modeled as follows: Level 1: Level : Y tij = π 0ij + π 1ij T tij + e tij π 0ij = β 00j + r ij π 1ij = β 10j Level 3: β 00j = γ 000 β 10j = γ γ 101 X j + u j Combined: Y tij = (γ r ij ) + (γ u j )T tij + γ 101 X j T tij + e tij, (1) where e tij ~N(0, σ e ), r ij ~N(0, τ r ), and u j ~N(0, τ u ). The r ij and u j represent the effects of participant and cluster, respectively, and the e tij represent a combination of measurement error and any random subject-by-time interaction (random slope) that may exist. The fixed effects are represented by the γ coefficients. More specifically, the pretest mean is represented by γ 000 ; the posttest mean is represented by γ γ 100 for the control condition and by γ γ γ 101 for the experimental condition. Thus γ 101 is the fixed treatment effect of interest. In this model the cluster-level random effect u j is multiplied by the dummy-coded time indicator (T tij ) in order to allow cluster-level variation in the posttest response but not in the pretest response (i.e., only when T tij = 1). The pretest response is not subject to cluster-level variation, so the random effect for cluster is multiplied by zero when entering the model for the pretest. The posttest response will have cluster-level variation for all subjects, regardless of their experimental condition (i.e., regardless of X j ), because the design is assumed to involve full EIC.

6 Methodology Center Tech Report Three-Level Modeling With EIC 6 Partial EIC with a Single Factor Where there is partial EIC, it is necessary to take cluster-level variation into account for participants in the experimental condition, but not for those in the control condition, because the latter are independent of other participants. However, standard multilevel model notation involves using the same number of subscripts (representing levels within the nesting structure) for each participant. To resolve this dilemma, Nahum-Shani, Dziak, and Collins (016) use a notational approach strategy based on that of Roberts and Roberts (005) and Bauer et al. (008). Specifically, if an individual is not assigned to a cluster (e.g., therapy group) for treatment administration purposes, then the individual is considered to consist of a trivial cluster of size one for analysis purposes. The response can still be written as Y tij, with t representing time, i representing participant, and j representing cluster. However, trivial clusters will contain only i=1. For example, the dataset might appear as follows: Cluster (j) Individual (i) Time (t) Within Cluster (0=Pre, 1=Post) X Y

7 Methodology Center Tech Report Three-Level Modeling With EIC 7 With Levels 1,, and 3 representing observation time, participant and cluster respectively, let Level 1: Level : Y tij = π 0ij + π 1ij T tij + e tij π 0ij = β 00j + r ij π 1ij = β 10j Level 3: β 00j = γ 000 β 10j = γ (γ u j )X j Combined: Y tij = (γ r ij ) + γ 100 T tij + (γ u j )X j T tij + e tij () where e tij ~N(0, σ e ), r ij ~N(0, τ r ), and u j ~N(0, τ u ). Here, the pretest mean is represented by γ 000 ; the posttest mean is represented by γ γ 100 for the control condition, and by γ γ γ 101 for the experimental condition. Thus γ 101 is the overall treatment effect. Model differs from Model 1 in that the cluster-level random effect u j is now multiplied by the condition and time dummy codes (X j T tij ), rather than by the time dummy code only (T tij ). Thus, Model allows a cluster-level variation only in the posttest responses of the clustered condition (i.e., only if both X j = 1 and T tij = 1) since these are the only responses for which meaningful clusters exist in the design under consideration. Model 1, on the other hand, allows cluster-level variation in the posttest response of both conditions. Optionally, the error variance σ e can be allowed to differ between pretest and posttest, and/or between the clustered and the unclustered conditions at posttest. The error variance parameter can be written as σ e,t,s where t is 0 for pretest and 1 for posttest, and s is 0 for clustered and 1 for unclustered. This notation may be unnecessarily complex. In particular, the

8 Methodology Center Tech Report Three-Level Modeling With EIC 8 clustered and unclustered conditions should not differ at pretest, although they might differ at posttest. Thus, it is reasonable to assume that the pretest error variances for the unclustered participants (denoted σ e,0,0 ) and the pretest error variance for the clustered participants (denoted σ e,0,1 ) are equal; hence we refer to both as σ e,0,0 for convenience. The posttest error variance for unclustered participants (σ e,1,0 ), and the posttest error variance for clustered participants (σ e,1,1 ) can each still differ from σ e,0,0 and from each other. Note that Models 1 and above assume that there is zero intraclass correlation (ICC) at pretest, as well as no systematic difference between control and experimental group responses at pretest. Thus, these models are appropriate only when participants are not only randomly assigned to conditions but also randomly assigned to clusters. If this is not the case, then the experiment might be better conceptualized as a between-clusters design with pre-existing clustering, as discussed in Dziak, Nahum-Shani, and Collins (01). Extension to Multiple Factors In the manuscript, we employ an approach that combines the advantages of both dummy coding and effect coding (see the discussion of Models 5 and 6 in Nahum-Shani, Dziak, and Collins, 016). This approach enables scientists to both conveniently model the random cluster effect for only clustered participants (by using dummy coding so that random effects could be multiplied by zero and hence cancel out where appropriate), but yet interpret the effects of multiple factors in a straightforward manner (by using effect coding for the fixed effects model). We continue to use this approach here. As in the manuscript, assuming three dichotomous factors for simplicity, let X 1, X and X 3 be the effect-coded representations (-1 for Off and +1 for On) of the first, second, and third factors. Also, let C be an indicator of whether the participant has been assigned to a nontrivial

9 Methodology Center Tech Report Three-Level Modeling With EIC 9 cluster (0 for no and 1 for yes). In the case of partial EIC, C is a dummy-coded version of the assumed cluster-generating factor X 1 ; that is, C = 1 if X 1 = +1, and C = 0 if X 1 = 1. In the case of full EIC, C = 1 for everyone since all participants are assigned to nontrivial clusters. As before, let Y tij be the response at time t for individual i in cluster j. With Levels 1,, and 3 representing observation time, participant and cluster respectively, we can model Y tij as follows: Level 1: Level : Y tij = π 0ij + π 1ij T tij + e tij π 0ij = β 00j + r ij π 1ij = β 10j Level 3: β 00j = γ 000 β 10j = γ u j C j + γ 101j X 1j + γ 10j X j + γ 103j X 3j Combined: Y tij = γ r ij + γ 100 T tij + u j C j T tij + γ 101 X 1j T tij (3) + γ 10 X j T tij + γ 103 X 3j T tij + e tij where e tij ~ N(0, σ e ), r ij ~N(0, τ r ), and u j ~N(0, τ u ); C and T are dummy coded (i.e., 0 or 1); and X 1, X, and X 3 are effect-coded (i.e., -1 or +1). As before, the error variance can differ between pretest (denote this as σ e,0,0 ), unclustered participants posttest (σ e,1,0 ), and clustered participants posttest (σ e,1,1 ). τ r represents the person-level variance in pretest to posttest change, and τ u represents the cluster-level variance in pretest to posttest change for the clustered conditions. Notice that if C j is a dummy-coded version of X 1j (i.e., the partial EIC setting), then Model 3 instead becomes a generalization of Model to multiple factors; if C j is 1 for all participants (i.e., the full EIC setting), then Model 3 becomes a multiple-factor generalization of Model 1. In other words, both Models 1 and are actually special cases of Model 3. Model 3 can

10 Methodology Center Tech Report Three-Level Modeling With EIC 10 be extended to include interactions, as in Model 6 in the Nahum-Shani, Dziak, and Collins (016) manuscript. Power Planning for Repeated Measures Experiments with EIC As discussed in the manuscript, one can estimate the power for a test of a main effect or an interaction in a linear mixed model using the noncentral F distribution. That is, we assume that the power can be approximated by the probability that a noncentral F 1,ν variate, having noncentrality parameter λ, exceeds the critical value κ of the test to be performed. Here, κ is the value such that a central F 1,ν variate has only a probability α of exceeding κ under H 0, and γ λ = Var(γ ) (4) where γ is the regression parameter in question. As in the manuscript, we recommend that in the full EIC design the degrees of freedom ν should be calculated as the number of clusters minus the number of regression coefficients to be estimated. In planning experiments with partial EIC, the number of nontrivial clusters minus the number of regression coefficients would be appropriate as a conservative initial estimate of ν. However, for analysis purposes in partial EIC designs it is better that ν be empirically estimated using Satterthwaite s approximation (see Roberts & Roberts, 005; Nahum-Shani, Dziak, & Collins, 016). The formulas for Var(γ ) in Model 3, and informal derivations thereof, are provided below for full and partial factorial designs. Sampling Variance for Full EIC with Repeated Measures (Model 3; C j 1) The three-level repeated measures or "time condition" (Murray, 1998) approach of Model 3 is equivalent, assuming balance, to an analysis of gain scores (see Dimitrov & Rumrill, 003). That is, the treatment effect of interest is the difference between conditions in the mean

11 Methodology Center Tech Report Three-Level Modeling With EIC 11 change score (posttest minus pretest; see Murray, 1998, p. 181). Therefore, let G ij = Y 1ij Y 0ij denote the difference in response between pretest and posttest for an individual i in cluster j. Then, define μ (X k =L) to equal the sample estimate of E(G X k = L). Given this new notation, and following the logic discussed in Appendix B of the manuscript, we use the identities γ k = 1 (μ (X k =+1) of μ (X k =+1) μ (X k = 1) ) and Var(γ k) = 1 Var (μ (X k =+1) ). Asymptotically, the sampling variance should be the same as that of the sampling variance of G ij:x k =+1, that is, the average of the G ij over the participants with X k = +1. The random-effects part of G ij is u j + e 1ij e 0ij, each term of which is independent of the others. Assuming balance, in G ij:xk =+1, there are J/ independent subject-level u j terms being averaged together, nj/ independent observation-level e 0ij terms being averaged together, and nj/ independent observation-level e 1ij terms being averaged together, where J represents the total number of clusters summed across all conditions, and n is the number of members per cluster. Thus, assuming that pretest and posttest have the same error variance σ e, it then follows that Var(γ k) = Var ( μ (X k =+1) μ (Xk = 1) ) (+1) ( 1) = 1 Var 4 (μ (X k =+1) = 1 Var (μ (X k =+1) ) ) + 1 Var 4 (μ (X k = 1) ) = 1 ( τ u J = τ u + σ e J + σ e nj nj + σ e nj ) (5) where τ u is the cluster-level variance in posttest. Comparing expression 5 superficially to the corresponding formula without pretest (see Table 4 in the manuscript), it seems as though including the pretest has increased sampling error.

12 Methodology Center Tech Report Three-Level Modeling With EIC 1 Specifically, the formula for Var(γ k) in the two-level model without pretest was τ u + σ e, but J nj with pretest as a repeated measure the formula is now τ u + σ e. However, this would be a J misleading conclusion. In fact, σ e as defined in the three-level model is likely to be much smaller than σ e as defined in the posttest-only model. This is because the subject-level variance in the three-level model is partitioned into τ r, which is the person-level variance in pretest to posttest change (random but stable subject-level differences) and σ e (random subject-by-time interaction plus random measurement error). That is, the posttest variance after adjusting for any cluster or treatment effects but not adjusting for pretest is σ Y = σ e + τ r. The second term, τ r, is removed in the three-level model by effectively subtracting away the pretest. In contrast, in the posttest-only two-level model, all of the subject-level random variability was assigned to σ e and none was removed. Thus, σ e = σ Y in the posttest-only model, but σ e = (1 ρ pre,post )σ Y in the repeated measures model, where nj ρ pre,post = = Corr(Y 0, Y 1 X, u) Cov(Y 0, Y 1 X, u) Var(Y 0 X, u) Var(Y 1 X, u) = τ r σ Y and where σ Y in both cases represents within-cluster posttest variance, that is, posttest variance after adjusting for any cluster or treatment effects but not adjusting for pretest. Thus, the pretest is worthwhile if ρ pre,post > 1 (just as in classical unclustered experiments; see Frison & Pocock, 199; Vickers, 001). Following the logic presented in Appendix B of the manuscript, Var(γ k) can be re-expressed in terms of the pretest-posttest correlation correlation ρ pre,post, the posttest intraclass correlation ρ Y1 and the within-cluster marginal variance σ Y. Specifically, we already saw that σ e = (1 ρ pre,post )σ Y. Also, the

13 Methodology Center Tech Report Three-Level Modeling With EIC 13 posttest covariance between two members i and i within the same cluster is τ r, and the total marginal variance of the random effects at pretest, including random cluster effects, is τ r + τ u + σ e = τ u + σ Y. Therefore ρ Y1 = τ u τ u +σ and we can write τ ρ Y 1 u = σ Y Y (1 ρ Y 1 ). Then expression 5 is equivalent to Var(γ k) = σ Y ( ρ Y1 + (1 ρ pre,post) (1 ρ Y 1 )J Jn ). (6) In (4), if γ k is specified as a multiple of σ e, then σ Y cancels out from the numerator and denominator, and only the correlations still need to be specified. Past literature may help researchers find realistic candidate values for ρ Y1 and ρ pre,post. Sampling Variance for Partial EIC with Repeated Measures (Model 3; C j is 0 or 1 depending on X 1 ). We now consider the partial EIC case. We assume that participants are assigned to clusters if they receive a treatment condition with X 1 = +1, but left unclustered if they receive a treatment condition with X 1 = 1. Analogously to the full EIC case, we replace the analysis with an analysis of change scores G ij to argue that Var(γ k) = 1 4 Var (μ (X 1 =+1) ) Var (μ (X 1 = 1) ). The random effects part of G ij is C j u j + e 1ij e 0ij. Thus, Var (μ (X 1 =+1) ) = τ u + σ e,0,1 J 1 nj 1 + σ e,1,1 nj 1 and

14 Methodology Center Tech Report Three-Level Modeling With EIC 14 Var (μ (X 1 = 1) ) = σ e,0,0 J 0 + σ e,1,0 J 0 where σ e,0,0, σ e,0,1, σ e,1,0 and σ e,1,1 are the error variances for unclustered pretests, clustered pretests, unclustered posttests and clustered posttests respectively, and τ u is the cluster-level variance in pretest to posttest change for the clustered conditions. Therefore, Var(γ k) = τ u + σ e,0,0 4J 1 4J 0 + σ e,1,0 4J 0 + σ e,0,1 + σ e,1,1. 4nJ 1 4nJ 1 J 1, J 0 and n above represent the number of clusters, number of unclustered individuals, and number of members per nontrivial cluster respectively, so that J 1 n is the number of clustered individuals and J 0 + J 1 n is the total number of participants. σ e,0,0, σ e,0,1, σ e,1,0 and σ e,1,1 denote the error variances for unclustered pretests, clustered pretests, unclustered posttests and clustered posttests respectively; and τ u is the cluster-level variance in pretest to posttest change for the clustered conditions. Assuming that the pretest error variances σ e,0,0 (for unclustered individuals) and σ e,0,1 (for individuals to be clustered) are equal, we can refer to both as σ e,0,0, and combine their terms. Further simplification is possible if we assume that pretest and posttest error variances are equal, such that σ e,0,0 = σ e,0,1 = σ e,1,0 = σ e,1,1 = σ e. In this case, following the same reasoning as in the previous section, Var(γ k) can be re-expressed in terms of correlations and marginal variances ρ Y Var(γ k) = σ Y ( + 1 ρ pre,post 4(1 ρ Y )J 1 J 1 n + 1 ρ pre,post J 0 ), where ρ Y is the posttest ICC for clustered individuals, adjusting for treatment; and ρ pre,post represents the pretest-posttest correlation after adjusting for treatment for unclustered

15 Methodology Center Tech Report Three-Level Modeling With EIC 15 individuals, and the pretest-posttest correlation after adjusting for treatment and cluster for clustered individuals.

16 Methodology Center Tech Report Three-Level Modeling With EIC 16 References Allison, P. D. (1990). Change scores as dependent variables in regression analysis. Sociological Methodology, 0, Bauer, D. J., Sterba, S. K., & Hallfors, D. D. (008). Evaluating group-based interventions when control participants are ungrouped. Multivariate Behavioral Research, 43, Dimitrov, D. M., & Rumrill, P. D. (003). Pretest-posttest designs and measurement of change. Work, 0, Dziak, J. J., Nahum-Shani, I., & Collins, L. M. (01). Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations. Psychological Methods, 17: Frison, L., & Pocock, S. J. (199). Repeated measures in clinical trials: analysis using mean summary statistics and its implications for design. Statistics in Medicine, 11, Janega, J. B., Murray, D. M., Varnell, S. P., Blitstein, J. L., Birnbaum, A. S., & Lytle, L. A. (004). Assessing intervention effects in a school-based nutrition intervention trial: Which analytic model is most powerful? Health Education & Behavior, 31, Murray, D. M. (1998). Design and analysis of group-randomized trials (Vol. 9). Oxford University Press. Nahum-Shani, I., Dziak, J. J., and Collins, L. M. (016). Multilevel Factorial Designs With Experiment-Induced Clustering. Submitted to Psychological Methods. Oakes, J. M., & Feldman, H. A. (001). Statistical power for nonequivalent pretest-posttest designs. The impact of change-score versus ANCOVA models. Evaluation Review, 5, 3-8.

17 Methodology Center Tech Report Three-Level Modeling With EIC 17 Roberts, C., & Roberts, S. A. (005). Design and analysis of clinical trials with clustering effects due to treatment. Clinical Trials,, Vickers, A. J. (001). The use of percentage change from baseline as an outcome in a controlled trial is statistically inefficient: a simulation study. BMC Medical Research Methodology, 1:6. Accessed at Vickers, A. J., & Altman, D. G. (001). Analysing controlled trials with baseline and follow up measurements. BMJ, 33,