Moving Beyond the Average Treatment Effect: A Look at Power Analyses for Moderator Effects in Cluster Randomized Trials Jessaca Spybrook Western Michigan University December 3, 015 (Joint work with Ben Kelcey, University of Cincinnati, Nianbo Dong, University of Missouri) *This work was funded by a grant (DGE-143769) from the National Science Foundation.
Overview n Background n Purpose n -level CRT n 3-level CRT n Implications Slide
Background n What works? Slide 3
Background n What works? n Under what conditions? Slide 4
Background n What works? n Under what conditions? n For whom? Slide 5
Purpose n What works? n Under what conditions? n For whom? Slide 6
Purpose n Designs -level cluster randomized trial (-level CRT) 3-level cluster randomized trial (3-level CRT) Slide 7
-level CRT n Suppose a team of researchers are planning an impact study to determine the effectiveness of a new 5 th grade science curriculum. They are planning a -level CRT with students nested within schools and schools are randomly assigned to the new curriculum or the current curriculum. Slide 8
-level CRT what works? n What works? What is the effect of the new science curriculum relative to the current curriculum on science achievement for fifth graders? Slide 9
-level CRT what works? n Model yij = γ 00 + γ 01T j + r0 j + e ij e r ij 0 j ~ ~ N N ( 0, σ ) ( 0, τ ) Main effect of Treatment Slide 10
-level CRT what works? n Model yij = γ 00 + γ 01T j + r0 j + e ij e r ij 0 j ~ ~ N N ( 0, σ ) ( 0, τ ) Main effect of Treatment Set τ ρ = τ + σ δ = γ 01 τ + σ Slide 11
-level CRT what works? n Minimum detectable effect size (MDES) (Bloom, 1995) MDES = M J 4 ( ρ + ( 1 ρ) / n) J M J- is multiplier that approaches.8 as J increases, two-tail test with 0.05 level of significance J is total number of sites n is number of individuals per site ρ is the intraclass correlation Slide 1
-level CRT what works? n To increase precision Common practice to include a cluster-level covariate School-level pretest n Model: y + ij = γ 00 + γ 01T j + γ 0W j + r0 j e ij Main effect of Treatment Slide 13
-level CRT Question What works? Effect of Interest Main effect of treatment Main effect of treatment w/ covariate MDES ( ρ + ( 1 ρ) n) 4 / M J J M J 3 4 ( 1 ) ( (1 R ) ρ + ρ / n) W J Slide 14
-level CRT Minimum Detectable Effect Size What Works? Main Effect J=30 n=10 0.39 n=50 0.9 n=100 0.7 J=60 n=10 0.7 n=50 0.0 n=100 0.19 Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80, ρ = 0.3, RW = 0.75. Slide 15
-level CRT under what conditions? n Under what conditions? Is there a difference in the mean effects by school type (urban vs. suburban)? Slide 16
-level CRT under what conditions? n Model: y + ij = γ 00 + γ 01T j + γ 0S j + γ 03T js j + γ 04W j + r0 j e ij Cluster level moderator effect Slide 17
-level CRT Question What works? Effect of Interest Main effect of treatment MDES MDESD ( ρ + ( 1 ρ) n) 4 / M J J Main effect of treatment w/ covariate M J 3 4 ( 1 ) ( (1 R ) ρ + ρ / n) W J Under what conditions? Cluster-level moderator M J 5 16 ( 1 ) ( (1 R ) ρ + ρ / n) SW J Slide 18
-level CRT Minimum Detectable Effect Size J=30 J=60 What Works? Main Effect Under what conditions? CL Mod n=10 0.39 0.75 n=50 0.9 0.53 n=100 0.7 0.49 n=10 0.7 0.5 n=50 0.0 0.37 n=100 0.19 0.34 Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80, ρ = SW 0.3, R W = 0.75, R = 0.80. Slide 19
-level CRT for whom? n For whom? Is there a difference in the mean effect for boys and girls? Slide 0
-level CRT for whom? n Model: y + ij = γ 00 + γ 01T j + γ 10 X ij + γ 11T j X ij + r0 j e ij Individual level moderator effect Slide 1
-level CRT Question What works? Under what conditions? For whom? Effect of Interest Main effect of treatment Main effect of treatment w/ covariate Cluster-level moderator Individual-level moderator M MDES MDESD ( ρ + ( 1 ρ) n) 4 / M J J M M J 3 J 5 n* J ( J ) 4 16 ( 1 ) ( (1 R ) ρ + ρ / n) W J ( 1 ) ( (1 R ) ρ + ρ / n) 16 SW J ( 1 ρ) ( (1 R ) / n) X J Slide
-level CRT Minimum Detectable Effect Size J=30 J=60 What Works? Main Effect Under what conditions? CL Mod For whom? Ind Mod n=10 0.39 0.75 0.51 n=50 0.9 0.53 0.3 n=100 0.7 0.49 0.16 n=10 0.7 0.5 0.36 n=50 0.0 0.37 0.16 n=100 0.19 0.34 0.11 Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant n per cluster, power = 0.80, ρ = X 0.3, R W = 0.75, R SW = 0.80, and R = 0.0. Slide 3
3-level CRT n Suppose a team of researchers are planning an impact study to determine the effectiveness of a school-wide science curriculum. They are planning a 3-level CRT with students nested within teachers nested within schools and schools are randomly assigned to the new curriculum or the current curriculum. Slide 4
3-level CRT what works? n What works? What is the effect of the new science curriculum relative to the current curriculum on science achievement for all students? Slide 5
3-level CRT under what conditions? n Under what conditions? Is there a difference in the mean effects by school type (urban vs. suburban)? Is there a difference in the mean effects by teacher experience level (0-3 years vs. more than 3 years)? Slide 6
3-level CRT for whom? n For whom? Is there a difference in the mean effect for boys and girls? Slide 7
3-level CRT Question Effect of Interest What works? Main effect of treatment M K 3 MDES [ ] {( 1 R ) ρ + ρ + ( 1 ρ ρ )/ n / J} 4 W L3 L L3 L K Slide 8
3-level CRT Question Effect of Interest What works? Main effect of treatment M K 3 MDES MDESD [ ] {( 1 R ) ρ + ρ + ( 1 ρ ρ )/ n / J} 4 W L3 L L3 L K Under what conditions? Cluster-level moderator M K 5 [ ] {( 1 R ) ρ + ρ + ( 1 ρ ρ )/ n / J} 16 SW L3 L L3 L K Slide 9
3-level CRT Question Effect of Interest What works? Main effect of treatment M K 3 MDES MDESD [ ] {( 1 R ) ρ + ρ + ( 1 ρ ρ )/ n / J} 4 W L3 L L3 L K Under what conditions? Under what conditions? Cluster-level moderator Teacher-level moderator M K 5 M u [ ] {( 1 R ) ρ + ρ + ( 1 ρ ρ )/ n / J} 16 SW L3 L L3 L K {[ ( ) ( ) ] 1 R ρ + 1 ρ / n / J} 16 M L L3 ρl K Note: u=j*k-(k-) Slide 30
3-level CRT Question Effect of Interest What works? Main effect of treatment M K 3 MDES MDESD [ ] {( 1 R ) ρ + ρ + ( 1 ρ ρ )/ n / J} 4 W L3 L L3 L K Under what conditions? Under what conditions? For whom? Cluster-level moderator Teacher-level moderator Individuallevel moderator M K 5 M u M v {( 1 R ) ρ + ρ + ( 1 ρ ρ )/ n / J} {[ ( ) ( ) ] 1 R ρ + 1 ρ / n / J} 16 M L L3 ρl K {[ / n] / J} ( 1 R )( 1 ρ ) 16 X L3 ρl K [ ] 16 SW L3 L L3 L K Note: u=j*k-(k-), v=n*j*k-(j*k)-(k-) Slide 31
J=5 3-level CRT Minimum Detectable Effect Size J=30 What Works? Main Effect n=10 0.4 n=30 0. n=10 0.19 n=30 0.18 Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools, 0.15, ρ = 0.08, R = 0.75, R = 0.80, R = 0.30, and R ρ L3 = L W S P X = 0.0. Slide 3
3-level CRT Minimum Detectable Effect Size What Works? Main Effect Under what conditions? CL Mod J=5 n=10 0.4 0.45 n=30 0. 0.41 J=30 n=10 0.19 0.34 n=30 0.18 0.33 Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools, 0.15, ρ = 0.08, R = 0.75, R = 0.80, R = 0.30, and R ρ L3 = L W S P X = 0.0. Slide 33
3-level CRT Minimum Detectable Effect Size What Works? Main Effect Under what conditions? CL Mod Under what conditions? TL Mod J=5 n=10 0.4 0.45 0.9 n=30 0. 0.41 0.3 J=30 n=10 0.19 0.34 0.1 n=30 0.18 0.33 0.09 Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools, 0.15, ρ = 0.08, R = 0.75, R = 0.80, R = 0.30, and R ρ L3 = L W S P X = 0.0. Slide 34
3-level CRT Minimum Detectable Effect Size What Works? Main Effect Under what conditions? CL Mod Under what conditions? TL Mod For whom? Ind Mod J=5 n=10 0.4 0.45 0.9 0.18 n=30 0. 0.41 0.3 0.11 J=30 n=10 0.19 0.34 0.1 0.08 n=30 0.18 0.33 0.09 0.04 Assumptions: Two-tail significance =0.05, equal allocation of clusters, constant J per cluster, constant n per teacher, 40 schools, 0.15, ρ = 0.08, R = 0.75, R = 0.80, R = 0.30, and R ρ L3 = L W S P X = 0.0. Slide 35
Implications n Under what conditions? Cluster level moderator n Challenging given current study sizes n Effectiveness studies rather than efficacy studies n If priority, need to consider in design stage 60 60 60 30 S 30 U 40S 0U 40S 0U 15 T 15C 15 T 15C 0T 0C 10T 10C 10T 30C 0T 0C 30 T 30 C 30 T 30 C 30 T 30 C
Implications n Under what conditions? Teacher level moderator n Number of teachers per school matters n If priority, need to consider in design phase Slide 37
Implications n For whom? Individual level moderator n In many cases, reasonable given current study sizes n If priority, need to consider in design phase Slide 38
Implications n Extensions Multisite studies R code available jessaca.spybrook@wmich.edu Some parts implemented in PowerUP! - http://web.missouri.edu/~dongn/ Add to Optimal Design Plus Slide 39
Thank You! Questions? Please email me: jessaca.spybrook@wmich.edu Slide 40