The Power of the Test in Three-Level Designs

Transcription

1 DISCUSSION PAPER SERIES IZA DP No. 4 The Power of the Test in Three-Level Designs Spyros Konstantopoulos Otober 006 Forshungsinstitut zur Zukunft der Arbeit Institute for the Study of Labor

2 The Power of the Test in Three-Level Designs Spyros Konstantopoulos Northwestern University and IZA Bonn Disussion Paper No. 4 Otober 006 IZA P.O. Box Bonn Germany Phone: Fax: iza@iza.org Any opinions expressed here are those of the author(s) and not those of the institute. Researh disseminated by IZA may inlude views on poliy, but the institute itself takes no institutional poliy positions. The Institute for the Study of Labor (IZA) in Bonn is a loal and virtual international researh enter and a plae of ommuniation between siene, politis and business. IZA is an independent nonprofit ompany supported by Deutshe Post World Net. The enter is assoiated with the University of Bonn and offers a stimulating researh environment through its researh networks, researh support, and visitors and dotoral programs. IZA engages in (i) original and internationally ompetitive researh in all fields of labor eonomis, (ii) development of poliy onepts, and (iii) dissemination of researh results and onepts to the interested publi. IZA Disussion Papers often represent preliminary work and are irulated to enourage disussion. Citation of suh a paper should aount for its provisional harater. A revised version may be available diretly from the author.

3 IZA Disussion Paper No. 4 Otober 006 ABSTRACT The Power of the Test in Three-Level Designs * Field experiments that involve nested strutures may assign treatment onditions either to entire groups (suh as lassrooms or shools), or individuals within groups (suh as students). Sine field experiments involve lustering, key aspets of their design inlude knowledge of the intralass orrelation struture and the sample sizes neessary to ahieve adequate power to detet the treatment effet. This study provides methods for omputing power in three-level designs, where for example, students are nested within lassrooms and lassrooms are nested within shools. The power omputations take into aount lustering effets at the lassroom and at the shool level, sample size effets (e.g., number of students, lassrooms, and shools), and ovariate effets (e.g., pre-treatment measures). The methods are generalizable to quasi-experimental studies that examine group differenes in an outome, or assoiations between preditors and outomes. JEL Classifiation: C9 Keywords: nested designs, statistial power, randomized trials, treatment effets, random effets, experiments, lustering Corresponding author: Spyros Konstantopoulos Shool of Eduation and Soial Poliy Northwestern University 0 Campus Drive Evanston, IL 6008 USA spyros@northwestern.edu * This material is based upon work supported by the National Siene Foundation under Grant No Any opinions, findings, and onlusions or reommendations expressed in this material are those of the author and do not neessarily reflet the views of the National Siene Foundation. I thank Larry Hedges for his valuable omments.

4 Many populations of interest in psyhology, eduation, and the soial sienes exhibit multilevel struture. For example, students are nested within lassrooms and lassrooms are nested within shools, employees are nested within departments, whih are nested within firms, individuals are nested within neighborhoods, whih are nested within ities. Beause individuals within aggregate units are often more alike than individuals in different units, this nested struture indues an intralass orrelation struture (often alled lustering in the sampling literature, see, e.g., Kish, 965) that needs to be taken into aount. In partiular, experiments that involve populations with nested strutures must take this struture into aount in both experimental design and analysis. Field experiments that involve nested population strutures, may assign treatment onditions either to individuals or to entire groups (suh as lassrooms, or shools). Beause preexisting aggregate groups often exhibit statistial lustering, designs that assign intat groups to treatment are often alled luster randomized or group randomized designs. Treatments are sometimes assigned to groups beause the treatment is naturally administered to intat groups (suh as a urriulum to a lassroom or a management system to a firm). In other ases, the assignment of treatments to groups is a matter of onveniene, being muh easier to implement than assignment to individuals. Analyti strategies for the analysis of designs involving nesting have long been available (see, e.g., Kirk, 995 and referenes therein). Suh designs involve nested fators (or bloks), whih are often onsidered to have random effets. Cluster randomized trials have also been used extensively in publi health and other areas of prevention siene and mediine (see, e.g., Donner and Klar, 000; and Murray, 998). The use of luster randomized experiments has inreased reently in eduational researh, mainly beause of the inreased interest in experiments to evaluate eduational interventions (see, e.g., Mosteller and 4

5 Boruh, 00). An example of luster randomized trials in eduation is Projet STAR, a large sale randomized experiment, where within eah shool students were randomly assigned to lassrooms of different sizes (see Nye, Hedges, and Konstantopoulos, 000). One of the most ritial issues in designing experiments is to ensure that the design is sensitive enough to detet the intervention effets that are expeted if the researhers hypotheses were orret. In other words, a ritial task in planning experimental studies involves making deisions about sample sizes to ensure suffiient statistial power of the test for the treatment effet. Power is defined as the probability of deteting a treatment effet when it exists. There is an extensive literature on the omputation of statistial power, (e.g., Cohen, 988; Kraemer & Thiemann, 987; Lipsey, 990; Murphy & Myors, 004). Muh of this literature involves the omputation of power in studies that use simple random samples and hene ignore lustering effets. Reently, software for omputing statistial power in single-level designs has beome widely available (Borenstein, Rothstein, & Cohen, 00). However, the omputation of statistial power in designs that involve lustering entails two signifiant hallenges. First, sine nested fators are usually taken to have random effets, power omputations usually involve variane omponents strutures (often expressed via intralass orrelations) of those random effets. Seond, there is not one sample size, but sample sizes at eah level of nesting, and these sample sizes at different levels of nesting will affet power differently. For example, in eduation, the power of the test used to detet a treatment effet depends not only on the sample size of the students within a lassroom or a shool, but on the sample size of lassrooms or shools as well. This design involves nesting at the shool or at the lassroom level, and is alled a two-level design. The last deade the development of advaned statistial methods that aount for lustering has equipped soial siene researhers 5

6 with appropriate tools for handling data with nested strutures (Goldstein, 003; Raudenbush & Bryk, 00). In addition, statistial theory for omputing power in two-level designs has been well doumented and statistial software for two-level designs is urrently available (e.g., Raudenbush, 997; Raudenbush & Bryk, 00; Raudenbush & Liu, 000, 00; Snijders & Bosker, 993, 999). The work on power analysis for nested designs with random effets has foused thus far solely on designs with two levels of lustering, for example students within shools, or repeated measurements over time within individuals (see Raudenbush & Liu, 000, 00). However, designs and data have often more ompliated strutures that may involve three levels of nesting. For example, in eduation, students are nested within lassrooms, and lassrooms are nested within shools. In mediine, patients are nested within wards, and wards are nested within hospitals. These examples are three-level designs where nesting ours naturally at two levels (lassrooms and shools) regardless of whether both soures of lustering are taken into aount in the design and the analysis part of the study. For example, an eduational researher may hoose to ignore one level of lustering and ondut an experimental study in eduation following a two-level design (students nested within lassrooms, or students nested within shools), and analyze the data obtained using two-level models. However, a more appropriate design and analysis would involve three levels, sine lustering effets exist naturally at the lassroom and at the shool level. Ignoring the three-level struture will result in an inaurate estimate of power of the test for the treatment effet in the design stage, and an inorret estimate of the standard error of the treatment effet in the analysis. In three-level designs the omputation of power is even more omplex than in two-level designs, sine there are lustering effets at the seond or middle level (e.g., lassrooms) and lustering effets at the third of top 6

7 level (e.g., shools). The power in these designs is, among other things, a funtion of three different sample sizes: the number of students within a lassroom, the number of lassrooms within a shool, and the overall number of shools that partiipate in the experiment. The power is also a funtion of two different intralass orrelations (at the shool and at the lassroom level) and the effet size (e.g., the magnitude of the treatment effet). Thus far, methods for omputing power and sample size requirements in experiments that involve three-level designs are not available. In addition, the analyses of nested data do not always involve two levels. For example, it is not unommon in eduation to ondut analyses assuming three-level models (e.g., Bryk & Raudenbush, 988; Nye, Konstantopoulos, & Hedges, 004). It is ruial that the design and analyses of experimental studies are in ongruene, sine ignoring or inluding lustering effets will impat power differently. Analyses that ignore lustering effets provide inorret estimates of the standard errors of the regression oeffiients. Findings from previous work that disuss power in two-level designs (e.g., Hedges & Hedberg, 006; Raudenbush, & Liu, 000) indiate that the proportion of the variane in the outome between shools or lusters is inversely related to the power of the test for the treatment effet. In other words, the larger the lustering effet, the lower the power of the test for the treatment effet. Now onsider the ase where one more level is added to the hierarhy (e.g., lassrooms within shools). If we assume that the lustering effet at the lassroom level is different than zero, and the lustering effet at the shool level is virtually unhanged, then the total lustering effets are larger than in two-level designs and hene the power has to be smaller. This paper provides methods that failitate the omputation of statistial power of tests for treatment effets in three-level designs. We provide examples drawn from the field of 7

8 eduation to illustrate the power omputations. The remaining of the paper is strutured as follows. First, we review the effets of lustering on power. Seond, we outline the three-level designs that will be disussed, and we explain the notation we use throughout the paper. Then, we present methods and examples for omputing power separately for eah design. Finally, we summarize the usefulness of the methods and draw onlusions. Clustering in Three-Level Designs Previous methods for power analysis in two-level designs involved the omputation of the non-entrality parameter of the non-entral F-distribution (Raudenbush & Liu, 000). The power is a funtion of the non-entrality parameter and higher values of this parameter orrespond to higher values of statistial power. The non-entrality parameter in turn is a funtion of the lustering effet, whih is typially expressed as an intralass orrelation. For example, suppose that in a two-level design the total variane of the outome in a population with nested struture (students nested within shools) is σ T. The total variane is deomposed into a between-shool variane ω and a within-shool variane σ e, so that σ = σ + ω. Then T e ρ = ω σ is the intralass orrelation and indiates the proportion of the variane in the / T outome between shools. The same logi holds for three-level designs. The power is a funtion of the nonentrality parameter, whih is a funtion of the lustering effets. The only differene is that lustering in three-level designs ours in more than one level. Consider a three-level struture where students are nested within lassrooms, and lassrooms are nested within shools. The total variane in the outome is now deomposed into three varianes: the between-student-withinlassroom and shool variane, σ e, the between-lassroom-within-shool variane, τ, and the 8

9 between-shool variane, ω. Then, the total variane in the outome is defined as σ σ τ ω T = e + +. Hene, in three-level designs we define two intralass orrelations: τ ρ = σ T () at the lassroom level and ω ρs = σ T () at the shool level. These intralass orrelations indiate the effets of lustering at the lassroom and at the shool level respetively. The subsripts and s indiate the levels of the hierarhy (e.g., lassroom or shool). The omputation of statistial power in luster randomized trials requires knowledge of the intralass orrelations. One way to obtain information about reasonable values of intralass orrelations is to ompute these values from luster randomized trials that have been already onduted. Murray and Blitstein (003) reported a summary of intralass orrelations obtained from 7 artiles reporting luster randomized trials in psyhology and publi health and Murray, Varnell, and Blitstein (004) give referenes to 4 very reent studies that provide data on intralass orrelations for health related outomes. Another strategy is to analyze sample surveys that have used a luster sampling design. Gulliford, Ukoumunne, and Chinn (999) and Verma and Lee (996) presented values of intralass orrelations based on surveys of health outomes. 9

10 A reent study provided plausible values of lustering for eduational outomes using reent large-sale studies that surveyed national probability samples of elementary and seondary students in Ameria (Hedges & Hedberg, 006). The present study uses intralass orrelation values that are reported in Hedges and Hedberg (006). Three-Level Designs In this study we use examples from eduation to illustrate the nested struture of the designs. Hene, we use students, lassrooms, and shools to define the three levels of the hierarhy. We make use of the terminology of eduation beause it makes the differentiation between smaller and larger units, and the hierarhial struture more obvious. We also assume that the outome of interest is aademi ahievement. Nonetheless, the designs and methods disussed here an be used for any hierarhial design with three levels of nesting and it is not restrited to eduational designs or data. In two-level designs either students within shools or entire shools are randomly assigned to one of two or more treatment onditions. This indiates that random assignment an our either at the individual or at the shool level. In three-level designs however, the nested struture is more omplex, sine either students within lassrooms, lassrooms within shools, or entire shools an be randomly assigned to treatment onditions. This study disusses three-level designs where randomization an our at any level: student, lassroom, or shool. In the first design the random assignment is at the shool level (the largest unit), that is, shools are assigned to a treatment and a ontrol group. In the seond design the random assignment is that lassroom level (the seond largest unit), that is, within eah shool lassrooms are randomly assigned to a treatment and a ontrol group. In the third design the random assignment is at the student level (the smallest unit), that is, within eah shool and lassroom students are random assigned to a 0

11 treatment and a ontrol group. In eah design we illustrate the general ase, whih inludes ovariates at all levels of the hierarhy, and the restrited ase where ovariates are not inluded in the model. For simpliity, in eah design we assume that there is one treatment and one ontrol group. We also assume that the designs are balaned. In the first design, entire shools are randomly assigned to two groups (treatment and ontrol). We represent the number of shools within eah group (treatment or ontrol) by m, the number of lassrooms within a shool by p, and the number of students within eah lassroom by n. Then, the sample size for the treatment and the ontrol groups is N = N = mpn and the total sample size is N = N + N = mpn. In t the seond design, lassrooms within shools are assigned to treatment and ontrol onditions. In this ase we represent the total number of shools aross groups by m, the number of lassrooms within eah group (treatment or ontrol) within eah shool by p, and the number of students within eah lassroom by n. The sample size for the treatment and the ontrol groups is again N = N = mpn and the total sample size is N = N + N = mpn. In the third design, students t within lassrooms and shools are assigned to treatment and ontrol onditions. In this ase we represent the total number of shools aross groups by m, the number of lassrooms within eah shool by p, and the number of students within eah group (treatment or ontrol) within eah lassroom by n. The sample size for the treatment and the ontrol groups is again N = N = mpn and the total sample size is N = N + N = mpn. In addition, in eah design, we t onsider the ase where q shool-level ovariates, w lassroom-level ovariates, and r studentlevel ovariates are inluded in the analysis. This indiates that in design one for example q (0 q < (m )) shool-level ovariates, w (0 w < mp q ) lassroom-level ovariates, and r (0 r < mpn q w - ) student-level ovariates an be inluded in the analysis. We assume t t t

12 that the ovariates at the student and at the lassroom level are entered around their lassroom and shool means respetively (group-mean entering). This ensures that preditors explain variation in the outome only at the level at whih they are introdued. Sine inluding ovariates in the model will adjust the intralass orrelations and the treatment effet we use the subsript A to indiate adjustment due to ovariates. We also use the subsript R to indiate residual variation. Defining the Effet Size Parameter In power analysis it is ommon to define an effet size parameter (or standardized mean differene) that is independent of sample size or the sale of the outome to examine whether the null hypothesis is true or not. The effet size typially used is Cohen s d (δ for a population parameter), and is expressed in standard deviation units, namely δ = (α α )/σ where α and α are the treatment effet parameters from the usual analysis of variane (ANOVA) model (defined preisely below) and σ is the population standard deviation. In luster randomized experiments, there are several possible hoies for the definition of σ sine there are multiple standard deviations (e.g., σ T, ω, τ, and σ e ) whih ould be used for the standardization. In aademi ahievement data, typially, most of the variation in ahievement is typially between students within lassrooms, and hene the total variation in ahievement, σ T, is loser to the error variane in ahievement at the student level, σ e. The between lassroom variation, τ, and the between shool variation, ω, in ahievement are typially muh smaller than σ T. For example, let s assume that 70 perent of the variation is between students within lassrooms, 5 perent between lassrooms within shools, and 5 perent between shools (roughly these are estimates from projet STAR). Then, the student-level error term variane is σ = 0.84σ, and the e T

13 varianes of the shool and lassroom random effets are ω =τ =0.39σ. In this study, we assume that ahievement is the potential outome and use the total standard deviation in ahievement to standardize the mean differene in ahievement between the two groups. The standard deviation σ T has the advantage that it is the total standard deviation in the population. Hene, we define the effet size parameter as T α - α δ =. (3) σ T We follow Cohen s (988) rules of thump about what onstitutes a small and a medium effet size. For example, Cohen onsidered effet sizes of 0.0 (the treatment effet is /5 of a standard deviation) and 0.50 (the treatment effet is ½ of a standard deviation) as small and medium respetively. In this study we ompute power for small and medium effet sizes. We fous on the power of treatment ontrasts, not omnibus (multiple-degree of freedom) treatment effets, beause in our experiene, multilevel designs are hosen to ensure the power of partiular treatment ontrasts. Even when several treatments are being ompared, there is typially one ontrast that is most important and the design is hosen to ensure adequate sensitivity for that ontrast. Design I: Treatment is Assigned at the Shool Level In this design, shools are nested within treatment, and lassrooms are nested within shools and treatment (see Kirk, 995, p. 488). In the disussion that follows, we assume that both shools and lassrooms are random effets. 3

14 A strutural model for a student outome Y shool in the i th treatment an be desribed in ANCOVA notation as ijkl, the l th student in the k th lassroom in the j th Y = µ+α + θ X + θ Z + θ Ψ + β + γ + ε, (4) T T T ijkl Ai I ijkl C ijk S ij A(i)j A(ij)k A ijk l ( ) where µ is the grand mean, α Ai is the (fixed) effet of the i th T treatment (i =,), θ I = (θ I,, θ Ir ) T is a row vetor of r individual-level ovariate effets, θ C = (θ C,, θ Cw ) is a row vetor of w T lassroom-level ovariate effets, θ S = (θ S,, θ Sq ) is a row vetor of q shool-level ovariate effets, X ijkl is a olumn vetor of r lassroom mean-entered individual-level ovariates (e.g., student harateristis) in the k th lassroom in the j th shool in the i th treatment, Z ijk is a olumn vetor of w shool mean-entered lassroom-level ovariates (e.g., lassroom or teaher harateristis) in the j th shool in the i th treatment, Ψ ij is a olumn vetor of q shool level ovariates (e.g., shool harateristis) in the i th treatment, and the last three terms are shool, lassroom, and student random effets respetively. Speifially, β Ai () j is the random effet of shool j (j =,,m) within treatment i, γ Aijk ( ) is the random effet of lassroom k (k =,,p) within shool j within treatment i, and ε Aijk ( ) is the error term of student l (l =,,n) within l lassroom k, within shool j, within treatment i. The subsript A indiates adjustment due to ovariate effets. The treatment effet may be adjusted by the effets of the ovariates. In priniple however, assuming randomization works, the treatment effet is orthogonal to the ovariates and the error term, and the adjustment should be zero. The lassroom and shool random effets are adjusted by lassroom and shool-level ovariates respetively and the student error term is adjusted by student-level ovariates. We assume that the adjusted student 4

15 error term as well as the adjusted lassroom and shool random effets are normally distributed with a mean of zero and residual varianes σ, τ and ω respetively. Re R, In a multi-level framework the above ANCOVA model an be expressed as R Y = u + u X + e, T jkl 0 jk rjk rjkl Ajkl the level two model for the interept is u = π + π Z + ζ 0 T 0 jk 00 j 0wj wjk A jk, and the level three model for the interept is π δ δ ξ T 00 j = A00TREATMENT + δ00q Ψ qj + A00 j where TREATMENT i is a dummy variable (treatment is, otherwise zero), ζ A0jk is the lassroom random effet adjusted by the effets of the lassroom preditors Z, ξ A00k is the shool random effet adjusted by the effets of the shool preditors Ψ, and e jkl is a student error term adjusted by the effets of the student preditors X. The student level ovariates are treated as fixed, namely u = π, π = δ. Similarly the lassroom level ovariates are treated as fixed, rjk rwj rwj rwq namely π 0wj δ0 wq. = The student residual as well as the adjusted lassroom, and shool random effets are normally distributed with a mean of zero and residual varianes Re,, R σ τ and ω R respetively. With the appropriate onstraints on the ANCOVA model (i.e., setting α i = 0 for the 5

16 ontrol group), these two models are idential and there is a one to one orrespondene between the parameters and the random effets in the two models. That is, µ = δ000, α = δ A00, θ I = δ rwq, θ C = δ0wq, θs δ00q =, β A = ξ A, γ = A ζ A, and ε A = ea. Indexes of Clustering: The Adjusted Intralass Correlation When ovariates are inluded in the model the four varianes of the outome are defined σ Re, τr, ω R σ σ = as, and + + ω RT, where RT σre τ R R. The subsript R indiates residual varianes (smaller in magnitude beause of the adjustment of the ovariates). There are two parameters that summarize the assoiations between these four varianes and indiate the lustering effets at the lassroom and at the shool level: the adjusted intralass orrelation at the lassroom level R RT ρ = τ A (5) σ and the adjusted intralass orrelation at the shool level ρ As ω = σ R RT, (6) where the subsript A indiates that the intralass orrelations are adjusted for the effets of the ovariates and are omputed using the residual varianes (subsript R). The unadjusted intralass orrelations indiate the effets of lustering, while the adjusted intralass orrelations indiate the remaining effets of lustering, net of the effets of ovariates. 6

17 Estimation In balaned designs (that is equal sample sizes for the experimental and the ontrol group) the estimate of the treatment effet is the differene in the means between the treatment and the ontrol group namely A effet is defined as Y... YA.... The standard error of the estimate of the treatment SE ( Y A Y ) = A pn R n R Re RT ( pn ) As ( n ) mpn + + = mpn ω τ σ σ ρ ρ A, where m is the number of shools within eah ondition, p is the number of lassrooms within eah shool, n is the number of students within eah lassroom, and all other terms have been defined previously. Hypothesis Testing The objetive is to examine the statistial signifiane of the treatment effet net of the possible effets of ovariates, whih means to test the hypothesis H 0 : α = A α A or α A α A = 0 or equivalently H 0 : δ A 00 = 0. Suppose that the researher wishes to test the hypothesis and arries out the usual t-test. This involves omputing the test statisti t A = mpn Y S ( A... Y A... ) A, 7

18 where S A is defined by S A = m m ( YAi ii YAiii ) + ( YAiii YAiii ) pn i= i= m q, and Y Aijii is the adjusted mean of the j th shool in the i th treatment group, and Y Aiiii is the adjusted mean of the i th treatment group. When the null hypothesis is true, the test statisti t has a Student s t-distribution with m-q- degrees of freedom. Computing Power When the null hypothesis is false, the test statisti t A has the non-entral t-distribution with m-q- degrees of freedom and non-entrality parameter λ A. The non-entrality parameter is defined as the expeted value of the estimate of the treatment effet divided by the square root of the variane of the estimate of the treatment effet namely αa αa mpn λ A =. σ + ρ + ρ ( pn ) ( n ) RT As A Notie that the new adjusted effet size parameter may not be known when omputing the power. Typially, the effet size estimate that is expressed in units of the unadjusted standard deviation, that is α α / σ T, is more likely to be known than α α σ A A / RT A A. In priniple, in randomized trials inluding ovariates should not adjust the treatment effet estimate sine the treatment is orthogonal to all ovariates. This means that α α = α α. However that the 8

19 standard deviation (the denominator) hanges when ovariates are inluded, and is now a residual standard deviation. This means that the effet size is not the same as the unadjusted effet size in equation 3. The ovariates will adjust the varianes at eah level and ultimately the total variane. Hene, we express the non-entrality parameter as a funtion of the unadjusted effet size and the unadjusted intralass orrelations, namely mpn α α σt λa = = σt σrt + ( pn ) ρas + ( n ) ρa (7) mpn = δ. η + η η ρ + η η ρ ( pn ) ( n ) e s e s e where ηs = ωr / ω, η = τr / τ, ηe = σre / σe. (8) The η s indiate the proportion of the varianes at eah level of the hierarhy that is still unexplained (perentage of residual variation). For example when η e = 0.5, this indiates that the variane at the student level dereased by 75 perent due to the inlusion of ovariates suh as pre-treatment measures. In the ase where no ovariates are inluded the non-entrality parameter redues to mpn λ = δ, + + ρ ( pn ) ρ ( n ) s (9) 9

20 sine η = η = η =. e s When the lustering at the lassroom level, ρ, is zero, then equations 7 and 9 redue to the non-entrality parameter used to ompute power in two-level designs. For example, when the lustering at the lassroom level is zero, ρ = 0, the two-level design involves students nested within shools where the treatment is assigned at the shool level. The equations 7 and 9 indiate that the non-entrality parameter of the t-test is a funtion of the number of students, lassrooms, shools, and the intralass orrelations at the lassroom and at the shool level. Power is inversely related to the lustering effets, that is the larger the intralass orrelations the smaller the non-entrality parameter, and the power. The fator ( pn ) ρ ( n ) + + ρ in equation 9 an be thought of as a design effet sine it reflets the s fator by whih the variane of the mean differene from a lustered sample exeeds the variane omputed from a simple random sample of the same total sample size. Notie that when the intralass orrelations are zero (no lustering) equation 9 redues to λ = mpn / δ and this holds for all three designs onsidered in this study. The power of the one-tailed t test at level α is p = Η[(α, m-q-), (m-q-), λ A ], where (α, ν) is the level α one-tailed ritial value of the t-distribution with ν degrees of freedom [e.g., (0.05,0) =.7], and Η(x, ν, λ) is the umulative distribution funtion of the non-entral t-distribution with ν degrees of freedom and non-entrality parameter λ A. The power of the two-tailed test at level α is p = Η [(α/, m-q-), (m-q-), λ A ] + Η [-(α/, m-q-), (m-q-), λ A ]. In the ase of no ovariates (that is q, w, r = 0) the degrees of freedom for the t-test are slightly hanged, and so is the non-entrality parameter λ A = λ. Alternatively and equivalently, the test for the treatment effet and statistial power an be omputed using the F-statisti. In this ase the F-statisti has a non-entral F-distribution with 0

21 degree of freedom in the numerator and m q degrees of freedom in the denominator and non-entrality parameter λ A or λ. How Does Power Depend on Student, Classroom, and Shool Units? In three-level designs, the number of units at different levels of the hierarhy will have different effets on power. In two-level designs we know that the number of shools have a larger impat on power than the number of students within shools (see Raudenbush & Liu, 000). Similarly, in three-level designs shools, lassrooms, and students will impat power differently. One way to examine this impat is to ompute the non-entrality parameter when the number of units at eah level of the hierarhy gets infinitely large. The power is a diret funtion of the non-entrality parameter, and hene, other things being equal, when the non-entrality parameter onverges to a real number, the power is smaller than one, and when the nonentrality parameter gets infinitely large the power approahes one. We illustrate the effet of the number of students, lassrooms, and shools on statistial power in figures to 3. Figure illustrates that, as the number of students in eah lassroom beomes larger, power inreases, but then levels off at a value that is smaller than one. This suggests that for a fixed number of shools and lassrooms, inreasing student sample size beyond a ertain point has only a small effet on statistial power. In fat, as the number of students in eah lass beomes indefinitely large, the value of the non-entrality parameter tends to λ = mp δ ( max) st ( pη ρ + η ρ ) s s and in the ase of no ovariates to

22 λ = mp δ ( max) st ( pρ + ρ ) s. Therefore the maximum power for a design with m shools per ondition and p lassrooms per shool is the power assoiated with λ ( max )st, whih is less than one. Figure illustrates power omputations for one tailed t-tests at the 0.05 level for small and medium effet sizes (0. and 0.5 SD) with and without ovariates as a funtion of the number of students per lassroom holding onstant the number of lassrooms per shool and the number of shools per ondition (p =, m = 8). Figure illustrates that, as the number of lassrooms in eah shool beomes larger, power inreases, but then levels off at a value that is smaller than one. This suggests that for a fixed number of shools and students in eah lassroom, inreasing lassroom sample size beyond a ertain point has a small effet on statistial power. In fat, as the number of lassrooms in eah shool beomes indefinitely large, the value of the non-entrality parameter tends to λ ( max) = mn δ nη ρ s s and in the ase of no ovariates to λ ( max) = mn δ nρ. s

23 Therefore the maximum power for a design with m shools per ondition and n students in eah lassroom is the power assoiated with λ ( max), whih is less than one. Figure illustrates power omputations for one tailed t-tests at the 0.05 level for small and medium effet sizes (0. and 0.5 SD) with and without ovariates as a funtion of the number of lassrooms per shool holding onstant the number of students per lassroom and the number of shools per ondition (n = 0, m = 8). Figure 3 illustrates that, as the number of shools beomes larger, power inreases dramatially, and approahes one. This suggests that for a fixed number of lassrooms in eah shool and students in eah lassroom, inreasing shool sample size has a substantial effet on statistial power. In fat, as the number of shools beomes indefinitely large, the value of the non-entrality tends to infinity λ ( max) s. Therefore the maximum power for a design with p lassrooms per shool and n students in eah lassroom is the power assoiated with λ ( max )s, whih tends to one. This indiates that the number of shools impats power muh more than the number of lassrooms and the number of students. Notie that the number of shools influenes power via the degrees of freedom of the test statisti as well, and this holds for all three designs. Corresponding results hold for two-level designs (e.g., students are nested within shools, and shools are assigned to treatments). The power again depends muh more on the number of shools than the number of students in eah shool. Raudenbush and Liu (000) have demonstrated empirially that for a fixed number of shools, the power did not tend to one when the number of students beomes large. In ontrast, for a fixed number of students in eah shool, the power tends to one when the number of shools beomes large. Figure 3 illustrates power omputations for one tailed t-tests at the 0.05 level for small and medium effet sizes (0. and 3

24 0.5 SD) with and without ovariates as a funtion of the number of shools holding onstant the number of students per lassroom and the number of lassrooms per shool (n = 0, p = ). To selet plausible values of lustering at the shool level, we follow the findings of a reent study that omputed a large amount of intralass orrelations in two-level designs (where students are nested within shools) using reent large-sale studies that surveyed national probability samples of elementary and seondary students in Ameria (Hedges & Hedberg, 006). The results indiated that most of the shool-level intralass orrelations ranged between 0. and 0.. Evidene from two-level analysis (where students are nested within shools) of the National Assessment of Eduational Progress (NAEP) trend data, and Projet STAR data also points to shool-level intralass orrelations between 0. and 0.. Hene, we ompute power using two values for the intralass orrelation at the shool level: 0. and 0.. In addition, evidene from NAEP main assessment and Projet STAR using three-level models (where students are nested within lassrooms and lassrooms are nested within shools) indiate that the lustering at the lassroom level is nearly /3 as large as the lustering at the shool level. Hene, we ompute power using two values for the intralass orrelation at the lassroom level: and Inluding ovariates also influenes power. Speifially, the larger the proportion of the variation explained at the student, lassroom, and shool level, the larger the value of the nonentrality parameter and in turn the higher the power. Again, following Hedges and Hedberg (006) we assume that five ovariates are inluded at eah level of the hierarhy (e.g, q = w = r = 5), and that the ovariates explain 50 perent of the variation in ahievement at eah level. Hene we ompute power for η = η = η =, and for η = η = η = 0 5. The power e s e s. omputations illustrated are for one-tailed tests for all three designs disussed. It is plausible to 4

25 hypothesize that the treatment effet is positive and that the mean outome of the treatment group will be larger than the mean outome of the ontrol group (one-diretional researh hypothesis). The power omputations of two-tailed tests yield smaller values of power. The following patterns are onsistent in figures to 3. First, the power inreases as the effet size inreases, other things being equal. For example, onsider a design with a total sample size of 640, where there are 0 students per lassroom, two lassrooms per shool, and eight shools per treatment ondition. When no ovariates are inluded, the lustering effets of shools and lassrooms are respetively ρ s = 0. and ρ = 0.067, and the effet size is δ = 0. SD, the power of a one-tailed test is 0.6, and nearly four times larger, 0.66, when the effet size is δ = 0.5 SD. Seond, the power dereases as the intralass orrelations inrease, other things being equal. In the previous example, when the effet size is δ = 0.5 SD, the intralass orrelations at the shool and lassroom level are respetively ρ s = 0. and ρ = 0.34, and all else is equal, the power of a one-tailed test is 0.4 (an absolute derease of 4 perent from 0.66). Third, the number of lassrooms has a larger impat on power than the number of students, other things being equal. For example, when the total number of shools is 6, there are eight lassrooms per shool and five students per lassroom, no ovariates are inluded, the intralass orrelations at the shool and at the lassroom level are respetively ρ s = 0. and ρ = 0.34, and the effet size is δ = 0.5 SD, the power of a one-tailed test is This indiates a relative inrease in power of 7 perent (from 0.4 to 0.49). Fourth, the number of shools influenes power muh more than the number of lassrooms and the number of students. For example, when the total number of shools is 3, there are two lassrooms per shool, 0 students per lassroom, no ovariates are inluded, the intralass orrelations at the shool and at the lassroom level are respetively ρ s = 0. and ρ = 5

26 0.34, and the effet size is δ = 0.5 SD, the power of a one-tailed test is 0.70 (a 67 perent relative inrease from 0.4 to 0.70). When the total number of shools is 3, there is one lassroom per shool, 0 students per lassroom, no ovariates are inluded, the intralass orrelations at the shool and at the lassroom level are respetively ρ s = 0. and ρ = 0.34, and the effet size is δ = 0.5 SD, the power of a one-tailed test is 0.6 (nearly a 50 perent relative inrease from 0.4 to 0.6). When the total number of shools is, there are two lassrooms per shool, 0 students per lassroom, and the lustering effets for lassrooms and shools are respetively ρ s = 0. and ρ = 0.067, the power of a one-tailed test is just above 0.80, the threshold established by Cohen (99). Fifth, as one would expet, the power is higher when ovariates are inluded in the model. For example, when the total number of shools is 6, there are two lassrooms per shool, 0 students per lassroom, five ovariates are inluded at eah level, 50 perent of the variane is explained by the ovariates at eah level, the intralass orrelations at the shool and at the lassroom level are respetively ρ s = 0. and ρ = 0.067, and the effet size is δ = 0.5 SD, the power of a one-tailed test is This indiates a 35 perent relative inrease in power from 0.66 (no ovariates) to Overall, the larger the proportion of variane explained at eah level, the higher the power, other things being equal. Also, the smaller the number of ovariates at the shool level the larger the power, other things being equal Insert Figures to 3 About Here

27 Design II: Treatment is Assigned at the Classroom Level In this design, shools and treatments are rossed, and lassrooms are nested within treatments and shools (see Kirk, 995, p. 49). Within eah shool, lassrooms are randomly assigned to a treatment and a ontrol group. Sine lassrooms within shools are assigned to treatment and ontrol groups, p in this design is the number of lassrooms in eah ondition in eah shool. In the disussion that follows, we assume that shools, lassrooms, and the treatment by shool interation are random effets. The strutural model in ANCOVA notation is Y = µ+α + θ X + θ Z + θ Ψ + β + αβ + γ + ε, (0) T T T ijkl Ai I ijkl C ijk S j Aj Aij A(ij)k A ijk l ( ) where the last four terms represent shool, treatment by shool, lassroom, and student random effets respetively and all other terms are as defined previously. The new term, αβ, is the treatment by shool random effet and follows a normal distribution with a mean of zero and a variane ω Rt (the subsript t indiates treatment). In a multi-level framework the model beomes Aij Y = u + u X + e, T jkl 0 jk rjk rjkl Ajkl the level two model for the interept is u = π + π Treatment + π Z + ζ 0, T 0 jk 00 j A0j jk 0wj wjk A jk 7

28 and the level three model for the interept and the treatment effet is π = δ + δ Ψ + ξ T 00 j q qj A00 j π = δ + δ Ψ + ξ. T A0 j A00 0q qj A0 j where ξ A0 j is the treatment by shool random effet, and all other parameters are as defined in the first design. All ovariates are treated as fixed as in the first design. Estimation As in the first design, the estimate of the treatment effet is the differene in the means Y... between the treatment and the ontrol group, A YA of the treatment effet in the seond design is defined as.... and the standard error of the estimate SE ( Y A Y ) = A pn Rt n R Re RT ( pn Rs ) As ( n ) A mpn + + = mpn ω τ σ σ ϑ ρ ρ, where ω Rt is the treatment by shool random effet residual variane, and ϑ = ω ω is the Rs Rt / R proportion of the treatment by shool random effet residual variane to the total between-shool ϑ Rs residual variane and 0, m is the total number of shools, p is the number of lassrooms within eah ondition in eah shool, n is the number of students within eah lassroom, and all other terms have been defined previously. Hypothesis Testing As previously we test the hypothesis 8

29 H 0 : α = A α A or α A α A = 0 or equivalently H 0 : δ A00 = 0. In this design, an exat test does not exist. However, two modified tests for the treatment effet an be omputed (see Kirk, 995). First, when the null hypothesis is true and the variane of the treatment by shool random effet ( αβ A jij ) is zero, the test statisti t that examines the signifiane of the treatment effet has a Student s t-distribution with m(p-)-w degrees of freedom. Seond, when the null hypothesis is true and the variane of the lassroom random effet ( γ A(ij)k ) is zero, the test statisti t that examines the signifiane of the treatment effet has a Student s t-distribution with m-q- degrees of freedom. However, the restritions in both tests are quite strong. That is, even if the treatment effet is onsistent aross shools and the variane of the treatment by shool interation is not signifiant, it does not mean that it is exatly zero. Similarly, it is highly unlikely that the variane of the lassroom random effet is exatly zero. Hene an approximate t-test is employed instead within the maximum likelihood estimation framework. This t-test involves dividing the estimate of the treatment effet by its standard error. When the null hypothesis is true, this ratio has approximately a t-distribution with m-q- degrees of freedom. Computing Power () When the null hypothesis is false the test statisti that examines the signifiane of the treatment effet has approximately a non-entral t-distribution with m-q- degrees of freedom and a non-entrality parameter λ A. As in design one, we express the non-entrality parameter as a funtion of the unadjusted intralass orrelations and the unadjusted effet size, namely, 9

30 λ A = mpn δ ηe + ( pnϑrsηs ηe) ρs + ( nη ηe) ρ. () The adjusted intralass orrelation at the shool level is defined as ρ = ω / σ = ω / ϑ σ, As Rs RT Rt Rs RT the adjusted intralass orrelation at the lassroom level is as defined in equation 5, and the ηs are as defined previously in equation 8. In the ase where no ovariates are inluded the nonentrality parameter beomes λ = mpn δ + ϑ ρ + ρ ( pn ) ( n ) s s, () where ϑ = ω ω is the proportion of the treatment by shool random effet variane to the s t / s total between-shool variane. The power in design two should typially be larger than the power in design one for two reasons. First, ϑ s, ϑ Rs are typially less than one, that is, the between-shool variane of the treatment effet is typially smaller than the total between-shool variane. Seond, the term p*n in design two is always a smaller number than in design one, sine the number of lassrooms within shools in design one is larger than the number of lassrooms per ondition in design two and the number of students per lassroom remains the same. The power of the one-tailed t test at a signifiane level α is p = Η[(α, m-q-), (m-q- ), λ A ], where (α, ν) is the level α one-tailed ritial value of the t-distribution with ν degrees of freedom [e.g., (0.05,0) =.7], and Η (x, ν, λ) is the umulative distribution funtion of the non-entral t-distribution with ν degrees of freedom and non-entrality parameter λ A. The power 30

31 of the two-tailed t test at level α is p = Η[(α/, m--q), (m-q-), λ A ] + Η[-(m--q), (m-q-), λ A ]. When no ovariates are inluded at any level (that is q, w, r = 0) the degrees of freedom for the t-test are slightly hanged, and λ A = λ. Alternatively and equivalently, the test for the treatment effet and statistial power an be omputed using the F-statisti. In this ase the F- statisti has a non-entral F-distribution with degree of freedom in the numerator and m q degrees of freedom in the denominator and non-entrality parameter λ A or How Does Power Depend on Student, Classroom, and Shool Units? As in design one, we illustrate the effet of the number of students, lassrooms, and shools on statistial power in figures 4 to 6. Figure 4 illustrates that, as the number of students in eah lassroom beomes larger, power inreases, but then as in design one it levels off at a value that is less than one. Nonetheless the trajetory is steeper than that in design one. This suggests that for a fixed number of shools and lassrooms, inreasing student sample size beyond a ertain point has a small effet on statistial power. In fat, as the number of students in eah lass beomes indefinitely large, the value of the non-entrality parameter tends to λ. λ = mp δ ( max) st ( pϑ ηρ + ηρ ) Rs s s and in the ase of no ovariates to λ = mp δ ( max) st ( pϑ ρ + ρ ) Rs s. 3

32 Therefore the maximum power for a design with m shools and p lassrooms per ondition per shool, is the power assoiated with λ ( max) st, whih is smaller than one. Figure 4 illustrates power omputations for one tailed t-tests at the 0.05 level when the effet size is 0. or 0.5 SD as a funtion of the number of students per lassroom holding onstant the number of lassrooms per ondition per shool and the number of shools (p =, m = 6). Figure 5 illustrates that, as the number of lassrooms in eah shool beomes larger, power inreases, but then levels off at a value that is less than one. This suggests that for a fixed number of shools and students in eah lassroom, inreasing lassroom sample size beyond a ertain point has a small effet on statistial power. Nonetheless the trajetory is steeper than that in design one. In fat, as the number of lassrooms in eah shool beomes indefinitely large, the value of the non-entrality parameter tends to λ ( max) = mn δ nϑ ηρ Rs s s and in the ase of no ovariates to λ ( max) = mn δ nϑ ρ. Rs s Therefore the maximum power for a design with m shools and n students in eah lassroom is the power assoiated with λ ( max), whih is less than one. Figure 5 illustrates power omputations for one tailed t-tests at the 0.05 level when the effet size is 0. or 0.5 SD as a funtion of the 3

33 number of lassrooms per ondition per shool holding onstant the number of students per lassroom and the number of shools (n = 0, m = 6). Figure 6 illustrates that, as the number of shools beomes larger, power inreases dramatially, and approahes one. This suggests that for a fixed number of lassrooms per ondition per shool and students in eah lassroom, inreasing shool sample size has a substantial effet on statistial power. In fat, as the number of shools beomes indefinitely large, the value of the non-entrality tends to infinity λ ( max) s. Therefore the maximum power for a design with p lassrooms per ondition per shool and n students in eah lassroom is the power assoiated with λ ( max) s, whih tends to one. This indiates that as in design one the number of shools impats power more than the number of lassrooms and the number of students. Nonetheless in design two the impat of students and lassrooms is more evident than in design one. Figure 6 illustrates power omputations for one tailed t-tests at the 0.05 level when the effet size is 0. or 0.5 SD as a funtion of the number of shools holding onstant the number of students and lassrooms per ondition (n = 0, p = ). The patterns observed in figures 4 to 6 are generally similar to those in figures to 3. Overall, other things being equal, the power in the seond design is higher than the power in the first design sine the between-shool variane of the treatment effet is typially smaller than the overall between-shool variane, that is, ϑ s or ϑ Rs is typially smaller than one. Evidene from projet STAR indiates that the between-shool variane of the treatment effet is nearly /7 of the overall between-shool variane (and this is the estimate used in our omputations). It appears that in the seond design the number of lassrooms impats power muh more than the number of students (ompared to the first design). For example, assume a design with a total sample size of 600, where there are overall 0 shools (m = 0), one lassroom within eah 33