Power. Week 8: Lecture 1 STAT: / 48

Transcription

1 Power STAT:5201 Week 8: Lecture 1 1 / 48

2 Power We have already described Type I and II errors. Decision Reality/True state Accept H o Reject H o H o is true good Type I error H o is false Type II error good Power is the probability of rejecting the null when the null is false (something we really hope to do). We want high power. We commonly use α to represent the Type I error rate, and β to represent the Type II error rate. Then, power is 1 β. Power (or 1 β) is a bit more difficult to deal with than the type I error rate or α. 2 / 48

3 Power There is a trade-off between Type I and Type II errors. Decreasing our likelihood of a Type I error means we are less likely to make rejections. In designing experiments and choosing sample sizes, we should consider power... * If the power is too low, then you re wasting your time and resources running experiments with very little chance of detecting something interesting. * If the power is too high, then you are spending resources in this experiment that might be better spent somewhere else. Like hitting a fly with nuclear weapon, too much... In practice, we tend to be content with a power of 0.80 or This provides high likelihood to detect something interesting when it s really there. 3 / 48

4 Refresher: sample size calculation for one-sample test with σ 2 known, find n H o : µ = µ o vs. H A : µ > µ o (one-sided) Choose α type I error rate Choose β type II error rate (i.e. set power =1 β) Get cut-off to control type I error rate at α level: P(Ȳ > c H o true) = P( Ȳ µo σ/ n > c µo σ/ n H otrue) = P(z > c µ o σ/ ) = α n }{{} c = µ o + z α σ n z α 4 / 48

5 Refresher: sample size calculation for one-sample test with σ 2 known, find n Setting our cut-off for significance at c = µ o + z α σ n controls the Type I error rate at the α level. Now, for a chosen β and a given µ A under H A true, we have P(Ȳ > c H A true) = P( Ȳ µ A σ/ n > c µ A σ/ n H A true) So, z α = P(z > (µo+zα σ n ) µ A σ/ ) n = P(z > z α d σ/ ) = 1 β (where d = µ A µ o) n }{{} (to control Type II error) z 1 β d σ/ n = z 1 β n = ( ) σ(zα z1 β ) 2 d sample size 5 / 48

6 Refresher: sample size calculation for one-sample test with σ 2 known, find n Here, we can see the standardized threshold of z α controlling the Type I error rate at the α level for sample size n. Then, given the true mean of µ A, we see that if we reject for standardized values greater than z α error rate of β, or power of 1 β. d σ/ n, then we have a Type II Standardized Original scale Threshold for significance 6 / 48

7 Refresher: sample size calculation for one-sample test with σ 2 known, find n If using the unstandardized values, P(Ȳ > µ o + z α σ n H o true) = α P(Ȳ > µ o + z α σ n H A true) = 1 β Power is the area to the right of the critical value under the shifted-right normal curve. 7 / 48

8 Refresher: sample size calculation for one-sample test with σ 2 unknown, find n H o : µ = µ o vs. H A : µ > µ o (one-sided) Test utilizes a t-test, more complex because σ 2 not known. Get cut-off to control type I error rate at α level: P(Ȳ > c H o true) = P( Ȳ µo s/ n > c µo s/ n H otrue) = P(t > c µ o s/ ) = α n }{{} c = µ o + t α,n 1 s n t α,n 1 8 / 48

9 Refresher: sample size calculation for one-sample test with σ 2 unknown, find n Setting our cut-off for significance at c = µ o + t α,n 1 σ n controls the Type I error rate at the α level. Now, for a chosen β and a given µ A under H A true, we have P(Ȳ > c H A true) = P( Ȳ µ A s/ n > c µ A s/ n H A true) = 1 β P(t > (µo+t α,n 1 s n ) µ A s/ ) = 1 β n P(t > t α,n 1 (µ A µ o ) s/ ) = 1 β n }{{} t 1 β,n 1 So, t 1 β,n 1 = t α,n 1 d s/ n n = ( ) s(tα,n 1 t 1 β,n 1 ) 2 d Notice that n is on both sides of the equation 9 / 48

10 Refresher: sample size calculation for one-sample test with σ 2 unknown, find n One option is an iterative procedure... 0) We need a value for s, often this comes from a pilot study. 1) Assume we know σ and start by using z-values. n (0) = ( ) 2 s(zα z 1 β ) d 2) Plug into unknown σ 2 formula. n (1) = ( s(tα,n ) (0) 1 t 1 β,n (0) 1 ) 2 d 3) Continuing to use the formula in 2), iterate until convergence. 10 / 48

11 Refresher: sample size calculation for one-sample test with σ 2 unknown, using R Example (Sample size calculation in R) ############## one-sample hypothesis test ################# ## Use iterative process for a one-sided test: alpha < beta < # --> gives power of 90% m.0 <- 4 # --> null mean m.a <- 6 # --> alternative mean s <- 3 # --> sample standard deviation ## Get initial n based on a known sigma, so use z-dist instead of t-distribution: n.0 <- ((s*(qnorm(alpha,lower.tail=false)- qnorm(1-beta,lower.tail=false)))/(m.a-m.0))^2 n.0 [1] / 48

12 Refresher: sample size calculation for one-sample test with σ 2 unknown, using R Example (Sample size calculation in R) ############## one-sample hypothesis test ################# ## Utilize sample size equation for unknown sigma ## and t-distribution to get next n. n.1 <- ((s*(qt(alpha,n.0-1,lower.tail=false)-qt(1-beta,n.0-1,l n.1 [1] ## Iterate until convergence: n.old <- n.0 n.new <- n.1 iter < / 48

13 Refresher: sample size calculation for one-sample test with σ 2 unknown, using R Example (Sample size calculation in R) ############## one-sample hypothesis test ################# ## Iterate until convergence: while(abs(n.old-n.new)>=1){ cat("iteration = ",iter,"\n") n.old <- n.new n.new <- ((s*(qt(alpha,n.old-1,lower.tail=false)- qt(1-beta,n.old-1,lower.tail=false)))/(m.a-m.0))^2 print(n.new) iter <- iter+1 } #[1] Use n=21 as your sample size. This was an iterative process when σ 2 unknown. 13 / 48

14 Sample size calculation for one-sample test with σ 2 unknown using Lenth applet U of Iowa Professor Russ Lenth has a widely used applet available for power and sample size calculations: rlenth/power/ Example (Lenth sample size website) We can use these same parameters from the previous example in the applet to see that n = / 48

15 Sample size calculation for one-sample test with σ 2 unknown using SAS Same sample size calculation done in SAS. Example (SAS sample size) 15 / 48

16 Sample size calculation for one-sample test with σ 2 unknown using SAS Example (SAS sample size) 16 / 48

17 Sample size calculation for Confidence Interval (CI) for one-sample scenario with σ 2 unknown s (1 α)100% CI for µ: x ± t α/2,n 1 n }{{} half-width of CI or MOE For a given MOE, n = ( ) tα/2,n 1 s 2 MOE We can again use an iterative process to get n. *round-up to get n at the end. 17 / 48

18 Sample size calculation for Confidence Interval (CI) for one-sample scenario with σ 2 unknown, using R Example (Sample size calculation in R for CI) ############# 95% confidence interval ############## alpha=0.05 s=0.5 w=0.1 #half-width n.0 <- (qnorm(alpha/2,lower.tail=false)*s/w)^2 n.0 [1] n.1 <- (qt(alpha/2,n.0-1,lower.tail=false)*s/w)^2 n.1 #[1] / 48

19 Sample size calculation for Confidence Interval (CI) for one-sample scenario with σ 2 unknown, using R Example (Sample size calculation in R for CI) ############# 95% confidence interval ############## n.old <- n.0 n.new <- n.1 while (abs(n.old-n.new)>=1){ n.old <- n.new n.new <- (qt(alpha/2,n.old-1,lower.tail=false)*s/w)^2 print(n.new) } [1] Use n=99 as your sample size. 19 / 48

20 Sample size calculation for Confidence Interval (CI) for one-sample scenario with σ 2 unknown, using Lenth website Example (Sample size calculation for CI using Lenth website) 20 / 48

21 Refresher: sample size calculation for two-sample t-test (equal sample sizes, constant variance) H o : µ 1 µ 2 = 0 vs. H A : µ 1 µ 2 > 0 (one-sided) Get cut-off to control type I error rate at α level: P(Ȳ 1 Ȳ 2 > c H o true) = α c P(t > ) = α S p 2/n }{{} t α,2n 2 c = t α,2n 2 S p 2/n Then for given β: P(Ȳ 1 Ȳ 2 > c H A true) = 1 β P(t > c d ) = 1 β (where d = µ 1 µ 2) S p 2/n }{{} t 1 β,2n 2 21 / 48

22 Refresher: sample size calculation for two-sample t-test (equal sample sizes, constant variance) t α,2n 2 S p 2/n d S p 2/n = t 1 β,2n 2 n = 2 ( ) Sp(tα,2n 2 t 1 β,2n 2 ) 2 d And again, an iterative procedure can be used. 22 / 48

23 Sample size calculation for two-sample t-test in R (equal sample sizes, constant variance) Example (Sample size calculation in R for 2-sample t-test) ############# two-sample hypothesis test ############# ## For a one-sided test: alpha < beta < #(gives power of 90%) d <- 2 ## difference between means. s <- 1 ## same for both groups (pooled) ## Get initial n based on a known sigma: n.0 <- 2*(s*(qnorm(alpha,lower.tail=FALSE)- qnorm((1-beta),lower.tail=false))/d)^2 n.0 [1] / 48

24 Sample size calculation for two-sample t-test in R (equal sample sizes, constant variance) Example (Sample size calculation in R for 2-sample t-test) ############# two-sample hypothesis test ############# ## Utilize equation for unknown sigma to get next n. n.1 <- 2*(s*(qt(alpha,2*n.0-2,lower.tail=FALSE)- qt((1-beta),2*n.0-2,lower.tail=false))/d)^2 n.1 [1] # Iterate until convergence: n.old <- n.0 n.new <- n.1 iter < / 48

25 Sample size calculation for two-sample t-test in R (equal sample sizes, constant variance) Example (Sample size calculation in R for 2-sample t-test) ############# two-sample hypothesis test ############# while (abs(n.old-n.new)>=1){ cat("iteration = ",iter,"\n") n.old <- n.new n.new <- 2*(s*(qt(alpha,2*n.old-2,lower.tail=FALSE)- qt((1-beta),2*n.old-2,lower.tail=false))/d)^2 print(n.new) iter=iter+1 } [1] Use n=6 as your sample size (6 in each group). 25 / 48

26 Sample size calculation for two-sample t-test, Lenth website (equal sample sizes, constant variance) Example (Sample size calculation for 2-sample t-test, Lenth website) For n = 6 power is , for n = 5 power is / 48

27 Sample size calculation for 2-sample t-test in SAS SAS allows you to consider a variety of differences in means and sample sizes at once to calculate power for each. Example (Sample size calculation for 2-sample t-test in SAS) 27 / 48

28 Sample size for Confidence Interval (CI) on a contrast Recall for our contrast γ = a i=1 c iµ i we have the estimate: a ˆγ = c i Ȳ i. i=1 95% CI for γ: ˆγ ± t α/2,n a MSE a i=1 c 2 i n i } {{ } MOE For a given half-width MOE, what sample size do we need? If we assume n i = n for all i (i.e. balanced design), then ( ) tα/2,n a 2 n = MOE MSE a i=1 c2 i where N = na. Again, you can use an iterative procedure to find n. 28 / 48

29 Power and Sample Size for 1-way ANOVA What about power for a 1-way ANOVA? We consider power in the overall F-test... H o : µ 1 = µ 2 =... = µ a vs. H A : not H o Test statistic F o = SSE red SSE full df MSE full = SStotal SSE full (a 1) MSE full = SStrt (a 1) MSE full = MStrt MSE full Reduced model: only 1 mean Full model: a means 29 / 48

30 Power and Sample Size for 1-way ANOVA Under H o true, F o follows a central F -distribution or F o F (a 1,N a,φ=0) where φ is the non-centrality parameter. This information sets us up with a cut-off for significance for controlling the Type I error rate at the α level / 48

31 Power and Sample Size for 1-way ANOVA What about the Type II error rate? What about power? As always, to compute power, we must provide the true state of the means under H A true. For the two-sample t-test, we only needed to state the difference between the two means. Now, with a means, there are many different configurations of the means where H o is false. This suggests we need to consider numerous configurations. 31 / 48

32 Power and Sample Size for 1-way ANOVA Under H A true, F o follows a non-central F -distribution or F o F (a 1,N a,φ) where φ > 0. We establish a cut-off for significance c = F (α,a 1,N a,φ=0) based on a given Type I error rate. Then, the power is the probability that F o is greater than or equal to c under H A true. 32 / 48

33 Power and Sample Size for 1-way ANOVA What is the value of φ? The non-centrality parameter? This depends on the particular configuration of a means. φ = a i=1 n i α 2 i σ 2 where α i = µ i µ and i n iα i = 0 The non-centrality parameter φ measures how far the treatment means are from being equal relative to the squared-standard error of a single mean or σ2 n i Recall, E[MS TRT ] = σ 2 + a i=1 n i α 2 a 1 With E[MSE] = σ 2 we can see how φ works with these values to form our F -statistic for Treatment. 33 / 48

34 Power and Sample Size for 1-way ANOVA φ = a i=1 n i α 2 i σ 2 Larger differences in means provides a larger φ and provides a larger power (all things being equal). In general... φ increases if you increase sample sizes. φ increases if the error variance is smaller. φ increases if the means µ i are farther apart. Central and non-central F(2,15,ncp=phi) F(2,15,ncp=0) 0.05 in upper tail of null F(2,15,ncp=51.09) x 34 / 48

35 Power and Sample Size for 1-way ANOVA Non-central F(2,15,ncp=phi) F(2,15,ncp=10) F(2,15,ncp=20) F(2,15,ncp=30) x 35 / 48

36 Power and Sample Size for 1-way ANOVA If there are three means, which configuration is easier to detect? Scenario 1 Scenario 2 µ 1 = 1 µ 1 = 0 µ 2 = 0 µ 2 = 0 µ 3 = 1 µ 3 = 2 φ = a i=1 n i (µ i µ) 2 σ 2 Assuming n i = n, φ 1 = n a i=1 (µ i 0) 2 = n 2 σ 2 σ 2 φ 2 = n a i=1 (µ i 2/3) 2 = n 2 2 σ 2 3 σ 2 Larger φ is easier to detect in scenario / 48

37 Power and Sample Size for 1-way ANOVA Example (Computing power for a given scenario) a = 3, n = 6, α = 0.05, σ 2 = µ 1 = 2.82 µ 2 = 3.89 µ 3 = 3.04 µ = 3.25 cut-off for significance is F (0.05,2,15,φ=0) = 3.68 α 1 = 0.43 α 2 = 0.64 α 3 = 0.21 φ = 6 3 i=1 α2 i = Power=P(F (2,15,φ=51.09) > 3.68) = *Very small MSE relative to α i s gave very large power. 37 / 48

38 Power and Sample Size for 1-way ANOVA Example (1-way ANOVA sample size in SAS) 38 / 48

39 Power and Sample Size for 1-way ANOVA Software will compute power for you. Depending on the software you use, it may ask for something that doesn t seem like the non-centrality parameter described above, but is actually a slight transformation of it. R will ask for between group variance and within group variance instead of the non-centrality parameter. ( (µi ) µ) 2 φ = n a i=1 α2 i σ 2 = n(a 1) a 1 σ 2 between group variance = n(a 1) within group variance So, between group variance = α 2 i a 1 and within group variance= σ2. 39 / 48

40 Power and Sample Size for 1-way ANOVA In the previous example, between group variance = α 2 i a 1 = , and within group variance= Example (1-way ANOVA sample size in R) ########### power in 1-way ANOVA ################# > power.anova.test(groups=3,n=6,between.var=0.3193, within.var=0.075,sig.level=0.05) Balanced one-way analysis of variance power calculation groups = 3 n = 6 between.var = within.var = sig.level = 0.05 power = NOTE: n is number in each group 40 / 48

41 Power and Sample Size for 1-way ANOVA The Lenth website requests SD[TREATMENT] which is between group variance and SD[WITHIN] which is within group variance. Example (1-way ANOVA sample size Lenth applet) 41 / 48

42 Power and Sample Size for 1-way ANOVA Example (1-way ANOVA power calculation) Another example... What power will I have with the following set-up? a = 4, n = 5, α = 0.05, σ 2 = 1.2 µ 1 = 9.5 µ 2 = 10.3 µ 3 = 11 µ 4 = 12.4 Get the non-centrality parameter... find power. 42 / 48

43 Power and Sample Size for 1-way ANOVA Checking with the Lenth applet: ˆσ = and α 2 i a 1 = Example (1-way ANOVA power calculation) 43 / 48

44 Power and Sample Size for 1-way ANOVA Checking with the R: σ 2 = 1.2 and α 2 i a 1 = 1.51 Example (1-way ANOVA power calculation in R) ############# power in 1-way ANOVA ################## > power.anova.test(groups=4,n=5,between.var=1.51, within.var=1.2,sig.level=0.05) Balanced one-way analysis of variance power calculation groups = 4 n = 5 between.var = 1.51 within.var = 1.2 sig.level = 0.05 power = NOTE: n is number in each group 44 / 48

45 Power and Sample Size for 1-way ANOVA φ = a i=1 n(µ i µ) 2 σ 2 In the previous example with 4 groups, we had µ 1 = 9.5, µ 2 = 10.3, µ 3 = 11, and µ 4 = 12.4 which lead to α 2 i = Actually, any configuration of means giving α 2 i = 4.54 with all the same other values (n, σ, type I error rate) will have the same power. When working with clients, they may be more comfortable hypothesizing what the range of µ i s is rather than specifically stating what each µ i is. Let D = µ max µ min and assume n i = n (balanced design). 45 / 48

46 Power and Sample Size for 1-way ANOVA φ = a i=1 n(µ i µ) 2 σ 2 Given that the range of the means is D, what do you think is the most favorable configuration of means? (i.e. the easiest to detect, or reject H o ). Given that the range of the means is D, what do you think is the least favorable configuration of means? (i.e. the hardest to detect, or lowest power). 46 / 48

47 Power and Sample Size for 1-way ANOVA To make power analysis useful, you must be able to specify some scientifically or practically meaningful set of alternative means, and you must be able to guess as to how large the error variance is σ 2. Perhaps the client can think about... If this were true, I would want to know about it, and then design for those interesting alternatives. Examples: The doubling of a mutation rate is practically significant, so I want to design for that. An increase in MPG of 1 is relevant, so I will design for that. A decrease in time-to-pain-relief of 20 minutes is meaningful to the patient, so I will design for that. Many granting agencies (NSF, NIH, etc.) require a power analysis before funding a proposal. 47 / 48

48 Power and Sample Size for a contrast of interest Rejection of the overall F-test can arise from any departure from the null hypothesis of equal means. In some situations, you may be interested in one or two particular contrasts (and less interested in others). You can design an experiment with the power of rejection of a given contrast in mind. H o : γ = 0 vs. H A : γ 0 where γ = a i=1 c iµ i and ˆγ = a i=1 c iȳi. Under H o true, F = SScontrast MSE F (1,N a) The non-centrality parameter under H A true is i=1 c i α i ) 2 σ 2 ( ) a c 2 = ( a i=1 c i µ i ) 2 i i=1 σ n 2 ( ) a c 2 i i i=1 n i φ = ( a And you find the power P(F (1,N a,φ) > F (α,1,n a) ). 48 / 48