Power Analysis Dr. J. Kyle Roberts Southern Methodist University Simmons School of Education and Human Development Department of Teaching and Learning When testing a specific null hypothesis (H 0 ), we have four possible outcomes. True State Decision Made H 0 = True H 0 = False Reject Type I error Correct Decision Fail to Reject Correct Decision Type II error In the above table, α is referred to as our Type I error rate. Therefore, if we set α = 0.05, then we have a 5% chance of making a Type I error.
Power Definition cont. From the table on the previous slide, we call the probability of making a Type II error, β. Based on these definitions for α and β, we then define power as the probability of rejecting the H 0 when it is false, or (1 β). Recall that this is specifically what we often want to do. Power is making a correct decision. Therefore, we maximize power when we minimize β. Power will have bounds 0 (1 β) 1 with larger values meaning more power. These can also be thought of as a percent. Factors Affecting Power 1. α Level As the α level increases (say from 0.05 to 0.10), β will decrease, meaning that power will go up (1 β). Conversely, as the α level decreases (say from 0.05 to 0.01), β will increase, meaning that power will go down (1 β). 2. Sample Size As sample size increases, the standard error of measurement will likewise decrease, all other things being equal since σ X = σ X / n. 3. Effect Size As the effect size increases, so does the power of the analysis. All other things remaining constant, as the size of the effect increases, the relative power associated with rejecting the null also increases.
Power for the Single Sample t test Suppose that we have a hypothetical dataset where µ = 100, X = 105, SD X = 15, and n = 30. In this case we can compute t as: t = X µ σ/ n 105 100 = 15/ 30 = 1.826 We can then compute the probability of t given our data and likewise β as: > pt(qt(0.975, 29), 29, 1.826) [1] 0.5770932 Graphical Representation of Heuristic Data > curve(pt(x, 29, ncp = 1.826), from = 0, to = 6) > abline(v = qt(0.975, 29)) pt(x, 29, ncp = 1.826) 0.0 0.2 0.4 0.6 0.8 1.0 0 1 2 3 4 5 6 x
In order to compute power, we need to know a few things about our data. 1. n 2. SD 3. Effect Size (or difference between means) 4. α level Knowing these four things will then let you compute the missing parameter (1 β), or power. > power.t.test(delta = 5, sd = 15, n = 30,, + type = "one.sample") One-sample t test power calculation n = 30 delta = 5 sd = 15 power = 0.4228091 alternative = two.sided Computing n based on Power Suppose that you are conducting a single sample experiment and you want to know how many people you should sample in order to attain a power of 0.85. We can compute this by the work backwards principle since n is the only missing piece of data. > power.t.test(delta = 5, sd = 15, power = 0.85,, + type = "one.sample") One-sample t test power calculation n = 82.74835 delta = 5 sd = 15 power = 0.85 alternative = two.sided This means that we need to sample 83 people to obtain a power of 0.85 given our study parameters.
Ways of Improving Power Reduce α from 0.05 to say 0.10. Consider methods for reducing the within-group variability (thus increasing the effect size). Increase the number of individuals in each group. Power for the One-Way ANOVA For the ANOVA, we can calculate relative group sample size given our estimated power, α, effect-size, and K. > library(pwr) > pwr.anova.test(f = 0.28, k = 4,, + power = 0.8) Balanced one-way analysis of variance power calculation k = 4 n = 35.75789 f = 0.28 power = 0.8 NOTE: n is number in each group
Post-Hoc Power for the One-Way ANOVA Likewise for the ANOVA, we can calculate post-hoc power based on α, effect-size, the average n, and K. > pwr.anova.test(f = 0.28, k = 4, n = 20, ) Balanced one-way analysis of variance power calculation k = 4 n = 20 f = 0.28 power = 0.5149793 NOTE: n is number in each group Other Types of Tests For all types of statistical tests, we can compute power or the needed sample size based on desired power. Almost all of the power calculations for statistical tests you will run are covered in Cohen (1988) Statistical Power Analysis for the Behavioral Sciences.