Hypothesis Testing and Confidence Intervals (Part 2): Cohen s d, Logic of Testing, and Confidence Intervals

Transcription

1 Hypothesis Testing and Confidence Intervals (Part 2): Cohen s d, Logic of Testing, and Confidence Intervals Lecture 9 Justin Kern April 9, 2018

2 Measuring Effect Size: Cohen s d Simply finding whether a hypothesis test is significant or not tells us very little. It only tells whether or not an effect exists in a population. It does not tell us how much of an effect there actually is! Definition: Effect size is a statistical measure of the size of an effect how far the sample statistic and a population parameter in a population. This allows researchers to describe how far scores have shifted in the population, or the percent of variability in scores that can be explained by a given variable.

3 Measuring Effect Size: Cohen s d One common measure of effect size is Cohen s d. d = x μ ҧ σ The direction of shift is determined by the sign of d. d < 0 means that the scores are shifted to the left of μ. d > 0 means that the scores are shifted to the right of μ. Larger absolute values of d mean that there is a larger effect. There are standard conventions for describing effect size, but take these with a grain of salt. Small effect: d < 0.2 Medium effect: 0.2 < d < 0.8 Large effect: d > 0.8

4 Example A developmental psychologist claims that a training program he developed according to a theory should improve problemsolving ability. For a population of 7-year-olds, the mean score μ on a standard problem-solving test is known to be 80 with a standard deviation of 10. To test the training program, 26 7-year-olds are selected at random, and their mean score is found to be 82. Let s assume the population of scores is normally distributed. Can we conclude, at an α =.05 level of significance, that the program works? Assume the sd of scores after the training program is also 10. We hypothesize that the test improves problem-solving. What are the null and alternative hypotheses? H 0 : μ = 80 vs. H 1 : μ > 80 The data are normally distributed, so തX~N μ, σ2 A z-statistic can then be formed: z = ҧ x μ 0 σ = n 26 Critical value method: α =.05 z α = n Since z = < = z α, then we cannot reject H 0. Substantively, this means that a mean score of 82 is likely to occur by chance, so we cannot say that the training program improved problem-solving skills in 7-year-olds. What is the effect size as measured by Cohen s d? d = x μ ҧ = = 0.20 σ 10 Small effect size

5 Basic Hypothesis Testing Consider a very popular computer game played by millions of people all over the world. The average score of the game is known to be μ = 5000 and the standard deviation is known to be σ = Suppose the game developers have just created a one-week tutorial to help players increase their performance. In order to find out if it actually works, they administer the tutorial to a random sample of n = 100 players, whose average score is calculated to be x ҧ = 5200 after one week. Does the tutorial actually work, or did those players happen to get an average score as high as 5200 just by chance? o H 0 : μ = 5000 o H 1 : μ > 5000 o z = x μ ҧ = = 2 σ/ n 1000/ 100 o p-value = P Z > z = P Z > 2 = o Interpretation of p-value: Given that the tutorial actually doesn t work (H 0 ), there is only a 2.28% chance that that a random sample of 100 players gets an average score as high as 5200 (or higher). o How low should the p-value be before we are convinced that the tutorial does work (reject H 0 and accept H 1 )? This consideration is somewhat arbitrary, but common standards are α = 0.05 or α = 0.01 Note that α is the cutoff p-value (below which we reject H 0, above which we fail to reject H 0 ) If we use α = 0.05, then (p-value = ) < (α = 0.05), so reject H 0 and accept H 1 (tutorial does work) If we use α = 0.01, then (p-value = ) > (α = 0.01), so fail to reject H 0 and accept H 0 (tutorial doesn t work)

6 Type I Error Let s say that in reality H 0 is true (i.e., H 1 is false, or tutorial doesn t work) Suppose we repeat the experiment over and over again (repeatedly draw random samples of 100 players and put them through the tutorial) and conduct a hypothesis test each time using α = 0.05 We would falsely reject H 0 (mistakenly decide that the tutorial works) 5% of the time just due to chance In other words, α is the probability of rejecting H 0 when in reality it is true (incorrectly deciding that the tutorial works when it actually doesn t) o o Type I Error: rejecting H 0 when, in fact, it is true α = P(Type I Error) Ultimately, α is the probability of Type I Error we are willing to live with z α = α μ σ/ n α = z α σn + μ α = 0.05 z α = z 0.05 = α = = / = = =

7 Type II Error Let s say that in reality H 0 is false (i.e., H 1 is true, or tutorial does work) Under the false premise of H 0, the population mean of players who take the tutorial is μ 0 = 5000 (tutorial doesn t work) and the standard deviation of the sampling distribution is σ n = = 100 Suppose that, in fact, the tutorial actually increases a player s score by 300 points on average Under the true premise of H 1, the population mean of players who take the tutorial is μ 1 = 5300 (tutorial increases mean score by 300), but assume the standard error stays the same at σ n = = 100 H 0 H 1 μ 0 = 5000 μ 1 = 5300 σ/ n = 100 σ/ n = 100

8 Type II Error Type II Error: failing to reject H 0 when, in fact, it is false (incorrectly deciding tutorial doesn t work when it actually does) β = P(Type II Error) α = z α σn + μ = = z β = α μ 1 = σ/ n 100 = β = P Z < z β = P Z < = H 0 H 1 μ 0 = 5000 μ 1 = 5300 σ/ n = 100 σ/ n = 100 β = α = =

9 Power Power: probability of rejecting H 0 when, in fact, it is false (correctly deciding tutorial works when it actually does) Power is the complement of β Power = 1 β = = H 0 H 1 Power = μ 0 = 5000 μ 1 = 5300 σ/ n = 100 σ/ n = 100 β = =