What p values really mean (and why I should care) Francis C. Dane, PhD

Transcription

1 What p values really mean (and why I should care) Francis C. Dane, PhD

2 Session Objectives Understand the statistical decision process Appreciate the limitations of interpreting p values Value the use of statistics regarding effect size

3 Inferential Statistics Test of fit between data and model Usually, model = null hypothesis There is no difference There is no relationship It s all measurement error and individual differences Usually, we hope to reject null hypothesis But sometimes we want to accept it

4 Statistical Decision Making Unknowable State of the World We (almost) never get to measure the entire population State of the World Wrong

5 Statistical Decision Making Decision about Fit between Data and Model We decide whether or not our data fit the chosen model Truth Decision about Fit Accept Model (Data fit) Reject Model (Data do not fit) (1-α) Error Type I α (aka p value) Wrong Error Type II β (1-β) power

6 Statistical Decision Making Four Possible Decision Outcomes Two ways to be correct, and two to be wrong State of the World Wrong Decision about Fit Accept Model (Data fit) Reject Model (Data do not fit) (1-α) Error Type I α (aka p value) Error Type II β (1-β) power

7 Statistical Decision Making We might incorrectly reject the model Observe an effect in what is really random variation State of the World Decision about Fit Accept Model (Data fit) Reject Model (Data do not fit) (1-α) Type I Error α p value Wrong Type II Error β (1-β) Power

8 Statistical Decision Making We might incorrectly accept the model Miss an effect hidden within random variation State of the World Wrong Decision about Fit Accept Model (Data fit) Reject Model (Data do not fit) (1-α) Type I Error α p value Type II Error β Decision (1-β) power

9 Statistical Decision Making We might correctly reject the model Observe an effect that exists State of the World Wrong Decision about Fit Accept Model (Data fit) Reject Model (Data do not fit) (1-α) Type I Error α p value Type II Error β (1-β) Power power

10 Statistical Decision Making We might correctly accept the model Decision about Fit Accept Model (Data fit) Reject Model (Data do not fit) State of the World (1-α) Type I Error α p value Wrong Type II Error β (1-β) Power

11 Statistical Decision Making State of the World Wrong Decision about Fit Accept Model (Data fit) Reject Model (Data do not fit) (1-α) Type I Error α p value Type II Error β (1-β) Power

12 Sample Size Dependence Both α and β are inversely dependent upon sample size Larger samples less error Larger samples more power Function of Standard Error S x = s n s = sample standard deviation n = sample size

13 Standard Error of Mean (s 10) Sample Size

14 Degrees of Freedom All model-fitting statistics are based on standard error Or an approximation (nonparametric statistics) Usually represented by degrees of freedom Number of estimable data points before last data points are determined by the model Student s t as example df = n-1 (per group)

15 p Value for t as a Function of Degrees of Freedom p(t = 1.0) p(t = 1.5) p(t = 2.0) p(t = 2.5) p(t = 3.0) p(t = 3.5)

16 Limitations of p <.05 (or <.0001) Smaller p value does NOT mean larger effect It is an indication of lack of fit between model and data p value is not relevant to the truth of the conclusion p <.05 (or any other number) is not a magic limen significance should not be used as sole basis for a conclusion, policy, treatment, etc. p <.05 is basis for further consideration

17 Effect Size Measures how much data deviate from model How large is the mean difference? How strong is the relationship between variables? How well does X predict Y? How much variance is accounted for?

18 Which is the Larger Effect? (Both are significant ) Mean = 10 Mean = 11 Mean = 10 Mean = 13

19 Measures of Effect Size Mean Differences Cohen s d Partial η 2 Relationships r 2 or R 2 β for regression OR, RR, etc.

20 Summary p value is probability of incorrectly rejecting the notion that the data fit the selected model p value is not an indication of the truth or validity of conclusions p value is an inadequate standard for deciding on importance Despite common usage, there is no mathematically justified p value for significant Effect size is an important part of interpreting results

21 Additional Reading Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2 ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Dane, F. C. (2011). Evaluating Research: Methodology for People Who Need to Read Research. Thousand Oaks, CA: Sage Publications. Trafimow, D., & Marks, M. (2015). Editorial. Basic & Applied Social Psychology, 37(1), 1-2. doi: / Wasserstein, R. L. & Lazar, N. A. (2016). The ASA's statement on p- values: Context, process, and purpose. The American Statistician. DOI: / Weisberg, H. I., Carver, R. P., Chachkin, N. J., & Cohen, D. K. (1979). Correspondence. Harvard Educational Review, 49,

22 Questions? (office) (mobile)