Philosophy and History of Statistics

Transcription

1 Philosophy and History of Statistics YES, they ARE important!!! Dr Mick Wilkinson Fellow of the Royal Statistical Society

2 The plan (Brief) history of statistics Philosophy of science Variability and Probability Estimating Testing Clinical/practical meaning vs statistical significance

3

4 Potted history Beginnings 431 BC Peloponnesian war 400 BC King Rtuparma of India AD 7 Census of Judea

5 Potted history Mathematical foundations 1560 axioms of probability 1749 Statistiks 1808 Normal bell-shaped distribution 1894 standard deviation 1908 t distribution for small samples 1933 conventional approach 1935 Significance coined by Fisher

6 Science Theory Prediction Generalisable facts / laws Hypotheses Data How do we decide if a theory is true?

7 All swans are white

8 Is the theory proven?

9 Inductive reasoning supporting obs = plausibility right? Wrong!!

10

11

12 So how CAN we use observations to prove theories?? WE CAN T!!! But we can use observations to DISPROVE theories!

13 All swans are white

14 Deductive reasoning From a single observation we can DEDUCE that all swans are white is FALSE and the theory is DISPROVEN

15 Seek to falsify everything If a prediction of a theory is falsified, amend or bin the theory Get closer to the truth by refinement of theories or rejection of bad ones Prefer the theory that explains the most and makes the most testable / falsifiable predictions If a theory does not make falsifiable predictions, it is not science

16 Truth in science is probabilistic Science is a way to understand how some thing gets to be known. As much as anything can be known because nothing is known for certain Richard Feynman Physicist Nobel Laureate Truth and proof are for priests and politicians, the rest of us have to deal with probability Carl Sagan Astromomer

17 What does probability mean? Objective interpretation (Richard von Mises, 1928) Subjective interpretation (Reverend Thomas Bayes, 1764)

18 Objective (long-run) probability Applies only to a precisely defined collective / population A hypothetical series of infinite repeats of the sampling procedure and data collection Does not and cannot apply to a any single event or finite number of events or hypotheses!

19 We use just one of an infinite number of possible samples

20 Problem A single sample can t give a definite estimate about the population from which it came Every sample will be different And here is another example

21 Standard error

22 Standard error Standard deviation of multiple sample means A measure of how far off the observed (sample) mean might be from the true (population) mean Might means with a certain probability or a particular level of confidence

23 But we won t have more than one sample what then?

24 Visualising a confidence interval probability As a normal curve Area = 0.95 lower likely limit probability distribution of Pop value, given the sample value sample value upper likely limit negative 0 positive value of effect statistic As an interval likely range of true value negative 0 positive value of effect statistic

25 What is a confidence interval? Sample statistic ± Where standard error determines the level of confidence 1 = 68% likely 2 = 95% likley 3 = 99% likely Can never be 100% confident A probable range in which the true population parameter will lie on % of occasions An Estimate of the probable SIZE of the true effect

26 Using Statistical Inference to examine theories Estimation using confidence intervals NHST conventional (though flawed) approach

27 Null-hypothesis-significance testing The most bone-headedly misguided procedure ever institutionalized in the rote training of science students (Rozeboom, 1997)

28 Concepts Hypothesis testing Rejection regions Type I error Type II error

29 Philosophy of NHST We can disprove, but not prove, things. Therefore, we need something to disprove. Let's assume the true effect is zero: the null hypothesis. If the value of the observed effect is unlikely under this assumption, we reject (disprove) the null hypothesis. "Unlikely" is related to (but not equal to) a probability or P value. P < 0.05 is regarded as unlikely enough to reject the null hypothesis (i.e., to conclude the true effect is not zero). We say the effect is statistically significant at the 0.05 or 5% level. P > 0.05 means not enough evidence to reject the null. We say the effect is statistically non-significant. Most scientists wrongly accept the null and conclude "no effect".

30 What does the P mean? P (D H0) Probability of your data or data more extreme cropping up in the long run if the true average effect was actually zero Confused??

31 The Hypothesis test H0 : parameter = specified value H1 : parameter that value Mutually exclusive so if H0 then NOT H1 H1 NOT H0

32 But because of sampling variation if H0 then probably NOT H1 H1 probably NOT H0 Infallible logic? If you are a woman, you are probably not the Queen of England This woman is the Queen of England Therefore, she is probably not a woman!!!

33 What about all the other H1s? So H0 is unlikely Does that mean that H1 is the ONLY alternative explanation? What about all the other possible explanations? NONE of them have been tested, so we can never know!!!! Fallacy of the transposed conditional

34 Highly falsifiable prediction?!! A is correlated with B

35 More falsifiable prediction A is positively correlated with B

36 The Hypothesis test an example For a two sample t test of population mean difference (μ) H0: μ = 0 H1: μ 0

37

38 The decision rule α

39 Two possible errors 0 17

40 0 17

41 0 17

42 0 17

43 0 17

44 Flaws with NHST Can t disprove your hypothesis it s never tested!! Sizeless outcome Arbitrary dichotomous decision With continuous measures, zero-point null is always false anyway Confuses strength of evidence against a hypothesis with probability of evidence occurring

45 Can NHST be Popperian?

46 Popperian approach with estimation I predict a new drug will decrease SBP by 20 mmhg But not more than 35 mmhg -unrealistic And not less than 5 mmhg not worthwhile Design experiment so if true effect is >35 or < 5 we can detect it

47 Clinically worthwhile effect Lower realistic limit Predicted effect Upper realistic limit Theory falsified Theory falsified Theory not falsified / corroborated Still conservative interval must contain worthwhile effects but exclude all others for a clear conclusion

48 For practical / clinical significance, interpret confidence limits in relation to the smallest practically/clinically beneficial and harmful effects. These are usually equal and opposite in sign. They define regions of beneficial, trivial, and harmful effects. harmful smallest clinically harmful effect trivial beneficial smallest clinically beneficial effect negative 0 positive value of effect statistic The slide after next is the key to clinical or practical importance. Need two things: the confidence interval and a sense of what is important (e.g., beneficial and harmful).

49 Problem: what's the minimum important effect? Quit the field / get your coat! Same problem with NHST - determines sample size you need to test the null properly. The default for most populations and effects is Cohen's set of smallest values. Apply to clinical, practical and/or mechanistic importance Standardized changes or differences in the mean: 0.20 of the between-subject standard deviation Test-retest error also sometimes used i.e. smallest worthwhile effect measurement error Smallest detectable effect

50 Put the confidence interval and these regions together to make a decision about practically meaningful, clear or decisive effects. Clinically harmful trivial beneficial decisive? negative 0 positive value of effect statistic Statistically significant? Yes: use it. Yes Yes: use it. Yes Yes: use it. No Yes: depends. No Yes: don't use it. Yes Yes: don't use it. No Yes: don't use it. No Yes: don't use it. Yes Yes: don't use it. Yes No: need more No research. Why hypothesis testing is unethical and impractical!

51 Why are statistics important? Turn lots of numbers into fewer numbers Make sense of variability Allows us to quantify how uncertain we are!! Lets us use data to examine theories i.e. essential part of science Just makes sure you use the right approach for the task!!!