Hypotheses and Errors

Transcription

1 Hypotheses and Errors Jonathan Bagley School of Mathematics, University of Manchester Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 1/22

2 Overview Today we ll develop the standard framework for answering statistical questions Hypothesis Testing and discuss the sorts of results it can yield. Hypothesis testing: intro via a model problem Two sorts of alternatives: the number of sides The probability of errors: false positives, wrongful convictions and other Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 2/22

3 Hypothesis testing Sketch version of Hypothesis Testing: Choose some default explanation for the data you are about to collect (frequency of spec-wearing is the same as the national average, experimental drug has no consequence...) and call it the null hypothesis. Choose some other explanation that you would like to test for: call this the alternative hypothesis. Compute some test statistic for the actual data. Work out the probability of getting the observed value of the test statistic if the null hypothesis were true. Reject the null hypothesis if the observed value is too unlikely. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 3/22

4 Other vocabulary for errors Consider a screening test, where the null hypothesis is that the patient is healthy Screening Test Result: Negative Positive Patient is: Healthy True Negative False Positive Ill False Negative True Positive Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 4/22

5 Hypotheses in the courtroom Consider a juror s job. What is the null hypothesis and what is the appropriate version of the error table? Answer: Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 5/22

6 Hypotheses in the courtroom Consider a juror s job. What is the null hypothesis and what is the appropriate version of the error table? Answer: The defendant is presumed innocent until proven guilty so the null hypothesis is innocence: Defendant is really: Acquit Juror s choices: Convict Innocent Justice is served Wrongful imprisonment Guilty Psycho on the streets Justice is served Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 5/22

7 Testing proportions You survey 50-student s from Optometry schools across the country, seeking to learn whether your fellow students are more likely to need spectacles than the average person drawn from a similar age profile. Assume and ask yourself: Pspecs = P 0 = 0.25 (a) How would you decide whether a group had an unusual number (either too many or two few) of spec-wearers? (b) How would you decide if a group had an unusually large number of spec-wearers? Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 6/22

8 Unusual numbers PDF Combined area = Observed p specs Reject the null hypothesis when measured frequency p a is implausible large or small: (p a P 0 ) > z σ 0 Test is a z-score where P 0 is the frequency associated with the null hypothesis and σ 0 = p P 0 (1 P 0 )/N is the standard deviation of the distribution of results we d expect to see if the null were true. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 7/22

9 Unexpectedly large numbers PDF Area = Observed p specs Reject the null hypothesis when measured frequency p a is too large: (p a P 0 ) σ 0 > z 0.05 Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 8/22

10 Possibilities for error One may come to the wrong conclusion in a hypothesis test... Our decision: Accept Reject Null is: True Correct Decision Type I Error 1 α α False Type II Error Correct Decision β (1 β) a.k.a. the power Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 9/22

11 Undeserved rejection: α What is the risk a Type I Error: rejecting the null hypothesis when it is really true? It is usually called α and is entirely under the analyst s control; it does not depend on the size, only on the desired level of confidence; it depends only on the null hypothesis and properties of the distribution of the test statistic under the null. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 10/22

12 Unfounded acceptance: β What is risk of a Type II error accepting the null hypothesis when it is false? it is called β: the quantity (1 β) is the power of the test; it depends on how false the null is, which depends on the size; this not under the data analyst s control. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 11/22

13 Errors for the one-sided test Back to our example: Null hypothesis was that local frequency equals national one, P a = P 0 ; Alternative hypotheses is that local frequency exceeds national one, P a > P 0 ; size was N = 50; reject null hypothesis in favour of alternative (too many specy types) with 95% confidence when measured local frequency exceeds Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 12/22

14 Small differences mean big P = β P = 0.3 a α : Prob. of Type I error β : Prob. of Type II error P 0.35 crit α Normal approximations to distribs of results under null and (one particular) alternative hypothesis. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 13/22

15 Details Bump at left shows (approximate) distribution of results we would get if the null hypothesis were true: P a = P 0. Bump at right shows (approximate) distribution of results we would get if the null hypothesis were false, but almost true : P a = 0.3 while P 0 = Shaded areas show how to calculate probabilities of error. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 14/22

16 Larger difference, smaller risk P = β P = 0.5 a α : Prob. of Type I error β : Prob. of Type II error P 0.35 crit α As previously, but now the null hypothesis is more false : P a = 0.5 while P 0 = Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 15/22

17 Huge difference, tiny risk P = α : Prob. of Type I error P 0.35 crit α P = 0.75 a Here the null hypothesis is manifestly false : P a = 0.75 while P 0 = Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 16/22

18 Probabilities of errors This table summarizes the shaded areas (probabilities of error) shown on the previous slides. Confidence Limit Panel P a β α (1 α) Top % 5% 95% Middle % 5% 95% Bottom % 5% 95% Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 17/22

19 Crank up the size P = β P = 0.3 a α α : Prob. of Type I error β : Prob. of Type II error P = β P = 0.3 a α α : Prob. of Type I error β : Prob. of Type II error P = P = 0.5 a P = P = 0.5 a α β α P = 0.75 a P = P = 0.75 a P = α α P 0.35 crit P 0.3 c Increasing the size reduces β by narrowing the distributions: at left N = 50, at right N = 200. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 18/22

20 Probabilities of error, bigger This table summarizes the shaded areas (probabilities of error) shown on the right panel (N = 200) of the previous slide. Confidence Limit Plot P a β α (1 α) First % 5% 95% Second % 5% 95% Third % 5% 95% Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 19/22

21 Trading risk: in pictures Prob. of Error α p = 0.24 c p = 0.26 c p = Critical value p c c β β: Prob. of Type II Error (0.05, 0.5) Coordinates are (α, β) (0.37, 0.11) (0.63, 0.03) α: Prob. of Type I Error Using a larger critical value p c decreases α, but at the cost of increased β (figures are for N = 200). Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 20/22

22 Trading risk: explicit formulae For a fixed critical value p c one finds the probability of a Type I Error using a z-score based on the null hypothesis: ( ) (pc P 0 ) α = 1 Φ and the risk of a Type II Error using a z-score based on a specific alternative hypothesis: ( ) (pc P a ) β = Φ. Here P 0 and σ 0 are the mean and std. deviation of the distrib. arising from the null while P a and σ a are those arising from the alternative hypothesis. σ a σ 0 Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 21/22

23 Check: hypothesis testing True or false? In a large- comparison between two groups, increasing the size will: (a) decrease the chance of a Type I error; (b) decrease the chance of a Type II error; (c) increase the power of the test against a given alternative; (d) make the null hypothesis less likely to be true. Answer: Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 22/22

24 Check: hypothesis testing True or false? In a large- comparison between two groups, increasing the size will: (a) decrease the chance of a Type I error; (b) decrease the chance of a Type II error; (c) increase the power of the test against a given alternative; (d) make the null hypothesis less likely to be true. Answer: Only items (b) and (c) are true. Jonathan Bagley, September 23, 2005 Hypotheses & Errors - p. 22/22