STAT 430 (Fall 2017): Tutorial 2

Transcription

1 STAT 430 (Fall 2017): Tutorial 2 A review of statistical power analysis Luyao Lin September 19/21, 2017 Department Statistics and Actuarial Science, Simon Fraser University

2 Hypothesis Testing A statistical hypothesis is an assumption about a population parameter (often denoted by θ). Hypothesis testing refers to the formal procedures used by statisticians to accept or reject statistical hypotheses (θ = θ 0 ). 1

3 Hypothesis Testing A statistical hypothesis is an assumption about a population parameter (often denoted by θ). Hypothesis testing refers to the formal procedures used by statisticians to accept or reject statistical hypotheses (θ = θ 0 ). Example: (a) Pregnancy Tests, Medical Exams, Clinical Studies, (b) A NFL example at http: // (c) Crash test in automobile industry. 1

4 Hypothesis Testing A statistical hypothesis is an assumption about a population parameter (often denoted by θ). Hypothesis testing refers to the formal procedures used by statisticians to accept or reject statistical hypotheses (θ = θ 0 ). Example: (a) Pregnancy Tests, Medical Exams, Clinical Studies, (b) A NFL example at http: // (c) Crash test in automobile industry. What it really means: to answer questions raised with data and probability. 1

5 A simple example: Patient: Am I pregnant? Fact: Normal hcg levels in non-pregnant women are less than 5.0 miu/ml. It raises to over 50.0 miu/ml during pregnancy. H 0 : θ 50 versus H a : θ > 50 (Why not the other way around?) 2

6 A simple example: Patient: Am I pregnant? Fact: Normal hcg levels in non-pregnant women are less than 5.0 miu/ml. It raises to over 50.0 miu/ml during pregnancy. H 0 : θ 50 versus H a : θ > 50 (Why not the other way around?) Innocent until proven guilty! 2

7 A simple example: Patient: Am I pregnant? Fact: Normal hcg levels in non-pregnant women are less than 5.0 miu/ml. It raises to over 50.0 miu/ml during pregnancy. H 0 : θ 50 versus H a : θ > 50 (Why not the other way around?) Innocent until proven guilty! Possible errors: Type I error (α): Prob(Reject H 0 H 0 is true) Type II error (β): Prob(Accept H 0 H a is true) 2

8 What qualifies as good hypothesis testing? We perform hypothesis testing by constraining probability of Type I error (α), say set α = One might also want to maximize statistical power, defined by 1 β. 3

9 What qualifies as good hypothesis testing? We perform hypothesis testing by constraining probability of Type I error (α), say set α = One might also want to maximize statistical power, defined by 1 β. Under H 0 : θ = θ 0 versus H a : θ = θ 1 (σ known), consider test statistic Z = X θ 0 σ/ n We already learnt that under H 0, Z N(0, 1). Reject region: Z > Z 1 α, s.t. Prob(Z > Z 1 α H 0 ) = α. 3

10 What qualifies as good hypothesis testing? We perform hypothesis testing by constraining probability of Type I error (α), say set α = One might also want to maximize statistical power, defined by 1 β. Under H 0 : θ = θ 0 versus H a : θ = θ 1 (σ known), consider test statistic Z = X θ 0 σ/ n We already learnt that under H 0, Z N(0, 1). Reject region: Z > Z 1 α, s.t. Prob(Z > Z 1 α H 0 ) = α. On the other hand, β = Prob(Z Z 1 α θ = θ 1 ). Since Z N( θ1 θ0 σ/ n, 1) under H a, we can get 1 β = 1 Φ(Z 1 α θ 1 θ 0 σ/ n ) = Power 3

11 Example of other Test Statistics (One sided tests) T Test Statistics: T = X θ 0 s/ n Under H 0: T T n 1 Under H a: T T n 1,λ Power = 1 β = 1 F (T θ 1 θ 0 s/ n ) F Test Statistics: F = MSR MSE under H 0, F F ν 1,n ν under H a, F F ν 1,n ν,φ Power = 1 β = 1 F (F 1 α), where F 1 α is the 100(1 α)-th percentile of F ν 1,n ν, and F ( ) is the cdf function for noncentral F ν 1,n ν,φ. Note: power is NOT a function of data (sample)! 4

12 Back to the Z statistic example: H 0 : θ = 50 versus H a : θ = 100 Reject H 0 when X > Z 0.95 σ/ n + 50 Power = 1 β = 1 Φ(Z 1 α θ1 θ0 σ/ n ) Question: What do you think affects power? 5

13 Effect of α Power = 1 β = 1 Φ(Z 1 α θ1 θ0 σ/ n ) (a) α = 0.05 (b) α = 0.1 6

14 Effect of n Power = 1 β = 1 Φ(Z 1 α θ1 θ0 σ/ n ) (a) n = 2 (b) n = 5 7

15 Effect of θ 1 θ 0 Power = 1 β = 1 Φ(Z 1 α θ1 θ0 σ/ n ) (a) θ 1 = 100 (b) θ 1 = 120 8

16 Effect of σ Power = 1 β = 1 Φ(Z 1 α θ1 θ0 σ/ n ) (a) σ = 25 (b) σ = 30 9

17 How to get power using SAS proc power; onesamplemeans test=t /*one sample t test */ nullmean = 0 /* theta_0 */ mean = 2 /* theta_1 */ stddev = 5 /* sigma */ ntotal = 20 /* sample size*/ power =. ; run; (more examples in SAS code) 10

18 If power is too hard to compute, what would you do? More popular than you think: simulation! When the distribution of the test statistic under H a is not available: (i) Specify θ 0, θ 1, σ, experimental design, and α. (ii) By simulating ɛ i N(0, σ 2 ) for each run in the design, and assigning θ = θ 1, obtain each y i for i = 1, 2,..., n. (iii) Fit model with simulated data from (ii) by testing H 0 : θ = θ 0 versus H a : θ = θ 1. Record if the results are significant. (iv) Repeat step (ii) - (iii) for a large number of times, say M, and count number of significant results, say K. The power is then K/M, i.e., proportion of times that H 0 is rejected knowing that it should be rejected (due to step (ii)). 11