The Welch-Satterthwaite approximation

Transcription

1 The Welch-Satterthwaite approximation Aidan Rocke Charles University of Prague October 19, 2018 Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

2 Overview 1 Motivation 2 Student s problem 3 Testing for equal variances 4 Welch generalisation 5 Concrete example in R 6 Questions Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

3 Motivation Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

4 A hypothetical experiment 1. We have a study where 38 participants were assigned to condition X, and 22 participants were assigned to condition Y. 2. The average score of both groups is the same on an experimental study(i.e. µ 1 = µ 2 = 0) so there is no effect. 3. The standard deviations of both groups differ: σ 1 = 1.11 and σ 2 = We would like to test equality of population means at the 5% significance level. 5. How can we do so while minimizing bias due to unequal sample sizes and different standard deviations? Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

5 Student s problem Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

6 Student s problem 1. How to test that two population means are equal? { H o : µ 1 = µ 2 H a : µ 1 µ 2 (1) 2. We assume i.i.d. normal samples: { X i N (µ 1, σ 2 1 ) Y i N (µ 2, σ 2 2 ) (2) 3. Let s further assume equal population variances: σ 1 = σ 2 4. Test statistic, assuming equal variances: T = X Ȳ s p 1 N N 2 sp 2 = (N 1 1)s1 2+(N 2 1)s2 2 N 1 +N 2 2 (3) 5. Problem: how is T distributed? Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

7 Student s solution 1. Note that T is distributed as follows: χ 2 v v T N (0, 1) χ 2 v v where N (0, 1) 2. Exact formula for Student s pdf with degrees of freedom v: (4) v+1 Γ( 2 ) f (t) = vπγ( v 2 3. Two-sided test with significance level α: t2 v 1 + ) ( 2 ) (5) )(1 v T > t 1 α/2,v (6) 4. Exact degrees of freedom if we assume equal variances: v = N 1 + N 2 2 (7) Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

8 Checking for equal variances Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

9 Checking for equal variances 1. We can specify the F-statistic in terms of the sample variances: F = s2 1 s For large samples, F 1 since: s1 2 P σ2 1 s2 2 P σ2 2 (8) (9) so we may use the F distribution to construct a statistical test. The problem is that the power of this test depends on min{n 1, N 2 }, which is presumably small, so we accept H o : σ 1 = σ 2 much too often! 3. Levene s test is slightly more robust, but suffers a similar weakness. Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

10 The Welch generalisation Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

11 Welch generalisation part I 1. Let s assume σ 1 σ 2 2. We can no longer pool variances so we have: T W = X Ȳ (10) s 2 1 N 1 + s2 2 N 2 3. Problem: How is T W distributed? (Hint: This depends on v.) Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

12 Welch generalisation part II 1. Welch noted that if σ and v are carefully chosen: s1 2 + s2 2 χ 2 σ 2 v N 1 N 2 v 2. To have agreement of the first moments in (11), we must solve: (11) E( s2 1 + s2 2 ) = E(χ 2 σ 2 v N 1 N 2 v ) (12) 3. In order to have agreement of the second moments: Var( s2 1 + s2 2 ) = Var(χ 2 σ 2 v N 1 N 2 v ) (13) 2. By solving both (12) and (13) we can theoretically approximate v as follows: ( σ2 1 N v = 1 + σ2 2 N 2 ) 2 (14) 1 N 1 1 ( σ2 1 N 1 ) N 2 1 ( σ2 2 N 2 ) 2 Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

13 Welch generalisation part III 1. Let s remember that: s1 2 P σ2 1 s2 2 P σ2 2 (15) 2. (15) suggests the following practical estimate of (14): v 3. Note that by inspection of (10) and (11): ( s2 1 N 1 + s2 2 N 2 ) 2 (16) 1 N 1 1 ( s2 1 N 1 ) N 2 1 ( s2 2 N 2 ) 2 where N (0, 1) χ 2 v v. T W N (0, 1) χ 2 v v (17) Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

14 Two-sample two-sided Welch test procedure 1. Define hypothesis: { H o : µ 1 = µ 2 H a : µ 1 µ 2 (18) 2. Choose significance level α, i.e. power of statistical test. 3. Calculate T W. 4. Calculate ˆv using Welch formula. 5. Perform two-sided test with signficance level α: T W > t 1 α/2,ˆv (19) Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

15 Concrete example in R Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

16 Experimental details 1. N 1 = 38 participants are in group X, and N 2 = 22 participants are in group Y. 2. The average score of both groups is the same on an experimental study(i.e. µ 1 = µ 2 = 0) so there is no effect. 3. The population standard deviations differ: σ 1 = 1.11 and σ 2 = We would like to test equality of population means at the 5% significance level(i.e. 95% confidence level). 5. We shall compare the analyses of the two-sample Welch test and two-sample Student s test using 5000 simulations. Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

17 Experiment settings in R require ( car ) # Car package required for Levene s test n1 <-38 # sample size for group X n2 <-22 # sample size for group Y sd1 <-1.11 # sd of group X sd2 <-1.84 # sd of group Y m1 <-0 m2 <-0 trued < -(m2-m1 )/( sqrt (((( n1-1 )*(( sd1^2 ))) + (n2-1 )*(( sd2^2 )))/(( n1+n2)-2 ))) nsims < # number of simulated experiments p1 <-numeric ( nsims ) p2 <-numeric ( nsims ) pvaluelevene <- numeric ( nsims ) # create variables for dataframe catx <- rep ("x",n1) caty <- rep ("y",n2) condition <- c(catx, caty ) Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

18 Run 5000 simulations # define random seed set. seed (0) # run simulations for (i in 1: nsims ){ # for each simulated experiment sim _x<- rnorm (n = n1, mean = m1, sd = sd1) # simulate condition X sim _y<- rnorm (n = n2, mean = m2, sd = sd2) # simulate condition Y # perform Student and Welch t - test ## perform the student t - test and store p - value p1[i]<-t.test (sim _x,sim _y, alternative = "two. sided ", var. equal = TRUE )$p. value # perform the welch t - test and store p - value p2[i]<-t.test (sim _x,sim _y, alternative = "two. sided ", var. equal = FALSE )$p. value # create dataframe for levene s test xy <- c( sim _x,sim _y) alldata <-data. frame (xy, condition ) # perform Levene s test pvaluelevene [i]<- levenetest ( alldata$xy ~ alldata$condition, data = alldata )$"Pr(>F)"[1:1] } Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

19 Power of Levene s test 1. Levene s test result: observedpowerlevene <-sum ( pvaluelevene < 0.05)/ nsims * The observed power for the Levene test is Even though the variances in our simulation are not equal, Levene s test only reveals this difference in 63% of the tests. Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

20 Histograms of p-values: student s test Note that more than half the p-values are below 0.5, indicating a deviation from the uniform distribution. Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

21 Histograms of p-values: Welch test Note that this distribution is approximately uniform which is to be expected as p-values should be uniformly distributed under the null hypothesis. Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

22 Remarks 1. When min{n 1, N 2 } is small, Levene s test is unreliable for checking equality of variances. 2. Student s two sample test is unreliable when the variances differ significantly. 3. If variances are equal, Welch s t-test and Student s t-test return the same p-value. 4. In most cases, Welch is a good default option with respect to type I error and statistical power. 5. R uses the Welch test by default. Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

23 Questions Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

24 References: 1. Miller Jr, R. G. (1997). Beyond ANOVA: basics of applied statistics. Chapman and Hall/CRC. Chapter Student. The Probable Error of a Mean. Biometrika 6, 1-25, Delacre, M., Lakens, D., & Leys, C. (in press). Why psychologists should by default use Welchs t-test instead of Students t-test. International Review of Social Psychology. Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25

25 The End Aidan Rocke (CU) Welch-Satterthwaite October 19, / 25