Practical Solutions to Behrens-Fisher Problem: Bootstrapping, Permutation, Dudewicz-Ahmed Method

Transcription

1 Practical Solutions to Behrens-Fisher Problem: Bootstrapping, Permutation, Dudewicz-Ahmed Method MAT653 Final Project Yanjun Yan Syracuse University Nov. 22, 2005

2 Outline Outline 1 Introduction 2 Problem Description 3 4 Results and Discussion 5 Future Work

3 Introduction Behrens-Fisher Problem Behrens-Fisher Problem is to test the composite null hypothesis H 0 : µ 1 = µ 2, for two normal distributions when the means µ 1, µ 2 and the variances σ1 2, σ2 2 are all unknown and hence possibly unequal. There are three exact solutions, and the Dudewicz-Ahmed Method [1, 2] is shown to be the most efficient procedure with the least requirement on the sample size most of the cases [3]. However, all three exact procedures need a second stage sampling, and if the sample size is limited, these procedures may not achieve their optimal performance.

4 Introduction Practical Behrens-Fisher Problem Q: If the sample size is limited such as n x = n y = 10 or 15, yet we still want to tell whether the two populations differ in mean, what can we do? A: For the D-A method, we may use the first n 1 observations as the first stage sampling and control the c value to make the second stage sampling take only 1 more observation. Or we may use resampling techniques such as bootstrapping and permutation to make the decisions. Q: After the practical modifications, what are the achievable power β given the required level α then? This is exactly the goal of this project.

5 Problem Description Problem Description Goal Given the sample drawn from certain distribution, set the desired level α, and use a two-tail decision rule to make a decision. By Monte-Carlo simulation, when the true 0, the proportion of all refusals is the estimated power β. Parameters For fair comparison, this project used the same parameters as in [3]. n = 10, 15, σ1 2 = 0.1, 1, 10, α = 0.01, 0.05, = 0.25, 0.5, 1, 1.5, 2, σ2 2/σ2 1 = 1, 1.5, 2, 5, 10, 20, 100,.

6 Problem Description Monte-Carlo Trial Number Given N samples from the same normal distribution, i = 1... N, let { 1 if reject H 0 z i = (1) 0 if not to reject H 0 then the probability for z i = 1 is α, each trial (sample) is Bernoullie(α), and z 1 + z z N is Binormial B(N,α). ˆα = z = B(N,α) n α. Approximate B(N,α) n by normal N (α, α(1 α) N ). Now that max α(1 α) = 1 4, the standard deviation of ˆα should satisfy SD(ˆα) /N. When N = 10, 000, ±2SD(ˆα) = ±.01; when N = 100, 000, ±2SD(ˆα) = ±.001, which is what we hope when α is as small as.01. So N = 100, 000 is a good choice of trial number. 1 4

7 Problem Description Flow Chart For n=10, 15 The For loops are nested from up to down. For simplicity, the nest is not completely shown For α =0.01, 0.05 Set B and allocation 2 For σ 1 =0.1, 1, 10 For = 0, 0.25, 1, 1.5, 2 For σ 2 /σ 2 =1, 1.5, 1 2 2, 5, 10, 20, 100, 10^10 For N_index = 1:N read x0, y0 x=x0(1:n)* σ y=(y0(1:n)+ )* 1 σ 2 Bootstrapping allocation (B times) Permutation allocation (B times) Fixed sample sized D-A method Fixed sample sized natural mean D-A method Pseudo distribution, Comparison, Decision Pseudo distribution, Comparison, Decision Interpolated β, Decision Interpolated β, Decision Save decision and β. βˆ Save βˆ for each parameter combination

8 Four Decision Procedures 1 Permutation 2 Bootstrapping 3 Fixed-Sample-Sized D-A Method 4 Fixed-Sample-Sized Natural Mean D-A Method All the four methods are defined for a fixed and limited sample size. Given one method and a set of parameters, the decision procedure is the same, but each sample has a different critical value for the decision, thus the decision rule for each sample is essentially different.

9 Random Number Generators The uniform random numbers are generated by URN22 in [4]. The normal random numbers are generated based on the Box-Muller Transformation in the form of Theorem on page 180 in [4]: X 1 = 2ln(U 1 )sin(2πu 2 ) Samples from other distributions are supposed to use the generalized lambda distribution (GLD) with four parameters: location, scale, skewness and kurtosis as introduced in [4] on page 208.

10 Permutation Permutation Permutation is resampling from the original samples without replacement. The permutation is implemented by sampling the shuffled observations from both x and y to construct a new resample x and y and calculate a t like test statistic T for mean comparison. Repeat this resampling many times and construct the pseudo population of T. Determine the critical values based on level α, and make a decision. This procedure assumes that the null hypothesis is true thus the two populations can be mixed up since the two populations are assumed to have the same means.

11 Permutation Permutation Another explanation of the shuffle, from the calculation aspect, is that there are two original samples x and y, and the test statistic is the t-like statistic as T = d s d / n ; if x is randomly sampled exclusively from x, and y is randomly sampled exclusively from y, the T will be bounded by unchanged d, 1 and only scaled by s d /, which is not random. n The d is defined as d = y x since in reality we don t know the sign of, so we should use a two tail t-like test.

12 Bootstrapping Bootstrapping In bootstrapping, the resamples are sampled from the original sample with replacement. So x can be either sampled exclusively from x, and y is sampled exclusively from y; or x and y are sampled from the pool of x and y. For fair comparison with the permutation though, it s assumed that the null hypothesis H 0 is true, so the bootstrapping also mixes up the original two samples, and x and y are sampled from the pool of x and y with replacement.

13 Bootstrapping Bootstrapping vs Permutation Although the procedures of bootstrapping and permutation are similar in this project except for replacement, the underling implications differ a lot. Bootstrapping is usually used for parameter estimation and confidence interval approximation; but permutation is usually used for hypothesis testing even without the need to define a parameter. If the hypothesis is about a parameter, the parameter estimation such as from bootstrapping may be further interpreted for hypothesis testing. In this project, we are interested in hypothesis testing on the mean comparison, so the bootstrapping and permutation methods are implemented and interpreted similarly.

14 Bootstrapping Pseudo Population Generation and Decision Making In both permutation and bootstrapping, we do the resampling B times for each single sample to generate a pseudo population from this sample. From the pseudo population, an empirical distribution of the mean comparison statistic T can be generated. Based on the distribution of T, determine the two tail critical values t α/2 and t 1 α/2, where the probability P (T < t α/2 ) = α/2 and P (T t 1 α/2 ) = α/2. Calculate the t value of the original sample, compare it to t α/2 and t 1 α/2. The decision rule is as follows. z i = { 1 if t < t α/2 or t > t 1 α/2 0 if t α/2 t t 1 α/2 (2)

15 Fixed-Sample-Sized D-A Method Fixed-Sample-Sized D-A Method In order for fair comparison, we set the first stage sample size in D-A method as n 1, and specify the constants appropriately to make both N 1 and N 2 to be n. Thus the smaller c = min(c 1, c 2 ) should be selected to satisfy both conditions. { c 1 n/(s 1 (S 1 + S 2 )) c 2 n/(s 2 (S 1 + S 2 )) (3)

16 Fixed-Sample-Sized D-A Method Fixed-Sample-Sized D-A Method After n and α are specified, h can be evaluated from Table I in paper [2]. For the parameters in this project, the h values are listed in the table below. Table: h values used in this project p h n Then the critical value of the D-A decision rule is K = h/c.

17 Fixed-Sample-Sized D-A Method Fixed-Sample-Sized D-A Method The decision rule is defined as { 1 if x ȳ > K z i = 0 if x ȳ K (4) where x = a 1 x a n x n, ȳ = b 1 y b n y n, since the common c is used for both samples, and N 1 = N 2 = n. From the exact solution in paper [5], set a 1 =... a n 1 = a, then a n = 1 (n 1)a. The same relation holds for b s. The coefficients can be derived from the following equations. { n i=1 a2 i = 1 1 S 1 (S 1 +S 2 ) c 2 n i=1 b2 i = (5) 1 1 S 2 (S 1 +S 2 ) c 2

18 Fixed-Sample-Sized D-A Method Fixed-Sample-Sized D-A Method a = 1 n + r 1 n+ (n 1)n S 1 (S 1 +S 2 )c 2 (n 1)n a n = 1 (n 1)a b = 1 n + r 1 n+ (n 1)n S 2 (S 1 +S 2 )c 2 (n 1)n b n = 1 (n 1)b These coefficients are utilized in calculating x and ȳ. In the original D-A method, the constant Q or c is determined from the required power β, but here since the c is determined from sample size requirement, hence Q = c is also determined, thus the theoretical β value is varied.

19 Fixed-Sample-Sized D-A Method Fixed-Sample-Sized D-A Method The different Q = c values for different β, when α and n are set, are listed in the table below based on Table II in paper [2]. The inverse relation is pursued here. Table: The Q values for different β when α and n are set Line # α β n Q

20 Fixed-Sample-Sized D-A Method Fixed-Sample-Sized D-A Method β A scatter plot of β vs Q is illustrated below. Smaller α or n tend to have a bigger Q value. Many curve fitting algorithms, such as polynomial, exponential, logarithm or spline, are tried to approximate this nonlinear trend. The well fitted and simplest cubic polynomial curve fitting is adopted in the following form. β = p 1 Q 3 + p 2 Q 2 + p 3 Q + p 4 (6) 1 The β values for different Q when α and n are set Q value Line 3: α=0.05, n=10 Line 4: α=0.05, n=15 Line 1: α=0.01, n=10 Line 2: α=0.01, n=15

21 Fixed-Sample-Sized D-A Method Fixed-Sample-Sized D-A Method β The fitted curves are shown below. The curve fitting of β values for different Q when α and n are set beta vs. y3 fit 1 beta vs. y4 fit 2 beta vs. y1 fit 3 beta vs. y2 fit Q value

22 Fixed-Sample-Sized Natural Mean D-A Method Fixed-Sample-Sized Natural Mean D-A Method As an approximate D-A method, x and ȳ can be replaced by the natural mean x and ȳ. The critical value K are kept the same as in the fixed-sample-sized D-A method.

23 Results and Discussion β Empirical Power of the Four Procedures on Normal Data When n = 10, α = 0.01, B = 5n/α = 5000, σ1 2 = 0.1, = 2, try = [1, 1.5, 2, 5, 10, 20, 100, ]: σ 2 2 σ Empirical Power β when = Bootstrap Permutation D A mean D A ^ σ /σ1 2

24 Results and Discussion β Empirical Power of the Four Procedures on Normal Data When n = 10, α = 0.01, B = 5n/α = 5000, σ1 2 = 0.1, = 0.25, try σ2 2 = [1, 1.5, 2, 5, 10, 20, 100, ]: σ Empirical Power β when = Bootstrap Permutation D A mean D A ^ σ /σ1 2

25 Results and Discussion Empirical Power of the Four Procedures on Uniform Data When n = 10, α = 0.01, B = 5n/α = 5000, σ1 2 = 1, = 1, try = [1, 1.5, 2, 5, 10, 20, 100, ]: Interesting enough, all four σ 2 2 σ 2 1 methods achieve a 100% power, though the data distribution does have overlap between the two populations as shown below: Two Uniform Populations x y x 10 5

26 Future Work Future Work 1 Given a set of distribution parameters without the constraint on the sample size, try the original D-A method and compare it with the practical solutions in this project. Take n = 5, σ 2 1 = 0.1, α = 0.01, = 1.5, β = 0.95, σ2 2 /σ2 1 = 1.5, it s known that E(N1) or E(N2) is according to Dr. Dudewicz s previous paper [3]. We may use this result to double check whether our functions work. 2 We ve tried to use the Neyman Pearson Decision Rule as a benchmark on how well the practical decision rules can be, based on the known distributions of the populations. However, current calculation hasn t achieved a consistent pattern between the benchmark and the practical solution, so the result was not shown in this presentation, but more work is expected in this area as well.

27 Future Work Future Work 3 Given the requirement on level α, we may test the achievable ˆα by evaluating the power function β( ) at = 0. However, even though the procedures are the same for different samples, the critical values for each sample is different. Namely the decision rule differs sample by sample, and the power function differs accordingly, thus there s no meaningful way to evaluate it at = 0. 4 If the samples are drawn from other popular distributions, such as β(r, s) or Uniform(a, b), these procedures may perform drastically differently, since D-A method is designed for the normal data. It ll be interesting to see the difference.

28 References REFERENCES E. J. Dudewicz and S. U. Ahmed, New exact and asymptotially optimal solution to the behrens-fisher problem, with tables, American Journal of mathematical and management sciences, vol. 18, pp , E. J. Dudewicz and S. U. Ahmed, New exact and asymptotially optimal heteroscedastic statistical procedures and tables, ii, American Journal of mathematical and management sciences, vol. 19, pp , E. J. Dudewicz, Y. Ma, E. S. Mai, and H. Su, Exact solutions to the behrens-fisher problem: Asymptotially optimal and finite sample efficient choice among, in Plenary Talk of International Conference on Statistics, Combinatorics and Related Areas SCRA (10th International Conference of the Forum for Interdisciplinary Mathematics), October Z. A. Karian and E. J. Dudewicz, Modern Statistical, Systems, and GPSS Simulation. USA: CRC Press, M. Aoshima, H. Hyakutake, and E. J. Dudewicz, An asymptotically optimal fixed-width condifence interval for the difference of two normal means, Sequential Analysis, vol. 15, no. 1.

29 References Thank you!