Using the Bootstrap for Estimating the Sample Size in Statistical Experiments

Transcription

1 Journal of Modern Appled Statstcal Methods Volume 1 Issue 1 Artcle Usng the Bootstrap for Estmatng the Sample Sze n Statstcal Experments Maher Qumsyeh Unversty of Dayton Follow ths and addtonal works at: Part of the Appled Statstcs Commons, Socal and Behavoral Scences Commons, and the Statstcal Theory Commons Recommended Ctaton Qumsyeh, Maher (013) "Usng the Bootstrap for Estmatng the Sample Sze n Statstcal Experments," Journal of Modern Appled Statstcal Methods: Vol. 1 : Iss. 1, Artcle 9. DOI: 10.37/jmasm/ Avalable at: Ths Regular Artcle s brought to you for free and open access by the Open Access Journals at DgtalCommons@WayneState. It has been accepted for ncluson n Journal of Modern Appled Statstcal Methods by an authorzed edtor of DgtalCommons@WayneState.

2 Usng the Bootstrap for Estmatng the Sample Sze n Statstcal Experments Cover Page Footnote Ths research s supported by Unversty of Dayton Seed Grant. Ths regular artcle s avalable n Journal of Modern Appled Statstcal Methods: ss1/9

3 Journal of Modern Appled Statstcal Methods Copyrght 013 JMASM, Inc. May 013, Vol. 1, No. 1, /13/$95.00 Usng the Bootstrap for Estmatng the Sample Sze n Statstcal Experments Maher Qumsyeh Unversty of Dayton, Dayton OH Efron s (1979) Bootstrap has been shown to be an effectve method for statstcal estmaton and testng. It provdes better estmates than normal approxmatons for studentzed means, least square estmates and many other statstcs of nterest. It can be used to select the actve factors - factors that have an effect on the response - n expermental desgns. Ths artcle shows that the bootstrap can be used to determne sample sze or the number of runs requred to acheve a certan confdence level n statstcal experments. Key words: Efron s bootstrap, expermental factors, statstcal estmaton, confdence level. Introducton Tradtonal methods of fndng sample szes depend on knowng the underlyng dstrbuton. For example, to determne a sample sze that wll result n (1-α) 100% confdence that the sample mean s wthn E unts from the populaton mean the followng s used: zα/σ n = (1) E assumng normalty, or usng the central lmt theorem, and determnng an approxmate value for σ. For a multple regresson model y = β0 + β1x1+ βx + + βkxk + ε () a (1 α) 100% confdence nterval for β j s gven by β ± t j 1 R. (1 R )( N p) / j (3) Maher Qumsyeh s an Assstant Professor n the Department of Mathematcs. Pror to jonng the Unversty of Dayton he was a Professor at Bethlehem Unversty for over 5 years, wth 10 years as Department Char. Emal hm at: qumsyeh@udayton.edu. where, R s the multple coeffcent of determnaton and R j s the same when x j s predcted from the remanng k 1 regressors. Usng equaton (3), the sample sze to predct the standardzed coeffcent wthn E unts of the true value (replacng t wth normal) s gven by z / (1 R ) n = + p. (4) E ( 1 ) R j However wth ths n there s approxmately a 50% chance that the nterval wll be longer than E (Kelly & Maxwell, 003). Hahn and Meker (1991) provde a value for N for whch the confdence s (1-δ) 100% that the nterval obtaned s of a length less than or equal E. The value of such N s ( n ) z / ) 1 (1 R χ δ N = + p. E ( 1 ) R n p j (5) where n s the value found n equaton (4). An alternatve s to use the bootstrap method to determne f the sample sze calculated usng equatons (1) and (5) s necessary or f t s larger than what s needed to acheve a certan confdence. The bootstrap has been shown to provde better than normal estmates of dstrbuton functons of studentzed 45

4 BOOTSTRAP USE FOR ESTIMATING SAMPLE SIZE IN STATISTICAL EXPERIMENTS statstcs (Sngh, 1981; Bckle & Freedman, 1980; Babu and Sngh, 1983, 1984). Qumsyeh (1994) showed that bootstrap approxmaton for the dstrbuton of the studentzed least square estmate s asymptotcally better, not only than the normal approxmaton, but also than the twoterm Edgeworth expanson. Lahr (199) showed the superorty of the bootstrap for approxmatng the dstrbuton of M-estmators. Bhattacharya and Qumsyeh (1989) preseneted an L p -comparson between the bootstrap and Edgeworth expansons. Fnally, Qumsyeh and Shaughnessy (008, 010) showed that the bootstrap can be used to determne the actve factors n two level desgns and how to estmate mssng responses n those desgns. In ths study the bootstrap was appled to three data sets; SAS and the SQL procedure n SAS were used to perform calculatons and resamplng. Data Set 1 Data set 1 s comprsed of 1,000 randomly selected samples of sze 61 each from a normal dstrbuton wth mean 0 and standard devaton (61 s the number n obtaned usng Equaton (1) wth E = 0.5 and α = 0.05). An example of one such sample of sze 61 s: The mean for ths sample was and the standard devaton was Data Set Data set s real data that correlates the GPA y (out of 4) of 194 students from Bethlehem Unversty, wth ther hgh school Math (x 1 ) and Englsh (x ) scores (out of 100 ponts). The frst few observatons are shown n Table 1. The model for data set two s: y = β + β x + β x + ε Table 1: Data Set Example y x 1 x Data Set 3 Data set 3 s an example provded by Bsgaard and Fuller (1995). It s a 4 full factoral experment to determne f blade sze (A), centerng (B), levelng (C) and speed (D) had an effect on the occurrence of undesrable marks on a steel sample. The desgn matrx s shown n Table, where Y represents the number of defectve (undesrable marks) among 0 samples at each settng and P s the proporton of defects at each settng. The Bootstrap The bootstrap was used to analyze the three data sets. SAS programmng and the SQL procedure n SAS were used to perform the analyses. Resamplng wth replacement was conducted 1,000 tmes based on Efron and Tbshran (1993) fndng that 1,000-,000 works best. The SAS program used for data set 1 46

5 MAHER QUMSIYEH Table : Data Set 3 Desgn Matrx Run A B C D Y P s provded n Appendx A; due to the length of the programs for data sets two and three they are not provded. A dfferent procedure was used for each data set. Data Set 1 For the frst example the sample sze was 61, ths s the sample sze necessary for 95% confdence that the sample mean s wthn 0.5 unts from the populaton mean usng the zα/σ equaton: n =. Usng resamplng and E takng a random sample of sze 61 from a N(0, ) dstrbuton, t s resampled 1,000 tmes wth replacement, the mean of each of the 1,000 samples s calculated and half the dfference between the.5 and 97.5 percentles of the 1,000 means s found. Ths should be the value of E. One sample of sze 61 from a N(0, ) dstrbuton to another the value of such E wll vary to a great degree, thus, ths procedure s repeated several tmes (500 n ths case) and an nterval for the values of such E s s lsted. The sample sze contnued to decrease and the values of E contnued to be recorded. Results are shown n Table 3. Table 3: Data Set 1 Results n Table 3 shows that a sample of sze 61 was not necessary; 48 would have been suffcent. The bootstrap was repeated 500 E

6 BOOTSTRAP USE FOR ESTIMATING SAMPLE SIZE IN STATISTICAL EXPERIMENTS tmes, each resamplng 1,000 tmes usng a dfferent 61 randomly selected data ponts wth replacement from a N(0, ) dstrbuton and the values of all 500 replcatons were n the nterval gven whch s for n = 61. (See Appendx A for the SAS program used for data set 1 wth n = 48.) Data Set The second data set, whch correlates the unversty GPA to hgh school Englsh and math scores, the model s: y= β 0 + β 1 x 1 + β x +ε. The ntal results usng SAS for the whole data set are shown n Table 4. Frst usng equaton (4), z / (1 R ) n = + p, E ( 1 ) R j to fnd an ntal n usng α as 0.05 and E as 0. for the standardzed betas (the value of E depends on the type of data at hand). Usng all 194 data ponts and the values of R and R j from the data set (R and R j ) and equaton (4), the value of s 60 (approxmatng to the next nteger), however, wth ths n there s approxmately a 50% chance that the nterval wll be longer than E. (Kelly & Maxwell, 003). By contrast, usng equaton (5), ( n ) z / ) 1 (1 R χ δ N = + p, E ( 1 ) R n p j δ = 0.05 and j = the value of N s determned to be 78; ths results n 95% confdence that the nterval obtaned s of a length less than or equal E (Hahn & Meker, 1991). Note that the value of E used s for the standardzed betas; thevalue of E for the non-standardzed betas (EN) wll be approxmately S y EN E = = S x Next, the half-length of the confdence nterval for β s determned usng the bootstrap method and a random sample of sze 78 from the 194 orgnal observatons. The procedure s as follows: 1. Select 78 ponts usng random samplng wthout replacement from the 194; name ths as subset and perform a regular regresson procedure obtanng (X 1,1, X,1, Y 1,E 1 ), (X 1,, X,, Y,E ),..., (X 1,78, X,78, Y 78,E 78 ). Here, E = Y Y, s the resdual for the th observaton.. Select 1,000 samples wth replacement from the subset, ths s the bootstrap sample. Each sample has 78 ponts and samples are desgnated as {sample 1 }, {sample },..., {sample 1000 }. 3. Examne {sample 1 }; n=78 ponts taken wth replacement from the subset. We have sets of ponts (X 1,1 *, X,1 *, Y 1 *, E 1 *), (X 1, *, X, *, Y *, E *),..., (X 1,78 *, X,78 *, Y 78 *,E 78 *). Each (X 1,j *, X,j *, Y j *, E j *) can be any of the (X 1,1, X,1, Y 1,E 1 ), (X 1,, X,, Y, E ),..., (X 1,78, X,78, Y 78, E 78 ) wth probablty 1/ Fnd the average of the E * s, and name ths ME 1. Due to the fact that the mean of the errors s assumed to be 0, standardze the errors by subtractng ME 1 from each of them. 5. Stll usng {sample 1 }, the new Y s are obtaned by the addng the respectve standardzed error term to the predcted values and these are termed as new Y j s. 6. Contnue to examne {sample 1 }; usng the least-square method, fnd the slope and the ntercept, (slope 1, slope, ntercept 1 ), based on (X 1,1 *, X,1 *,, newy 1 * ), (X 1, *, X, *, newy * ),..., (X 1,78 *,X,78 *, newy 78 *). 7. Repeat steps 3-6 for the other 999 samples to obtan 1,000 estmates for the ntercept β 0 and the slopes β 1 and β. Interest s n β. 8. Estmate the value of β by averagng the 1,000 estmates of β and calculate a 95% CI for β by fndng the.5 and 97.5 percentle 48

7 MAHER QUMSIYEH of those 1,000 values. Half the length of ths nterval, E *, wll be compared wth the EN = prevously obtaned. Based on ths procedure, how s t known that there s 95% confdence that half the length of the nterval wll not exceed EN? The answer s that by repeatng steps 1-8, 1,000 tmes to obtan 1,000 EN s and then fndng the top 95 percentle, t should not exceed EN. The estmate for β from one random subset of 78 ponts was = and a 95% CI for β was (0.0089, 0.04). Ths assumes that all condtons, such as normal resduals and constant varances, hold; n addton, half the length of the nterval s , whch s smaller than expected (0.0097). The n = 78 guarantees that 95% of the cases wll result n smaller half lengths. Usng the bootstrap method dscussed results n a mean half-length of 1,000 runs of the bootstrap method of and a 95% confdence nterval of ( , 0.010): ths s wthout any assumptons on the model. An estmate for β was calculated as an average of the 1,000 bootstrap sample estmate for β t was = for one run wth a 95% CI of ( , ). The bootstrap was repeated 1,000 tmes and the average value for the estmated values of β was wth a 95% CI of ( ). Wthout any assumpton on the model, the bootstrap produced an estmate for β that was close to that produced assumng the regular model assumptons hold, n addton, the length of the 95% CI was a lttle shorter than expected usng a sample of sze 78 ( vs ). It s mportant to note that the calculatons were carred out wthout assumng the error terms to be normal, however, t s valuable to understand what wll happen f the error terms n the example are exactly normal. Table 4: SAS Results for Data Set Model: Model1 Dependent Varable: y Number of Observatons Read: 194 Number of Observatons Used: 194 Analyss of Varance Source DF Sum of Mean Squares Square F Value Pr > F Model < Error Corrected Total Root MSE R-Square Dependent Mean Adj R-Sq Coeff Var Parameter Estmates Varable DF Parameter Standard Estmate Error t value Pr > t 95% Confdence Lmts Intercept x < x <

8 BOOTSTRAP USE FOR ESTIMATING SAMPLE SIZE IN STATISTICAL EXPERIMENTS nd Data Set (Normal Errors) For a random sample of sze 78, the predcted values of the y s, y were calculated and a new varable w was defned as: w = y + cε where ε s randomly chosen from a N(0,1) dstrbuton. The new w s wth the orgnal x s wll have normal errors wth constant varance. Note that the varance of the w s must be the same as the orgnal y s to be able to compare the length of the confdence nterval for the new β wth the prevous one. The solver functon n Mcrosoft Excel was used to provde a value for c to acheve ths; ths value of c was calculated to be Ths mproved the prevous results and half the length of the 95% CI for β usng the bootstrap method was much smaller ( vs ). Ths shows that usng the bootstrap requres a smaller sample sze than the prevous estmate of n = 78. Data Set 3 Bsgaard and Fuller (1995) provded a table that gves estmated sample szes (n) for the number of runs at each settng for two level full factoral experments usng proportons as a response. Ther estmate for n whch represents the number of runs needed to detect an error of sze Δ n the untransformed scale s gven by n ( z z β ) Nδ / = (6) where N s the total number of basc runs n a k factoral experment (4, 8, 16, ), α and β are the probabltes of type I and type II errors, 0.05 and 0.1 respectvely, and δ s the expected value of the effect (Bsgaard & Fuller, 1995). Bsgaard and Fuller s table presents values of Δ that vary from 10% to 90% of the proporton of defectve (p 0 ) and shows that sample sze depends on the average defectve level. If the average defectve level s low, for example 5%, a larger sample sze s needed to ndcate that a change has truly occurred. For the 3 rd data set the current level of defectve (p 0 ) was not known, t was approxmated wth the average proporton of defectve n the sample at each settng whch s Because n s gven n ths experment as 0, the method descrbed n Bsgaard and Fuller (1995) or the table they provde can be used to determne the mnmum sze of detectable error. For α to be at most 0.05, the mnmum sze s Δ >0.185; calculatng the effect sze of each factor, t was found that factors A, B and the AB nteracton have effect szes larger than ths (0.984, 0.08, 0.47 respectvely). Ths agrees wth the half normal plot (Danel, 1959) whch states that factors B and the AB nteracton appear to be slghtly actve (not very clear) and that factor A s a defntely actve factor (see Fgure 1). In the calculatons descrbed t s not certan that a 95% confdence nterval for the effect sze wll have ts lower bound less than for those factors (A, B and AB). Qumsyeh and Shaughnessy (008, 010) showed that the bootstrap can be used (under no assumptons) to determne actve factors n factoral experments, to estmate the sze of the effect and to determne a confdence nterval for the effect sze. The method can be descrbed wth the followng steps usng factor A for llustraton purposes: 1. Sample N/ responses wth replacement from data at the +1 level of the gven factor A.. Sample N/ responses wth replacement from data at the 1 level of the gven factor A. 3. Estmate the effect of that factor usng the dfference between +1 level and 1 level. 4. Repeat the samplng procedure a large number of tmes (1,000 n ths example). 5. Fnd the average of the 1,000 values; ths s an estmate of the effect sze of factor A. Determne the upper (1 α/) and lower α/ percentle ponts of the resampled effect values found n step 4. Use these values to construct the 50

9 MAHER QUMSIYEH Fgure 1: Half-Normal Plot for the Effects n Data Set 3.5 A CD ACD ABD AC BD C BCD D BC ABCD ABC AD B AB effect sze. If the confdence nterval doesn t contan 0 then ths factor s an actve factor a factor that has an effect on the response. Usng the procedure descrbed prevously for data set 3, the followng were determned: All confdence ntervals for the effect sze for all factors except factor A contaned 0, therefore they must be assumed as nactve factors. For factors A, B and the AB whch appear to be effectve usng the normal plot and are reported as actve factors by Bsgaard and Fuller (1995), usng the proporton of defectves, the results were as follows: Factor A: The mean effect sze for the 1,000 runs was and a 95% confdence nterval for the effect sze was (0.6188, ). Factor B: The mean effect sze for the 1,000 runs was and a 95% confdence nterval for the effect sze was ( 0.313, ). Factor AB (The AB nteracton): The mean effect sze for the 1,000 runs was and a 95% confdence nterval for the effect sze was ( 0.406, ). Results show that only factor A can be consdered actve. If ths s the case, the confdence nterval for effect A has a lower bound of whch leads to a much hgher value than the least expected of Ths ndcates that a sample sze smaller than 0 would have been suffcent. 51

10 BOOTSTRAP USE FOR ESTIMATING SAMPLE SIZE IN STATISTICAL EXPERIMENTS Concluson The bootstrap method can be used to determne sample szes n statstcal experments and to check whether a certan sample sze used s more than s needed by examnng the length of the confdence nterval resultng from usng the bootstrap method. The bootstrap s also good for selectng actve factors and n constructng confdence ntervals for effect sze. The avalablty of computers and statstcal software make usng re-samplng (bootstrap) easy and fast and provdes good predctons. Acknowledgment Ths research s supported by Unversty of Dayton Seed Grant. References Babu, G., & Sngh, K. (1983). Inference on means usng the bootstrap. The Annals of Statstcs, 11, Babu, G. J., & Sngh, K. (1984). On one term Edgeworth correcton by Efrons bootstrap. Sankhya, 46, Seres A, Bhattacharya, R. N., & Qumsyeh, M. (1989). Second order and L p - comparson between the bootstrap and emprcal Edgeworth expanson methodologes. Annals of Statstcs, 17, Bckel, P. J., & Freedman, D. A. (1980). On Edgeworth expansons for the bootstrap. Unpublshed. Bsgaard, S., & Fuller, H. (1995). Sample sze estmate for k-p desgns wth bnary responses. Qualty Engneerng, 7(4), Danel, C. (1959). Usng of half normal plots n nterpretng factoral two-level experments. Technometrcs, 1, Efron, B. (1979). Bootstrap methods: Another look at jacknfe. The Annals of Statstcs, 7, 1-6. Efron, B., & Tbshran, R. (1993). An ntroducton to the bootstrap. New York, NY: Chapman and Hall. Hahn, G. J., & Meeker, W. Q. (1991). Statstcal ntervals: A gude for practtoners. New York, NY: Wley. Kelley, K., & Maxwell, E. (003). Sample sze for multple regresson: Obtanng regresson coeffcents that are accurate, not smply sgnfcant. Psychologcal Methods, 8(3), Lahr, S. (199). Bootstrappng M- estmators of a multple lnear regresson parameter. The Annals of Statstcs, 0(3), Qumsyeh, M. (1994). Bootstrappng and emprcal Edgeworth expansons n multple lnear regresson models. Communcatons n Statstcal Theory and Methods, 3(11), Qumsyeh, M., & Shaughnessy, G. (008). Usng the bootstrap to select actve factors n unreplcated factoral experment. In JSM Proceedngs, Statstcal Computng Secton. Alexandra, VA: Amercan Statstcal Assocaton. Qumsyeh, M., & Shaughnessy, G. (010). Bootstrappng Un-replcated two-level desgns wth mssng responses. Journal of Statstcs: Advances n Theory and Applcatons, 4, Sngh, K. (1981). On the asymptotc accuracy of Efron s bootstrap. The Annals of Statstcs, 9, Appendx A: SAS Program Used for Data Set 1 %macro rep; %do rep=1 %to 500; data ma (drop=); *%let num=48; *do =1 to &num; ; do =1 to 48; x=*rannorm(56367)+0; output; end; %macro numberng(n); data numberng; do =1 to &N; output; end; qut; data ma1; set numberng; set ma; %mend; %numberng(48); %macro repeat; 5

11 MAHER QUMSIYEH Appendx A (contnued): SAS Program Used for Data Set 1 %do repeat=1 %to 1000; %macro dstnct(nthrow, N); data table; do j = 1 to &nthrow; pt = nt(nt(ranun(0) * &N) + 1.5); set ma1 pont=pt; output; end; stop; * requred for pont= ; %mend; %dstnct(48,48); proc means data=table n mean noprnt; var x; output out=table3 n=n mean=mx; qut; proc sql; create table tablef as select a1.mx as mx1 from table3 as a1; proc append data=tablef base=summary force; %end; %mend; %repeat; proc unvarate data=summary noprnt; var mx1; output out=z1 mean=tm pctlpts =.5, 97.5 pctlname=p5 p975 pctlpre = mx1; data E; set z1; E=(mx1p975-mx1p5)/; proc append data=e base=e1 force; proc sql; drop table E ; drop table Ma ; drop table Ma1 ; drop table Numberng ; drop table Summary ; drop table Table ; drop table Table3 ; drop table Tablef ; drop table z1 ; qut; %end; %mend; %rep; proc unvarate data=e1 noprnt; var E; output out=length mean=tm pctlpts =.5, 97.5 pctlname=p5 p975 pctlpre = E; qut; 53