The Bootstrap, Jackkife, Radomizatio, ad other o-traditioal approaches to estimatio ad hypothesis testig Ratioale Much of moder statistics is achored i the use of statistics ad hypothesis tests that oly have desirable ad well-kow properties whe computed from populatios that are ormally distributed. While it is claimed that may such statistics ad hypothesis tests are geerally robust with respect to o-ormality, other approaches that require a empirical ivestigatio of the uderlyig populatio distributio or of the distributio of the statistic are possible ad i some istaces preferable. I istaces whe the distributio of a statistic, coceivably a very complicated statistic, is ukow, o recourse to a ormal theory approach is available ad alterative approaches are required. I. Hypothesis Testig A. Normal Theory Approach For illustratio cosider Studet's t - test for differeces i meas whe variaces are ukow, but are cosidered to be equal. The hypothesis of iterest is that H 0 : µ = µ. While several possible alterative hypotheses could be specified, for our purposes H A : µ < µ. Give two samples draw from populatios ad, assumig that these are ormally distributed populatios with equal variaces, ad that the samples were draw idepedetly ad at radom from each populatio, the a statistic whose distributio is kow ca be elaborated to test H 0 : x s + x s, () where x, x, s, s,, are the respective sample meas, variaces ad sample sizes. Whe the coditios stated above are strictly met ad H 0 is true, () is distributed as Studet's t with ( + - ) degrees of freedom. As + ), t (0,). ( N The reasos for makig the assumptios specified above is to allow the ivestigator to make some statemet about the likelihood of the computed t - value, ad
to make a decisio as to whether to profess belief i either H 0 or H a. The percetiles of the t distributio with the computed degrees of freedom ca be iterpreted as the coditioal probability of observig the computed t value or oe larger (or smaller) give that H 0 is true: P ( t T H 0 ) = α.. Therefore we would probably wish to profess belief i H 0 for suitably large values of α ad disbelief for suitably small values. Embedded i this probability we must also iclude the distributioal assumptios metioed above P ( t T H, N( µ, σ ), ( µ, σ ), σ = σ ) = α. 0 N If a specific alterative hypothesis had bee stated, for example H A : µ = µ -, the uder the assumptio of ormality ad equal variaces, the t - statistic could be recomputed give the ew estimate of µ uder the alterative hypothesis µ = + x. The coditioal probability of obtaiig the observed differece i t values as computed uder the ull ad alterative hypotheses (t 0 - t a ), give specified α, the observed variaces, ad the particular alterative hypothesis could be computed: P ( t 0 t A δ T α, σ, σ, H A ) = β. This probability is also coditioed o the assumptio that both populatios are ormally distributed ( N ( µ, σ ), N ( µ, )). σ Recall that of these two coditioal probabilities α is the Type I error rate, the probability of rejectig the tested ull hypothesis whe true, ad that βis the Type II error rate, the probability of failig to reject the tested ull hypothesis whe false. I preset this review to emphasize that the estimatio of each of these probabilities, which are iterpreted as error rates i the process of makig a decisio about ature, i the course of iterpretig a specific statistical test, is totally cotiget o assumig specific forms for the distributio of the uderlyig populatios. To kow these error rates exactly requires that all the coditios of these tests be met. B. A distributio-free approach Oe way to avoid these distributioal assumptios has bee the approach ow called o - parametric, rak - order, rak - like, ad distributio - free statistics. A series of tests may of which apply i situatios aalogous to ormal theory statistics have bee elaborated (see ref,, 3 for expaded treatmets of these procedures).
The key to the fuctio of these statistics is that they are based o the raks of the actual observatios i a joit rakig ad ot o the observatios themselves. For example the Wilcoxo distributio - free rak sum test ca be applied i place of the sample or separate groups Studet's t test. The observatios from both samples combied are raked from least to greatest ad the sum of the raks assiged to the observatios from either sample is computed. If both samples are comprised of observatios that are of similar magitude, the the raks assiged to sample should be similar to the raks assiged to sample. For fixed sample sizes a fixed umber of raks are possible, for = ad = 7, raks will be assiged. Uder the ull hypothesis that the locatio of each populatio o the umber lie is idetical, the sum of the raks assiged to either sample should equal the sum obtaied from radomly assigig raks to the observatios i each sample. I this example there are C ways of assigig raks to sample ad C 7 or way to assig raks to sample after assigig the raks to sample. The total umber of possible arragemets of the raks is the C ad it is possible for each arragemet to compute the sum of the raks. From this we ca eumerate the distributio of the sum of the raks, W. W i this example ca rage from to 0. If we divide the umber of arragemets with w = W C the we have the probability of observig a particular value of w equal to P ( w = W H 0 ). To obtai the coditioal probability that w > W give H 0, we simply tabulate the cumulative probabilities. The oly additioal assumptio embedded i this approach is that the observatios are idepedet, but this is also a assumptio of the ormal theory approach. Notice we make o assumptio about the forms of the uderlyig populatios about which we wish to make ifereces, ad that the exact distributio of the test statistic, W, is kow because it is eumerated. These distributio - free statistics are usually criticized for beig less "efficiet" tha the aalogous test based o assumig the populatios to be ormally distributed. It is true that whe the uderlyig populatios are ormally distributed the the asymptotic relative efficiecies (ratio of sample sizes of oe test to aother ecessary to have equal power relative to a broad class of alterative hypotheses for fixed α) of distributio - free tests are geerally lower tha their ormal theory aalogs, but usually ot markedly so. I istaces where the uderlyig populatios are o-ormal the the distributio - free tests ca be ifiitely more efficiet that their ormal theory couterparts. I geeral, this meas that distributiofree tests will have higher Type II error rates (β) tha ormal theory tests whe the ormal theory assumptios are met. Type I error rates will ot be affected. However, if 3
the uderlyig populatios are ot ormally distributed the ormal theory tests ca lead to uder estimatio of both Type I ad Type II error rate. C. Radomizatio So far we have used two approaches to estimatig error rates i hypothesis testig that either require the assumptio of a particular form of the distributio of the uderlyig populatio, or that require the ivestigator to be able to eumerate the distributio of the test statistic whe the ull hypothesis is true ad uder specific alterative hypotheses. What ca be doe whe we either wish to assume ormality or ca we eumerate the distributio of the test statistic? Recall the aalogy I used whe describig how to geerate the expected sum of raks assiged to a particular sample uder the ull hypothesis of idetical populatio locatios o a umber lie. The aalogy was to a process of radomly assigig raks to observatios idepedet of oe's kowledge of which sample a observatio is a member. A radomizatio test makes use of such a procedure, but does so by operatig o the observatios rather tha the joit rakig of the observatios. For this reaso, the distributio of a aalogous statistic (the sum of the observatios i oe sample) caot be easily tabulated, although it is theoretically possible to eumerate such a distributio. From oe istace to the ext the observatios may be of substatially differet magitude so a sigle tabulatio of the probabilities of observig a specific sum of observatios could ot be made, a differet tabulatio would be required for each applicatio of the test. A further problem arises if the sample sizes are large. I the example metioed previously there are oly C = 79 possible arragemets of values so the exact distributio of the sum of observatios i oe sample could coceivably have bee eumerated. Had our sample sizes bee 0 ad the over 3. millio arragemets would have bee possible. If you have had ay experiece i combiatorial eumeratio the you would kow that this approach has rapidly become computatioally impractical. With high-speed computers it is certaily possible to tally 3. millio sums, but developig a efficiet algorithm to be sure that each ad every arragemet has bee icluded, ad icluded oly oce is prohibitive. What the? Sample. Whe the uiverse of possible arragemets is too large to eumerate why ot sample arragemets from this uiverse idepedetly ad at radom? The distributio of the test statistic over this series of samples ca the be tabulated, its' mea ad variace computed, ad the error rate associated with a hypothesis test estimated. Table cotais samples of = 0, ad =, obtaied from samplig from populatios with µ = 00, σ = 40 ad µ = 90, σ = 40, respectively. A ormal theory t - test applied to these data yields a t = 3.36, df = 3, 0.000 < p < 0.00. The same data examied by the distributio - free Wilcoxo's rak sum test yields W* =.773, 0.006 < p < 0.0030. Applyig this radomizatio approach with 000 iteratios of samplig without replacemet first 0 ad the observatios ad computig the t statistic for each of these samples yields the distributio depicted i Figure. The 4
ormal theory t distributio is depicted as a smooth curve. Accordig to the radomizatio procedure the probability of observig a t value greater tha or equal to that actually observed (3.36) is 0.00 < p < 0.006. Remember that this samplig procedure, ulike a eumeratio, allows each possible arragemet of values to be sampled more tha oce. The probability that o ay iteratio a particular arragemet will be chose is i this istace C or approximately 3. x 0-7. After 000 such 0 radomizatios it is quite possible that some arragemets have bee sampled more tha oce, but there is o reaso to believe that particular arragemets yieldig either low or high t - values should be systematically icluded or excluded from the 000 radomizatios. This approach is obviously a empirical approach to learig somethig about the distributio of a test statistic uder specified coditios. This Mote Carlo sample procedure would have to be performed aew for each ew set of observatios. Oe aspect that may be a advatage of this approach over ormal theory approaches is that ay ad hoc test statistic ca be elaborated sice a direct empirical ivestigatio of its distributioal properties accompaies each test. For example we could just use the differece i the sample meas x x as oe test statistic. Figure shows the distributio of x x over the same 000 radomizatios. The actual differece betwee sample meas is 8.9 ad uder the radomizatio approach the probability of observig a differece this large or larger is 0.004 < p < 0.00. The computed probability of observig the t or x x actually observed compares favorably with the ormal theory estimates. Figure 3 ad 4 illustrate the same procedure applied to two populatios whose uderlyig distributios are expoetial. Table presets the sample data geerated from two populatios with λ = 0 ad λ = 00, respectively. A ormal theory test o these data yields t = -.0874, df = 3, 0.0l <p < 0.0, while the radomizatio approach yielded a probability of obtaiig the observed value of t or oe greater of 0.07 < p < 0.08, ad a probability of obtaiig the observed or a greater differece i meas of p > 0.0. Here we see the ormal theory test breakig dow ad covergece i the results obtaied from the distributio - free ad radomizatio tests. Is all of this kosher? We ca see the parallel developmet of the distributio-free ad the radomizatio tests, yet is the radomizatio test actually yieldig a meaigful result? The aswer is a resoudig well-maybe-er-i-do't-kow. The radomizatio procedure essetially asks the questio, give observed samples ad, if we assume that these samples came from the same uderlyig populatio whose distributio F is give by the + sampled values, with probability mass /( + ) for each observatio, what is the chace of partitioig the observatios ito groups of the size observed that have meas that differ by a amout as large as that observed? Is this the best empirical estimate of the distributio of a test statistic?
D. The Bootstrap The Bootstrap is aother empirical approach to uderstadig the distributioal properties of a test statistic, but is also useful as a meas of estimatig statistics ad their stadard errors. The bootstrap is very similar to the radomizatio procedure outlied above. The observed distributio of sample values is used as a estimate of the uderlyig probability distributio of the populatio F. The, the distributio of a statistic for fixed sample sizes is obtaied by repeatedly samplig from the distributio F, with each value receivig probability mass /( + ), but samplig values with replacemet, so that istead of idividual partitios of the data havig the potetial to occur more tha oce, the idividual values themselves may appear repeatedly i a sigle sample. Uder this resamplig algorithm the umber of possible sample arragemets is much greater tha for the radomizatio approach. For example with a total sample size m =, with compoet samples of size 7 ad, 7 x = 8.96004 x 0 arragemets are possible. For m =, ad = 0, =, 0 x = 8.88784 x 0 34 arragemets are possible, factors of 0 0 ad 0 8 more arragemets, respectively, compared to the radomizatio approach. Ay test statistic averaged across a series of say 000 samples uder this algorithm will have a larger stadard error sice sub-samples of F ca deviate from F more tha uder the radomizatio algorithm. Figures ad 6 illustrate the distributio of t values ad x x for 000 bootstrap samples of the empirical probability distributio preseted i Table. For the ormal populatios the bootstrap estimates the probability of the observed t or oe greater to be 0.00 < p < 0.003 which, surprisigly is somewhat less tha the radomizatio approach. This compariso is reversed whe examiig the differeces betwee meas. The bootstrap estimates the probability of the observed mea differece or oe greater as 0.0 < p < 0.03, which is a order of magitude greater tha that estimated by the radomizatio approach, or for that matter for the t - statistic from the same group of bootstrap samples. Figures 7 ad 8 provide similar data for the samples derived from expoetially distributed populatios preseted i Table. The bootstrap is more coservative tha either the ormal theory approach or the radomizatio approach whe examiig the t value obtaied for the expoetial populatios. This is the result I would geerally expect i a compariso of the bootstrap ad radomizatio. Which approach is best? While the radomizatio approach ca be see to be aalogous to the eumeratio of distributios that characterizes distributio - free statistics, it is urealistic i that the distributio of a test statistic across a series of radomized samples is restricted to sub-samples that cotai exactly the same observatios as the true samples, oce each. I some istaces this may be the appropriate procedure, but i geeral radomizatio may give urealistically small stadard errors for test statistics, so that the true Type I error rates will be greater tha omially stated ad Type II error rates also will be greater tha omially stated. However, i all the examples preseted above the empirical radomizatio ad bootstrap approaches compare favorably with the ormal theory approach. 6
II. Estimatio A. The Jackkife I the course of applyig each of the empirical techiques i the costructio of hypothesis tests we could also have estimated test statistics ad a suite of characteristics of the test statistics ad the empirical distributios. I the test or meas we obviously could estimate the meas, the variaces (or stadard errors), ad the medias (their stadard errors), etc. We could also estimate the bias associated with each of these estimators. Defie a estimator θˆ of the parameter θ, E ( θ) ˆ = θ + c the the bias of the estimator is c. The sample mea, x, is a ubiased estimator of µ because E (x) = µ, eve though differet samples may give differet estimates x i of µ they are all ubiased estimates. I geeral, however, most estimators are biased, ad the bias ca be depicted as a Taylor series expasio of the estimator. So the bias of θˆ is ( ˆ a a a3 E θ θ) = + + 3 +... If we defie J θ ˆ θˆ ( ) θˆ, i =,,..., = i to be a ew estimator of θ, the the bias of J θˆ is ˆ a a + a E( θ J θ) = 3..., which is less tha the bias of θˆ sice it elimiates the term of order /. I practice the J estimator θˆ is computed as ˆ J θ = i= θˆ i = θˆ ( ) i= θˆ i where i =,.... This is the first order jackkife estimator. It is useful i that it is a less biased estimator although beig somewhat more variable tha the u-jackkifed estimator, but this icreased variability is at maximum ˆ a SE( θ J ) = SE( θˆ)( + ). 7
Sice the stadard error of a estimator decreases as by a factor of, the J estimator θˆ has dispersio greater by a factor of / tha θˆ, but usually oly -3/ J greater tha θˆ. Therefore the reductio i bias achieved by usig θˆ is ot offset by a similar icrease i the magitude of the estimator's variace. If we depict the bootstrap estimator asθˆ B the the jackkife estimator of the stadard error of θˆ is σ ϑ ( ˆ B = VAR θ L )( F ) where F B is the empirical bootstrap probability distributio of the radom variables ad θˆ is a liear approximatio of the estimator θˆ o the empirical bootstrap probability L B distributio. This implies that the bootstrap estimator θˆ has a stadard error that is [ / - ] / J times less tha the jackkife estimator θˆ. The jackkife estimator has bias β ϑ = [ E ( θˆ ( F B ) θ) ] q Where θˆ q is a quadratic approximatio of the estimator θˆ o the distributio F B ad E idicates the expectatio with respect to bootstrap samplig. This implies that the bootstrap estimate of bias is /( - ) times less tha the jackkife estimate of bias (see ref. 4). I geeral the the bootstrap will provide estimators with less bias ad variace tha the jackkife. Table 3 shows a data set geerated by samplig from two ormally distributed populatios with µ = 00, σ = 60, ad µ = 00 ad σ = 30. To test the hypothesis that the variaces of these populatios are equal, that is H 0 :σ = σ versus the alterative that H, :σ σ A we could use the ormal theory approach, which is agai coditioed, o the assumptios metioed earlier ad elaborate the test statistic based o the sample estimates of σ s s, 8
which is F distributed with - umerator degrees of freedom ad - deomiator degrees of freedom. Alteratively we could use a jackkife or a bootstrap estimate of the same or a similar test statistic. The F statistic computed uder ormal theory assumptios is F 9,4 = 8.86, p <0.00, while the bootstrap estimate of the probability of obtaiig the observed F or oe greater is 0.07 < p < 0.08. The jackkife test is performed o the atural logs of the jackkifed variaces rather tha the variaces themselves. A full descriptio of the computatios is give i referece (3). The test statistic for the jackkife test o variaces is B A Q =, V + V where B ad A are the averages of the atural logs of the variaces across the ad jackkifed estimates ad V ad V are the variaces of the jackkifed estimates of the variaces. For large samples ( + > 0), Q is N(0,), but for small equal size samples it follows studet's t distributio with + - degrees of a freedom. For this example Q = -.8883, 0.094 < p < 0.03. Figure 9 presets the distributio of the bootstrap estimates of F, ad Table 4 presets the jackkifed pseudo-values their stadard errors ad bias. Table ad Figure 0 provide a similar test for two expoetial populatios. Uder the assumptio of ormality F 9,4 =.783, 0.0 < p < 0.0. The jackkife test, however, yields Q = 3.09, 0.0009 < p < 0.00, ad the bootstrap yields p > 0.0. I these istaces the jackkife is the most powerful test. III. Prospectus So far I have preseted i a o-rigorous fashio a umber of computatioally expesive, empirical approaches to estimatio ad hypothesis testig. The theory uderlyig some of these approaches is well developed ad I refer you to the referece list for that material. However, much of what I have preseted has o rigorous theoretical uderpiigs, but ca be show to be quite useful particularly i situatios where the assumptio of ormality is suspect. The progosis amog statisticias is that theory will catch up to our computatioal prowess, so that may of these procedures will be justified ad should be adopted. I the iterim, however, should you choose to employ oe of the more radical of these procedures be prepared for cosiderable disagreemet over its validity ad usefuless. The prospects for further developmet of these kids of procedures, ad work to establish their limitatios, advatages, ad care ad maiteace is cosiderable. At preset, however, the burde of ivestigatig the properties of oe of these procedures, i its applicatio to a particular situatio ad test statistic rests with the ivestigator. Refereces Bradley, J.U. 968. Distributio-free statistical tests. Pretice-Hall, Ic: Eglewood Cliff, N.J. Coover, W.J. 980. Practical Noparametric statistics. Joh Wiley ad Sos: New York. 9
Hollader, M. ad D.A. Wolfe. 973. Noparametric Statistical Methods. Joh Wiley ad Sos: New York. Efro, B. ad G. Gog. 983. A leisurely look at the bootstrap, the jackkife, ad crossvalidatio. The America Statisticia 37: 36-48. Other Readable Literature Miller, R.G. 974. The jackkife - a review. Biometrika 6: -. Peters, S.C. ad D.A. Freedma. 984. Some otes o the Bootstrap i regressio problems. Joural of Busiess ad Ecoomic Statistics : 406-409. Efro, B. 979. Bootstrap Methods: aother look at the jackkife. Aals of Statistics 7: -6. Other Not So Readable Literature Arvese, J.N. 969. Jackkifig U-Statistics. Aals of Mathematical Statistics 40: 076-00. Miller, R.G. 964. A trustworthy jackkife. Aals of Mathematical Statistics 3: 94-60. Miller, R.G. 968. Jackkifig variaces. Aals of Mathematical Statistics 39: 67-8. Queouille, M.H. 96. Notes o bias i estimatio. Biometrika 43: 33-360. Some Applicatios Zahl, S. 977. Jackkifig a idex of diversity. Ecology 8: 907-93. Heltshe, J.F. ad N.E. Forrester. 98. Statistical evaluatio of the jackkife estimate of diversity whe usig quadrat samples. Ecology 66: 07-. Routledge, R.D. 980. Bias i estimatig the diversity of large ucesused commuities. Ecology 6: 76-8. 0