NONPARAMETRIC PREDICTIVE INFERENCE FOR REPRODUCIBILITY OF TWO BASIC TESTS BASED ON ORDER STATISTICS

Transcription

1 REVSTAT Statistical Journal Volume 16, Number 2, April 2018, NONPARAMETRIC PREDICTIVE INFERENCE FOR REPRODUCIBILITY OF TWO BASIC TESTS BASED ON ORDER STATISTICS Authors: Frank P.A. Coolen Department o Mathematical Sciences, Durham University, Durham, UK rank.coolen@durham.ac.uk Hana N. Alqiari Department o Mathematics, Qassim University, Buraidah, Saudi Arabia Received: February 2017 Revised: June 2017 Accepted: July 2017 Abstract: Reproducibility o statistical hypothesis tests is an issue o major importance in applied statistics: i the test were repeated, would the same overall conclusion be reached, that is rejection or non-rejection o the null hypothesis? Nonparametric predictive inerence (NPI provides a natural ramework or such inerences, as its explicitly predictive nature its well with the core problem ormulation o a repeat o the test in the uture. NPI is a requentist statistics method using relatively ew assumptions, made possible by the use o lower and upper probabilities. For inerence on reproducibility o statistical tests, NPI provides lower and upper reproducibility probabilities (RP. In this paper, the NPI-RP method is presented or two basic tests using order statistics, namely a test or a speciic value or a population quantile and a precedence test or comparison o data rom two populations, as typically used or experiments involving lietime data i one wishes to conclude beore all observations are available. Key-Words: lower and upper probabilities; nonparametric predictive inerence; precedence test; quantile test; reproducibility. AMS Subject Classiication: 60A99, 62G99, 62P30.

2 168 Frank Coolen and Hana Alqiari

3 Reproducibility o Tests Based on Order Statistics INTRODUCTION Testing o hypotheses is one o the main tools in statistics and crucial in many applications. While many dierent tests have been developed or a wide range o scenarios, the aspect o reproducibility o tests has long been neglected: the question addressed is whether or not a test, i it were repeated under the same circumstances, would lead to the same overall conclusion, that is rejection or non-rejection o the null hypothesis. Recently, this topic has started to gain attention, in particular through the publication o a handbook on reproducibility [4] which provides a collection o papers on the issue. Nevertheless, whilst hypothesis testing is mainly seen as a requentist statistics procedure, the classic requentist ramework is not suited or inerence on reproducibility as this is neither an estimation nor a testing problem. The very nature o reproducibility is predictive, namely given the results o one test one wishes to predict the outcome o a possible uture test. Coolen and Bin Himd [11] presented nonparametric predictive inerence (NPI or reproducibility o some basic tests, with more attention to this topic in the PhD thesis o Bin Himd [8], these publications also provide a critical discussion o earlier methods or reproducibility presented in the literature. This paper contributes to development o NPI or reproducibility by considering two tests based on order statistics, namely a one sample quantile test and a two sample precedence test. Central to these inerences are NPI results or uture order statistics [12]. This paper provides a concise presentation o NPI or the quantile and basic precedence test, urther details, examples and discussion are included in the PhD thesis o Alqiari [1]. This paper is organized as ollows. Section 2 provides a brie introduction to NPI, including key results on NPI or uture order statistics as used in this paper. Section 3 discusses aspects o reproducibility o statistical tests and explains the NPI perspective on such inerences. Section 4 presents the NPI approach to reproducibility o a basic quantile test. Section 5 considers a precedence test used or comparison o two populations. Some concluding remarks are given in Section 6. All computations in this paper were perormed using the statistical sotware R. 2. NONPARAMETRIC PREDICTIVE INFERENCE Nonparametric predictive inerence (NPI [5, 10] is a statistical ramework which uses ew modelling assumptions, with inerences explicitly in terms o uture observations. For real-valued random quantities attention has thus ar been

4 170 Frank Coolen and Hana Alqiari mostly restricted to a single uture observation, although multiple uture observations have been considered or some NPI methods, e.g. in statistical process control [2, 3]. Assume that we have real-valued ordered data x (1 < x (2 < < x (n, with n 1. For ease o notation, deine x (0 = and x (n+1 =, or at other known lower and upper bounds o the range o possible values or these random quantities. The n observations create a partition o the real-line into n + 1 intervals I j = (x (j 1, x (j or j = 1,..., n + 1. We assume throughout this paper that ties do not occur. I we wish to allow ties, also between past and uture observations, we could use closed intervals [x (j 1, x (j ] instead o these open intervals I j, the dierence is rather minimal and to keep presentation easy we have opted not to do this here. We are interested in m 1 uture observations, X n+i or i = 1,..., m. We link the data and uture observations via Hill s assumption A (n [17], or, more precisely, via A (n+m 1 (which implies A (n+k or all k = 0, 1,..., m 2; we will reer to this generically as the A (n assumptions, which can be considered as a post-data version o a inite exchangeability assumption or n + m random quantities. The A (n assumptions imply that all possible orderings o the n data observations and the m uture observations are equally likely, where the n data observations are not distinguished among each other and neither are the m uture observations. Let S j = #{X n+i I j, i = 1,..., m}, then the A (n assumptions lead to ( n+1 ( n + m 1 (2.1 P {S j = s j } = n j=1 where s j are non-negative integers with n+1 j=1 s j = m. Another convenient way to interpret the A (n assumptions with n data observations and m uture observations is to think that n randomly chosen observations out o all n + m real-valued observations are revealed, ollowing which you wish to make inerences about the m unrevealed observations. The A (n assumptions then imply that one has no inormation about whether speciic values o neighbouring revealed observations make it less or more likely that a uture observation alls in between them. For any event involving the m uture observations, Equation (2.1 implies that we can count the number o such orderings or which this event holds. Generally in NPI a lower probability or the event o interest is derived by counting all orderings or which this event has to hold, while the corresponding upper probability is derived by counting all orderings or which this event can hold [5, 10]. In NPI, the A (n assumptions justiy the use o resulting inerences directly as predictive probabilities. Using only precise probabilities, such inerences cannot be used or many events o interest, but in NPI we use the act, in line with De Finetti s Fundamental Theorem o Probability [14], that corresponding optimal bounds can be derived or all events o interest [5]. These bounds are lower and upper probabilities in the theory o imprecise probability [6]. NPI provides

5 Reproducibility o Tests Based on Order Statistics 171 exactly calibrated requentist inerences [18], and it has strong consistency properties in theory o interval probability [5]. In NPI the n observations are explicitly used through the A (n assumptions, yet as there is no use o conditioning as in the Bayesian ramework, we do not use an explicit notation to indicate this use o the data. The m uture observations must be assumed to result rom the same sampling method as the n data observations in order to have ull exchangeability. NPI is totally based on the A (n assumptions, which however should be considered with care as they imply e.g. that the speciic ordering in which the data appeared is irrelevant, so accepting A (n implies an exchangeability judgement or the n observations. It is attractive that the appropriateness o this approach can be decided upon ater the n observations have become available. NPI is always in line with inerences based on empirical distributions, which is an attractive property when aiming at objectivity [10]. Let X (r, or r = 1,..., m, be the r-th ordered uture observation, so X (r = X n+i or one i = 1,..., m and X (1 < X (2 < < X (m. The ollowing probabilities are derived by counting the relevant orderings and use o Equation (2.1. For j = 1,..., n + 1 and r = 1,..., m, (2.2 P ( ( ( ( j + r 2 n j m r n + m 1 X (r I j =. j 1 n j + 1 n For this event NPI provides a precise probability, as each o the ( n+m n equally likely orderings o n past and m uture observations has the r-th ordered uture observation in precisely one interval I j. As Equation (2.2 only speciies the probabilities or the events that X (r belongs to intervals I j, it can be considered to provide a partial speciication o a probability distribution or X (r, no assumptions are made about the distribution o the probability masses within such intervals I j. Analysis o the probability in Equation (2.2 leads to some interesting results, including the logical symmetry P(X (r I j = P(X (m+1 r I n+2 j. For all r, the probability or X (r I j is unimodal in j, with the maximum probability assigned to interval I j with ( r 1 m 1 (n + 1 j ( r 1 m 1 (n A urther interesting property occurs or the special case where the number o uture observations is equal to the number o data observations, so m = n. In this case, P(X (r < x r = P(X (r > x r = 0.5 holds or all r = 1,..., m. This act can be proven by considering all ( 2n n equally likely orderings, where clearly in precisely hal o these orderings the r-th uture observation occurs beore the r-th data observation due to the overall exchangeability assumption. The special case m = n plays an important role in this paper as it naturally occurs in analysis o reproducibility o statistical hypothesis tests.

6 172 Frank Coolen and Hana Alqiari 3. REPRODUCIBILITY OF STATISTICAL TESTS Statistical hypothesis testing is used in many application areas and normally results in either non-rejection o the stated null hypothesis or its rejection in avour o a stated alternative, at a predetermined level o signiicance. Whilst this procedure is embedded in the successul long-standing tradition o statistics, a related aspect that had received relatively little attention in the literature until recently is the reproducibility o such tests: i the test were repeated, would it lead to the same overall conclusion? Attention to problems with reproducibility, including problems with understanding o concepts by practitioners in application areas, was raised by Goodman [16] and Senn [21]. Methods or addressing reproducibility, proposed in the literature since then, have mainly shown that the classical requentist ramework o statistics may not be immediately suitable or inerence on test reproducibility (see [11] or a discussion o such proposed methods. Recently, many aspects o reproducibility, including some attention to statistical methods, have been discussed in a volume dedicated to this topic [4]. The reproducibility probability (RP or a test is the probability or the event that, i the test is repeated based on an experiment perormed in the same way as the original experiment, the test outcome, that is either rejection o the null-hypothesis or not, will be the same. In practice, ocus may oten be on reproducibility o tests in which the null-hypothesis is rejected, or example because signiicant eects tend to lead to new treatments in medical applications. However, also i the null-hypothesis is not rejected it is important to have a meaningul assessment o the reproducibility o the test. Note that RP is assessed knowing the outcome o the irst, actual experiment, which consists o the actual observations, so not only the value o a suicient test statistic or even just the conclusion on rejection or non-rejection o the null-hypothesis. This is important as the RP will vary with dierent experiment outcomes, which is logical and will lead to higher RP i the data supported the original test conclusion more strongly. A suicient test statistic, i o reduced dimension compared to the ull data set, does not provide suitable input or the NPI method, hence the use o the ull data set is required or the inerences considered in this paper. Coolen and Bin Himd [11] introduced NPI or RP, denoted by NPI-RP, by considering some basic nonparametric tests: the sign test, Wilcoxon s signed rank test, and the two sample rank sum test. For these inerences NPI or Bernoulli quantities [9] and or real-valued observations [5] were used. This did not lead to precise valued reproducibility probabilities but to NPI lower and upper reproducibility probabilities, denoted by RP and RP, respectively. For these tests analytic methods were presented to calculate the NPI lower and upper probabilities or test reproducibility. To enable NPI or more complex test scenarios, the NPI-bootstrap method can be used, as introduced and illustrated by Bin Himd [8] or the Kolmogorov Smirnov test.

7 Reproducibility o Tests Based on Order Statistics 173 This paper presents NPI-RP or two classical tests which are based on order statistics, namely a one sample quantile test (Section 4 and a two sample precedence test (Section 5. For these inerences, NPI or uture order statistics [12] is used, as briely reviewed in Section 2. We assume that the irst, actual experiment led to ordered real-valued observations x (1 < x (2 < < x (n. As we consider an imaginary repeat o this experiment, we use NPI or m = n uture ordered observations, henceorth denoted by X (1 < X (2 < < X (n, with the superscript used to emphasize that we consider uture order statistics. 4. QUANTILE TEST The quantile test is a basic nonparametric test or the value o a population quantile [15]. Let κ p denote the 100 p-th quantile o an unspeciied continuous distribution, or 0 p 1. On the basis o a sample o observations o independent and identically distributed random quantities X i, i =1,..., n, we consider the one-sided test o null-hypothesis H 0 : κ p = κ 0 p versus alternative H 1 : κ p > κ 0 p, or a speciied value κ 0 p. We restrict attention in this paper to NPI or reproducibility o this one-sided quantile test. The corresponding methodology or the two-sided test ollows the same steps and is included in the PhD thesis o Alqiari [1], where also some more discussion and examples are given o the tests presented in this paper. Actually, there is an interesting issue about two-sided tests in such scenarios, that requires some urther thought. I the original test leads to rejection o the null hypothesis due to a relatively small value o the test statistic, would one consider the test result to be reproduced i a uture test leads to rejection due to a relatively large value o the test statistic, so in the other tail o the statistic s distribution under H 0? Technically perhaps this is the case, but on the basis o the combined evidence o the two tests one would probably want to investigate the whole setting urther and not regard the second test as conirming the results o the irst test. This is let as a topic or consideration. Under H 0, κ 0 p is the 100 p-th quantile o the distribution unction o the X i, so P(X i κ 0 p H 0 = p. Deine the random variable K as the number o X i in the sample o size n that are less than or equal to κ 0 p, that is K = n i=1 1 { X i κ 0 } p with 1{A} = 1 i A is true and 1{A} = 0 i A is not true. A logical test rule is to reject H 0 i X (r > κ 0 p, where X (r is the r-th ordered observation in the sample (ordered rom small to large, or a suitable value o r corresponding to a chosen signiicance level, so i K r 1. For signiicance level α, r is the largest integer

8 174 Frank Coolen and Hana Alqiari such that P ( X (r > κ 0 r 1 ( n p H 0 = p i (1 p n i α. i i=0 For large sample sizes the Normal distribution approximation to the Binomial distribution can be used in order to determine the appropriate value o r. For a given data set x 1,..., x n, the test statistic o the one-sided quantile test as deined above is the number o observations less than or equal to κ 0 p, denoted by n k = 1 { x i κ 0 p}. i=1 For the value r derived as discussed above, H 0 is rejected i and only i k r 1. Based on such data and the result o the actual hypothesis test, that is whether the null hypothesis is rejected in avour o the alternative hypothesis or not, NPI can be applied to study the reproducibility o the test. First we consider the case where k r 1, so the original test leads to rejection o H 0. Reproducibility o this test result is thereore the event that, i the test were repeated, also leading to n observations, then that would also lead to rejection o H 0. Using the notation or uture observations introduced in Section 3, this would occur i the r-th ordered observation o the uture sample exceeds κ 0 p. The NPI lower and upper reproducibility probabilities or this event, as unction o k r 1, are RP(k = P ( X (r > κ0 p k = n+1 1 { x j 1 > κ 0 } ( p P X (r I j j=1 and RP(k = P ( X (r > κ0 p k n+1 = 1 { x j > κ 0 } ( p P X (r I j, j=1 respectively. Note that the dependence o these lower and upper probabilities on the value k is not explicit in the notation used or the terms on the righthand side, but is due to the number o data x j that exceed κ 0 p. It is easily shown that P(X (r > κ0 p k = P(X (r > κ0 p k+1, which leads to RP(k = RP(k +1 or values o k leading to rejection o H 0. I the original test does not lead to rejection o H 0, so i k r, then reproducibility o the test is the event that the null hypothesis would also not get rejected in the uture test. The NPI lower and upper reproducibility probabilities or this event, as unction o k r, are RP(k = P ( X (r κ0 p k = n+1 1 { x j κ 0 } ( p P X (r I j j=1

9 Reproducibility o Tests Based on Order Statistics 175 and RP(k = P ( X (r κ0 p k = n+1 1 { x j 1 κ 0 } ( p P X (r I j, j=1 respectively. It is easily seen that RP(k = RP(k 1 or values o k such that k 1 leads to H 0 not being rejected. I an actual observation in the original test is exactly equal to the speciied value κ 0 p, then the NPI method would actually provide a precise reproducibility probability. We do not consider this urther as the test hypotheses must always be speciied without consideration o the actual test data, hence this case is extremely unlikely to occur; or some urther discussion see [1]. The minimum value that can occur or the NPI lower reproducibility probabilities or this one-sided quantile test, ollowing either rejection or non-rejection o the null hypothesis in the original test, is equal to 0.5. This ollows directly rom the ormulae or the NPI lower reproducibility probabilities given above, together with P(X (r < x r = P(X (r > x r = 0.5 as explained in Section 2. The NPI upper reproducibility probabilities can be equal to one. This occurs when all observations in the original test are less than κ 0 p, so k = n, in which case the original test let to H 0 not being rejected or all values o r (so or all order statistics considered, hence or any level o signiicance; this relects that, with no evidence in the original data in avour o the possibility that the data values can actually exceed κ 0 p, one cannot exclude the possibility that no uture observations could exceed this value. Note that the corresponding NPI lower reproducibility probability will be less than one, relecting that the original data set only provides limited inormation, this lower probability will increase towards one as unction o n. The upper reproducibility probability is also equal to one i all observations in the original test are greater than κ 0 p, so k = 0, or which case the reasoning is similar to that above but o course now with H 0 being rejected. Example 1. Suppose that the original test has sample size n = 15 and we are interested in testing the null hypothesis that the third quartile, so the 75% quantile, o the underlying distribution is equal to a speciied value κ against the alternative hypothesis that this third quartile is greater than κ , tested at signiicance level α = Using the Binomial distribution or the classical quantile test, this leads to the rule that H 0 is rejected i x (8 > κ and H 0 is not rejected i x (8 < κ Note that we do not discuss the case x (8 = κ which is slightly dierent as the NPI approach leads to precise probabilities instead o lower and upper probabilities (see [1], it is also o little practical relevance. Table 1 presents the NPI lower and upper reproducibility probabilities or all values o k, which is the number o observations in the original test which are less than κ I k 7 then the original test leads to H 0 being rejected while it is not rejected or k 8. Hence, the NPI lower and upper reproducibility probabilities are or the events X (8 > κ and X (8 < κ0 0.75, respectively. This

10 176 Frank Coolen and Hana Alqiari table illustrates the logical act that the worst reproducibility is achieved or k at the threshold values 7 and 8, with increasing RP values when moving away rom these values, leading to maximum NPI-RP values or k = 0 and k = 15. Because or this test the threshold between rejecting and not rejecting H 0 is between k = 7 and k = 8 out o n = 15 observations, the NPI-RP values are symmetric, that is the same or k = j and k = 15 j or j = 0, 1,...,7 in Table 1. Table 1: NPI-RP or third quartile, n = 15 and α = k RP(k RP(k k RP(k RP(k k RP(k RP(k Table 2 presents NPI-RP values or the quantile test considering the median, so the 50% quantile, again with sample size n = 15 and testing the null hypothesis that the median is equal to a speciied value κ against the one-sided hypothesis that it is greater than κ 0 0.5, at level o signiicance α = This leads to the test rule that H 0 is rejected i the number k o observations that are smaller than κ is less than or equal to 3, and H 0 is not rejected i k 4. Note that throughout this paper, precise values 0.5 and 1 are presented without additional decimals, so the values are actually less than 1 but rounded upwards. O course, these NPI-RP values are not symmetric, and reproducibility becomes very likely or initial test results with a substantial number o observations less than κ But rejection o H 0, which occurs or k 3 and is oten o main practical relevance, has relatively low NPI-RP values. Table 2: NPI-RP or median, n = 15 and α = k RP(k RP(k k RP(k RP(k k RP(k RP(k Tables 3 and 4 present the NPI-RP results or the same one-sided quantile test on the third quartile or n = 30, at signiicance levels α = 0.05 and α =

11 Reproducibility o Tests Based on Order Statistics , respectively. Using the Normal distribution approximation, the test rule or α = 0.05 is to reject H 0 that this third quartile is equal to κ in avour o the alternative hypothesis that it is greater than κ i k 18 and not to reject it i k 19, where k is again the number o observations less than κ For α = 0.01, H 0 is rejected i k 16 and not rejected i k 17. The change in level o signiicance α leads obviously to change o the rejection threshold, with H 0 being rejected or a smaller range o values k in case o smaller value o α. Comparison o Table 3 with Table 1 shows that the larger sample size tends to lead to slightly less imprecision, that is the dierence between corresponding upper and lower probabilities, this is e.g. shown by considering the upper probabilities RP(k or the values o k next to the rejection thresholds, so corresponding to RP(k = 0.5. Table 3: NPI-RP or third quartile, n = 30 and α = k RP(k RP(k k RP(k RP(k k RP(k RP(k Table 4: NPI-RP or third quartile, n = 30 and α = k RP(k RP(k k RP(k RP(k k RP(k RP(k

12 178 Frank Coolen and Hana Alqiari 5. PRECEDENCE TEST As a second example o NPI or reproducibility o a statistical test based on order statistics we consider a basic nonparametric precedence test. Such a test, irst proposed by Nelson [19], is typically used or comparison o two groups o lietime data, where one wishes to reach a conclusion beore all units on test have ailed. The test is based on the order o the observed ailure times or the two groups, and typically leads to, possibly many, right-censored observations at the time when the test is ended. Balakrishnan and Ng [7] present a detailed introduction and overview o precedence testing, including more sophisticated tests than the basic one considered in this paper. NPI or precedence testing was presented by Coolen-Schrijner et al. [13], without consideration o reproducibility. It should be emphasized that we consider here the NPI approach or reproducibility o a classical precedence test, so not o the NPI approach to precedence testing [13]. We consider the classical scenario with two independent samples. Let X (1 < X (2 < < X (nx be the ordered real-valued observations in a sample o size n x drawn randomly rom a continuously distributed population, which we reer to as the X population, with a probability distribution depending on location parameter λ x. Similarly, let Y (1 < Y (2 < < Y (ny be the ordered real-valued observations in a sample o size n y drawn randomly rom another continuously distributed population (the Y population with a probability distribution which is identical to that o the X population except or its location parameter λ y. The hypothesis test or the locations o these two populations considered here is H 0 : λ x = λ y versus H 1 : λ x < λ y, which is to be interpreted such that, under H 1, observations rom the Y population tend to be larger than observations rom the X population. The precedence test considered in this paper, or this speciic hypothesis test scenario, is as ollows. Given n x and n y, one speciies the value o r, such that the test is ended at, or beore, the r-th observation o the Y population. For speciic level o signiicance α, one determines the value k (which thereore is a unction o α and o r such that H 0 is rejected i and only i X (k < Y (r. The critical value or k is the smallest integer which satisies P ( X k < Y r H 0 = ( nx + n y n x 1 r 1 ( ( j + k 1 ny j + n x k α. j n y j j=0 Note that the test is typically ended at the time T = min(x (k, Y (r, with the conclusion that H 0 is rejected in avour o the one-sided alternative hypothesis H 1 speciied above i T = X (k and H 0 is not rejected i T = Y (r. It is o interest to emphasize this censoring; continuing with the original test would make no dierence at all to the test conclusion, but urther observations would make a dierence or the NPI reproducibility results, as will be discussed later.

13 Reproducibility o Tests Based on Order Statistics 179 The NPI approach or reproducibility o this two-sample precedence test considers again the same test scenario applied to uture order statistics, and derives the lower and upper probabilities or the event that the same overall test conclusion will be derived, given the data rom the original test. This involves the earlier described NPI approach or inerence on the r-th uture order statistic Y (r out o n y uture observations based on the data rom the Y population, and similarly or the k-th uture order statistics X (k out o the n x uture observations based on the data rom the X population, where the values o r and k are the same as used or the original test (as we assume also the same signiicance level or the uture test. Note, however, that there is a complication: or ull speciication o the NPI probabilities or these uture order statistics, we require the ull data rom the original test to be available. But, as mentioned, the data resulting rom the original precedence test typically has right-censored observations or at least one, but most likely both populations, and these are all just known to exceed the time T at which the original test had ended. Beore we proceed, we discuss this situation in more detail as it is important or the general idea o studying reproducibility o tests. We should emphasize that we have not come across this issue beore in the literature, but it seems to be important and more details are provided by Alqiari [1]. There are two perspectives on the study o reproducibility o such precedence tests. First, one can study the test outcome assuming that actually complete data were available, so all n x and n y observations o the X and Y populations, respectively, in the original test are assumed to be available. Secondly, one can consider inerence or the realistic scenario with the actual data rom the original test, so including rightcensored observations at time T. The irst scenario is the most straightorward or the development o NPI-RP, and we start with this scenario. Then we explain how this irst scenario, without additional assumptions, leads to NPI-RP or the second scenario. The starting point or NPI-RP or the precedence test is to apply NPI or n x uture observations, based on the n x original test observations rom the X population, which are assumed to be ully available, and similarly or n y uture observations based on the n y observations rom the Y population. Using the results presented in Section 2, with notation adapted to indicate the speciic populations, the ollowing NPI lower and upper reproducibility probabilities are derived. First, i H 0 is rejected in the original test, so x (k < y (r, then RP = P ( X (k < Y (r RP = P ( X (k < Y (r n x+1 = n y+1 j x=1 j y=1 n x+1 = n y+1 j x=1 j y=1 1 { } ( x (jx < y (jy 1 P X (k ( Ix j x P Y (r Iy j y, 1 { x (jx 1 < y (jy } P ( X (k Ix j x P ( Y (r Iy j y.

14 180 Frank Coolen and Hana Alqiari I H 0 is not rejected in the original test, so x (k > y (r, then RP = P ( X (k > Y (r RP = P ( X (k > Y (r n x+1 = n y+1 j x=1 j y=1 n x+1 = n y+1 j x=1 j y=1 1 { } ( x (jx 1 < y (jy P X (k ( Ix j x P Y (r Iy j y, 1 { } ( x (jx < y (jy 1 P X (k ( Ix j x P Y (r Iy j y. The ollowing general results or this NPI lower and upper reproducibility probabilities are easily derived [1]. Both in case o rejecting and not rejecting H 0, the maximum possible value o the NPI upper reproducibility probability is 1. I H 0 was rejected this occurs i x (nx < y (1, while i H 0 was not rejected this occurs i x (1 > y (ny, so both cases lead to maximum reproducibility i the original test data were entirely separated in the sense that either all observations rom X population occurred beore all observations rom the Y population, or the other way around. In both cases o rejecting or not rejecting H 0 in the original test, the minimum value o the NPI lower reproducibility probability is I H 0 was rejected, this occurs i y (r 1 < x (1 and x (k < y (r and y (ny < x (k+1. I H 0 was not rejected, this occurs i x (k 1 < y (1 and y (r < x (k and x (nx < y (r+1. Both these smallest possible values or RP result rom data orderings that, whilst leading to a test conclusion, are least supportive or it, together with the act that P(X (k < x (k = P(X (k > x (k = 0.5 (and similar or Y (r as discussed in Section 2. The eect o local changes to the combined ordering o the data o the two populations in the original test is important. Suppose that, or given data or the X and Y populations or the original test, observations y (u and x (v are such that y (u < x (v and in the combined ordering o all n x + n y data they are consecutive. Now suppose that we change these observations, and denote them by ỹ (u and x (v, respectively, such that they keep their order in the data rom their own population but between them change their order, so x (v < ỹ (u. Then this local change to the combined ordering o the data leads to increase o both the NPI lower and upper probabilities or the event X (k < Y (r, that is P ( X (k < Y (r y (u < x (v < P ( X(k < Y (r x (v < ỹ (u, P ( X (k < Y (r y (u < x (v < P ( X(k < Y (r x (v < ỹ (u. This implies that the NPI-RP inerences or the precedence test depend monotonically on the combined ordering o the original test data, which is an important property to derive such inerence or actual tests including right-censored observations, as discussed ater the next example.

15 Reproducibility o Tests Based on Order Statistics 181 Example 2. Nelson [20] presents data consisting o six groups o times (in minutes to breakdown o an insulating luid subjected to dierent levels o voltage. To illustrate NPI-RP or the basic precedence test as discussed above, we assume that sample 3 provides data rom the X population and sample 6 rom the Y population, these times are presented in Table 5. Both samples are o size 10, and we assume that the precedence testing scenario discussed in this section is ollowed, so we assume that the population distributions may only dier in location parameters, with H 0 : λ x = λ y tested versus H 1 : λ x < λ y. We assume that r = 6, so the test is set up to end at the observation o the sixth ailure time or the Y population. We discuss both signiicance levels α = 0.05 and α = 0.1. The missing values in Table 5 are only known to exceed Table 5: Times to insulating luid breakdown. X sample * Y sample * * * * For signiicance level α = 0.05, the critical value is k = 10, while or α = 0.1 this is k = 9. Thereore, the provided data will lead, in this precedence test, to rejection o H 0 at 10% level o signiicance but not to rejection o H 0 at 5% level o signiicance. For both scenarios, the NPI lower and upper reproducibility probabilities are presented in Table 6, or all o the possible orderings o the right-censored observations. Note that in total 15 observations are available, with 1 value o the X sample and 4 values o the Y sample only known to exceed Table 6: NPI-RP or precedence test on insulating luid breakdown data. rank o x 10 α = 0.05 α = 0.1 RP RP RP RP In this table, we give the rank, rom the combined ordering o all 20 observations, o the right-censored observation x (10, or example when this is 17 it implies that y (7 < x (10 < y (8. Table 6 presents both the results or α = 0.05, in which case H 0 was not rejected in the original test, hence reproducibility is achieved i H 0 is also not rejected in the uture test, and the results or α = 0.05, in which case

16 182 Frank Coolen and Hana Alqiari H 0 was rejected so reproducibility also implies rejection o H 0 in the uture test. Note that or α = 0.1 we still assume that y (6 = 3.83 was actually observed, even though the test could have been concluded at time x (9 = 2.57 because x (9 < y (6 was conclusive or the test in this case. Table 6 shows that the NPI-RP values are increasing in the combined rank o x (10 or α = 0.05 and decreasing or α = 0.1, which illustrates the monotonicity o these inerences with regard to changes in ranks o the data as discussed above, as increasing combined rank o x (10 provides more evidence in support o H 0, hence in avour o reproducing the original test result or α = 0.05 but against doing so or α = 0.1. We notice that the actual rank that x (10 would have among the 20 combined observations has substantial inluence on the NPI-RP values. In this example, the imprecision RP RP is large. This is due to the relatively small data sets and the act that two groups o data are compared, with imprecision or the predictive inerences or both groups through the A (n assumptions or each group. Thus ar, we have studied reproducibility o the basic precedence test rom the perspective o having the complete data available, in Example 2 this was illustrated by considering all possible orderings or the right-censored data in the two samples. However, a more realistic perspective is to only use the actual test outcome, without any assumptions on the ordering o the right-censored observations. Using lower and upper probabilities, this can be easily achieved by deining RP as the minimum o all NPI lower probabilities or reproducibility over all possible orderings or the right-censored observations, and similarly by deining RP as the maximum o all NPI upper probabilities or reproducibility over all possible orderings or the right-censored observations. Hence, in Example 2, this leads to RP = and RP = or α = 0.05, and RP = and RP = or α = 0.1. O course, this leads to increased imprecision compared to every possible speciic ordering o the right-censored observations, but it is convenient as no urther assumptions about those right-censored observations are required. Furthermore, to derive the NPI-RP values or this perspective one does not need to calculate the corresponding values or each possible combined ordering o right-censored observations, due to the above discussed monotonicity o these inerences. Hence, we always know or which speciic ordering o rightcensored observations these NPI-RP values are obtained, that is either with all right-censored observations rom the X sample occurring beore all right-censored observations rom the Y sample, or the other way around, depending on the actual outcome o the original test. This perspective is illustrated urther in Example 3. Example 3. We consider again NPI-RP or the precedence test as presented in this section, so with one-sided alternative hypothesis H 1 : λ x < λ y. Suppose that n x = 10 units o the X population and n y = 8 units o the Y population are put on a lie test, where one wants at most two Y units to actually ail, so the value r = 2 is chosen. Testing at signiicance level α = 0.05, the critical value is k = 7, so H 0 is rejected i x (7 < y (2 while H 0 is not rejected i y (2 < x (7.

17 Reproducibility o Tests Based on Order Statistics 183 Note that, with the test ending at time min(x (7, y (2, there are least 3 rightcensored X observations and at least 6 right-censored Y observations; this leads to large imprecision in the NPI-RP values. Table 7 presents the NPI lower and upper reproducibility probabilities or this test, or all possible data in the original test, which are indicated through the rankings o all observations until the test is ended, in the combined ranking o the X and Y samples. As indicated, the columns to the let relate to the cases where H 0 is not rejected while the columns to the right relate to the cases where H 0 is rejected. All these NPI-RP values are calculated using the monotonicity with regard to the combined ranks o the right-censored observations, as explained above. These results illustrate the earlier discussed maximum value 1 or RP and minimum value 0.25 or RP. It is particularly noticeable that the NPI lower reproducibility probabilities or this test tend to be small, which is not really surprising due to the large number o right-censored observations resulting rom the choice r = 2. Table 7: NPI-RP or precedence test with n x = 10, n y = 8, r = 2, k = 7 and α = H 0 not rejected H 0 rejected X ranks Y ranks RP RP X ranks Y ranks RP RP 1, , , , ,7, ,2 3, , ,3 2, , ,3 1, ,2, , , , , , , CONCLUDING REMARKS The NPI approach to reproducibility o tests provides many research challenges. It can be developed or many statistical tests, while or some data types (e.g. multivariate data irst NPI requires to be developed urther. The test scenarios studied or particular tests may require careul attention, as illustrated by the dierent perspectives discussed or the precedence test in Section 5. As mentioned, the precedence test scenario discussed in this paper is very basic. Balakrishnan and Ng [7] present a detailed introduction and overview o precedence testing, including more sophisticated tests than the basic one considered

18 184 Frank Coolen and Hana Alqiari in this paper. In practice, it is important or such tests, and also in general, to also consider the power o the test; thus ar this has not yet been considered in the NPI approach or reproducibility o testing. With urther development o this approach, we are aiming at guidance on selection o test methods which, or speciied level o signiicance, have good power and good reproducibility properties. This may oten require more test data than needed ollowing traditional guidance, but the assurance o good reproducibility is important or many applications and may lead to savings in the longer run by reducing processes, such as development o new medication, to continue on the basis o alse test results which may later turn out not to be reproduced in repeated tests under similar circumstances. Further details, examples and discussion o the tests presented in this paper are given in the PhD thesis o Alqiari [1]. ACKNOWLEDGMENTS This work was carried out while Hana Alqiari was studying or PhD at the Department o Mathematical Sciences, Durham University, supported by the Ministry o Higher Education in Saudi Arabia and Qassim University. The authors grateully acknowledge detailed comments by three anonymous reviewers which led to improved presentation o the paper. The authors also thank Proessor Sat Gupta or the kind invitation to present this work at the International Conerence on Advances in Interdisciplinary Statistics and Combinatorics at The University o North Carolina at Greensboro (October 2016, and the subsequent invitation to submit this paper or the conerence special issue o this journal. REFERENCES [1] Alqiari, H.N.(2017. Nonparametric Predictive Inerence or Future Order Statistics, PhD Thesis, Durham University (available rom [2] Arts, G.R.J. and Coolen, F.P.A. (2008. Two nonparametric predictive control charts, Journal o Statistical Theory and Practice, 2, [3] Arts, G.R.J.; Coolen, F.P.A. and van der Laan, P. (2004. Nonparametric predictive inerence in statistical process control, Quality Technology and Quantitative Management, 1, [4] Atmanspacher, H. and Maasen, S. (Eds. (2016. Reproducibility: Principles, Problems, Practices and Prospects, Wiley, Hoboken, New Jersey.

19 Reproducibility o Tests Based on Order Statistics 185 [5] Augustin, T. and Coolen, F.P.A. (2004. Nonparametric predictive inerence and interval probability, Journal o Statistical Planning and Inerence, 124, [6] Augustin, T.; Coolen, F.P.A.; de Cooman, G. and Troaes, M.C.M. (Eds. (2014. Introduction to Imprecise Probabilities, Wiley, Chichester. [7] Balakrishnan, N. and Ng, H.K.T. (2006. Precedence-Type Tests and Applications, Wiley, Hoboken, New Jersey. [8] Bin Himd, S. (2014. Nonparametric Predictive Methods or Bootstrap and Test Reproducibility, PhD Thesis, Durham University (available rom [9] Coolen, F.P.A. (1998. Low structure imprecise predictive inerence or Bayes problem, Statistics and Probability Letters, 36, [10] Coolen, F.P.A. (2006. On nonparametric predictive inerence and objective Bayesianism, Journal o Logic, Language and Inormation, 15, [11] Coolen, F.P.A. and Bin Himd, S. (2014. Nonparametric predictive inerence or reproducibility o basic nonparametric tests, Journal o Statistical Theory and Practice, 8, [12] Coolen, F.P.A.; Coolen-Maturi, T. and Alqiari, H.N. (2018. Nonparametric predictive inerence or uture order statistics, Communications in Statistics Theory and Methods, 47, [13] Coolen-Schrijner, P.; Maturi, T.A. and Coolen, F.P.A. (2009. Nonparametric predictive precedence testing or two groups, Journal o Statistical Theory and Practice, 3, [14] De Finetti, B. (1974. Theory o Probability (2 volumes, Wiley, London. [15] Gibbons, J.D. and Chakraborti, S. (2010. Nonparametric Statistical Inerence (5th ed., Chapman & Hall, Boca Raton, FL. [16] Goodman, S.N. (1992. A comment on replication, p-values and evidence, Statistics in Medicine, 11, [17] Hill, B.M. (1968. Posterior distribution o percentiles: Bayes theorem or sampling rom a population, Journal o the American Statistical Association, 63, [18] Lawless, J.F. and Fredette, M. (2005. Frequentist prediction intervals and predictive distributions, Biometrika, 92, [19] Nelson, L.S. (1963. Tables or a precedence lie test, Technometrics, 124, [20] Nelson, W.B. (1982. Applied Lie Data Analysis, Wiley, Hoboken, New Jersey. [21] Senn, S. (2002. Comment on A comment on replication, p-values and evidence, by S.N. Goodman (Letter to the editor, Statistics in Medicine, 21, With author s reply, pp