A nonparametric test for trend based on initial ranks

Transcription

1 Journal of Statistical Computation and Simulation Vol. 76, No. 9, September 006, A nonparametric test for trend based on initial ranks GLENN HOFMANN* and N. BALAKRISHNAN HSBC, USA ID Analytics, Avenue of Science, San Diego, CA 918, USA Received 8 December 004; in final form 8 May 005) We compare the power of several nonparametric statistics for testing the hypothesis of an i.i.d. sequence against the alternative of an increasing trend in location or scale. We introduce a new test based on initial ranks and display it to perform consistently as good as or better than the classical tests due to Kendall and Spearman. The same is true for a test based on inverse-normally transformed ranks. Keywords: Hypothesis of randomness; Trend; Location trend; Nonparametric tests; Scale trend; Ranks; Initial ranks; Power simulation 1. Introduction In many practical situations, it is important to test for the randomness of a series of observations, i.e. the hypothesis that X 1,...,X n are independent and identically distributed i.i.d.) There is a large class of relevant alternatives and no single test is likely to be efficient against all of them. We will focus on the alternative of an increasing trend in the series. Let U 1,...,U n be i.i.d. with unknown distribution function F and let X i = U i + βti), i = 1,...,n, β R, 1) where ti) is a strictly increasing function. Then, β = 0 corresponds to the null hypothesis of randomness, and β>0 marks an increasing trend in location. The trend function ti) can have different shapes. Linear, logarithmic and polynomial trends are the most commonly considered. For positive random variables with means related to scale changes, we may also be interested in scale trends, corresponding to β>0 in the model X i = U i 1 + βti)), i = 1,...,n, β R, ) and once again ti) is strictly increasing. Several nonparametric test statistics have been proposed for such trend alternatives. Let R i i = 1,...,n) be the rank of X i in X 1,...,X n *Corresponding author. glennhofmann@yahoo.com Journal of Statistical Computation and Simulation ISSN print/issn online 006 Taylor & Francis DOI: /

2 830 G. Hofmann and N. Balakrishnan when ordered from smallest to largest. Mann [1] considered Kendall s statistic Q = signr j R i ), where 1 i<j n 1 if x>0 signx) = 1 if x<0, 3) 0 ifx = 0 which is related to Kendall s rank correlation coefficient τ by τ = Q/nn 1). To this date, Q remains the omnipotent nonparametric trend test of choice. It has high power for a variety of underlying distributions. Daniels [] compared Q with Spearman s rank correlation coefficient 1 r S = i n + 1 ) R nn i n + 1 ) ) 1 nn + 1) = ir 1) nn i. 1) 4 4) He pointed out that Q and r S do not measure the same aspect of dependence and showed that r S is optimal for testing against linear trends ti) = i in 1)) in the class of linear combinations of { 1 R j >R i I {Rj >R i } = 0 otherwise. Stuart [3, 4] compared the asymptotic relative efficiencies of several trend tests in case of an underlying normal distribution. He found Q and r S to be the most powerful nonparametric options. Aiyar et al. [5] gave a more extensive asymptotic comparison of rank tests for trend alternatives. They introduced several other rank statistics which are asymptotically optimal for certain distributions and trends. Most notably, they presented M = i n + 1 ) ) 1 Ri n + 1 and B = i n + 1 ) sign R i n + 1 ), 6) which are the asymptotically optimal linear rank statistics for a linear trend with an underlying normal and double-exponential distributions, respectively. Here, denotes the standard normal distribution function. Foster and Stuart [6] introduced a nonparametric trend test based on records. Let Nn UN n L) be the number of upper lower) records in the sequence X 1,...,X n, where X i is an upper lower) record if and only if X i > <)X j for all 1 j<i. Similarly, let Nn U,N L n be the numbers of upper and lower records, respectively, in the reverse sequence X n,...,x 1. Foster and Stuart [6] defined their test statistic as D = Nn U N n L U ) Nn 5) Nn L ). 7) See Arnold et al. [7, chapter 5) for further details. According to Stuart [3], for location trends in normal distributions, its asymptotic efficiency vanishes when compared with Kendall s Q.

3 A nonparametric test for trend based on initial ranks 831 On the other hand, Diersen and Trenkler [8] showed that for polynomial location trends in uniform and exponential distributions, the asymptotic efficiency of D is infinitely greater than that of Q. However, this asymptotic statement is not readily reflected in small or moderate sample sizes. Our own simulations reveal for n 100 exponential) and n 50 uniform), Q in fact to be more powerful than D. Although these asymptotic comparisons have been made in the literature for test statistics mentioned earlier, we are not aware of any work considering finite sample sizes. In this article, we propose a different nonparametric trend test and compare its small-sample performance to the procedures mentioned earlier. In section, we will introduce initial ranks and derive a new class of test statistics on the basis of this concept. We will see that this approach allows us to look at the previously mentioned statistics from a different point of view, which will enable us to combine their advantages. In section 3, we use Monte Carlo simulations to perform a power comparison for various distributions, trends and sample sizes. On the basis of this extensive simulation study, it turns out that M as well as our new statistic G are consistently more powerful or at least as good as the Kendall s Q and Spearman s r S statistics. In section 4, we apply several trend test statistics to monthly ozone data from Los Angeles and observe their performance.. A class of test statistics based on initial ranks In a sequence of observations X 1,...,X n, let ρ k ={number of X i X k, 1 i k}, 1 k n, 8) denote the initial rank of X k, i.e. the rank of X k when it first appears. Further, let ρ n+1 k ={number of X i X k,k i n} 9) be the initial rank of X k in the reverse sequence X n,...,x 1, where it is the n + 1 k)th observation. Note that the two types of initial ranks determine the rank through the identity R k = ρ k + ρn+1 k 1, k = 1,...,n. 10) There is a one-to-one relation between the sequence {R k } of ranks and the sequence {ρ k } of initial ranks. As large initial ranks are directly related to increasing trends, they provide an intuitive tool for the derivation of test statistics. Let us first express Kendall s Q, Spearman s r S and Foster and Stuart s D in terms of the ρ k s. Note that equation 3) can be written as Q = i<j I {Xi <X j } i<j I {Xi >X j }, where I { } equals 1 if { } holds and zero otherwise. Resolving the first sum over i and the second sum over j, we readily have Q = = n 1 ρ j 1) ρn+1 i 1) j= ρ k ρk. 11)

4 83 G. Hofmann and N. Balakrishnan Next, in order to express r S using initial ranks, observe that equation 10) implies nn + 3) = R k + 1) = ρ k + Using equations 10) and 1), we can express r S in equation 4) in the form r S = 1 nn 1) { k n + 1 ) ρ k ρk. 1) k n + 1 ) } ρk. 13) Finally, as lower records are characterized by having initial rank one, and upper records by having maximal initial rank, equation 7) can be written as D = a k ρ k ) a k ρk ), where 1 ifρ = k a k ρ) = 1 if ρ = 1. 14) 0 otherwise From equations 11), 13) and 14), we readily observe that Q, r S and D all have the form b k ρ k ) b k ρk ), where b k is some function which is nondecreasing for Q and D. This makes intuitive sense because both large ρ k and small ρ k indicate an increasing trend. The expression b kρ k ) implicitly includes two components. First, it weighs the observation number linearly in Q, because ρ k {1,...,k} can be larger for larger k, and quadratically in r S because of an additional linear factor. Secondly, it depends on a nondecreasing function of the initial rank within the observation number, i.e. its actual values on {1,...,k}. This will become clear in a moment, as we separate the two components in order to modify them independently. We standardize the initial ranks to be in the interval [ 1, 1] γ 1 = γ1 = 0, γ k = ρ k k + 1)/), γk k 1/) = ρ k and consider test statistics of the class wk 1)gγ k ) k + 1)/), k =,...,n, 15) k 1/) wk 1)gγk ), 16) where w is the weight function of the observation number and g the influence function of the initial rank. As later observations have more information about trends, and higher initial ranks indicate upward behaviour, both w and g should typically be increasing functions.

5 A nonparametric test for trend based on initial ranks 833 Straightforward calculations show that Q, r S and D are in the class 16) with the following components: Q: w Q x) = 1 x, g Qx) = x, [ 6 r S : w r x) = x n 1 ] x, g nn r x) = x, 17) 1) 1 if x = 1 D: w D x) = 1, g D x) = 1 if x = 1. 0 otherwise To combine the advantages of these statistics, we can use an influence function g that accommodates the linear case Q, r S ), the weighing of extremes D) and any intermediate situation, by means of a parameter. Possible choices are or g 1 x; a) = arctanhax) arctanha) log1 + ax)/1 ax)) =, a 0, 1) log1 + a)/1 a)) g x; a) = 1 ax + 1)/), a 0, 1). 1 a + 1)/) As both give very similar results, we prefer g 1 for its ease of computation. For small a,g 1 is close to linear, whereas for a 1, g 1 x; a) g D x). We tried several weight functions, obtaining the best overall power with cubic weights wx) = x 3. On the basis of these observations, we propose the test statistic Ga) = )) 1 + a 1 log k 1) log 3 1 a 1 + aγk 1 aγ k ) 1 + aγ )) log k 1 aγk. 18) The parameter a 0, 1) will be chosen according to n, where a larger a typically works better for a larger n. The subsequent Monte Carlo simulations and the example of section 4 illustrate some choices of a. Table 4 gives recommended parameter values. For the simulations in the following section, we use a = 0.8 for n = 10 and a = for n = Monte Carlo power comparison We performed Monte Carlo power simulations for n = 10 and n = 100 using the test statistics given previously. An initial simulation under the null hypothesis determined the critical values at the α = 0.05 level of significance. We then simulated the power under the alternative hypotheses 1) and ) with linear, logarithmic and quadratic trend functions. For the location hypothesis 1), we used the following underlying models for the U i s: standard normal, standard exponential, standard uniform, standard Laplace, standard logistic and skew-normal with density fx; λ = 5) = λx)φx) [9, 10], where and φ are the standard normal c.d.f. and density, respectively. The scale alternative ) is only appropriate for trend tests if the variables are positive. We considered it for the standard exponential and uniform distributions when n = 100. For n = 10, even for large trends β, none of the tests were able to detect the scale alternative with reasonable power. The only exception is the quadratic trend in a uniform model. The results are summarized in tables 1 and. For a good standard comparison, we chose the trend coefficient β, such that Kendall s Q achieves power between 0.5 and 0.6. The true

6 834 G. Hofmann and N. Balakrishnan Table 1. Power comparison for n = 100. Power Q r S D B M G0.995) Model standard form) Trend Type β Normal Linear Location Normal Log Location Normal Quadratic Location Exponential Linear Location Exponential Log Location Exponential Quadratic Location Uniform Linear Location Uniform Log Location Uniform Quadratic Location Exponential Linear Scale Exponential Log Scale Exponential Quadratic Scale Uniform Linear Scale Uniform Log Scale Uniform Quadratic Scale Skewed normal λ = 5) Linear Location Skewed normal λ = 5) Log Location Skewed normal λ = 5) Quadratic Location Double exponential Linear Location Double exponential Log Location Double exponential Quadratic Location Logistic Linear Location Logistic Log Location Logistic Quadratic Location Table. Power comparison for n = 10. Power Q r S D B M G0.8) Model standard form) Trend Type β Normal Linear Location Normal Log Location Normal Quadratic Location Exponential Linear Location Exponential Log Location Exponential Quadratic Location Uniform Linear Location Uniform Log Location Uniform Quadratic Location Uniform Quadratic Scale Skewed normal λ = 5) Linear Location Skewed normal λ = 5) Log Location Skewed normal λ = 5) Quadratic Location Double exponential Linear Location Double exponential Log Location Double exponential Quadratic Location Logistic Linear Location Logistic Log Location Logistic Quadratic Location

7 A nonparametric test for trend based on initial ranks 835 α s are given underneath the symbol of each test statistic. To assure the given accuracy of the simulated power, we used 1,000,000 simulations for n = 10 and 58,000 for n = 100. We performed the runs in S-Plus on a Pentium IV machine, where each run took between 9 n = 100) and 43 n = 10) min. From tables 1 and, we observe that G and M have consistently higher or equal power when compared with the Q and r S statistics. G and M have similar performances, with G having higher power for n = 100 in the exponential and uniform location cases. In addition, G and M have the advantage of being more continuous than the Q and r S statistics and consequently, one can achieve a desired α-level more closely. 4. Example Let us consider the data set of monthly averages of hourly ozone readings in downtown Los Angeles from 1955 to 197 figure 1), as given by Box et al. [11]. It shows a decreasing trend that is somewhat obscured by seasonal effects. All test statistics clearly identify the trend p-values < 0.001), when the complete data set is used. Hence, using the complete data set makes it difficult to compare among them. We can, however, look at shorter intervals of consecutive observations where the trend will be harder to detect, and a more apparent test statistic comparison can be made. We used consecutive k-year intervals k = 4,...,10) from the 18 years of data, each starting in January. Hence, there are 19 k such intervals starting in years 1955, 1956,...,1973 k. For each test statistic, we obtained the median p-values over all intervals with the same k) because this robust measure can give us a clear idea of how each test statistic behaves on shorter stretches of the sample data. Results are given in table 3. We estimated the null distributions for all test statistics null hypothesis of β = 0 in equations 1) and ), implying i.i.d. observations) from 5,000 Monte Carlo simulations. The G0.99), G0.995) and G0.999) statistics show Figure 1. Monthly averages of hourly ozone readings in downtown Los Angeles from 1955 to 197, as given by Box et al. [11].

8 836 G. Hofmann and N. Balakrishnan Table 3. Decreasing trend alternative in monthly ozone data figure 1): median p-values for the 19 k) possibilities of consecutive k-year intervals. Number Subset Number of of k-year length in ozone readings intervals years k) n = 1k) 19 k) Q r S D B M G0.99) G0.995) G0.999) Table 4. Percentiles of Ga) that can be used as critical values in tests. Percentiles of Ga) Recommended n values of a 90% 95% 97.5% 99% 99.5% ,000 19,000,500 6,500 9, ,800 7,700 85, , , , , ,000 31,000 57, ,000 35, , , , , ,000 78,000 96,000 1,00, ,000 1,10,000 1,340,000 1,570,000 1,730, ,80,000 1,640,000 1,960,000,300,000,550, ,910,000,450,000,930,000 3,470,000 3,830, ,760,000 3,530,000 4,00,000 4,980,000 5,50, ,00,000 6,410,000 7,640,000 9,110,000 10,000, ,350,000 1,000,000 14,300,000 16,900,000 18,700,000 Note: Ga) has a symmetric distribution around 0. consistently lower p-values than all other test statistics and hence appear to be better able to detect the trend. Recommendations for the parameter a are given in the following section, in conjunction with table Conclusions We introduced a new test statistic Ga) for location and scale trend alternatives to the null hypothesis of a random series of observations Ga) = )) 1 + a 1 log k 1) log 3 1 a 1 + aγk 1 aγ k ) 1 + aγ )) log k 1 aγk, 19) where γ 1 = γ 1 = 0, γ k = ρ k k + 1)/)/k 1)/), γ k = ρ k k + 1)//k 1)/) k =,...,n)and ρ, ρ are the initial ranks defined in equations 8) and 9), respectively. For a variety of trends and distributions, Ga) is at least as or more powerful as the classical tests of Kendall and Spearman or other tests considered here. Therefore, Ga) can be used as an omnipotent statistic for detecting trends. Recommended values of a and selected percentiles

9 A nonparametric test for trend based on initial ranks 837 of Ga) are given in table 4 for some values of n. Forn>150,a = can be used. The table was obtained from 100,000 simulations of the null distributions, which allow reasonable accuracy to the three given significant digits. References [1] Mann, H.B., 1945, Nonparametric test against trend. Econometrica, 13, [] Daniels, H.E., 1950, Rank correlation and population models. Journal of Royal Statistical Society Series B, 1, [3] Stuart, A., 1954, Asymptotic relative efficiencies of distribution-free tests of randomness against normal alternatives. Journal of American Statistical Association, 49, [4] Stuart, A., 1956, The efficiencies of tests of randomness against normal regression. Journal of American Statistical Association, 51, [5] Aiyar, R.J., Guillier, C.L. andalbers, W., 1979,Asymptotic relative efficiencies of rank tests for trend alternatives. Journal of American Statistical Association, 74, [6] Foster, F.G. and Stuart, A., 1954, Distribution-free tests in time series based on the breaking of records. Journal of Royal Statistical Society Series B, 16, 1. [7] Arnold, B.C., Balakrishnan, N. and Nagaraja, H.N., 1998, Records New York: John Wiley). [8] Diersen, J. and Trenkler, G., 1996, Records tests for trend in location. Statistics, 8, 1 1. [9] Azzalini, A., 1985, A class of distributions which includes the normal ones. Scandinavian Journal of Statistics, 1, [10] Johnson, N.L., Kotz, S. and Balakrishnan, N., 1994, Continuous Univariate Distributions nd edn), Vol. 1 New York: John Wiley). [11] Box, G.E.P., Jenkins, G.M. and Reinsel, G.C., 1994, Time Series Analysis Forecasting and Control New Jersey: Prentice Hall).