Bias-corrected goodness-of-fit tests for Pareto-type behavior

Transcription

1 Bias-corrected goodness-of-fit tests for Pareto-type behavior Yuri Goegebeur University of Southern Denmark, Department of Statistics JB Winsløws Ve 9B DK5000 Odense C, Denmark Jan Beirlant Catholic University of Leuven, Department of Mathematics Celestinenlaan 200B B300 Heverlee, Belgium Tertius de Wet University of Stellenbosch, Department of Statistics and Actuarial Science Private Bag X 7602 Matieland, South Africa tdewet@sunacza Introduction Extreme value theory focuses on characteristics related to the tail of a distribution function such as indices describing tail decay, extreme quantiles and small tail probabilities In the process of making inferences about the far tail of a distribution function, it is necessary to extend the empirical distribution function beyond the available data Procedures for such inferences on distributional tails depend on one or more conditions on the tail behavior, and successful application of these procedures often depends critically on the validity of the assumptions For an up to date overview of the extreme value methodology we refer to eg Beirlant et al 2004) Despite the widespread interest in the estimation of tail characteristics, not much thought has been given to testing the underlying assumptions Given the semi-parametric nature of these conditions, testing them has a complex nature, which partly explains the scarcity of testing procedures However, an important recent contribution to testing such a broad class of conditions has been made by Dietrich et al 2002) Our contribution is in the spirit of their paper Consider random variables X,, X n independent and identically distributed iid) according to some distribution function F and let X,n X n,n denote the corresponding ascending order statistics If for sequences of constants a n > 0) n and b n ) n ) ) lim P n Xn,n b n a n x = lim n F n b n + a n x) = Gx) at all continuity points of G, for G some non-degenerate distribution function, then G has to be of the generalized extreme value GEV) type: { exp + γx) /γ ), + γx > 0, γ 0, 2) G γ x) = exp exp x)), x R, γ = 0 Note that the behavior of this distribution function is governed by the single parameter γ, called the extreme value index If F satisfies )-2), then it is said to belong to the max-domain of attraction of G γ, denoted F DG γ ) An important subclass of the max-domain of attraction of the GEV distribution is the class of the Pareto-type models These are characterized by heavy tailed distribution

2 functions with infinite right endpoints, having γ > 0 For Pareto-type distributions it can be shown that the first order condition ) can be expressed in an equivalent way in terms of the survival function F : 3) F x) = x /γ l F x), x > 0, where l F denotes a slowly varying function at infinity, ie 4) l F λx) l F x) as x for all λ > 0 In terms of the tail quantile function U, defined as Ux) = inf{y : F y) /x}, x >, we then have that 5) Ux) = x γ l U x), where l U again denotes a slowly varying function at infinity Gnedenko, 943) Pareto-type tails are systematically used in certain branches of non-life insurance, as well as in finance stock returns), telecommunication file sizes, waiting times), geology diamond values, earthquake magnitudes), and many others In the analysis of heavy tailed distributions the estimation of γ, and the subsequent estimation of extreme quantiles, assume a central position Several estimators for γ have been proposed in the literature, and their asymptotic distributions established, usually under a second order condition on the tail behavior see eg Beirlant et al, 2004, and de Haan and Ferreira, 2006) This condition specifies the rate of convergence of ratios of the form lλx)/lx), with l a slowly varying function, to their limit see Bingham et al, 987) Second order condition R l ) A slowly varying function l satisfies a second order condition if there exists a real constant ρ < 0 and a rate function b satisfying bx) 0 as x, such that for all λ, as x, lλx) lx) bx)λρ ρ In the context of estimation of γ, it is then typically assumed that the slowly varying function l U in 5) satisfies a second order condition In this paper we consider testing that the underlying distribution is of Pareto-type together with a second order condition holding Formally, our null hypothesis is H 0 : F is of Pareto-type with l U satisfying R l It is well known that the log-transform of a strict) Pareto random variable has an exponential distribution This fact is used by many authors in developing inferential procedures under the Pareto-type assumption Our approach to testing H 0 is also to exploit this fact by considering goodness-of-fit tests for exponentiality as possible test statistics The literature on goodness-of-fit tests for the exponential distribution is quite elaborate, see eg Henze and Meintanis 2005) for a recent overview of this literature Such tests often take the form of the ratio of two estimators for the exponential scale parameter In a similar way, we will construct our test statistics as ratios of two estimators for the extreme value index γ It should be noted that a number of recent papers evaluate the fit of the generalized Pareto distribution to a sample of exceedances over a fixed threshold, see eg Davison and Smith 990), Choulakian and Stephens 200), Marohn 2002), or of the GEV distribution to a sample of maxima,

3 see eg Kinnison 989) Related to this, as an application in the area of botany, Portnoy and Willson 993) apply stochastic diffusion processes to develop a statistical model for the tail of the seed dispersal distribution Their model includes two types of qualitative tail behavior, algebraic and exponential tails, and hence allows to test hypothesis about the nature of tail behavior All these statistics test the fit of the parametric model under investigation, and not the broader assumptions, as we do Our approach is semi-parametric in nature, involving excesses over a random threshold corresponding to an intermediate order statistic, and hence is in the spirit of Dietrich et al 2002), where a tail version of the Cramér-von Mises statistic is considered, and Koning and Peng 2005), who study tail versions of the Kolmogorov, Berk-Jones and score tests for assessing Pareto-type behavior Other related work can be found in Drees et al 2006), Neves and Fraga Alves 2006), Neves et al 2006), and Beirlant et al 2006) An excellent and up-to-date overview of the approaches to testing extreme value conditions is given in Fraga Alves and Neves 2006) A maor advantage of our approach is that the second order tail condition can be easily incorporated in the test statistics, yielding bias-corrected test procedures A kernel goodness-of-fit statistic As mentioned in the introduction, we will exploit the relationship between the strict Pareto and the exponential distribution, and the properties of the latter, to construct our test statistic Consider X,, X n iid P a/γ) random variables, where P a/γ) denotes the strict Pareto distribution with Pareto index /γ, with corresponding ascending order statistics X,n X n,n Then the ratios Y,k = X n k+,n /X n k,n, =,, k, are ointly distributed as the order statistics of a random sample of size k from the P a/γ) distribution Consequently, Y,k = log Y,k behave as Exp/γ) order statistics, where Exp/γ) denotes the exponential distribution with scale parameter γ In case the data originate from a Pareto-type distribution these properties hold approximately above a sufficiently high threshold Exponential goodness-of-fit test statistics are quite often a ratio of two estimators for the exponential scale parameter eg Lewis, 965, Jackson, 967, de Wet and Venter, 973) Inspired by this and based on the above properties of P a/γ) order statistics, we apply a similar ratio to the k largest order statistics, leading to the following test statistic ) k k K k+ Z, H k,n with K denoting a kernel function, Z = log X n +,n log X n,n ), and where H k,n is the Hill estimator for γ Hill, 975), ie H k,n = k k Z, or, after appropriate centering, 6) H k,n k K ) Z, k + with K a kernel function satisfying 0 Ku)du = 0 Theorem Consider X,, X n iid random variables according to distribution function F, where F DG γ ) for some γ > 0 Assume l U satisfies R l and let Kt) = t t 0 uv)dv for some function u satisfying k /k )/k ut)dt f k+ ) for some positive continuous function f defined on 0, ) such that 0 log+ /w)fw)dw <, 0 Kw) 2+δ dw < for some δ > 0 and ) k k K k+ 0 as k Then as k, n, k/n 0 and kbn/k) c, ) L c ) K Z N Ku)u ρ du, K 2 u)du H k,n k k + γ 0 0

4 We will now focus on some important special cases of these kernel-type goodness-of-fit statistics Jackson kernel function We modify the Jackson statistic Jackson, 967), originally proposed as a goodness-of-fit statistic for testing exponentiality, in such a way that it measures the linearity of the k largest observations on the Pareto quantile plot Consider X,,, X n iid Expλ) random variables The Jackson statistic is given by 7) T J = t,nx,n X where t,n = λex,n ) = i= n i + ) The numerator is clearly a sum of cross products of order statistics and their expected values The denominator is introduced to lift up the dependence on the nuisance parameter λ The Jackson statistic can hence be considered as a correlation like statistic based on the exponential quantile plot For our purposes it is more convenient to express 7) in terms of the standardized spacings V = n + )X,n X,n ), =,, n From the Rényi representation these are known to be iid Expλ) Rearranging terms of 7), it can be shown that T J = C,nV V where C,n = and C,n = + t,n, = 2,, n The limiting distribution of T J was derived by Jackson 967): Theorem 2 Jackson, 967) Assume X,, X n iid Expλ) random variables, then for n ntj 2) L N0, ) We will now adust the Jackson statistic in such a way that it measures the linearity of the k upper order statistics on the Pareto quantile plot Consider a random sample X,, X n of Paretotype distributed random variables Application of the Jackson statistic to Y,k, =,, k, introduced above, yields, after rearranging terms, ) Z T J = k k K J H k,n k+ where K J u) = log u Since 0 K J u)du = 2 we have K Ju) = log u The function K J satisfies the conditions of Theorem with us) = 2 log s, and hence we can state the following proposition Proposition Assume X,, X n iid random variables according to distribution function F, where F DG γ ) for some γ > 0 and l U satisfying R l Then as k, n, k/n 0 and kbn/k) c, kt J 2) = H k,n k ) ) L cρ K J Z N k + γ ρ) 2, Bias-corrected Jackson kernel function To obtain a bias-corrected version of TJ, note that both numerator and denominator of T J are weighted) averages of the Z, =,, k Within the framework of Pareto-type tails and assuming

5 condition R l on l U holds, Beirlant et al 999) derived the following approximate representation for log-spacings of successive order statistics ) ρ 8) Z γ + b n,k + ε, =,, k, k + where b n,k = bn/k) and ε, =,, k, are zero centered error terms, or, equivalently ) ρ Z b n,k γ + ε, =,, k k + This then motivates the following bias-corrected Jackson statistic k TBCJˆρ) k K J k+) Z ˆb ) ) ˆρ LS,k ˆρ) k+ =, ˆγ LS,k ˆρ) with ˆρ denoting a consistent estimator for ρ Here ˆγ LS,k ρ) and ˆb LS,k ρ) are the least squares estimators for γ and b n,k, respectively, obtained from 8), taking ρ as fixed: 9) 0) ˆγ LS,k ρ) = k Z ˆb LS,k ρ) ρ, ˆbLS,k ρ) = ρ)2 2ρ) ρ 2 k ) ) ρ Z k + ρ Of course, in practice, the parameter ρ has to be replaced by a consistent estimator ˆρ Known estimators ˆρ k,n using k extreme data) share the consistency property when kbn, k For an elaborate discussion of the estimation of ρ we refer to Gomes et al 2002) and Fraga Alves et al 2003) Under a third order slow variation condition, estimators were proposed such that, as kbn, k, kbn, kˆρ k,n ρ) is asymptotically normal with After some straightforward manipulations it follows that kt ) BCJ ˆρ) 2) K BCJ ˆγ LS,k ˆρ) k k + ; ˆρ Z K BCJ u; ρ) = log u + 2ρ ρ u ρ ) ρ The limiting distribution of the normalized bias-corrected Jackson statistic can be derived since the kernel function K BCJ satisfies the conditions of Theorem with us) = 2 log s+2ρ )s ρ ρ) / ρ))/ρ Proposition 2 Assume X,, X n iid random variables according to distribution function F, where F DG γ ) for some γ > 0, l U satisfies R l and ρ is known, then as k, n, k/n 0 and kbn/k) c, ˆγ LS,k ρ) k ) ) ) K BCJ k + ; ρ L ρ 2 Z N 0, ρ

6 Note that the effect of the bias correction is nicely reflected in the limiting normal distribution: whatever c, the normal limit is centered at 0 When ρ is substituted by an estimator ˆρ k,n as dis- ) k k K BCJ k+ ; ˆρ Z, can be represented cussed above, then the normalized test statistic asymptotically by ) K BCJ ˆγ LS,k ρ) k k + ; ρ Z + kbn/k)ˆρ k,n ρ) ˆγ LS,k ρ) k + o P ), ˆγ LS,k ˆρ) k + ) ρ ρ K BCJ ) k + ; ρ so that the asymptotic distribution stated in Proposition 2 does not change after substitution of ρ, in the case kbn/k) c Lewis kernel function As a second example we study the Lewis goodness-of-fit statistic Consider a sample X,, X n of iid Expλ) random variables The Lewis statistic is given by T L = n n+ V n + n X Asymptotic normality of this statistic was proven in Lewis 965) Theorem 3 Lewis, 965) Assume X,, X n iid Expλ) random variables, then for n ntl 05) L N0, 2 ) In case of a random sample X,, X n of Pareto-type random variables, we can apply the Lewis statistic to Y,k, =,, k, yielding T L = k k k+ Z H k,n Clearly K L u) = u and K Lu) = u 05 The function K L satisfies the conditions of Theorem with us) = 2s 05, leading to the following proposition: Proposition 3 Assume X,, X n iid random variables according to distribution function F, where F DG γ ) for some γ > 0 and l U satisfying R l Then as k, n, k/n 0 and kbn/k) c, kt L 05) = H k,n k ) L cρ K L Z N k + 2γ ρ)2 ρ), ) 2 Note that for the same value of c, the absolute value of the bias of the Lewis statistic is smaller than the absolute bias of the Jackson statistic Bias-corrected Lewis kernel function We now adust the Lewis statistic by taking the second order tail behavior of a Pareto-type model into account The derivation is completely analogous to the case of the bias-corrected Jackson statistic, leading to k k k+ 2) Z ˆb ) ) ˆρ LS,k ˆρ) k+ T BCL ˆρ) = ˆγ LS,k ˆρ)

7 with ˆγ LS,k ρ) and ˆb LS,k ρ) as in 9) and 0), respectively In this case the kernel function is given by K BCL u; ρ) = u ρ) 2ρ) + u ρ ) 2 2ρ2 ρ) ρ Proposition 4 Assume X,, X n iid random variables according to distribution function F, where F DG γ ) for some γ > 0 and l U satisfying R l with ρ Then as k, n, k/n 0 and kbn/k) c, ) K BCL ˆγ LS,k ρ) k k + ; ρ L Z N 0, ) ) + ρ ρ Some remarks apply Firstly, unlike the uncorrected Lewis statistic, the bias-corrected version has a limiting distribution with mean zero, irrespective of the value of c Secondly, in case ρ = a degenerate limiting distribution is obtained for the bias-corrected Lewis statistic This follows immediately from its construction, in fact K BCL u; ) = 0 In this case ρ = ) ) k ˆγ LS,k ˆρ) k K BCL k+ ; ˆρ Z, is represented asymptotically as kbn/k)ˆρ k,n ρ) ˆγ LS,k ρ) k k + ) ρ ρ) ρ K BCL k + ρ= ; + o P ), so that the asymptotics of ˆρ dominates the limiting behavior of the bias-corrected Lewis statistic REFERENCES Beirlant, J, de Wet, T and Goegebeur, Y, 2006 A goodness-of-fit statistic for Pareto-type behaviour Journal of Computational and Applied Mathematics, 86, 99-6 Beirlant, J, Dierckx, G, Goegebeur, Y and Matthys, G, 999 Tail index estimation and an exponential regression model Extremes, 2, Beirlant, J, Goegebeur, Y, Segers, J and Teugels, J, 2004 Statistics of Extremes - Theory and Applications Wiley Series in Probability and Statistics Bingham, NH, Goldie, CM and Teugels, JL, 987 Regular Variation Cambridge University Press, Cambridge Choulakian, V and Stephens, MA, 200 Goodness-of-fit tests for the generalized Pareto distribution Technometrics, 43, Davison, AC and Smith, RL, 990 Models for exceedances over high thresholds with comments) Journal of the Royal Statistical Society, Series B, 52, de Haan, L and Ferreira, A, 2006 Extreme Value Theory: An Introduction Springer de Wet, T and Venter, JH, 973 A goodness-of-fit test for a scale parameter family of distributions South African Statistical Journal, 7, Dietrich, D, de Haan, L and Hüsler, J, 2002 Testing extreme value conditions Extremes, 5, 7-85 Drees, H, de Haan, L and Li, D, 2006 Approximations to the tail empirical distribution function with applications to testing extreme value conditions Journal of Statistical Planning and Inference, 36,

8 Fraga Alves, MI, Gomes, MI and de Haan, L, 2003 A new class of semi-parametric estimators of the second order parameter Portugaliae Mathematica, 60, Fraga Alves, MI and Neves, C, 2006 Testing extreme value conditions an overview and recent approaches International Conference on Mathematical and Statistical Modeling in Honor of Enrique Castillo June 28-30, 2006 Gnedenko, BV, 943 Sur la distribution limite du terme maximum d une série aléatoire Annals of Mathematics, 44, Gomes, MI, de Haan, L and Peng, L, 2002 Semi-parametric estimators of the second order parameter in statistics of extremes Extremes, 5, Henze, N and Meintanis, SG, 2005 Recent and classical tests for exponentiality: a partial review with comparisons Metrika, 6, Hill, BM, 975 A simple general approach to inference about the tail of a distribution Annals of Statistics, 3, Jackson, OAY, 967 An analysis of departures from the exponential distribution Journal of the Royal Statistical Society B, 29, Kinnison, R, 989 Correlation coefficient goodness-of-fit test for the extreme-value distribution The American Statistician, 43, Koning, AJ and Peng, L, 2005 Goodness-of-fit tests for a heavy tailed distribution Econometric Institute Report Lewis, PAW, 965 Some results on tests for Poisson processes Biometrika, 52, Marohn, F, 2002 A characterization of generalized Pareto distributions by progressive censoring schemes and goodness-of-fit tests Communications in Statistics - Theory and Methods, 3, Neves, C and Fraga Alves, MI, 2006 statistics in extremes Test to appear) Semi-parametric approach to Hasofer-Wang and Greenwood Neves, C, Picek, J and Fraga Alves, MI, 2006 The contribution of the maximum to the sum of excesses for testing max-domains of attraction Journal of Statistical Planning and Inference, 36, Portnoy, S and Willson, MF, 993 Evolutionary Ecology, 7, Seed dispersal curves: bahavior of the tail of the distribution ABSTRACT In this paper we review the goodness-of-fit problem for assessing whether a sample is consistent with the Pareto-type model To this end we introduce a general kernel goodness-of-fit statistic The derivation of the proposed class is based on the close link between the strict Pareto and the exponential distribution and puts some of the available goodness-of-fit procedures for the latter in a broader perspective The limiting distribution for this general kernel statistic will be derived under mild regularity conditions and some important special cases will be investigated in greater depth Keywords and phrases: extreme value index, kernel statistic, goodness-of-fit