ESTIMATING A TAIL EXPONENT BY MODELLING DEPARTURE FROM A PARETO DISTRIBUTION

Transcription

1 The Annals of Statstcs 999, Vol. 7, No., ESTIMATING A TAIL EXPONENT BY MODELLING DEPARTURE FROM A PARETO DISTRIBUTION BY ANDREY FEUERVERGER AND PETER HALL Unvesty of Toonto and Unvesty of Toonto and Austalan Natonal Unvesty We suggest two sempaametc methods fo accommodatng depatues fom a Paeto model when estmatng a tal exponent by fttng the model to exteme-value data. The methods ae based on appoxmate lkelhood and on least squaes, espectvely. The latte s somewhat smple to use and moe obust aganst depatues fom classcal exteme-value appoxmatons, but poduces estmatos wth appoxmately 6% geate vaance when conventonal exteme-value appoxmatons ae appopate. Relatve to the conventonal assumpton that the samplng populaton has exactly a Paeto dstbuton beyond a theshold, ou methods educe bas by an ode of magntude wthout nflatng the ode of vaance. They ae motvated by data on extema of communty szes and ae llustated by an applcaton n that context.. Intoducton. Estmatng the tal exponent, o shape paamete, of a dstbuton s motvated by a patculaly wde vaety of pactcal poblems, n aeas angng fom lngustcs to socology and fom hydology to nsuance. See, fo example, Zpf Ž 9, 99., Todoovc Ž 978., Smth Ž98, 989., NERC Ž 985., Hoskng and Walls Ž 987. and Rootzen and Tajvd Ž Many models and estmatos have been poposed, ncludng those of Hll Ž 975., Pckands Ž 975., de Haan and Resnck Ž 980., Teugels Ž 98., Csogo, Deheuvels and Mason Ž 985. and Hoskng, Walls and Wood Ž The goodness of ft of a Paeto model to the tal of a dstbuton can often be exploed vsually by smply plottng the logathms of exteme ode-statstcs aganst the logathms of the anks. The plot should be appoxmately lnea f the Paeto model apples ove the ange of those ode statstcs, and n ths case the negatve value of the slope of the lne s an estmate of the Paeto exponent. Moe effcent estmatos ae avalable, howeve, and those n use today ae geneally of two types, based on lkelhood and the method of moments, espectvely. Both these appoaches ae susceptble to eos n the assumed model fo the dstbuton tal. As Rootzen and Tajvd Ž 997. pont out, none of the standad appoaches s obust aganst depatues fom the assumpton that the tal of the dstbuton s appoxmated by a GP genealzed Paeto Receved Novembe 997; evsed Febuay 999. Key wods and phases. Bas educton, exteme-value theoy, log-spacngs, maxmum lkelhood, ode statstcs, peaks-ove-theshold, egesson, egula vaaton, spacngs, Zpf s law. 760

2 ESTIMATING A TAIL EXPONENT 76 dstbuton. If thee ae maked devatons fom a GP tal, the esults wll be msleadng. On the othe hand, f one confnes attenton to data that ae so fa out n the tal that the Paeto assumpton s vald, then the effectve sample sze can be small and the estmato of tal exponent may have elatvely lage vaance. Moeove, n some applcatons of the Paeto model, data that povde evdence of depatue fom the model ae of geate nteest than those to whch the model fts well. A case n pont s Zpf s Ž 9, 99. classc study of the dynamcs of communty szes. A lnea plot of the logathm of communty sze aganst the logathm of ank, as n the case of U.S. ctes dung most of the twenteth centuy see, e.g., Zpf Ž 9., Chapte 0; Hll Ž 975. has been suggested as evdence of stable ntanatonal equlbum ; whle nonlnea plots, fo example n the context of Austan and Austalan communtes 60 to 80 yeas ago, have been ntepeted as mplyng nstablty Zpf Ž 99.. As Zpf agues, t can be of geate socologcal nteest to analyze data fom countes o eas that depat fom the benchmak of ntanatonal equlbum, than t s to analyze those whch acheve the benchmak. In ths pape we popose a smple and effectve way of educng the bas that ases f one uses exteme-value data elatvely deeply nto the sample. We show that the man effects of bas may be accommodated by modellng the scale of log-spacngs of ode statstcs. Wth ths esult n mnd, we suggest two bas-educton methods, both developed fom a smple scale-change model fo log-spacngs, and based on lkelhood and on least squaes, espectvely. In the fst method, a thee-paamete appoxmaton to lkelhood s suggested fo the dstbuton of log-spacngs. One of the paametes s the desed tal exponent. In ou least-squaes appoach, log log spacngs play the ole of esponse vaables, the explanatoy vaables ae anks of ode statstcs and the egesson eos have a type-3 exteme-value dstbuton, wth exponentally lght tals. These technques educe bas by an ode of magntude, wthout affectng the ode of vaance. Theefoe, they can lead to sgnfcant eductons n mean squaed eo. Moeove, they allow us to model exteme-value data that depat makedly fom the standad asymptotc egme. Ths advantage wll be llustated n Secton 3 by applcaton to hghly non-paeto data on Austalan communty szes. We ae not awae of kenel o moment methods that ae compettve wth ou appoach; they would be awkwad to constuct n the context of a flexble and pactcable class of dstbutons that epesent depatues fom the Paeto model. Statstcal popetes of estmatos of tal exponents n Paeto models Ž sometmes efeed to as models fo Zpf s law. have been studed extensvely. Ealy wok n a paametc settng ncludes that of Hll Ž 970, 97., Hll and Woodoofe Ž 975. and Wessman Ž Smth Ž 985. ponts out the anomalous behavou of estmatos of cetan exponents n the context of genealzed Paeto dstbutons. Hall Ž 98, 990. and Csogo, Deheuvels and Mason Ž 985., among othes, consde the effect of choce of theshold on

3 76 A. FEUERVERGER AND P. HALL pefomance of tal-exponent estmatos. Rootzen and Tajvd Ž 997. compae the pefomances of dffeent appoaches to tal-paamete estmaton. Davson Ž 98. and Smth Ž 98. povde statstcal accounts of peaksove-theshold, o POT, methods. Those technques ae of couse not dentcal to methods based on exteme-ode statstcs, but the stong dualty between the two appoaches means, as Smth Ž 98. notes, that vtually all methods fo one context have vesons fo the othe. Ou bas-coecton methods ae no excepton. Howeve, fo the sake of bevty we shall dscuss them only n the case of exteme ode-statstcs. Secton wll ntoduce ou methods, and Secton 3 wll descbe the numecal popetes. Theoetcal pefomance wll be outlned n Secton, and techncal aguments behnd that wok wll be summazed n Secton 5.. Methodology... Modellng the souce of bas. We shall ntoduce methodology n the case whee the tal that s of nteest s at the ogn. Ou methods extend mmedately fom thee to the case of a tal at nfnty. Suppose the dstbu- ton functon F admts the appoxmaton F x Cx as x 0, o moe explctly, Ž.. FŽ x. Cx Ž x., whee C, ae postve constants and denotes a functon that conveges to 0as x 0. We wsh to estmate fom a andom sample X X,..., X n dawn fom the dstbuton F. Often, one would poceed by assumng the patcula Paeto model Ž.. F0 Ž x. Cx, assumed fo 0 x, say, athe than Ž.., and allevatng bas poblems caused by dscepances between Ž.. and Ž.. by usng only patculaly small ode-statstcs fom X. Howeve, ths appoach can have a detmental effect on pefomance, snce t gnoes nfomaton about that les futhe nto the sample. That nfomaton would be usable f we knew moe about the functon. In late wok we shall efe to the model at Ž.. as a petubed Paeto dstbuton, when t s necessay to dstngush t fom the Paeto dstbuton at Ž... One appoach to accessng the nfomaton s to model, fo example n the fashon Ž.3. Ž x. Dx ož x. as x 0, whee 0 and D ae unknown constants. In pncple we could dop the small oh emande tem at Ž.3., substtute the esultng fomula nto Ž.. and estmate the fou paametes C, D,, by maxmum lkelhood. Ths means, n effect, wokng wth the model. F x C x C x,

4 ESTIMATING A TAIL EXPONENT 763 assumed fo 0 x, say, whee C,, ae all postve and the model s made dentfable by nsstng that. The type of depatue fom a Paeto model suggested by Ž.3. o Ž.. agues that the tue dstbuton s, to a fst appoxmaton, a mxtue of two Paeto dstbutons. Ths s sometmes mplctly assumed n accounts of depatues fom the Paeto. See fo example Zpf s Ž 99., page 3 dscusson of the contbutons made by dstnct uban and ual communtes to the oveall dstbuton of communty szes, and the emaks n the Appendx n ths pape on data fom countes that have been fomed fom meges of autonomous states... Lkelhood and least-squaes appoaches. Ou methods ae based on the obsevaton that, to a good appoxmaton, the nomalzed log-spacngs of small ode-statstcs n the sample X ae vey nealy escaled exponental vaables, whee the scale change may be smply epesented n tems of the model at Ž... Specfcally, let Xn Xnn denote the ode statstcs fom X and defne U Ž log Xn, log X n.. Then, fo a functon that can be expessed n tems of the of Ž.., t may be shown that Ž.5. U Z Ž n. Z exp Ž n., whee and the vaables Z, Z,... ae ndependent and exponentally dstbuted wth unt mean. It can be poved that f satsfes Ž.. and Ž.3. then.6 y D y o y Ž. as y 0, whee and D C D. In vew of Ž.5., ths suggests egadng the vaables U as exponental wth mean exp D n, and estmatng, D, by maxmum lkelhood. In ths case the negatve log-lkelhood s Ý Ý Ž.7. log D Ž n. U exp D Ž n.. ŽHee, s the theshold o smoothng paamete, about whch we shall say moe n Secton... Dffeentatng wth espect to, equatng to zeo and solvng fo, we obtan TŽ D,., say. Substtutng back nto Ž.7. we conclude that Ž D ˆ, ˆ. should be chosen to mnmze Ý Ý Ž.8. LŽ D,. D Ž n. log U exp D Ž n.. An altenatve appoach to nfeence may be based on the fact that, by Ž.5., the log log spacngs V log U satsfy.9 V n,

5 76 A. FEUERVERGER AND P. HALL whee log, s the mean of the dstbuton of log Z Ž 0 0 thus, , the negatve of the value of Eule s constant. 0, and log Z Ž fo n. 0 may be ntepeted as an eo n the appoxmate egesson model Ž.9.. Gven that may be expessed by Ž.6., we may Ž fo fxed. compute explct estmatos Ž., Dˆ Ž. ˆ of the unknowns, D that mnmze Ý Ž.0. SŽ, D,. V D Ž n.. Despte the havng lage asymptotc vaance than maxmum lkelhood methods Ž see Secton., we found n numecal studes that the least-squaes appoaches wee sometmes less based and so could have supeo oveall pefomance Ž Secton 3.. The case whee s known, o s a known functon of Ž see Secton 3.., s of majo nteest. If we can educe the numbe of unknown paametes, then the vaances of ou estmates of those that eman should also be educed. Two canoncal cases deseve specal menton. In the fst, one may thnk of X as beng geneated n the fashon X Y, whee Y has a densty that s nonzeo and dffeentable at the ogn. Hee, and so. The second example s geneated n the fom F x G x, whee the dstbuton functon G s suppoted on the postve half-lne and Žcon- fned to that doman. has a densty that s nonzeo and dffeentable at the ogn. Hee, and so. In pactce one may obtan empcal evdence fo ethe of these cases by computng plot estmates of D,,. See Secton 3. fo an example..3. An exploatoy least-squaes appoach. To obtan a pelmnay appoxmaton to t s sometmes helpful to ft the sempaametc model at Ž.. and Ž.3. dectly to log-spacngs, as follows. Obseve fom Ž.5. and Ž.6. that log X log X Z D Ž n.. n, n Hence, fo j and wth x log j and y log X log X, j j n, nj Ž.. yj x j D expž x j. D 3, whee the constants D, D depend on j and n. Ž 3 The appoxmaton s geneally good, at least fo suffcently geate than j, f D3 s omtted. We do, howeve, need j modeately lage n ode to neglect the stochastc component.. We may ft the model at Ž.. by odnay o weghted leastsquaes fo values satsfyng j, say... Estmatos of. The analyss n Secton. suggests thee estmatos of, of whch the fst s based on maxmum lkelhood and the othes on least-squaes. The lkelhood-based estmato s found by takng Ž D ˆ, ˆ. to

6 ESTIMATING A TAIL EXPONENT 765 mnmze LD, Ž., defned at Ž.8., and puttng ˆ ˆ ˆ ˆ ˆ ž / Ý ½ 5 Ž.. T D, U exp D Ž n.. To deve the least-squaes estmatos, let V log U log Ž log Xn, log X. log, choose Ž, ˆ ˆ, Dˆ. to mnmze SŽ, D,. defned at Ž.0. n and let ˆ ½ ˆ n, n 5 Ž.3. W Ž log X log X. exp D Ž n., x W Ý W and 0 Hx 0 log xe dx Then two least- squaes estmatos of ae ˆ expž. 0 ˆ and ˆ 3 W. When s known, o estmated sepaately, we eplace ˆ by that value at all ts appeaances, fo example at Ž.. and Ž.3.. If we take Ž see Sectons. and 3.., then we should agan make the obvous changes to the algothm, maxmzng the lkelhood o mnmzng the sum of squaes unde the constant. Gven an estmato ˆ of, one may obtan an estmato of C qute smply, by substtutng nto the fomula Cˆ Ž X. ˆ n n. As expected, the estmatos of do not eque the value of n, whle those of C do. In pactce, when only exteme-ode statstcs ae ecoded, the value of n s usually unknown. The value of at Ž.8. and Ž.0. plays the ole of a theshold, o smoothng paamete, detemnng the depth nto the data that we ae pepaed to go when fttng the model defned by Ž.. and Ž.3.. Fo conventonal estmatos, such as those of Hll Ž 975. and also fo ou estmatos ˆj, nceasng esults n an ncease n bas, owng to depatue of Ž.. fom ts deal fom Ž.., but ths s accompaned by a decease n vaance. One vtue of ou appoach s that fo t, bas does not ncease so apdly wth nceasng, and so may be chosen an ode of magntude lage, poducng an mpovement n mean-squae pefomance by an ode of magntude. These popetes wll be demonstated numecally n Secton 3 and theoetcally n Secton ; see, fo example, Remak.. 3. Numecal popetes. 3.. Example: communty szes. Zpf Ž 99., page 39, showed gaphcally that data on lage Austalan communty szes n 9 depated makedly fom a Paeto dstbuton wth egulaly vayng tal at nfnty. He nethe tabulated hs data no gave a souce, but they wee appaently taken fom Wckens Ž 9.. See the Appendx of the pesent pape fo detals. Fgue gaphs the logathm of communty sze aganst the logathm of ank fo these data fo all 56 Austalan communtes that had 000 o moe nhabtants n 9. ŽZpf consdeed only communtes of moe than 3000 people.. The gaph would of couse be lnea f the samplng dstbuton wee Paeto.

7 766 A. FEUERVERGER AND P. HALL FIG.. Logathms of Austalan communty szes, I. Plot of logathm of communty sze aganst logathm of ank fo all 56 Austalan communtes havng 000 o moe nhabtants n 9. Nevetheless, t s plausble that the data wee geneated as X Y, whee X denotes the populaton of a andomly chosen Austalan communty and Y has the petubed Paeto dstbuton modelled by Ž.. and Ž.3.. We analyzed the data fom that vewpont and also as though they wee fom a petubed Paeto dstbuton wth an uppe bound at a fxed numbe N 0, say. That s, we subtacted each populaton sze fom N0 and egaded the new data Z N0 X as comng fom a populaton whose dstbuton functon had the fom descbed by Ž.. and Ž.3.. Ths appoach has appaently not been used befoe wth communty-sze data and would be nmcal to Zpf s deas, but thee ae nevetheless good empcal easons fo adoptng t. We shall epot hee only ths type of analyss, snce t poduced esduals whose empcal dstbuton was close to the exponental than was that of esduals obtaned usng the fst appoach. Fgue shows a plot of values of the negatve of the logathm of Z aganst the logathm of ank. It epesents the analogue of Fgue n ths context and would be lnea f the sngle-component Paeto dstbuton at Ž.. wee an appopate model fo depatues of communty sze fom an uppe bound. When applyng the exploatoy least-squaes appoach suggested n Secton.3, we found that f we took j 5 then a plot of y aganst expž x. j j, fo j 56 and wth chosen by least-squaes, gave vey nealy a

8 ESTIMATING A TAIL EXPONENT 767 FIG.. Logathms of Austalan communty szes, II. Plot of negatve value of the logathm of the dffeence between sze of 0th lagest communty and sze of the Ž 0. th lagest communty, fo 5, aganst the logathm of. staght lne wth negatve slope. Ths suggests that fo some N 0, the 0 dstbuton functon G of cty sze has vey nealy the popety, Ž 3.. GŽ N x. A logž A x. 0 A A x A x OŽ x., fo postve constants, A, A and small postve values of x. Ths s the petubed Paeto model suggested by Ž.. and Ž.3., wth. To appecate why Ž 3.. follows fom lneaty of the plot, note that lneaty suggests Ž that to a good appoxmaton, Xn, B exp B. fo constants B, B 0. Takng X G Ž n. n, and wtng x fo n, we obtan Ž x GB exp B3x.. Ths gves Ž 3.. exactly, wth. Theefoe, n ou subsequent analyss, we took atž.3., o equvalently, at Ž.6.. Ž Of couse, ths dffes fom that n the paagaph mmedately above.. We used the maxmum lkelhood Ž ML. and leastsquaes Ž LS. methods suggested n Sectons. and., wth thee constaned to equal. Fo bevty we took the LS estmato to be ˆ ; esults fo ˆ3 ae smla. Fo the sake of smplcty we took N 0 to equal the populaton of the smallest communty that was not used n the analyss. That s, f attenton was confned to communtes whose sze anked n the ange k k, then we took N0 to equal the sze of the kth lagest communty. Removal of lage communty szes was necessay to avod eos

9 768 A. FEUERVERGER AND P. HALL FIG. 3. Resdual q-q plot. Ranked values of the esduals Z ˆ, defned by Ž 3.., plotted aganst coespondng quantles of the exponental dstbuton wth unt mean. The lne y x s supemposed on the plot. Raw data wee those used to geneate Fgue. due to abtaness of communty boundaes n lage Austalan ctes. The defntons of Sydney and Sydney Noth, and Melboune and Melboune South as dstnct communtes ae among the obvous examples of ths poblem. The goodness-of-ft of the exponental dstbuton to esduals s appaent fom q q plots, of whch Fgue 3 s typcal. Thee we gaph log Zˆ Ž k. aganst log Ž. fo, whee Ž 3.. Zˆ U Ž. exp D Ž n. ˆ ˆ ˆ ½ 5 s ou estmate of the quantty Z appeang at Ž.5., the paamete estmates on the ght-hand sde of Ž 3.. wee obtaned by maxmum lkelhood unde the constant that and Zˆ Ž. denotes the th lagest value of Z. Fo Fgue 3 we took N to be the sze of the 0th lagest communty Ž 0 so that k 0. and used all emanng communtes of sze 000 o moe Žso that We do not of couse know the value of n n Ž 3.., but that s of no concen snce t s absobed nto the estmato of D. The esultng estmates of, fo dffeent values of and the cut-off k, ange fom about 0.9 to. and ae gven n Table. The ML estmates ae a lttle less vaable than those based on LS and also tend to be a lttle lage. Not much can be ead nto ths, howeve, snce all estmates wee computed fom the same data. Both ML and LS estmates ae stkngly esstant to

10 ESTIMATING A TAIL EXPONENT 769 substantal changes n. The man souce of vaaton appeas to be andom fluctuaton, athe than systematc vaaton wth. Ths eflects the vey low bas asng fom the excellent ft of the model when s constaned to equal ; see Fgue 3. By way of contast, the Hll Ž 975. estmate of nceases vtually monotoncally wth nceasng, and moe than doubles n sze Ž fom.57 to wthn the ange of Table. Ths substantal systematc eo eflects the vey poo ft to data offeed by the sngle-component Paeto model at Ž... The poo ft s exemplfed by a hghly nonlnea q q plot Ž analogous to that of Fgue 3 and not gven hee. fo esduals unde the sngle-component model. In the same settng as Fgue 3, the full Ž.e., wthout constaned. ML estmate of equals 0.6, and the estmate of s 0.5. To assess the sgnfcance of these esults we conducted a smulaton study usng data geneated fom the dstbuton at Ž 3.3., wth, n 0,000 and 50 and values of B that poduced Zpf plots havng modeate cuvatues, as n Fgue. ŽThe case of hghe cuvatue wll be teated n Secton 3... These values of n and wee chosen because they lead to Zpf plots boadly smla to that n Fgue. Fo the sake of smplcty we shall contnue to use the same n and n Secton 3.. TABLE Estmates of fo dffeent values of k and * k k ML LS Hll ML LS Hll ML LS Hll ML LS Hll ML LS Hll ML LS Hll * In each cell n the table, the maxmum lkelhood estmate Ždenoted by ML, and computed unde the constant. s lsted above the coespondng least-squaes estmate Ž LS., whch n tun s lsted above the Hll Ž 975. estmate.

11 770 A. FEUERVERGER AND P. HALL We found that Ž. the full ML and LS estmatos of have much hghe vaance than the constaned estmatos Ž.e., wth constaned to equal.; Ž. the dffeences n bas ae elatvely mno; Ž 3. the constaned ML estmato of s substantally moe accuate than the Hll Ž 975. estmato; and Ž. the full ML estmato of s based downwads by a facto close to, wth vaance beng less of a poblem. Ths bas appeas to be due to thd-ode effects, whch ae not captued by second-ode models such as the combnaton of Ž.. and Ž.3.. In vew of ths bas, we do not fnd the value 0.5, obtaned usng the full ML method, to be poblematcal. Oveall, the full ML estmato does not pefom well fo these data, snce depatue fom a sngle Paeto dstbuton s not suffcently geat; but the constaned ML method s effectve. We conclude that the tue value of s close to. Ths s the value clamed by Zpf Ž 99. fo countes such as the Unted States that satsfy hs law of ntanatonal equlbum, although of couse he was concened wth egula vaaton at nfnty, not at an uppe bound to cty sze. 3.. Summay of numecal popetes. When fttng mxtue models, t s geneally found that a multcomponent model poduces mpoved pefomance only f thee s clea evdence that moe than one component s necessay. When a sngle component s adequate, fttng a mxtue of two o moe components typcally leads to poo pefomance, because Ž. the addtonal nusance paametes use up nfomaton that would othewse be avalable fo estmatng the man paametes of nteest, and Ž. thee ae poblems of dentfablty when the components ae close. Ths gves se to elatvely poo pefomance of the full ML and LS methods, noted n Secton 3., when the Zpf cuve has only modeate cuvatue. Howeve, when fttng a sngle Paeto dstbuton to data such as those on whch Fgue s based, t s elatvely mpotant to detemne the theshold,, by empcal means. It has been obseved pevously that, n a ange of settngs, empcal choce of can ncease oot mean squaed eo by a facto of about when s unknown; see, fo example, Hall and Welsh Ž As shown n Secton 3. Ž Table., when fttng a Paeto mxtue the estmato of s elatvely obust aganst systematc effects esultng fom choce of. Theefoe, even f full o constaned ML o LS methods, fo fxed values of, poduce estmates that pefom smlaly to Hll s estmato when the latte s computed at an optmal theshold, the ML o LS methods can be supeo to the Hll estmato n pactce. Analyss of theshold-choce methods fo Hll s estmato s beyond the scope of ths pape. Snce s the paamete of a hgh-ode tem n the model, the lkelhood suface s elatvely flat nea the maxmzng value of. Fgue depcts a typcal gaph of LŽ D,., defned at Ž.8., and shows ths popety clealy. Smla behavo may be obseved fo the functon SŽ, D,., defned at Ž.0., and n ths case one must also take cae that the estmate of s not taken to be a pathologcal extemum at nfnty. These poblems ae not as

12 ESTIMATING A TAIL EXPONENT 77 FIG.. Lkelhood suface. Typcal plot of LŽ D,. fo data geneated n the smulaton study n Secton 3.. seous when data sets ae analyzed ndvdually, howeve. In ou smulaton study we used gd seach to appoxmate the mnmum, but due to the flatness of L and S t sometmes happened that the value we obtaned was a long way fom the tue mnmum. Addtonally, n some samples whee the Zpf plot was appoxmately lnea, despte the aveage Zpf cuve beng nonlnea, the estmate of was a lage dstance fom the tue value of. Fo these easons we use medan absolute devaton Ž MAD. nstead of mean squaed eo to descbe pefomance. We smulated data fom the dstbuton Ž 3.3. X U expž BU., whee, 0, B D, D and U had a Unfom dstbuton on the nteval 0,. The paametes,, D have the meanngs ascbed to them n Secton, and the value of C thee s now. In patcula, the petubed Paeto model defned at Ž.. and Ž.3. s vald. In the context of Ž 3.3., and fo n 0,000 and 50, Table gves medan absolute devatons of estmates computed by full o constaned ML and LS methods. The constaned estmatos ae geneally slghtly supeo, and the ML and LS methods pefom smlaly. Thoughout ths secton, the esults epoted epesent aveages ove 00 smulated ndependent samples. The MAD of estmatos suggested by Hll Ž 975., fo 50, exceeds that of full ML estmatos by a facto of between.3 and 3.6 acoss the ange of Table. If the Hll estmato s computed at the value of that gave t optmal MAD pefomance n the smulaton study, then ts MAD s geneally

13 77 A. FEUERVERGER AND P. HALL TABLE Values of mean absolute devaton* Ž B 500. Ž B 5,000. Ž0. ML 0. Ž0. ML 0.06 Ž0. LS 0.3 Ž0. LS 0.07 Ž0. LS 0.3 Ž0. LS 0.07 Ž. ML 0.0 Ž. ML 0.00 Ž. LS 0.0 Ž. LS Ž. LS 0.0 Ž B 300. Ž B 300. Ž0. Ž0. ML 0.30 ML 0.30 Ž0. Ž0. LS 0.36 LS 0.36 Ž0. Ž0. LS 0.35 LS 0.35 Ž. Ž. ML 0.3 ML 0.7 Ž. Ž. LS 0. LS 0.38 Ž. LS 0.3 Ž B 00. Ž B 30. Ž0. ML 0.87 Ž0. ML.0 Ž0. Ž0. LS 0.58 LS 0.93 Ž0. Ž0. LS 0.59 LS 0.9 Ž. Ž. ML 0.6 ML.0 Ž. Ž. LS 0.57 LS.0 Ž. LS 0.60 * The estmates ae ˆ Ž abbevated hee to ML., Ž LS. and Ž LS. ˆ ˆ3, and ae as defned n Secton.. The unconstaned foms, constaned foms subject to, and constaned foms subject to, ae ndcated by the supescpts Ž0., Ž. and Ž., espectvely. When the constant s, the dstncton between LS and LS s lost; thee, the LS estmato of s defned by mnmzng SŽ log,, D. wth espect to Ž, D.. 0 close to that of the full ML and LS estmates. Howeve, ths does not take nto account the need n pactce to choose empcally so as to acheve good pefomance. We made no attempt to optmze ou ML o LS estmates ove. When o, n patcula, pefomance can be mpoved sgnfcantly by usng smalle values of. Gudance as to the appopate may be ganed by examnng q q plots; see Fgue 3. In each case, B n Ž 3.. was chosen so that aveage values of Zpf plots showed cuvatue moe maked than that n Fgue and of the opposte sgn. Subject to ths constant, B was selected so that cuvatues wee vsually smla n all the settngs of the table. Fgue 5 depcts aveage values of Zpf plots n the case of the fst column of Table, each cuve epesentng the mean of 00 ndependent synthetc samples. If cuvatue s deceased then the pefomance of full ML and LS methods elatve to the constaned vesons deteoates, snce the elatve contbuton of stochastc eo to MAD nceases. As noted n Secton 3., the ML and LS estmatos ae elatvely obust aganst changes n.

14 ESTIMATING A TAIL EXPONENT 773 FIG. 5. Aveage Zpf plots. Plot of expected value of the negatve of the logathm of the th smallest smulated data value, aganst the logathm of. The paamete values fo panels Ž a., Ž b. and Ž c. ae those of the fst thee blocks, espectvely, n the fst column of Table.

15 77 A. FEUERVERGER AND P. HALL. Theoetcal popetes. Assume that Ž.. holds, whee 0 and the functon s twce-dffeentable on Ž 0,. and satsfes Ž. Ž. Ž.. x x Dx O x fo 0,, as x 0, wth 0 and D. Put and, and let,..., denote postve constants. Ž 6 In Remak. we shall gve the values of fo j,,, 5.. j Defne the estmatos ˆ, ˆ, ˆ 3 as n Secton.. THEOREM. Assume condton Ž.., and that Ž n. at a ate such Ž that n O n. fo some 0. If s estmated as pat of the lkelhood o least-squaes pocedue then, fo j,, 3, mnž,. 3 ˆj nj p Ž.. N O Ž n. Ž log n. as n, whee the andom vaable N s asymptotcally Nomal NŽ0,. nj j. If, n the estmaton pocedue, we substtute fo a andom vaable that s Ž wthn O of ts tue value, whee 0 O n. p fo some 0, then Ž.. emans tue povded Ž. we ntepet Nnj as an asymptotcally Ž Nomal N 0,. andom vaable, and Ž. j 3 we eplace the emande tem mnž,. by O n log n. p Ž. REMARK. Values of. Fo nteges j and k 0, defne j H j k x Ž log x. dx, jk 0 and put Ž.Ž , 0, 0 and Then, H x x log x dx Ž., and 0, whee s the vaance of the logathm of an exponental andom vaable. Thee s no such elementay elatonshp n the cases of 3 o 6, fo whch the fomulas ae patculaly complex and, fo bevty, ae not gven hee. Moe smply, howeve, Ž. and REMARK. Ž Bas educton.. Condton Ž.. asks that to fst ode, decease lke x as x 0, and to second ode, decease lke x. In these ccumstances the bases of moe conventonal estmatos of ae asymptotc to a constant multple of Ž n. ; see fo example Hall Ž 98. and Csogo, Deheuvels and Mason Ž Ou theoem shows that ths level of bas has been elmnated completely fom the estmatos ˆ and ˆ and that the new mnž,. bas s of ode only n, multpled by a logathmc facto. Ths epesents an mpovement by an ode of magntude.

16 ESTIMATING A TAIL EXPONENT 775 REMARK.3 Ž Vaance.. The vaance of conventonal estmatos s of sze Hall Ž 98.; Csogo, Deheuvels and Mason Ž It follows fom the theoem that ths level of vaance s peseved by ou bas-educed estmatos. Moeove, t may be poved that unde the assumpton that the model at Ž.. holds exactly fo x n some nteval 0,, the estmato ˆ has asymptotc mnmum vaance among all estmatos based on X n,..., X n. REMARK. Ž Mean squaed eo educton.. The theoetcally smallest ode of mean squaed eo s acheved by selectng the theshold,, so that squaed asymptotc bas s of the same sze as asymptotc vaance. In vew of the esults noted n Remaks. and.3, ths wll poduce a mean squaed eo that s an ode of magntude less fo ou estmatos ˆ, ˆ and ˆ 3 than n cases of conventonal methods. In the conventonal cases mentoned n Remaks. and.3, ths balance would be acheved wth a value of the same sze as Ž n., but fo ou estmatos, the optmal s an ode of magntude lage. REMARK.5 Ž Estmatos of C, D and.. By extendng aguments n Secton 5 t may be shown that the estmato Cˆ j Xn ˆj n has asymp- mnž,. totc vaance of sze log n and asymptotc bas of sze n multpled by a powe of log n Ž the latte dependng on j.. Ou estmatos of D and, deved by ethe maxmum lkelhood o least squaes, have vaances of sze n log n and Ž n., espectvely. Theefoe, these estmatos wll not be consstent unless Ž n.. In vew of Remak., ths eques to be an ode of magntude lage than would typcally be used fo conventonal estmatos of. Note, howeve, that despte ou lkelhood and least-squaes estmatos beng deved as functons of estmatos of D and, consstent estmaton of these quanttes s not equed fo consstent estmaton of. 5. Devaton of theoem. Fo bevty we teat only the case of ˆ,n the settng whee,, D ae estmated togethe. Let Z, Z,... be ndependent exponental andom vaables wth unt mean and defne n Ý S Z Ž n j. j n j and T expž S.. By Reny s epesentaton fo ode statstcs e.g., Davd 970, page 8, we may choose the Z s so that X F Ž T. n fo n. Obseve too that Ž 5.. S logž n. O Ž. unfomly n and that the epesentaton of F at. may equvalently be wtten as 5. log F x log x C x, p

17 776 A. FEUERVERGER AND P. HALL whee, C log C, and by., defnng mn,, C Dx O x as x 0. Futhemoe, S S Z. It follows fom the latte esult and Ž 5.. that U Ž log X log X. Z Ž T. Ž T., n, n whence, defnng E log Z, log, log Z and Ž. Ž. Ž. log Z T T, we have Ž 5.3. U Z expž.. Ž z. Ž. Put z e. Then by., Ž z. Oexp Ž z. 3 3 fo,, as z. In vew of Ž 5.., expž S. Ž n. O Ž. p unfomly n, and so, defnng Ž n., we have Ž 5.. Ž T. Ž T. Ž S. Ž S Ž. pž. Z S O Z. Moeove, by Ž 5.., Ž 5.5. S 3Ž. 3logŽ n. OpŽ.. Defne a logž n. Ž n. Ž n.. Combnng Ž 5.. and Ž , we de- duce that Ž T. Ž T. Za O 3 Z Ž Z.. p Theefoe, a O Ž Z. Ž n. p, unfomly n. Ž. Now, a C DŽ n. OŽ n.. Hence, p Ž. 5.6 D n O Z n, unfomly n, whee D C D. Substtutng Ž 5.6. nto Ž 5.3. we see that p Ž. 5.7 U Z exp D n O Z Z n, unfomly n. Fom ths pont t s convenent to wte 0, D 0, 0 athe than, D, fo the tue values of, D, and to wte, D, fo geneal canddates Ž 0. 0 fo, D,. In ths notaton, put, D D D D and 0 0 Ž 0, and note that n and mn,.. Now, D n D D D n exp log n D n D n log n D ½ 5 0 D O n log n,

18 unfomly n. Theefoe, by 5.7, Ž 5.8. Ž 5.9. ESTIMATING A TAIL EXPONENT 777 Ž n. V U exp D Ž D. p Z D 0 l O Q Ž D,., V Ž n. Z D l Z Ž n. 0 D O Q D, p unfomly n, whee l log n and Ž. Ž. ½ D Ž. 5 Q D, Z Z n log n. Fo j,, let Bj equal the aveage ove of the left-hand sdes of Ž 5.8. and Ž 5.9., espectvely, and let B3 equal the aveage ove of the left-hand sde of Ž 5.9. multpled by l. Let Z, Z and mz equal the aveages of Z, Z and mz, espectvely, ove the same ange. Put l l and m logž., and obseve that l l m. Fo nonnegatve nteges k, j k defne Ý m, and let q n Ž log n. jk D. In ths notaton we have, by Ž 5.8. and Ž 5.9., Ž 5.0. Also, 0 B Z D 0 D l 0 Op q, 0 B Z D 0 D l 0 Ý p Z Ž n. O Ž q., 0 B3 mz l Z D l 0 D Ž l l 0. Ý p Z Ž n. l O Ž q log n.. Ý 0 Ý Ý 0 Ý b n n, b n log n l n l. Consequently, 0 bb 0 Z D 0 D l 0 Ý p Z Ž n. O Ž q., 0 b B l 0 Z D 0 D l 0 Ý p Z Ž n. l O Ž q log n.,

19 778 A. FEUERVERGER AND P. HALL whence 0 bb B 0Z Z D 0 0 D Ž 5.. Ž 5.. O q pž log n., Ž 3. Ž 0. b B B l Z mz l Z l Ž l D l 0 0 l 0 D l l 0 Ž l 0. p½ 5 O q log n log n. Hee we have used the fact that, fo j 0,, j j Ý p½ 5 Ž Z. Ž n. l O Ž log n.. Let M m jk denote the symmetc matx wth m 0 0, m lž. and m lž 0. l Ž 0.. Then, det M Ž.Ž. Ž.. Defne Ž w, w, w EZ, Ž Z, mz., Ž W, W, W. Ž Z w, Z w, mz w. 3 3, A 0W W and A Ž l. W Ž W lw Let R, R,... denote genec andom vaables each of whch equals o Ž.. In ths notaton, Ž 5.3. Hence, Ž 5.. T W det M 0, l 0 M A, A Ž 0 0. W Ž. W Ž. W W Ž det M. W Ž 0. det M W Ž 0. W Ž 0 0. W 3. j Snce jk jk as n then the quantty at 5. s asymptotcally Nomal wth zeo mean and vaance, whee s as defned n Remak.. p

20 ESTIMATING A TAIL EXPONENT 779 Let p Ž n. denote the pat of q that does not nvolve D o. If, n the quantty on the fa left-hand sde of Ž 5.3., we eplace Ž A, A. by Ž A, A. A O Ž p., A O Ž p log n., p p then the net change to the fa ght-hand sde of Ž 5.3. s to add a tem O Ž t. p, whee t pž log n.. Hence, the net change to Ž 5.. s to add a tem of sze Ž n. Ž log n. 3. We clam that ths gves the clamed lmt theoem n the case of. To appecate why, let ˆ, ˆ ˆ D denote the vesons of D, n whch D, ae eplaced by D ˆ, ˆ, espectvely. Note that the functon LD, Ž., defned at Ž.3., s mnmzed when bb B 0 and bb B3 0. Theefoe, D ˆ, ˆ ae gven asymptotcally by the equatons fomed by settng the ght-hand sdes of Ž 5.. and Ž 5.. equal to zeo. Ths shows Ž ˆ ˆ. T Ž 0. T that, equals D M A, A D note the appeaance of M Ž A, A. T on the left n Ž 5.3., plus tems that ae ethe neglgble o of ˆ sze n log n. Moeove, ou lkelhood-based estmato ˆ of equals B B Ž D,., evaluated at Ž D ˆ, ˆ.. Usng the expanson Ž 5.0. of B we see that ˆ equals ž / 0 Z D, l ˆ, ˆ, 0 0 D plus tems that ae ethe neglgble o of sze Ž n. Ž log n. 3. Note the appeaance of Ž, l. on the left n Ž Hence, by Ž 5.3. Ž 0 0 modfed as suggested eale., we deduce that ˆ equals the quantty at Ž 5.., plus a tem of sze t. The desed cental lmt theoem fo follows. APPENDIX Notes on the data. The data analyzed n Secton 3. wee extacted fom tables pepaed followng the census of the Commonwealth of Austala on the nghts of 3 and Apl, 9. See Wckens Ž 9.. As defntons of communtes we took Muncpaltes fo New South Wales, Westen Austala and Tasmana, Ctes, Towns and Booughs fo Vctoa, Ctes and Towns fo Queensland, and Copoatons fo South Austala. Altenatve defntons, ncopoatng spase communtes n ual dstcts, would nclude data on She populatons n New South Wales, Vctoa and Queensland, Dstct Councls fo South Austala, and Road Dstcts fo Westen Austala. Howeve, ou choce appeas to epoduce exactly the data pesented gaphcally by Zpf Ž 99., page 39. The populatons of the two ntenal tetoes n 9, the Fedeal Captal Tetoy and the Nothen Tetoy, wee not tabulated by Wckens Ž 9. n a fom whch s eadly compaable wth that fo the states, although detaled geogaphc dstbutons wee gven. We judged fom the latte that the tetoes dd not nclude any communtes wth populatons exceedng 000 and so dd not nclude them n ou data set. T ˆ

21 780 A. FEUERVERGER AND P. HALL Explanatons altenatve to those of Zpf 9, 99 ae possble fo nonlnea plots of log-communty sze aganst log-ank. They nclude the hypothess that the data ae dawn fom mxtues of Paeto dstbutons. Fo example, the dstbuton of Austalan communty szes n 9 would have eflected the stctues of development n sx lagely autonomous Btsh colones, whch had been fedeated nto a mxtue only 0 yeas pevously. Acknowledgment. We ae patculaly gateful to Ms. R. Boyce fo obtanng the Austalan populaton data used n Secton 3. REFERENCES CSORGO, S., DEHEUVELS, P. and MASON, D. Ž Kenel estmates of the tal ndex of a dstbuton. Ann. Statst DAVID, H. A. Ž Ode Statstcs. Wley, New Yok. DAVISON, A. C. Ž 98.. Modellng excesses ove hgh thesholds, wth an applcaton. In Statstcal Extemes and Applcatons Ž J. Tago de Olvea, ed Redel, Dodecht. DE HAAN, L. and RESNICK, S. I. Ž A smple asymptotc estmate fo the ndex of a stable dstbuton. J. Roy. Statst. Soc. Se. B HALL, P. Ž 98.. On some smple estmates of an exponent of egula vaaton. J. Roy. Statst. Soc. Se. B 37. HALL, P. Ž Usng the bootstap to estmate mean squaed eo and select smoothng paamete n nonpaametc poblems. J. Multvaate Anal HALL, P. and WELSH, A. H. Ž Adaptve estmates of paametes of egula vaaton. Ann. Statst HILL, B. M. Ž Zpf s law and po dstbutons fo the composton of a populaton. J. Ame. Statst. Assoc HILL, B. M. Ž 97.. The ank fequency fom of Zpf s law. J. Ame. Statst. Assoc HILL, B. M. Ž A smple geneal appoach to nfeence about the tal of a dstbuton. Ann. Statst HILL, B. M. and WOODROOFE, M. Ž Stonge foms of Zpf s law. J. Ame. Statst. Assoc HOSKING, J. R. M. and WALLIS, J. R. Ž Paamete and quantle estmaton fo the genealzed Paeto dstbuton. Technometcs HOSKING, J. R. M., WALLIS, J. R. and WOOD, E. F. Ž Estmaton of the genealzed exteme-value dstbuton by the method of pobablty-weghted moments. Technometcs NERC Ž Flood Studes Repot. Natual Envonment Reseach Councl, London. PICKANDS, J. III Ž Statstcal nfeence usng exteme ode statstcs. Ann. Statst ROOTZEN, H. and TAJVIDI, N. Ž Exteme value statstcs and wnd stom losses: a case study. Scand. Actua. J SMITH, R. L. Ž 98.. Theshold methods fo sample extemes. In Statstcal Extemes and Applcatons Ž J. Tago de Olvea, ed Redel, Dodecht. SMITH, R. L. Ž Maxmum lkelhood estmaton n a class of nonegula cases. Bometka SMITH, R. L. Ž Exteme value analyss of envonmental tme sees: an applcaton to tend detecton n gound-level ozone. Statst. Sc TEUGELS, J. L. Ž 98.. Lmt theoems on ode statstcs. Ann. Pobab TODOROVIC, P. Ž Stochastc models of floods. Wate Resouce Reseach WEISSMAN, I. Ž Estmaton of paametes and lage quantles based on the k lagest obsevatons. J. Ame. Statst. Assoc

22 ESTIMATING A TAIL EXPONENT 78 WICKENS, C. H. Ž 9.. Census of the Commonwealth of Austala, 9. H. J. Geen, Govenment Pnte, Melboune. ZIPF, G. K. Ž 9.. Natonal Unty and Dsunty: the Naton as a Bo-Socal Ogansm. Pncpa Pess, Bloomngton, IN. ZIPF, G. K. Ž 99.. Human Behavo and the Pncple of Least Effot: an Intoducton to Human Ecology. Addson-Wesley, Cambdge, MA. DEPARTMENT OF STATISTICS CENTRE FOR MATHEMATICS UNIVERSITY OF TORONTO AND ITS APPLICATIONS 00 ST. GEORGE STREET AUSTRALIAN NATIONAL UNIVERSITY TORONTO, ONTARIO CANBERRA ACT 000 CANADA M55 3G3 AUSTRALIA halpstat@petty.anu.edu.au