PostScript le created: August 6, 2006 time 839 minutes

GEOPHYSICAL RESEARCH LETTERS, VOL.???, XXXX, DOI:10.1029/, PostScript le created: August 6, 2006 time 839 minutes Earthquake predictability measurement: information score and error diagram Yan Y. Kagan 1 1 Department of Earth and Space Sciences, University of California, Los Angeles, California, USA Yan Y. Kagan, Department of Earth and Space Sciences, University of California, Los Angeles, California, 90095-1567, USA (e-mail: ykagan@ucla.edu)

X-2 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT Abstract. I discuss two methods for measuring the eectiveness of earthquake prediction algorithms: the information score based on the likelihood ratio and error diagrams. For both of these methods, closed form expressions are obtained for the renewal process based on the gamma and lognormal distributions. The error diagram is more informative than the likelihood ratio and uniquely species the information score. I derive an expression connecting the information score and error diagrams. I then obtain the estimate of the region bounds in the error diagram for any value of the information score.

KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT X-3 1. Introduction In a recent article, Jordan (2006) argued that more objective, rigorous, quantitative methods for testing earthquake prediction schemes need to be developed. Particularly, he asked \What is the intrinsic predictability of the earthquake rupture process?" To contribute to this inquiry I discuss two methods currently used to measure the performance of earthquake prediction programs. The rst method is the likelihood ratio procedure which has long been used for statistical analysis of random processes. In particular, Kagan and Knopo (1976 1987), Kagan (1991), Ogata (1999), Kagan and Jackson (2000), Imoto (2004), Rhoades and Evison (2006), and Helmstetter et al. (2006) have applied this likelihood method for earthquake occurrence studies. Kagan and Knopo (1977) rst proposed calculating the information score for earthquake predictability based on the likelihood ratio. The second method is related to the Relative Operating Characteristic (ROC) used in weather prediction eorts (Jollie and Stephenson, 2003), where the success rate of an event prediction is compared against the false alarm rate (ibid., p. 69 see also Holliday et al., 2005). Since periodic (diurnal, annual) eects are strong in weather prediction, such a method has broad applications we can compare the above characteristics of a forecast system for one-day or one-year alarm periods. But in earthquake prediction, there is no natural time scale for forecasting, so the time interval is arbitrary. Therefore, if the alarm duration is increased, both criteria approach the trivial result: all events are predicted with no false alarms.

X-4 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT Molchan (1990) modied this method as an error diagram to predict random point processes. Molchan and Kagan (1992) and Molchan (1997 2003) also review the error diagram method and its applications. McGuire et al. (2005), Kossobokov (2006), and Baiesi (2006) recently used this method to evaluate earthquake prediction algorithms. Kagan and Jackson (2006) commented on Kossobokov's analysis and discussed the application of error diagrams to predicting earthquakes. 2. Information score Kagan and Knopo (1977) suggested measuring the eectiveness of earthquake prediction algorithm by rst evaluating the likelihood ratio to test how well does a model approximate earthquake occurrence. In particular, they estimated the information score, ^I, per one earthquake by ^I = ` ; `0 N = 1 N NX i=1 log 2 p i i (1) where ` ; `0 is the log-likelihood ratio, N is the number of earthquakes in a catalog, p i is the probability of earthquake occurrence according to a stochastic model, i is a similar probability for a Poisson process, and log 2 was used to obtain the score in bits of information. One information bit would mean that uncertainty of earthquake occurrence is reduced on average by a factor of 2 by the using a particular model. Here the `average' needs to be understood as a geometrical mean. For long catalogs (N!1) I = 1 N E log 2 p i i ^I (2) where E is the mathematical expectation (Vere-Jones, 1998 Daley and Vere-Jones, 2003).

KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT X-5 For a renewal (i.e., with independent intervals) process the information score can be calculated as (Daley and Vere-Jones, 2003, their equation 7.6.16) I = m (1; log m + H ) (3) where m is the intensity (rate) of a renewal process and the entropy function H is H = Z 1 0 f(x) logf(x) dx (4) where f(x) is a probability density function (pdf). The entropy function (4) has been calculated in closed form for two distributions, gamma and lognormal (Daley and Vere-Jones, 2004 Bebbington, 2005). Imoto (2004) obtained information score estimates for the lognormal, gamma, and several other distributions. For this purpose (4) was integrated numerically. The gamma distribution has the pdf f(x) = x ;1 ;() exp(;x) (5) where ; is the gamma function, is a shape parameter, is a scale parameter, and 0 <<1 0 <<1 0 x<1 (Evans et al., 2000). If = 1, then the process is the Poisson one, <1characterizes the occurrence of clustered events. For the gamma renewal process, normalized to have the mean equal to 1, i.e., =, the information score is (Daley and Vere-Jones, 2004, their equation 14) I() =[log +( ; 1) () ; ; log ;()]= log(2) (6) where is the digamma function (Abramowitz and Stegun, 1972). If = 1, I() = 0. The lognormal distribution has the pdf f(x) = p 1 exp ; 1 x 2 2 [log(x) ; 2 ]2 (7)

X-6 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT where is a shape parameter, is a scale parameter and 0 <<1 0 <<1 0 x<1 (Evans et al., 2000). For a renewal process normalized to have the mean equal to 1, the information score for the lognormal distribution is (Bebbington, 2005, p. 2303) I() = " 1+ 2 2 log(2) ; log 2 p 2 # : (8) The small -values correspond to a quasi-periodic process, the large values to a clustered one. The sequence with the parameter value = 1 is the closest to the Poisson process, its information score is at minimum, but is still non-zero, I = 0:117 bits. Fig. 1 displays a simulation of the lognormal renewal process for = 1:86. According to (8), the average information score for such a process would be 1 bit. An alarm with the duration w is declared after each event. If a new event occurs during a declared alarm, the alarm time is extended accordingly (cf. Stark, 1997). These alarms are shown at the bottom of the plot. 3. Error diagrams The error diagram for evaluating how well a prediction program performs was rst suggested by Molchan (1990). For any prediction algorithm, the diagram plots the fraction of failures to predict,, versus the fraction of alarm time,. The curve isconcave (Molchan, 1997 2003). The error diagram curve for a clustered renewal process can be calculated as (w) = Z w 0 f(x) dx (9) where w is the alarm duration and (w) = w [1; (w)] + Z w 0 xf(x) dx : (10)

KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT X-7 The rst right-hand term in (10) is the average alarm duration, if no event occurs in the w interval. The second term is the average alarm length. For the lognormal and gamma renewal processes the variables (w) and(w) canbe found in a closed form. For the gamma distribution (w) = ( w)=;() (11) and (w) = w [1; (w)]+(1 + w)=;(1 + ) (12) where is the incomplete gamma function (Abramowitz and Stegun, 1972). For the lognormal distribution (w) = 1 2 ( 1 + erf " log(w) ; 2 =2 p 2 #) (13) and (w) = w [1; (w)]+ 1 2 ( " 2 ; 2log(w) 1+erfc p 2 2 #) : (14) Here erf and erfc are the error function and its complementary function, respectively. Fig. 2 shows an example of the error diagram for the lognormal renewal process. The theoretical curves (13, 14) are compared to the simulation results for 33 choices of the alarm duration, w. The alarm duration starts with 0:001 and then increases logarithmically with the factor value 1.52 until it reaches the total length of a series (1000 units). I deliberately use relatively short simulated sequences to show random uctuations. 4. Interrelation between the error diagram and information score

X-8 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT The information score can be calculated for an error diagram curve as I = Z 1 0 log 2 @ @! d : (15) It is helpful to have an estimate of the region boundaries for curves corresponding to a specic value of the information score. Such an estimate can be obtained if the prediction scheme in the error diagram consists of two linear segments with the slopes for the rst segment and D 2 for the second segment D 1 = @ 1 @ 1 = 1 1 (16) D 1 1 and D 2 1 (17) given a curve concavity. D 1 = D 2 = D 0 = 1 is the random guess strategy (Molchan, 1990). For the assumed information score I the envelope curve is dened by the equation D 1 " 1 ; 1 ; =D 1 # 1; =2 I : (18) By solving this equation for any value of D 1, one obtains the -value for the contact point of two linear segments, = =D 1. As an example, Fig. 3 displays several two line segments that correspond to the prediction schemes with an information score that equals 1 bit. The envelope curve is also shown. This curve most likely delineates the lower boundary of all possible error curves with the information score I = 1 bit. The upper boundary is, in principle, the random guess line, D 0 : if a large D 1 is specied, the resulting curve can be as close to this line as needed.

KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT X-9 I also show in this diagram the results of simulating a mixture of two Poisson processes with the rates diering by a factor 44.4. This factor was adjusted to obtain the information score 1 bit for a renewal process in which intervals have been selected randomly from each sequence. The simulation results are similar to theoretical curves having two straight line segments. Fig. 4 displays the curves for the renewal processes with gamma and lognormal distributions. The information score again is taken to be 1 bit. Curves for both sequences, clustered and quasi-periodic, are shown. All four curves are within the region specied by (18) and the random guess line. When simulating or computing curves for quasi-periodic sequences, alarm declaration is reversed, i.e., it is declared after the elapsed w time period following an event. Alternatively, an alarm strategy is the same as in clustered sequences producing an `antipodal prediction' (Molchan and Kagan, 1992 Molchan, 1997). Then a curve is rotated 180 around the center of symmetry [ =1=2 =1=2]. 5. Discussion Clearly, from the theoretical and the simulation results described above, the error diagram represents a much more complete picture of the stochastic point process than does the likelihood analysis. Using the diagram curve one can calculate the information score for a sequence. The score also imposes some limits on the diagram region where curves are located, but Figs. 3 and 4 show that these limitations are rather broad. By specifying a more restricted class of point processes to approximate an earthquake occurrence, the interrelation between these two methods can likely be made more precise.

X-10 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT A few comments on how the discussed techniques might forecast real earthquake rupture are due here. First, more appropriate stochastic model for earthquake occurrence is not a renewal but a branching process (Hawkes and Oakes, 1974 Kagan and Knopo, 1976 Ogata, 1999) which captures the important feature of seismicity, its clustering. Moreover, earthquakes occur not only in time. Their spatial coordinates, earthquake size, and focal mechanisms need to be taken into account in actual prediction eorts. Introducing new variables complicates the calculation of the information score and the error diagram. Molchan and Kagan (1992) have done some preliminary work in determining error diagrams for multidimensional processes. Kagan and Jackson (2000) and Helmstetter et al. (2006) have shown how toevaluate the eectiveness of spatial smoothing for seismic hazard maps. Another challenge in dealing with earthquake prediction is the fractal nature of most distributions controlling earthquakes (Kagan, 2006). Since these distributions approach innity for small time and distance intervals, the value of the information score is not well dened (see Helmstetter et al., 2006). Similarly, the error diagram curve would start to approach the point of the ideal prediction ( =0 = 0) for earthquake catalogs of high location accuracy and extending to small time intervals after a strong earthquake. Finally,Iwould like to mention that equation 15 was derived, using heuristic arguments, in December 1991 { January 1992. Since that time I privately sent these preliminary results to many researchers interested in the problem. Recently, Harte and Vere-Jones (2005, their equation 18) published a similar formula for a model of the discrete-time

KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT X-11 point process. They did not explore the error diagram properties and any constraints the information score would impose on the diagram. Acknowledgments. I appreciate partial support from the National Science Foundation through grants EAR 04-09890, and DMS-0306526, as well as from the Southern California Earthquake Center (SCEC). SCEC is funded by NSF Cooperative Agreement EAR-0106924 and USGS Cooperative Agreement 02HQAG0008. Iamvery grateful to P. Stark of UC Berkeley who sent mematlab programs employed in his (Stark, 1997) paper. With appropriate modications these programs were used in some calculation reported above. The author thanks D. D. Jackson, I. V. Zaliapin, F. Schoenberg, and J. C. Zhuang of UCLA, D. Vere-Jones of Wellington University andp. Stark for very useful discussions. Publication 0000, SCEC. References Abramowitz, M. and I. A. Stegun (1972), Handbook of Mathematical Functions, Dover, NY, pp 1046. Baiesi, M. (2006), Scaling and precursor motifs in earthquake networks, Physica A, 360(2), 534-542. Bebbington, M. S. (2005), Information gains for stress release models, Pure Appl. Geophys., 162(12), 2299-2319. Daley, D. J., andvere-jones, D. (2003), An Introduction to the Theory of Point Processes, Springer-Verlag, New York, 2-nd ed., Vol. 1, pp. 469.

X-12 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT Daley, D. J., and Vere-Jones, D. (2004), Scoring probability forecasts for point processes: The entropy score and information gain, J. Applied Probability, 41A, 297-312. Evans, M., N. Hastings, and B. Peacock (2000), Statistical Distributions, 3rd ed., New York, J. Wiley, 221 pp. Harte, D., and Vere-Jones, D. (2005), The entropy score and its uses in earthquake forecasting, Pure Appl. Geophys., 162(6-7), 1229-1253. Hawkes, A. G. and Oakes, D. (1974), A cluster process representation of a self-exciting process, J. Appl. Prob., 11, 493-503. Helmstetter, A., Y. Y. Kagan, and D. D. Jackson (2006), Comparison of short-term and time-independent earthquake forecast models for southern California, Bull. Seismol. Soc. Amer., 96(1), 90-106. Holliday, J. R., K. Z. Nanjo, K. F. Tiampo, J. B. Rundle, D. L. Turcotte (2005), Earthquake forecasting and its verication, Nonlinear Processes Geophys., 12(6), 965-977. Imoto, M. (2004), Probability gains expected for renewal process models, Earth Planets Space, 56, 563-571. Jollie, I. T. and D. B. Stephenson, Eds. (2003), Forecast Verication: a Practitioner's Guide in Atmospheric Science, J. Wiley, Chichester, England, 240 pp. Jordan, T. H. (2006), Earthquake predictability, brick by brick, Seismol. Res. Lett., 77(1), 3-6. Kagan, Y. Y. (1991), Likelihood analysis of earthquake catalogues, Geophys. J. Int., 106, 135-148.

KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT X-13 Kagan, Y. Y. (2006), Why does theoretical physics fail to explain and predict earthquake occurrence?, in: Lecture Notes in Physics, P. Bhattacharyya and B. K. Chakrabarti, (Eds) Springer Verlag, accepted, see http://scec.ess.ucla.edu/ykagan/india index.html. Kagan, Y. Y., and D. D. Jackson (2000), Probabilistic forecasting of earthquakes, Geophys. J. Int., 143, 438-453. Kagan, Y. Y., and D. D. Jackson (2006), Comment on`testing earthquake prediction methods: \The West Pacic short-term forecast of earthquakes with magnitude MwHRV 5.8"' by V. G. Kossobokov, Tectonophysics, 413(1-2), 33-38. Kagan, Y., and L. Knopo (1976), Statistical search for non-random features of the seismicity of strong earthquakes, Phys. Earth Planet. Inter., 12(4), 291-318. Kagan, Y., and L. Knopo (1977), Earthquake risk prediction as a stochastic process, Phys. Earth Planet. Inter., 14(2), 97-108. Kagan, Y. Y., and L. Knopo (1987), Statistical short-term earthquake prediction, Science, 236, 1563-1567. Kossobokov, V. G. (2006), Testing earthquake prediction methods: \The West Pacic short-term forecast of earthquakes with magnitude MwHRV 5.8", Tectonophysics, 413(1-2), 25-31. McGuire, J. J., Boettcher, M. S., and Jordan, T. H. (2005), Foreshock sequences and shortterm earthquake predictability on East Pacic Rise transform faults, Nature, 434(7032), 457-461 Correction { Nature, 435(7041), 528.

X-14 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT Molchan, G. M. (1990), Strategies in strong earthquake prediction, Phys. Earth Planet. Inter., 61(1-2), 84-98. Molchan, G. M. (1997), Earthquake prediction as a decision-making problem, Pure Appl. Geoph., 149, 233-247. Molchan, G. M. (2003), Earthquake prediction strategies: A theoretical analysis, In: Keilis-Borok, V. I., and A. A. Soloviev, (Eds) Nonlinear Dynamics of the Lithosphere and Earthquake Prediction, Springer, Heidelberg, 208-237. Molchan, G. M., and Y. Y. Kagan (1992), Earthquake prediction and its optimization, J. Geophys. Res., 97, 4823-4838. Ogata, Y. (1999), Seismicity analysis through point-process modeling: A review, Pure Appl. Geophys., 155, 471-507. Rhoades, D. A., and F. F. Evison (2006), The EEPAS forecasting model and the probability of moderate-to-large earthquakes in central Japan, Tectonophysics 417(1-2), 119-130. Stark, P. B. (1997), Earthquake prediction: the null hypothesis, Geophys. J. Int., 131, 495-499. Vere-Jones, D. (1998), Probabilities and information gain for earthquake forecasting, Vyqislitel~na Sesmologi (Computational Seismology), 30, Geos, Moscow, 248-263.

KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT X-15 Figure 1. Parts of a realization of lognormal distributed renewal process. Solid line { cumulativenumber of events. Crosses { beginning of alarms, squares { end of alarms. In this example, 33 events are simulated with the shape parameter = 1:86 (see Eq. 7). After each event an alarm with duration w =0:25 is issued. 26 events fall into 7 alarms, i.e., they are successfully predicted. The duration of alarms is 41.7 % of the total time. Figure 2. Error diagrams for the lognormal renewal process with = 1:86. The straight solid line is the strategy curve corresponding to a random guess. The left solid curve is calculated using (13{14), circles are the result of simulations. The dashed and dotted curves are the rst and second right-hand terms in (14), respectively. Figure 3. Error diagrams for renewal processes. The thick straight solid line corresponds to a random guess. Thin solid lines are for the curves with the information score 1 bit. The D 1 -values (18) for the rst segment starting from the right (or the second segment starting from the bottom) are 2, 2.5, 3, 4, 6, 10, 20, 50, 100, 250, 1000, and 10000. The left thick solid line is an envelope curve for these two-segment curves. Circles stand for simulating the Poisson renewal process with two states (see text).

X-16 KAGAN : EARTHQUAKE PREDICTABILITY MEASUREMENT Figure 4. Error diagrams for renewal processes with the information score 1 bit. The straight solid line is the diagram curve corresponding to a random guess. The left solid line is an envelope curve for two-segment curves. Dashed curves with squares and with diamond signs are for the gamma distribution with = 0:329, and = 8:53, respectively. Solid curves with circles and with plus signs are for lognormal distribution with = 1:86, and = 0:35, respectively.

35 Fig. 1 30 Number of events 25 20 15 10 5 0 0 1 2 3 4 5 6 7 8 9 10 Time

1 Fig. 2 Fraction failures to predict, ν 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.2 0.4 0.6 0.8 1 Fraction alarm time, τ

1 Fig. 3 Fraction failures to predict, ν 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.2 0.4 0.6 0.8 1 Fraction alarm time, τ

1 Fig. 4 Fraction failures to predict, ν 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.2 0.4 0.6 0.8 1 Fraction alarm time, τ