Earthquake predictability measurement: information score and error diagram Yan Y. Kagan Department of Earth and Space Sciences University of California, Los Angeles, California, USA August, 00 Abstract 0 I discuss two methods for measuring the effectiveness of earthquake prediction algorithms: the information score based on the likelihood ratio and error diagrams. For both of these methods, closed form expressions are obtained for the renewal process based on the gamma and lognormal distributions. The error diagram is more informative than the likelihood ratio and uniquely specifies the information score. I derive an expression connecting the information score and error diagrams. I then obtain the estimate of the region bounds in the error diagram for any value of the information score. Yan Y. Kagan, Department of Earth and Space Sciences, University of California, Los Angeles, California, 00-, USA; (e-mail: ykagan@ucla.edu)
Introduction 0 0 In a recent article, Jordan (00) argued that more objective, rigorous, quantitative methods for testing earthquake prediction schemes need to be developed. Particularly, he asked What is the intrinsic predictability of the earthquake rupture process? To contribute to this inquiry I discuss two methods currently used to measure the performance of earthquake prediction programs. The first method is the likelihood ratio procedure which has long been used for statistical analysis of random processes. In particular, Kagan and Knopoff (; ), Kagan (), Ogata (), Kagan and Jackson (000), Imoto (00), Rhoades and Evison (00), and Helmstetter et al. (00) have applied this likelihood method for earthquake occurrence studies. Kagan and Knopoff () first proposed calculating the information score for earthquake predictability based on the likelihood ratio. The second method is related to the Relative Operating Characteristic (ROC) used in weather prediction efforts (Jolliffe and Stephenson, 00), where the success rate of an event prediction is compared against the false alarm rate (ibid., p. ; see also Holliday et al., 00). Since periodic (diurnal, annual) effects are strong in weather prediction, such a method has broad applications; we can compare the above characteristics of a forecast system for one-day or one-year alarm periods. But in earthquake prediction, there
0 is no natural time scale for forecasting, so the time interval is arbitrary. Therefore, if the alarm duration is increased, both criteria approach the trivial result: all events are predicted with no false alarms. Molchan (0) modified this method as an error diagram to predict random point processes. Molchan and Kagan () and Molchan (; 00) also review the error diagram method and its applications. McGuire et al. (00), Kossobokov (00), and Baiesi (00) recently used this method to evaluate earthquake prediction algorithms. Kagan and Jackson (00) commented on Kossobokov s analysis and discussed the application of error diagrams to predicting earthquakes. Information score Kagan and Knopoff () suggested measuring the effectiveness of earth- quake prediction algorithm by first evaluating the likelihood ratio to test how well does a model approximate earthquake occurrence. In particular, they estimated the information score, Î, per one earthquake by Î = l l 0 N = N N i= log p i π i, () where l l 0 is the log-likelihood ratio, N is the number of earthquakes in a catalog, p i is the probability of earthquake occurrence according to a stochastic model, π i is a similar probability for a Poisson process, and log was used to obtain the score in bits of information. One information bit
would mean that uncertainty of earthquake occurrence is reduced on average by a factor of by the using a particular model. Here the average needs to be understood as a geometrical mean. For long catalogs (N ) I = ( ) N E p i log π i Î, () where E is the mathematical expectation (Vere-Jones, ; Daley and Vere- Jones, 00). For a renewal (i.e., with independent intervals) process th
The gamma distribution has the pdf f(x) = λκ x κ Γ(κ) exp( κ x), () where Γ is the gamma function, κ is a shape parameter, λ is a scale parameter, and 0 < λ < ; 0 < κ < ; 0 x < (Evans et al., 000). If κ =, then the process is the Poisson one, κ < characterizes the occurrence of clustered events. For the gamma renewal process, normalized to have the mean equal to, i.e., λ = κ, the information score is (Daley and Vere-Jones, 00, their equation ) I(κ) = [ log κ + (κ ) ψ(κ) κ log Γ(κ) ]/log(), () 0 where ψ is the digamma function (Abramowitz and Stegun, ). If κ =, I(κ) = 0. The lognormal distribution has the pdf f(x) = { σx π exp } [ log(x) µ ], () σ where σ is a shape parameter, µ is a scale parameter and 0 < µ < ; 0 < σ < ; 0 x < (Evans et al., 000). For a renewal process normalized to have the mean equal to, the information score for the lognormal distribution is (Bebbington, 00, p. 0) I(σ) = [ + σ log() log ( σ π ) ]. ()
The small σ-values correspond to a quasi-periodic process, the large values to a clustered one. The sequence with the parameter value σ = is the closest to the Poisson process, its information score is at minimum, but is still non-zero, I = 0. bits. Error diagrams 0 The error diagram for evaluating how well a prediction program performs was first suggested by Molchan (0). For any prediction algorithm, the diagram plots the fraction of failures to predict, ν, versus the fraction of alarm time, τ. The curve is concave (Molchan, ; 00). The error diagram curve for a clustered renewal process can be calculated as ν(w) = w 0 f(x) dx, () where w is the alarm duration and τ(w) = w [ ν(w) ] + w 0 xf(x) dx. (0) The first right-hand term in (0) is the average alarm duration, if no event occurs in the w interval. The second term is the average alarm length. For the lognormal and gamma renewal processes the variables ν(w) and τ(w) can be found in a closed form. For the gamma distribution ν(w) = γ(κ, κw)/γ(κ), ()
and τ(w) = w [ ν(w) ] + γ( + κ, κw)/γ( + κ), () 0 where γ is the incomplete gamma function (Abramowitz and Stegun, ). For the lognormal distribution ν(w) = { [ log(w) σ ]} / + erf σ, () and τ(w) = w [ ν(w) ] + { [ σ ]} log(w) + erfc σ. () Here erf and erfc are the error function and its complementary function, respectively. Fig. shows an example of the error diagram for the lognormal renewal process. The theoretical curves (, ) are compared to the simulation results for choices of the alarm duration, w. The alarm duration starts with 0.00 and then increases logarithmically with the factor value. until it reaches the total length of a series (000 units). I deliberately use relatively short simulated sequences to show random fluctuations. Interrelation between the error diagram and information score The information score can be calculated for an error diagram curve as I = 0 ( ) ν log dν. () τ
It is helpful to have an estimate of the region boundaries for curves corresponding to a specific value of the information score. Such an estimate can be obtained if the prediction scheme in the error diagram consists of two linear segments with the slopes D = ν τ = ν τ, () for the first segment and D for the second segment; D, and D, () given a curve concavity. D = D = D 0 = is the random guess strategy (Molchan, 0). For the assumed information score I the envelope curve is defined by the equation [ ν ] ν = I. () D ν ν/d 0 By solving this equation for any value of D, one obtains the ν-value for the contact point of two linear segments, τ = ν/d. As an example, Fig. displays several two line segments that correspond to the prediction schemes with an information score that equals bit. The envelope curve is also shown. This curve most likely delineates the lower boundary of all possible error curves with the information score I = bit. The upper boundary is, in principle, the random guess line, D 0 : if a large D is specified, the resulting curve can be as close to this line as needed.
0 I also show in this diagram the results of simulating a mixture of two Poisson processes with the rates differing by a factor.. This factor was adjusted to obtain the information score bit for a renewal process in which intervals have been selected randomly from each sequence. The simulation results are similar to theoretical curves having two straight line segments. Fig. displays the curves for the renewal processes with gamma and lognormal distributions. The information score again is taken to be bit. Curves for both sequences, clustered and quasi-periodic, are shown. All four curves are within the region specified by () and the random guess line. When simulating or computing curves for quasi-periodic sequences, alarm declaration is reversed, i.e., it is declared after the elapsed w time period following an event. Alternatively, an alarm strategy is the same as in clustered sequences producing an antipodal prediction (Molchan and Kagan, ; Molchan, ). Then a curve is rotated 0 around the center of symmetry [ ν = /; τ = / ]. Discussion 0 Clearly, from the theoretical and the simulation results described above, the error diagram represents a much more complete picture of the stochastic point process than does the likelihood analysis. Using the diagram curve one can calculate the information score for a sequence. The score also imposes some limits on the diagram region where curves are located, but Figs.
0 0 and show that these limitations are rather broad. By specifying a more restricted class of point processes to approximate an earthquake occurrence, the interrelation between these two methods can likely be made more precise. A few comments on how the discussed techniques might forecast real earthquake rupture are due here. First, more appropriate stochastic model for earthquake occurrence is not a renewal but a branching process (Hawkes and Oakes, ; Kagan and Knopoff, ; Ogata, ) which captures the important feature of seismicity, its clustering. Moreover, earthquakes occur not only in time. Their spatial coordinates, earthquake size, and focal mechanisms need to be taken into account in actual prediction efforts. Introducing new variables complicates the calculation of the information score and the error diagram. Molchan and Kagan () have done some preliminary work in determining error diagrams for multidimensional processes. Kagan and Jackson (000) and Helmstetter et al. (00) have shown how to evaluate the effectiveness of spatial smoothing for seismic hazard maps. Another challenge in dealing with earthquake prediction is the fractal nature of most distributions controlling earthquakes (Kagan, 00). Since these distributions approach infinity for small time and distance intervals, the value of the information score is not well defined (see Helmstetter et al., 00). Similarly, the error diagram curve would start to approach the point of the ideal prediction (ν = 0, τ = 0) for earthquake catalogs of high location 0
accuracy and extending to small time intervals after a strong earthquake. Finally, I would like to mention that equation was derived, using heuristic arguments, in December January. Since that time I privately sent these preliminary results to many researchers interested in the problem. Recently, Harte and Vere-Jones (00, their equation ) published a similar formula for a model of the discrete-time point process. They did not explore the error diagram properties and any constraints the information score would impose on the diagram. Acknowledgments 0 I appreciate partial support from the National Science Foundation through grants EAR 0-00, and DMS-00, as well as from the Southern California Earthquake Center (SCEC). SCEC is funded by NSF Cooperative Agreement EAR-00 and USGS Cooperative Agreement 0HQAG000. I am very grateful to P. Stark of UC Berkeley who sent me matlab programs employed in his (Stark, ) paper. With appropriate modifications these programs were used in some calculation reported above. The author thanks D. D. Jackson, I. V. Zaliapin, F. Schoenberg, and J. C. Zhuang of UCLA, D. Vere-Jones of Wellington University and P. Stark for very useful discussions. Publication 0000, SCEC.
References [] Abramowitz, M. and I. A. Stegun (), Handbook of Mathematical Functions, Dover, NY, pp 0. [] Baiesi, M. (00), Scaling and precursor motifs in earthquake networks, Physica A, 0(), -. [] Bebbington, M. S. (00), Information gains for stress release models, Pure Appl. Geophys., (), -. [] Daley, D. J., and Vere-Jones, D. (00), An Introduction to the Theory of Point Processes, Springer-Verlag, New York, -nd ed., Vol., pp.. 0 [] Daley, D. J., and Vere-Jones, D. (00), Scoring probability forecasts for point processes: The entropy score and information gain, J. Applied Probability, A, -. [] Evans, M., N. Hastings, and B. Peacock (000), Statistical Distribu- tions, rd ed., New York, J. Wiley, pp. [] Harte, D., and Vere-Jones, D. (00), The entropy score and its uses in earthquake forecasting, Pure Appl. Geophys., (-), -. [] Hawkes, A. G. and Oakes, D. (), A cluster process representation of a self-exciting process, J. Appl. Prob.,, -0.
[] Helmstetter, A., Y. Y. Kagan, and D. D. Jackson (00), Compar- ison of short-term and time-independent earthquake forecast models for southern California, Bull. Seismol. Soc. Amer., (), 0-0. [0] Holliday, J. R., K. Z. Nanjo, K. F. Tiampo, J. B. Rundle, D. L. Turcotte (00), Earthquake forecasting and its verification, Nonlinear Processes Geophys., (), -. [] Imoto, M. (00), Probability gains expected for renewal process mod- els, Earth Planets Space,, -. 0 [] Jolliffe, I. T. and D. B. Stephenson, Eds. (00), Forecast Verification: a Practitioner s Guide in Atmospheric Science, J. Wiley, Chichester, England, 0 pp. [] Jordan, T. H. (00), Earthquake predictability, brick by brick, Seis- mol. Res. Lett., (), -. [] Kagan, Y. Y. (), Likelihood analysis of earthquake catalogues, Geophys. J. Int., 0, -. [] Kagan, Y. Y. (00), Why does theoretical physics fail to explain and predict earthquake occurrence?, in: Lecture Notes in Physics, P. Bhattacharyya and B. K. Chakrabarti, (Eds) Springer Verlag, accepted, see http://scec.ess.ucla.edu/ ykagan/india index.html.
[] Kagan, Y. Y., and D. D. Jackson (000), Probabilistic forecasting of earthquakes, Geophys. J. Int.,, -. [] Kagan, Y. Y., and D. D. Jackson (00), Comment on Testing earthquake prediction methods: The West Pacific short-term forecast of earthquakes with magnitude MwHRV. by V. G. Kossobokov, Tectonophysics, (-), -. [] Kagan, Y., and L. Knopoff (), Statistical search for non-random features of the seismicity of strong earthquakes, Phys. Earth Planet. Inter., (), -. 0 [] Kagan, Y., and L. Knopoff (), Earthquake risk prediction as a stochastic process, Phys. Earth Planet. Inter., (), -0. [0] Kagan, Y. Y., and L. Knopoff (), Statistical short-term earthquake prediction, Science,, -. [] Kossobokov, V. G. (00), Testing earthquake prediction methods: The West Pacific short-term forecast of earthquakes with magnitude MwHRV., Tectonophysics, (-), -. 0 [] McGuire, J. J., Boettcher, M. S., and Jordan, T. H. (00), Foreshock sequences and short-term earthquake predictability on East Pacific Rise transform faults, Nature, (0), -; Correction Nature, (0),.
[] Molchan, G. M. (0), Strategies in strong earthquake prediction, Phys. Earth Planet. Inter., (-), -. [] Molchan, G. M. (), Earthquake prediction as a decision-making problem, Pure Appl. Geoph.,, -. [] Molchan, G. M. (00), Earthquake prediction strategies: A theoretical analysis, In: Keilis-Borok, V. I., and A. A. Soloviev, (Eds) Nonlinear Dynamics of the Lithosphere and Earthquake Prediction, Springer, Heidelberg, 0-. 0 [] Molchan, G. M., and Y. Y. Kagan (), Earthquake prediction and its optimization, J. Geophys. Res.,, -. [] Ogata, Y. (), Seismicity analysis through point-process modeling: A review, Pure Appl. Geophys.,, -0. [] Rhoades, D. A., and F. F. Evison (00), The EEPAS forecasting model and the probability of moderate-to-large earthquakes in central Japan, Tectonophysics (-), -0. [] Stark, P. B. (), Earthquake prediction: the null hypothesis, Geo- phys. J. Int.,, -. 0 [0] Vere-Jones, D. (), Probabilities and information gain for earth- quake forecasting, Computational Seismology, 0, Geos, Moscow, -.
Captions Figure : Error diagrams for the lognormal renewal process with σ =.. The straight solid line is the strategy curve corresponding to a random guess. The left solid curve is calculated using ( ), circles are the result of simulations. The dashed and dotted curves are the first and second right-hand terms in (), respectively. 0 Figure : Error diagrams for renewal processes. The thick straight solid line corresponds to a random guess. Thin solid lines are for the curves with the information score bit. The D -values () for the first segment starting from the right (or the second segment starting from the bottom) are,.,,,, 0, 0, 0, 00, 0, 000, and 0000. The left thick solid line is an envelope curve for these two-segment curves. Circles stand for simulating the Poisson renewal process with two states (see text). 0 Figure : Error diagrams for renewal processes with the information score bit. The straight solid line is the diagram curve corresponding to a random guess. The left solid line is an envelope curve for two-segment curves. Dashed curves with squares and with diamond signs are for the gamma distribution with κ = 0., and κ =., respectively. Solid curves with circles and with plus signs are for lognormal distribution with σ =., and σ = 0., respectively.
Fig. 0. Fraction failures to predict, ν 0. 0. 0. 0. 0. 0. 0. 0. 0 0 0. 0. 0. 0. Fraction alarm time, τ Figure : Error diagrams for the lognormal renewal process with σ =.. The straight solid line is the strategy curve corresponding to a random guess. The left solid curve is calculated using ( ), circles are the result of simulations. The dashed and dotted curves are the first and second right-hand terms in (), respectively.
Fig. 0. Fraction failures to predict, ν 0. 0. 0. 0. 0. 0. 0. 0. 0 0 0. 0. 0. 0. Fraction alarm time, τ Figure : Error diagrams for renewal processes. The thick straight solid line corresponds to a random guess. Thin solid lines are for the curves with the information score bit. The D -values () for the first segment starting from the right (or the second segment starting from the bottom) are,.,,,, 0, 0, 0, 00, 0, 000, and 0000. The left thick solid line is an envelope curve for these two-segment curves. Circles stand for simulating the Poisson renewal process with two states (see text).
Fig. 0. Fraction failures to predict, ν 0. 0. 0. 0. 0. 0. 0. 0. 0 0 0. 0. 0. 0. Fraction alarm time, τ Figure : Error diagrams for renewal processes with the information score bit. The straight solid line is the diagram curve corresponding to a random guess. The left solid line is an envelope curve for two-segment curves. Dashed curves with squares and with diamond signs are for the gamma distribution with κ = 0., and κ =., respectively. Solid curves with circles and with plus signs are for lognormal distribution with σ =., and σ = 0., respectively.