IRREPRODUCIBLE SCIENCE

Transcription

1 SUMMER SCHOOL 2017 IRREPRODUCIBLE SCIENCE P-VALUES, STATISTICAL INFERENCE AND NON-ERGODIC SYSTEMS ALESSI A CAPONERA SUPERVISOR: MAXIMILIAN WERNER

2 What does research reproducibility mean? Reproducibility or replicability?

3 What does research reproducibility mean? Reproducibility or replicability? 1. Methods reproducibility 2. Results reproducibility 3. Inferential reproducibility Goodman et al., 2017

4 What does research reproducibility mean? Reproducibility or replicability? 1. Methods reproducibility 2. Results reproducibility 3. Inferential reproducibility Goodman et al., 2017 Reproducibility & Ergodicity A recipe for irreproducible result Peters and Werner, 2017

5 CSEP, what is it? How should scientific prediction experiments be conducted and evaluated? What is the intrinsic predictability of the earthquake rupture process? The Collaboratory for the Study of Earthquake Predictability (CSEP) seeks to provide answers to these questions developing international collaborations between the regional testing centers and accommodating a wide-ranging set of prediction experiments.

6 Four weeks... Many questions! We focused on: Conditional S-test Multinomial-based algorithm for catalog simulation Information gain: from I N A, B to I t A, B

7

8 Notation (Zechar et al., 2010a): Likelihood-based tests R: testing region M: binned magnitude range of interest S: binned spatial domain of interest R = M x S

9 Notation (Zechar et al., 2010a): Likelihood-based tests R: testing region M: binned magnitude range of interest S: binned spatial domain of interest R = M x S Earthquake forecast: Λ = { λ i, j i M, j S }

10 Likelihood-based tests Observed catalog: Ω = { ω i, j i M, j S} Simulated catalogs: Ω x = { ω x i, j i M, j S}, x = 1,, N rep Observed log - likelihood: L Ω; Λ Simulated log - likelihoods: L Ω x ; Λ, x = 1,, N rep

11 Forecasts comparison Λ A = { λ A i, j i M, j S } Λ B = { λ B i, j i M, j S }

12 Forecasts comparison Λ A = { λ A i, j i M, j S } Λ B = { λ B i, j i M, j S } Information gain per earthquake: I N A, B = L Ω; Λ A L Ω; Λ B N

13 Information gain per earthquake: from I N A, B to I t A, B I t A, B = L Ωt ; Λ A t L Ω t ; Λ B t N(t) t N t = σ i=1 N i = #{ observed earthquakes up to time t } L Ω t ; Λ i t is the log likelihood up to time t given forecast i, i = A, B

14 The data Earthquake catalog for the 2010 Canterbury earthquake series. The catalog covers the same 20 monthly time windows as the available forecasts.

15 Given N 1,, N t, we simulate catalogs consistent with forecast A and compare ε t x = 2t N t 1,.95 s t x N t, ε t obs = 2t N t 1,.95 s t obs (Rhoades et al., 2011) with the information gain empirical quantile spread obtained from simulations up to time t. N t

16

17 ETAS experiment The epidemic type aftershock sequence (ETAS) model describes an inhomogeneous point process with conditional intensity function λ t H t ) = μ + K i:t i <t e α m i m 0 c p 1 (p 1) t t i + c p, where H t = t i, m i : t i < t.

18 ETAS experiment The epidemic type aftershock sequence (ETAS) model describes an inhomogeneous point process with conditional intensity function λ t H t ) = μ + K i:t i <t e α m i m 0 c p 1 (p 1) t t i + c p, where H t = t i, m i : t i < t. Gutenberg-Richter: p m i Exp b ln 10 m 0 = 0

19 ETAS vs. homogeneous Poisson process mu <- 1 b <- 1 beta <- b*log(10) # 2.3 alpha <- 2.2 n <-.8 # avg number triggered eqks K <- f(n, alpha, beta) c <- 0.1 p <- 1.2 # p > 1

20 ETAS vs. homogeneous Poisson process

21

22 References 1. Goodman, S. N., Fanelli, D., and Ioannidis, J. P. (2016). What does research reproducibility mean?. Science translational medicine, 8(341), Peters, O. and Werner, M. (2017). A recipe for irreproducible results. arxiv. 3. Rhoades, D., Schorlemmer, D., Gerstenberger, M., Christophersen, A., Zechar, J., and Imoto, M. (2011). Efficient testing of earthquake forecasting models. Acta Geophysica, 59(4), Zechar, J. D., Gerstenberger, M. C., and Rhoades, D. A. (2010a). Likelihood-based tests for evaluating space rate magnitude earthquake forecasts. Bulletin of the Seismological Society of America, 100(3), Zechar, J. D., Schorlemmer, D., Liukis, M., Yu, J., Euchner, F., Maechling, P. J., and Jordan, T. H. (2010b). The Collaboratory for the Study of Earthquake Predictability perspective on computational earthquake science. Concurrency and Computation: Practice and Experience, 22(12),