Modeling daily precipitation in Space and Time

Transcription

1 Space and Time SWGen - Hydro Berlin 20 September 2017 temporal - dependence

2 Outline temporal - dependence temporal - dependence

3 Stochastic Weather Generator Stochastic Weather Generator (SWG) is a stochastic numerical that generates weather series - at the appropriate scales - with the same statistical properties as the observed ones. temporal - dependence

4 Stochastic Weather Generator Stochastic Weather Generator (SWG) is a stochastic numerical that generates weather series - at the appropriate scales - with the same statistical properties as the observed ones. As Prof Bardossy said: what properties of the weather generator should be reproduced? temporal - dependence

5 Stochastic Weather Generator Stochastic Weather Generator (SWG) is a stochastic numerical that generates weather series - at the appropriate scales - with the same statistical properties as the observed ones. As Prof Bardossy said: what properties of the weather generator should be reproduced? Weak sense similarity - certain functionals e.g. duration of wet/dry spells temporal - dependence

6 Stochastic Weather Generator Stochastic Weather Generator (SWG) is a stochastic numerical that generates weather series - at the appropriate scales - with the same statistical properties as the observed ones. As Prof Bardossy said: what properties of the weather generator should be reproduced? Weak sense similarity - certain functionals e.g. duration of wet/dry spells Extremes temporal - dependence

7 temporal are daily precipitation from 1/1/ /12/2004 provided by Swedish Meteorological and Hydrological Institute (SMHI). - dependence

8 Model At a fixed location the distribution of precipitation is of mixed type - a mass at zero and a continuous part. temporal - dependence

9 Model At a fixed location the distribution of precipitation is of mixed type - a mass at zero and a continuous part. Traditionally this has been dealt with, with: O t {0, 1}, occurence of dicator temporal - dependence

10 Model At a fixed location the distribution of precipitation is of mixed type - a mass at zero and a continuous part. Traditionally this has been dealt with, with: O t {0, 1}, occurence of dicator Z t R +, amount precipitation if there is precipitation temporal - dependence

11 Model At a fixed location the distribution of precipitation is of mixed type - a mass at zero and a continuous part. Traditionally this has been dealt with, with: O t {0, 1}, occurence of dicator Z t R +, amount precipitation if there is precipitation with O t Z t Y t = O t Z t temporal - dependence

12 Model, cont. For O t - usual choice is a 1-order Markov chain - fails to reproduce dry spells, Racsko et al. (1991), Guttorp (1995). temporal - dependence

13 Model, cont. For O t - usual choice is a 1-order Markov chain - fails to reproduce dry spells, Racsko et al. (1991), Guttorp (1995). We fit higher order M-C temporal - dependence

14 Model, cont. For O t - usual choice is a 1-order Markov chain - fails to reproduce dry spells, Racsko et al. (1991), Guttorp (1995). We fit higher order M-C For Z t, - usually Gamma and Weibull and mixtures of them. As a result the extreme values are usually underestimated. temporal - dependence

15 Model, cont. For O t - usual choice is a 1-order Markov chain - fails to reproduce dry spells, Racsko et al. (1991), Guttorp (1995). We fit higher order M-C For Z t, - usually Gamma and Weibull and mixtures of them. As a result the extreme values are usually underestimated. We fit a hybrid with the empirical below a fixed level and GP for the exceedances temporal - dependence

16 Order of Markov chain Markov chains of different orders have been fit to data on a monthly, bimonthly and seasonal basis. Bayesian Information Criterion (BIC), Akaike Information Criterion (AIC) and Generalized Maximum Flunctuation Criterion gave that 1 order M-C not the best. Order of M-C affects the distribution of the dry/wet spell length. When M-C is of order 1 or 2, the distribution of the first and the subsequent dry spells the same. This is not true for order greater than 2! Distribution of dry spells can be used to select the order of the M-C. temporal - dependence

17 Kolmogorov- Smirnov Order Estimator KS test D = sup m N + P k (D(X ) m) F emp(m) F emp(x) empirical distr. of first dry spell length P(D(X t) = m) = {u S k } π u {w S k :τ 2 (w)=(1,0)} P(τ k (X T ) = u τ k (X 0 ) = w) m P(X i = v k+i τ k (X i 1 ) = τ k (v k+i 1 )) i=1 temporal Season Model S1 S2 S3 S4 k = k = k = dependence Number of data sets that have passed the KS test at the 10% tail value for different orders of the Markov chain.

18 Long Dry Spells I A long dry spell has length k order of Markov chain probability Empirical 3 Markov 2 Markov Markov time (days) temporal - dependence Conditional distribution of Dry Spell given length 3 days for December-February.

19 Long Dry Spells II Model l = 1 l = 2 l = 3 k = k = k = Observed mean value Expected length of long dry spells for season Dec-Feb in Lund. temporal - dependence

20 Long Dry Spells II Model l = 1 l = 2 l = 3 k = k = k = Observed mean value Expected length of long dry spells for season Dec-Feb in Lund. O t if ed as a Markov chain it should be of order higher than 1. temporal - dependence

21 Amount Process We the amount precipitation process in two steps Model the dependence using 2-dim Gaussian copula Estimate the marginal distribution using a composite : Empirical distribution below a threshold Fit of distribution of excesses above the threshold temporal - dependence

22 Dependence Structure Amounts of precipitation during successive days are dependent. Dependence is computed using a 2-dim Gaussian copula,. f Yt,Y t+1 (x, y) = f 1(x)f 2(y)C(x, y) T(Y t+1 ) temporal - dependence T(Y t )

23 Marginal Distribution I The amount of precipitation process is led using a composite : F C(x; u) = F emp(x u) + (1 F emp(u))f u(x), where ( ( x u )) 1 ξ F u(x) = ξ GP σ The parameters are estimated using maximum likelihood. temporal - dependence

24 Marginal Distribution I The amount of precipitation process is led using a composite : F C(x; u) = F emp(x u) + (1 F emp(u))f u(x), where ( ( x u )) 1 ξ F u(x) = ξ GP σ The parameters are estimated using maximum likelihood. Choice of threshold u not obvious... High u leads to independent but possibly few extremes Declustering temporal - dependence

25 Marginal Distribution I The amount of precipitation process is led using a composite : F C(x; u) = F emp(x u) + (1 F emp(u))f u(x), where ( ( x u )) 1 ξ F u(x) = ξ GP σ The parameters are estimated using maximum likelihood. Choice of threshold u not obvious... High u leads to independent but possibly few extremes Declustering u a point of discontinuity temporal - dependence

26 Marginal Distribution I The amount of precipitation process is led using a composite : F C(x; u) = F emp(x u) + (1 F emp(u))f u(x), where ( ( x u )) 1 ξ F u(x) = ξ GP σ The parameters are estimated using maximum likelihood. Choice of threshold u not obvious... High u leads to independent but possibly few extremes Declustering u a point of discontinuity An AR(1) is imposed on the data temporal - dependence

27 Model Estimation Extreme Empirical Return Level GP Return Level Plot Probability Plot Return Period (Years) Extreme Empirical (mm) no of observations Quantile Plot GP (mm) Density Plot Amount of precipitation (mm) temporal - dependence Diagnostic plots for fixed threshold excess fitted to daily precipitation data.

28 Weather Indices temporal - dependence

29 Weather Indices Index Description Formula R10mm Heavy precipitation days 1{Zi >10} R20mm Very heavy precipitation days 1{Zi >20} RX1day Highest 1 day precipitation amount max i Z i RX5day Highest 5 day precipitation amount max 4j=0 i Z i+j CDD Max number of consecutive dry days max{j : τ j (X i ) = 0} CWD Max number of consecutive wet days max{j : τ j (X i ) = 1} R75p Moderate wet days 1{Zi >q 0.75 } R90p Above moderate wet days 1{Zi >q 0.90 } R95p Very wet days 1{Zi >q 0.95 } R95p Extremely wet days 1{Zi >q 0.99 } R75pTOT fraction due to R75p Zi 1 {Zi >q 0.75 } / Z i R90pTOT fraction due to R90p Zi 1 {Zi >q 0.90 } / Z i R95pTOT fraction due to R95p Zi 1 {Zi >q 0.95 } / Z i R99pTOT fraction due to R99p Zi 1 {Zi >q 0.99 } / Z i SDII Simple daily intensity index Zi / O i Prec90p 90%-quant. of thinned amount of precip. F 1 A (0.9) Table: Weather Indices and their mathematical expressions. The quantiles q ( ) have been estimated using the observed data. temporal - dependence

30 Maximum amount of rainfall over 1 and 5 days Index RX1day Model cumul. distr. 0 Empirical distr amount of prec. (mm) temporal - dependence

31 Maximum amount of rainfall over 1 and 5 days Index RX1day Model cumul. distr. 0 Empirical distr amount of prec. (mm) Index RX5day temporal - Model cumul. distr. Empirical distr amount of prec. (mm) dependence

32 Depend only on occurence process Index CDD Model cumul. distr. Empirical distr no. of days temporal - dependence

33 Depend only on occurence process 1 Index CDD 1 Index CWD Model cumul. distr. Empirical distr no. of days For example once every two years we expect a dry spell of length above 2 weeks and a wet spell of length approx. 12 days temporal Model cumul. distr. - Empirical distr Temporal 24 dependence no. of days dependence

34 Problems temporal - dependence

35 Problems The independence assumption seems unrealistic. It also leads to intermittency problems. temporal - dependence

36 Problems The independence assumption seems unrealistic. It also leads to intermittency problems. Extensions temporal - dependence

37 Problems The independence assumption seems unrealistic. It also leads to intermittency problems. Extensions Spatial dependence temporal - dependence

38 Problems The independence assumption seems unrealistic. It also leads to intermittency problems. Extensions Spatial dependence Smooth transitions of rainfall/drought temporal - dependence

39 Problems The independence assumption seems unrealistic. It also leads to intermittency problems. Extensions Spatial dependence Smooth transitions of rainfall/drought time temporal - dependence

40 Problems The independence assumption seems unrealistic. It also leads to intermittency problems. Extensions Spatial dependence Smooth transitions of rainfall/drought time space temporal - dependence

41 Latent Gaussian fields Z(p, t) temporal - dependence

42 Latent Gaussian fields Z(p, t) O(s, t) = 1 Z(s,t)>u Y (s, t) = T (Z(s, t) u)), temporal - dependence

43 Latent Gaussian fields Z(p, t) O(s, t) = 1 Z(s,t)>u Y (s, t) = T (Z(s, t) u)), temporal Observed process Transformed observed process Semi latent process zero level - dependence

44 Latent Gaussian fields Z(p, t) O(s, t) = 1 Z(s,t)>u Y (s, t) = T (Z(s, t) u)), temporal Observed process Transformed observed process Semi latent process zero level - dependence

45 The Meta-Gaussian Latent and transformed Gaussian { ψ s,t(z(s, t)), if W (s, t) > u (0), Y (s, t) = 0 if W (s, t) u (0), Z(s, t) - a Gaussian r.f at location s and time t ψ s,t - anamorphosis function W (s, t) - a r.f - when values are above level u something happens We suggest W = Z # of parameters is low Avoid edge effect - unrealistically large intensities generated near the bdry of dry areas Advantage: Compatible with any type - even discontinuous - of marginal distribution. Retains the marginal distribution at individual locations and allows for different types of spatial correlation between locations temporal - dependence

46 Model identification Anamorphosis function - hybrid with gamma below a threshold u and GP above temporal - dependence

47 Model identification Anamorphosis function - hybrid with gamma below a threshold u and GP above Mean temporal - dependence

48 Model identification Anamorphosis function - hybrid with gamma below a threshold u and GP above Mean Covariance - in time and space The level u now gives contour lines that can be thought of as the rain bdry - same difficulty as before when choosing u. temporal - dependence

49 Mean function Link between observed process Y (s, t) and Gaussian Z(s, t) 1 Y >0 = 1 Z>0 P(positive event) = E[1 Z>0 ] temporal - dependence

50 Mean function Link between observed process Y (s, t) and Gaussian Z(s, t) 1 Y >0 = 1 Z>0 P(positive event) = E[1 Z>0 ] = Φ(µ) Mean is estimated by numerically inverting above equation. temporal - dependence

51 Covariance A difficulty arises from the fact that the field is latent and transformed. Implicit calculations, and Lennartsson (2015), temporal - dependence

52 Covariance A difficulty arises from the fact that the field is latent and transformed. Implicit calculations, and Lennartsson (2015), Modified Maximum likelihood, Durban and Glasbey (2003) temporal - dependence

53 Covariance A difficulty arises from the fact that the field is latent and transformed. Implicit calculations, and Lennartsson (2015), Modified Maximum likelihood, Durban and Glasbey (2003) Hermite polynomial expansion, Guilliot (1999) temporal - dependence

54 Covariance A difficulty arises from the fact that the field is latent and transformed. Implicit calculations, and Lennartsson (2015), Modified Maximum likelihood, Durban and Glasbey (2003) Hermite polynomial expansion, Guilliot (1999) Excursion sets, Lantuéjoul (2002) temporal - dependence

55 Covariance A difficulty arises from the fact that the field is latent and transformed. Implicit calculations, and Lennartsson (2015), Modified Maximum likelihood, Durban and Glasbey (2003) Hermite polynomial expansion, Guilliot (1999) Excursion sets, Lantuéjoul (2002) temporal - dependence

56 Covariance C(h, τ) = η1 { h =0,τ=0} + 1 η b A θ,ɛ h 2 a τ + 1 e a τ +1 isotropic and stationary covariance - distance is anisotropic (linearly transformed coordinate system) A θ,ɛ h 2 = ɛ(x cos θ + y sin θ) ɛ ( x sin θ + y cos θ)2, Minimization using min η,a,b,η,θ n obs (i, j, τ) (ˆρ ij (τ) Cov M (Z(s i, t), Z(s j, t + τ))) 2 t i j temporal - dependence

57 temporal - dependence

58 How good is the stochastic generator? temporal spatial spatio-temporal temporal - dependence

59 How good is the stochastic generator? temporal spatial spatio-temporal Simulate 100 times 51 years of precipitation data temporal - dependence

60 Depends on u, GP parameters and temporal latent process conditional proportions of wet day given previous day dry conditional proportions of wet day given previous day wet temporal - dependence

61 Depends on u, GP parameters and temporal latent process conditional proportions of wet day given previous day dry conditional proportions of wet day given previous day wet P(wet dry) days temporal - dependence

62 Depends on u, GP parameters and temporal latent process conditional proportions of wet day given previous day dry conditional proportions of wet day given previous day wet P(wet wet) days temporal - dependence

63 II dry/wet spells - on latent field only - the time the field is below or above a fixed level weather indices temporal - dependence

64 II probablity of number of days with wet spell dry/wet spells - on latent field only - the time the field is below or above a fixed level weather indices Duration (days) temporal - dependence

65 II dry/wet spells - on latent field only - the time the field is below or above a fixed level weather indices RX5dayM1 Observed 90% konfidence interval temporal - dependence

66 Spatial dependence temporal - dependence

67 Spatial dependence proportions simultaneous occurrences of dry, wet days correlation of intensity # of simultaneously wet stations temporal - dependence

68 Spatial dependence proportions simultaneous occurrences of dry, wet days correlation of intensity # of simultaneously wet stations Proportions joint occurence MOnth1 Estimated proportion of observed data Estimated proportion of simulated data temporal - dependence

69 Spatial dependence proportions simultaneous occurrences of dry, wet days correlation of intensity # of simultaneously wet stations Correlation of daily intensity data, month:7 Estim correlations of observed data Estim correlations of simulated data temporal - dependence

70 Spatial dependence proportions simultaneous occurrences of dry, wet days correlation of intensity # of simultaneously wet stations Observed, thinned Observed, 0 padded Observed, 1 padded Model temporal - dependence relative frequency no. stations

71 temporal - dependence

72 pairwise lagged occurrences weather indices of spatially aggregated data temporal - dependence

73 Estimated proportion of observed data pairwise lagged occurrences weather indices of spatially aggregated data Proportions of one station dry and the other wet the following day month: Estimated proportion of simulated data temporal - dependence

74 pairwise lagged occurrences weather indices of spatially aggregated data PRCPTOT temporal Observed, dry padded Observed, wet padded 90% konfidence interval - dependence

75 References Allcroft, D. J., and C. A. Glasbey (2003), A latent Gaussian Markov random-field for spatiotemporal rainfall disaggregation, Biomathematics and statistics, 52, Durban, M., and C. A. Glasbey (2001), Weather ling using a multivariate latent Gaussian, Agricultural and Forest Meteorology, 109, G. Guillot (1999), Approximation of Sahelian rainfall fields with meta-gaussian random functions Part 1: definition and methodology, Stochastic Environmental Research and Risk assesment 13 (1999), C. Lantuéjoul (2002), Geostatistical simulations, Berlin: Springer, temporal - dependence

76 References Lennartsson, J., A., and D. Chen (2008), Modelling Sweden using multiple step Markov chains and a composite, Journal of Hydrology, 363(1 4), A., and Lennartsson, J., (2015), A precipitation generator based n a censorer latent Gaussian field, Water Resources Research A. and A. Lenzi, Probabilistic Prediction of wind power using latent fields,stochastic Environmental Reasearch and Risk Assessment, Doi: /s (2017) temporal - dependence