Hidden Markov Models for precipitation

Transcription

1 Hidden Markov Models for precipitation Pierre Ailliot Université de Brest Joint work with Peter Thomson Statistics Research Associates (NZ) Page 1

2 Context Part of the project Climate-related risks for energy supply and demand Important part is to assess future climate variability Based on past observations and expected behavior of ENSO, IPO, human-induced climate change,... Aim : developing a stochastic models to simulate realistic scenarios for the future climate. First step: Focus on rainfall Reproduce past conditions Simulated rainfall used as input in models of large-scale hydrology and river flows Estimate risks related to production of hydroelectricity (6% of NZ electricity) Page

3 Outline Data Basic Hidden Markov Model (HMM) HMM with truncated Gaussian distributions Conclusion Page 3

4 Data Data Rainfall data in New Zealand Daily rainfall K=7 locations Y t = ( Y t ( 1),, Y t ( K) )' Data Y t ( k) R + : rainfall (mm) during day t at location k 6 years Focus on April 3 1 Hydroelectric stations Page 4

5 Data Time structure (location 1) Time (year) Time (day) Marginal distribution Two components:. Y t ( k) = if no rainfall occurs. Y t ( k) > if a rainfall occurs % of dry days 4 6 Marginal distribution, location 1 Page 5

6 Data Spatial structure corr(y t ) Page 6

7 HMMs for rainfall: basic model Basic HMM: model description (Zucchini & Guttorp (1991)) Existence of weather types in meteorological time series Another meteorological time series Cyclonic 1 Anticyclonic Wind speed (in m/s), Brittany, January 1991 The weather type depends on the position of high pressure systems, frontal systems... Introduced as a hidden process { 1 Q} S t Page 7

8 HMMs for rainfall: basic model Basic HMM: model description (Zucchini & Guttorp (1991)) Existence of weather types in meteorological time series Introduced as a hidden process Time structure: HMM S t { 1 Q} PS ( t S t 1 = s t 1, Y t 1 = y t 1,, S = s, Y = y ) = PS ( t S t 1 = s t 1 ) PY ( t S t = s t S t 1 = s t 1, Y =,, S= s, Y= y ) = PY ( t S t = s t ), t 1 y t 1 Weather type (not observed) S t-1 S t S t+1 Precipitation (observed) Y t-1 Y t Y t+1...dynamics induced only by { }! S t Page 8

9 HMMs for rainfall: basic model Basic HMM: model description (Zucchini & Guttorp (1991)) Existence of weather types in meteorological time series Introduced as a hidden process Time structure: HMM...Dynamics induced only by { }! Spatial structure: conditional independence S t S t { 1 Q} py ( t ( 1), Y t ( K) S t = s t ) = p( Y t( k) S t = s t ) k { 1 K} Weather type (not observed) S t-1 S t S t+1 Y t-1 (1) Y t (1) Y t+1 (1) Precipitation (observed) Y t-1 (K) Y t (K) Y t+1 (K) Spatial dependence induced only by { }! S t Page 9

10 HMMs for rainfall: basic model Basic HMM: model description (Zucchini & Guttorp (1991)) Existence of weather types in meteorological time series Introduced as a hidden process Time structure: HMM...Dynamics induced only by { }! Spatial structure: conditional independence Spatial dependence induced only by { }! S t S t { 1 Q} py ( t ( 1), Y t ( K) S t = s t ) = p( Y t( k) S t = s t ) S t k { 1 K} py ( t ( k) dy S t = s) = ( ) s 1 p k p k s if dy ( ) γ y α ( s ) ( s) ( ;, β k k ) if dy 1 γ( y; α, β) = y α 1 Γα ( )β exp y α β -- ( s) ( s) ( s) p k [, 1], α k >, β k > Page 1

11 HMMs for rainfall: basic model Parameter estimation 3KQ + Q 1 parameters EM algorithm to compute the Maximum Likelihood Estimates Several random starting values to avoid local extrema Model selection First selection with AIC and BIC Q AIC BIC Focus on the model with Q = 4 Easier interpretation than models with Q 5 Produce more realistic synthetic time series than models with Q 3 Page 11

12 HMMs for rainfall: basic model Model validation Meteorological interpretability Emission probabilities Regime 1 Regime Regime 3 Regime 4 P(Y t > S t ) 9% 5% 3% 4% 7% % 6% 64% 7% 5% 38% 49% 45% 51% 73% 88% 88% 65% 97% 95% 89% 94% 97% 98% 99% 99% 96% 98% E[Y t S t,y t >] var(y t S t,y t >) Page 1

13 HMMs for rainfall: basic model Model validation Meteorological interpretability Emission probabilities Transition matrix, stationary distribution, mean durations Summary Regime 1: dry conditions, long persistence Regime and 3: intermediate patterns, regional differences, higher rainfall in regime 3, short persistence Regime 4: heavy rainfall Similar meteorological interpretation for other datasets Page 13

14 HMMs for rainfall: basic model Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution Data Model (+95% interval) Location 1 (Winchmore) Page 14

15 HMMs for rainfall: basic model Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution: OK Dynamics at the different locations Data Model (+95% interval) Autocorrelation (Location 1) Autocorrelation (Location 3) Page 15

16 HMMs for rainfall: basic model Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution: OK Dynamics at the different locations Data Model (+95% interval) Dry durations (Location 1) Dry durations (Location 3) Page 16

17 HMMs for rainfall: basic model Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution: OK Dynamics at the different locations: ~OK Spatial structure Data Model Page 17

18 HMMs for rainfall: basic model Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution: OK Dynamics at the different locations: ~OK Spatial structure: not good! Need for a better model! Existence of residual spatial structure within the weather types regime 1 regime Empirical correlation matrices in the different weather types (identified via the Viterbi algo.) regime regime Page 18

19 HMMs for rainfall: basic model Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution: OK Dynamics at the different locations: ~OK Spatial structure: not good! Need for a better model! Existence of residual spatial structure within the weather types Introduce spatial structure in the emission probabilities Need model for multivariate mixed discrete-continuous distributions...truncated Gaussian random fields Page 19

20 HMMs for rainfall: HMM with truncated Gaussian fields HMM with truncated Gaussian fields: model description If = s then S t Y t ( k) = if W t ( k) W t ( k) β ( s) ( k) if W t ( k) > = + with Z t NOI (, ) i.i.d W t m ( s) H ( s) Z t m ( s) R K, Σ ( s) = H ( s) ( H ( s) )' R K K and β s.4 ( ) R +* ( ) K.3..1 P[Y=] pdf of W N( 1.88, 3.78) pdf of Y = max( W, ) 1.81 Page

21 HMMs for rainfall: HMM with truncated Gaussian fields Assumptions on the covariance matrices HMMCI model ( Q 1 + 3QK parameters) ( s (, ) σ ) s s = i σ j with σ j Σ ( s) ij ( ) δ i, j ( ) > HMMdist model ( Q 1 + K3Q ( + 1) parameters) ( s (, ) σ ) ( s i σ ) j exp λ ( s) ( s) = ( dz ( i, z j )) with σ j > and λ s Σ ( s) ij ( ) > HMMloc model ( Q 1 + 4QK parameters) Model selection with AIC ( s (, ) σ ) ( s i σ ) ( s j exp λ ) ( s i λ ) ( s) s = ( j dz ( i, z j )) with σ j > and λ i Σ ( s) ij Q Basic HMM HMMCI HMMdist HMMloc HMMfull ( ) > Page 1

22 HMMs for rainfall: HMM with truncated Gaussian fields Model validation Meteorological interpretability Regime 1 Regime Regime 3 Regime 4 P(Y t > S t ) 4% 4% 11% 1% 8% 9% 11% % 8% 3% 7% 3% 31% 4% 56% 97% 76% 65% 6% 66% 59% 86% 73% 84% 87% 93% 9% 9% E[Y t S t,y t >] var(y t S t,y t >) corr(y t S t ) Page

23 HMMs for rainfall: HMM with truncated Gaussian fields Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution (Location 1) Truncation + Power transf. Mixing regime 1 regime regime 3 regime regime 1 regime regime 3 regime mixed data PW ( t ( 1) S t = s) PY ( t ( 1) S t = s) P( Y t( 1) S t = s)ps ( t = s) Page 3

24 HMMs for rainfall: HMM with truncated Gaussian fields Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution (Location 1).8.6 Data Model Page 4

25 HMMs for rainfall: HMM with truncated Gaussian fields Model validation Meteorological interpretability: OK Realism of artificial sequences simulated with the model Marginal distribution: OK Dynamics at the different locations: ~OK Spatial structure: OK Data Model Page 5

26 HMMs for rainfall: HMM with truncated Gaussian fields Parameter estimation Model structure Weather type: Not observed, finite values Conditionally Gaussian vector: Not observed, continuous values S t-1 S t S t+1 W t-1 W t W t+1 Precipitation: Observed, mixed continuous-discrete Y t-1 Y t Y t+1 Monte-Carlo EM algorithm Page 6

27 HMMs for rainfall: HMM with truncated Gaussian fields Need to compute the following smoothing probabilities for the M-step u t ( s) = ps ( t = sy 1T, θ n ) v t s, s' ( ) = ps ( t 1 = s, S t = s' y 1T, θ n ) EW t y 1T, S t [ = s, θ n ] EW [ t W t ' S t = s, y1 T, θ n ] Several algorithms can be used Generic algorithms : Gibbs sampler, particle filter,... More efficient algorithms if we take advantage of the specific structure of the model S t-1 S t S t+1 W t-1 W t W t+1 Y t-1 Y t Y t+1 Page 7

28 HMMs for rainfall: HMM with truncated Gaussian fields Need to compute the following smoothing probabilities for the M-step u t ( s) = ps ( t = sy 1T, θ n ) v t s, s' ( ) = ps ( t 1 = s, S t = s' y 1T, θ n ) EW [ t y 1T, θ n ] EW [ t W t ' y 1T, θ n ] Several algorithms can be used Generic algorithms : Gibbs sampler, particle filter,... More efficient algorithms if we take advantage of the specific structure of the model Forward-backward algorithm for the discrete component Monte-Carlo integration for the gaussian component ( φ pdf of NmΣ (, )) ], ] k ], ] k ], ] k φ( w 1,, w k, w k + 1,, w K ) dw 1 d w i φ( w 1,, w k, w k + 1,, w K ) dw 1 d w k w i w j φ( w 1,, w k, w k + 1,, w K ) dw 1 d w k w k Page 8

29 HMMs for rainfall: HMM with truncated Gaussian fields N=1 N=5 N~n*n 63.5 log(l) m (1) (1) Σ (1) (1,1) 4 β (1) (1) Number of iteration Number of iteration Page 9

30 Conclusion Conclusion Better description of the spatial structure of the data Some perspectives Add an autoregressive part to better describe the local dynamics Incorporate seasonal and inter-annual components in the model Some virtues of HMM Distributional versatility Ability to model diverse time scales Open structure which allows for more physical models Page 3