Diagnostic plots for dynamical systems

Transcription

1 Diagnosic plos for dynamical sysems Richard Wilkinson, Kamonra Suphawan, Theo Kypraios School of Mahemaical Sciences Universiy of Noingham UCM - July 2012

2 Inroducion Dynamical sysems and common ypes of error Predicive disribuions Scoring rules Diagnosic plos Toy example Less oyish example

3 Dynamical sysems simulaors Sae vecor x which evolves hrough ime. Le x 0:T denoe (x 0, x 1,..., x T ). Compuer model f which encapsulaes our beliefs abou he dynamics of he sae vecor x +1 = f (x, u ) + w where w repens a simulaor discrepancy erm (can depend on u and x). Trea f as a black-box Observaions y = h(x ) where h( ) usually conains some sochasic elemen We re no doing parameer esimaion.

4 UCM 2010 Wilkinson e al Assume Evaporanspiraion Rain Overland flow x +1 = f (x ) + δ(x ) Soil hsoil Percolaion Laeral subsuface flow River hriver River flow where δ(x) is a funcional simulaor discrepancy. We hen proposed a model for δ( ), parameric or oherwise, and aemped o learn i. Ground Waer hgw Percolaion o deep aquifers Baseflow This urned ou o be hard - we never observe x, only y, which may be lower dimensional and only provide limied informaion abou x.

5 UCM 2010 Wilkinson e al Evaporanspiraion Rain Overland flow Assume x +1 = f (x ) + δ(x ) Soil Ground Waer hsoil Percolaion hgw Percolaion o deep aquifers Laeral subsuface flow Baseflow River hriver River flow where δ(x) is a funcional simulaor discrepancy. We hen proposed a model for δ( ), parameric or oherwise, and aemped o learn i. This urned ou o be hard - we never observe x, only y, which may be lower dimensional and only provide limied informaion abou x. Simpler quesion: Given an imperfec saisical forecasing sysem, diagnose which aspec of he sysem is causing he error

6 All models are wrong... The forecasing sysem consiss of a simulaor, saisical model of simulaor discrepancy, observaion equaion and saisical model, inferenial scheme... call he enire sysem a saisical forecasing sysem There are various ypes of error we migh see Incorrec simulaor dynamics (simulaor discrepancy) Simulaor discrepancy erm mis-specified Measuremen process mis-specified (incorrec variance, incorrec h) Iniial condiion errors - poor choice of x o when iniialising forecass. Given an imperfec forecasing sysem, how do we know wha ype of error we are faced wih? we ve looked in various lierau, bu found lile

7 Sysem oupu We mus decide wha aspec of he sysem oupu o use, and hen how o judge i. Possible sysem oupus: Poserior predicive disribuions π(y rep y 1: ) or smoohed disribuions π(y y 1:T ) for 1 T Predicive disribuions π(y +k y 1: ) - k-sep-ahead forecas Poin esimaes from any of he above Noe: obaining hese disribuions ypically requi some work, e.g. Kalman filer and is varians, sequenial Mone Carlo mehods, ABC mehods. (needed?)

8 Sysem oupu We mus decide wha aspec of he sysem oupu o use, and hen how o judge i. Possible sysem oupus: Poserior predicive disribuions π(y rep y 1: ) or smoohed disribuions π(y y 1:T ) for 1 T Predicive disribuions π(y +k y 1: ) - k-sep-ahead forecas Poin esimaes from any of he above Noe: obaining hese disribuions ypically requi some work, e.g. Kalman filer and is varians, sequenial Mone Carlo mehods, ABC mehods. (needed?) poserior predicive disribuions: a debae over heir validiy Difficuly of inerpreaion - condiioning on he daa moves he sae predicion closer o he daa, making probabiliies harder o inerpre. p-values don have a U[0, 1] disribuion under he null Prior predicive p-values have been recommended insead by various auhors (e.g., Bayarri and Casellanos 2007)

9 k-sep ahead-predicive disribuions In he dynamical sysems seing, he prior predicive densiy is he k-sep-ahead forecas π(y +k y 1: ) Predicion vs explanaion (Shmueli 2011) base validaion and diagnosics solely on he forecasing sysem s abily o predic, no explain Link o Dawid s prequenial approach over-fiing less of a problem if we are only using predicive measu

10 k-sep ahead-predicive disribuions In he dynamical sysems seing, he prior predicive densiy is he k-sep-ahead forecas π(y +k y 1: ) Predicion vs explanaion (Shmueli 2011) base validaion and diagnosics solely on he forecasing sysem s abily o predic, no explain Link o Dawid s prequenial approach over-fiing less of a problem if we are only using predicive measu By looking a predicions a differen lead imes we can emphasise differen aspecs of he forecasing sysem:

11 How o judge: Numerical Sco Numerous numerical sco are used e.g., MSE, MAD, MAPE, and corponding skill-sco obained by comparing hese values o hose given for a reference forecasing sysem such as climaology or persisence, e.g., Nash-Sucliffe efficiency, Theil s U.

12 How o judge: Numerical Sco Numerous numerical sco are used e.g., MSE, MAD, MAPE, and corponding skill-sco obained by comparing hese values o hose given for a reference forecasing sysem such as climaology or persisence, e.g., Nash-Sucliffe efficiency, Theil s U. Theory of proper scoring rules undergoing a urgence in recen years. A scoring rule S akes forecas disribuion π and observed value y and reurns a numerical score S(π, x) R S is proper iff E q S(π, X ) = S(π, x)q(dx) is maximised a π = q. Tha is, if we believe q, hen we maximize our expeced score by reporing q.

13 How o judge: Numerical Sco Numerous numerical sco are used e.g., MSE, MAD, MAPE, and corponding skill-sco obained by comparing hese values o hose given for a reference forecasing sysem such as climaology or persisence, e.g., Nash-Sucliffe efficiency, Theil s U. Theory of proper scoring rules undergoing a urgence in recen years. A scoring rule S akes forecas disribuion π and observed value y and reurns a numerical score S(π, x) R S is proper iff E q S(π, X ) = S(π, x)q(dx) is maximised a π = q. Tha is, if we believe q, hen we maximize our expeced score by reporing q. Imporan o use proper sco, as improper sco are an inducemen o hedging Examples include he CRPS, logarihmic score, Brier score, Dawid score... Any proper score can be decomposed ino a reliabiliy erm and a sharpness erm.

14 Improper score vs proper score Two parameer model, boh scale parameers, 1-sep ahead forecass Simulaor discrepancy variance on x-axis - rue value a 1 Measuremen error variance on y-axis - rue value a 1 Score on z-axis MSE conour surface CRPS conour surface au^ au^ sigma^2 CRPS correcly idenifies region conaining rue value MSE can only find good raios of parameer values. sigma^2

15 Diagnosic plos We wan o be able o say where he problem lies, no jus ha here is a problem. Tesing and formal inference are hard problems for sae-space problems because of he high-dimensional naure of he missing daa. Tukey (1962, and many ohers) suggesed we use graphical mehods o highligh problems, raher han paramerizing ypes of deparure in advance and developing significance ess. Diagnosic plos seem o be a beer opion han deerminisic sco. Useful plos include Residuals ɛ +k = y +k E(y +k y ) vs ime + k Residuals ɛ +k vs poserior mean E(y +k y ) Auocorrelaion () plos a various lags CUSUM ype plos.

16 Aribue diagrams Hsu and Murphy 1986 Sequence of binary evens wih oucomes d 1, d 2,..., d T Sequence of forecas probabiliies f 1, f,..., f T. Pariion [0, 1] ino K inervals wih each inerval characerized by a repenaive forecas probabiliy f k. le d k be he relaive occurence of he even when he forecas was in inerval k. Aribue diagram is he plo of d k vs f k. We can add simple error bars using he Mone Carlo variance.

17 Aribue diagrams for coninuous random variables f[k:t] Aribue diagrams are defined for sequences of binary evens. We arificially creae suchseries a sequence by defining predicion Aribue diagram inervals I 4 cenral inerval I (c) = [a, b ] lef inerval I (l) = (, l ] righ inerval I (r) = [r, ) so ha P(y +k I ( ) +k y 1:) = p and hen define he sequence of binary evens o be observed relaive frequencies cenral inerval lef inerval righ inerval r (p) = Lag { 1 if y +k I ( ) +k 0 oherwise forecas probabiliies We can hen compare he relaive frequency r (p) wih p (perfecly calibraed forecas has r (p) = p). Repea for various p and plo o ge a version of he aribue diagram for coninuous random variables.

18 Illusraive example 1 To illusrae he paerns we expec o see for he various diagnosic plos as errors of differen ypes of error are inroduced, consider he simple linear Gaussian model X +1 = ax + b + N(0, σ 2 ) Y = cy + N(0, τ 2 ) We can use he Kalman filer o obain he filering disribuions and he k-sep ahead predicions in his case. We generae an observed daase wih a = b = c = τ = σ = 1 and hen inroduce errors one a a ime o illusrae he deviaions we expec from he null form of he various plos.

19 Null plos - 1 sep ahead f[k:t] Series a = 1, c = 1, sigma^sq = 1, au^sq = 1, k = 1 Aribue diagram observed relaive frequencies cenral inerval lef inerval righ inerval Trace plos should look like whie noise Residual plos a band of poins wih no noiceable paern plo show no correlaon a lag greaer han 1 Aribue diagrams show no significan depariure from y = x

20 Null plos - 5 sep ahead a = 1, c = 1, sigma^sq = 1, au^sq = 1, k = Series observed relaive frequencies cenral inerval lef inerval righ inerval f[k:t] Aribue diagram We now expec o see some correlaion as he iduals from predicions less han 5 ime poins apar will correlaed plo shows posiive correlaion for lags upo 5, and hen a period of negaive correlaion. Aribue diagrams show no significan depariure from y = x

21 Lag forecas probabiliies 0 50 a = 1100, c = 1150, sigma^sq 200 = 1, au^sq 10 = 50.1, k 0 = Incorrec measuremen error - oo small 4 k = k = Series Lag Series observed relaive frequencies observed relaive frequencies f[k:t] Aribue diagram cenral inerval lef inerval righ inerval f[k:t] cenral inerval lef inerval righ inerval forecas probabiliies Aribue diagram Lag 1 correlaion will be negaive Aribue diagram: cenral inerval - over-confiden for cenral inerval. For L/R inervals, under-confiden for p < 0.5 and over-confiden for p > 0.5. As he lead ime k increases, he raio 1, and so kσ 2 +τ 2 kσ 2 +τ 2 rue he aribue diagrams will begin o look OK

22 Incorrec measuremen error - oo large k = 1 Series Lag observed relaive frequencies f[k:t] Aribue diagram cenral inerval lef inerval righ inerval forecas probabiliies Wih τ 2 oo large, he paern is reversed. Cenral inerval will be under-confiden. L/R inervals will be over-confiden for p < 0.5, oherwise under-confiden. Aribue diagram looks beer as lead ime increases.

23 Lag forecas probabiliies 0 50a = 1100, c = 1150, sigma^sq 200 = 0.1, au^sq 5= 1, k 0 = Incorrec simulaor discrepancy error - oo small k = k = Series Lag Series observed relaive frequencies observed relaive frequencies f[k:t] Aribue diagram cenral inerval lef inerval righ inerval f[k:t] cenral inerval lef inerval righ inerval forecas probabiliies Aribue diagram plo will end o be posiive - oo much emphasis on model Aribue diagram shows same paern as when meas. error is oo small. However, as he lead ime k increases, he raio kσ2 +τ 2 1 kσ 2 +τrue 2 slowly increases, and so he aribue diagrams don improve (and may look worse)

24 Lag forecas probabiliies 0 50a = 1100, c = 1150, sigma^sq 200 = 10, au^sq 10 = 51, k 0 = Incorrec simulaor discrepancy error - oo large 4 k = k = Series Lag Series observed relaive frequencies observed relaive frequencies f[k:t] Aribue diagram cenral inerval lef inerval righ inerval f[k:t] cenral inerval lef inerval righ inerval forecas probabiliies Aribue diagram Negaive correlaion a lag 1 - iduals osciallae (chasing he daa) Aribue diagram shows same paern as when meas. error is oo large, bu he error doesn improve wih lead ime

25 Incorrec simulaor dynamics Harder o deal wih, as he variey of possible errors is larger. Similar o having oo small a simulaor discrepancy erm. Cenral inerval aribue diagrams will be over-confiden. However, here are some differences we may be able o spo. Paerns of correlaion in he idual plos plo posiive a all lags Error grows faser wih longer lead imes A difference eviden beween he lef and righ aribue diagram curves.

26 Incorrec simulaor dynamics: a = 1.1, lead ime k = f[k:t] Series a = 1.1, c = 1, sigma^sq = 1, au^sq = 1, k = 1 Aribue diagram observed relaive frequencies cenral inerval lef inerval righ inerval Residual plo sars o show he shape of he missing dynamics L/R aribue diagrams show a difference All s posiive

27 Incorrec simulaor dynamics: a = 1.1, lead ime k = 5 a = 1.1, c = 1, sigma^sq = 1, au^sq = 1, k = Series observed relaive frequencies cenral inerval lef inerval righ inerval f[k:t] Aribue diagram Residual plo paern becomes more obvious wih longer lead ime L/R/C aribue diagrams ge worse L/R inerval aribue curves are differen s large and posiive

28 Incorrec simulaor dynamics: a = 1.1, lead ime k = 10 a = 1.1, c = 1, sigma^sq = 1, au^sq = 1, k = Large difference beween L/R aribue diagrams f[k:t] Series Aribue diagram observed relaive frequencies cenral inerval lef inerval righ inerval

29 Incorrec simulaor dynamics: a = 0.5, lead ime k = 1 a = 0.5, c = 1, sigma^sq = 1, au^sq = 1, k = Larger simulaor error easier o spo f[k:t] Series Aribue diagram observed relaive frequencies cenral inerval lef inerval righ inerval

30 Incorrec simulaor dynamics: a = 0.5, lead ime k = 1 a = 0.5, c = 1, sigma^sq = 1, au^sq = 1, k = f[k:t] Larger simulaor error easier o spo If here are muliple ypes of error, plos give mixed messages. Paerns seem o exhibi behaviour corponding o dominan error. Series Aribue diagram observed relaive frequencies cenral inerval lef inerval righ inerval

31 Rainfall-runoff simulaor Daa from he Abercrombie Valley, Aus Evaporanspiraion Soil Ground Waer Rain hsoil Percolaion hgw Percolaion o deep aquifers Overland flow Laeral subsuface flow Baseflow River hriver River flow We have measuremens of he river flow, evoporanspiraion, and rainfall for a 2 year period. The measuremen error process is unknown, bu we ve assumed log(r() + λ) N(log(R() sim + λ), s 2 ) where R() is he river flow measuremen, and R() sim is he simulaor predicion + discrepancy. Discrepancy esimaed by ML previously.

32 k = 1, esimaed discrepancy Series Lag iduals observed relaive frequencies cenral inerval lef inerval righ inerval Aribue diagram forecased mean forecas probabiliies

33 k = 5, esimaed discrepancy Series Lag iduals observed relaive frequencies cenral inerval lef inerval righ inerval Aribue diagram forecased mean forecas probabiliies

34 k = 10, esimaed discrepancy/3 Series Lag iduals observed relaive frequencies Aribue diagram cenral inerval lef inerval righ inerval forecased mean forecas probabiliies

35 k = 1, esimaed discrepancy/3 Series Lag iduals observed relaive frequencies Aribue diagram cenral inerval lef inerval righ inerval forecased mean forecas probabiliies

36 k = 1, esimaed discrepancy 1.7, measuremen error/3 Series Lag iduals observed relaive frequencies Aribue diagram cenral inerval lef inerval righ inerval forecased mean forecas probabiliies

37 k = 10, esimaed discrepancy 1.7, measuremen error/3 Series Lag iduals observed relaive frequencies Aribue diagram cenral inerval lef inerval righ inerval forecased mean forecas probabiliies

38 Conclusions Paramerizing all errors and esimaing parameer values is fraugh wih difficulies. We hink a case can be made for using simpler diagnosic ools, a leas in he early sage of modelling. Predicive disribuions more useful for diagnosics han poserior-predicive disribuions (forecass no hindcass) Proper scoring rules needed if scale parameers are unknown Using differen lead imes emphasises differen aspecs of he problem Diagnosic plos more useful han numerical summaries for diagnosing errors Aribue diagrams wih arificial predicion inervals can be useful. However, i seems o be inherenly difficul o spo he source of errors, paricularly if hey are small unless long ime-series are available (compuaional ource?). Thank you for lisening!

39 To do Apply o more real examples Combinaions of differen ypes of errors - how do we spo and disenangle Theoreical properies Prove expeced sign of lags Prove expeced oher paerns in aribue diagrams ec Problems wih correlaion/dependence in he aribue plos - are he confidence bands correc? Somehing doesn seem quie righ. As we le T increase, he L/R inerval aribue diagrams don seem o work Theoreically, L/R curves should be he same when no simulaor bias, only incorrec simulaor or measuremen error. Even wih rue parameer values, he iduals plos doesn look like a null plo for large lags - eg lag 100 shows clear rend. Why? Is i correlaion in he iduals - should we hin o 1/k?