A re-sampling based weather generator

A re-sampling based weather generator Sara Martino 1 Joint work with T. Nipen 2 and C. Lussana 2 1 Sintef Energy Resources 2 Norwegian Metereologic Institute Stockholm 7th Dec. 2017 Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 1 / 31

Overview 1 Motivation 2 Resampling 3 The Dataset 4 Joining Segments 5 Results 6 Concluding remarks Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 2 / 31

Why a weather simulator? Hydrological and production planning models need weather variables as input Most important: temperature and precipitation. But also solar radiation and wind have increasing importance Observed series are sparse (in space) and short (in time) Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 3 / 31

What does a weather simulator do? Create random time series of weather variables that look realistic: in time in space in relation to each other Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 4 / 31

Is it easy to build a weather simulator? We want to simulate precipitation: Spatial Structure Type of precipitation Orography Temporal Structure Mass at zero Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 5 / 31

Is it easy to build a weather simulator? We want to simulate precipitation: Spatial Structure Type of precipitation Orography Temporal Structure Mass at zero And we want to simulate more than just precipitation!! Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 5 / 31

Spatial and temporal domain for the simulator We fokus on: Daily values for the weather variables The simulator is used for medium/long term planing Spatial domain includes Scandinavia and Northern Europe Energy market in Europe becomes more and more integrated Largest energy companies need to account also for what happens in other countries Renewable energy production increases in many countries in Europe Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 6 / 31

Need for a larger simulator Increased spatial domain Increased number of variables Precipitation Temperature Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 7 / 31

Need for a larger simulator Increased spatial domain Increased number of variables Precipitation Temperature Humidity, Radiation, Wind Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 7 / 31

Challenges The larger domain and the increased number of variables of interest creates many challenges The size of the spatial model Modeling the covariances spatial temporal inter-variables Many such characteristics are embedded in deterministic models used in numerical weather prediction There are dataset with stored NWP ensembles or hindcast ensembles. Such datasets are steadily increasing. Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 8 / 31

Challenges The larger domain and the increased number of variables of interest creates many challenges The size of the spatial model Modeling the covariances spatial temporal inter-variables Many such characteristics are embedded in deterministic models used in numerical weather prediction There are dataset with stored NWP ensembles or hindcast ensembles. Such datasets are steadily increasing. Idea: Try to re-sample from such data set! Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 8 / 31

Re-sampling idea Create a new weather sequence by joining randomly chosen 10-days segments We do not try to model the transition but look for an analog day in the database We do not model climate change and assume stationary weather for the next 5/10 years Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 9 / 31

Pros and Cons Advantages At daily scale: physical correlation between variables At daily scale: physical spatial correlation No need to model time transition The model is easy to understand and to implement It can be applied in any part of the world Disadvantages Need an algorithm to join sequences You get the physics that is in the forecast model! Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 10 / 31

The Dataset Based on ECMWF ensemble re-forecasts Twice a week, the ECMWF reruns their current operational model for a number of past days (only 5 ensemble) Time span 1997-2016 (growing) Spatial resolution 0.25 0.25 ( in Norway ca 14 27 Km) 10 days long forecast Total of ca 23000 10 days segments In contrast, a reanalysis dataset would have ca 20 365 days Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 11 / 31

The Dataset South of Norway as temporary spatial domain 8 variables (2m temperature (K), precipitation (mm), sea level pressure (Pa), cloud area fraction (0-1), zonal wind speed (m/s), meridional wind speed (m/s), incoming solar radiation (W/m2), relative humidity ) We are not directly interested in sea level pressure but use it for its large scale behavior Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 12 / 31

Seasonal Dataset The dataset is monthly based to ensure seasonal variation Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 13 / 31

Drift Correction Precipitation Temperature We found a drift in the forecast model Use QQ map to correct it (independently for each cell) Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 14 / 31

How to join segments I For the current segment y, for each candidate segment x in the relevant database Compute the score: Where S x = k ν=1 wν RMSD ( f (y 10 ν ), f (x 1 ν ) ) k is the number of variables that are included in the score w ν is a weight (depends also on the units of the variable) y 10 and x 1 are the 10th day in the current segment and the rst day in the candidate segment RMSD( ) is the root mean squared dierence f ( ) is a functional of the relevant map Randomly sample one of the top N segments Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 15 / 31

How to join segments I Illustration Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 16 / 31

How to join segments II Need to reduce dimension Aggregate over whole domain or use wavelets Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 17 / 31

How to join segments III Important choices to make: Which variables to include in the score Which weight to give to each variable in the score Which spatial resolution to choose Still working to nd the optimal set up Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 18 / 31

How to join segments III Important choices to make: Which variables to include in the score (Pressure and temperature) Which weight to give to each variable in the score Which spatial resolution to choose Still working to nd the optimal set up Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 18 / 31

How to join segments III Important choices to make: Which variables to include in the score Which weight to give to each variable in the score (Proportional to the sd of the variable) Which spatial resolution to choose Still working to nd the optimal set up Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 18 / 31

One year of simulated elds Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 19 / 31

Verication of the scenarios characteristics In out verication the truth is the day 0 forecast Smoothness of the time series Time structure Spatial structure Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 20 / 31

Smoothness of the scenarios Mean inter day jump near Oslo Even matching on the spatial mean gives reasonable results Wavelets improves the smoothness locally Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 21 / 31

Smoothness of the scenarios Cold and dry in winter season Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 22 / 31

Seasonal and Annual variation Realistic seasonal pattern Manage to create warm/cald, dry/wet years Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 23 / 31

Autocorrelation Autocorrelation for temperature and precipitation Cell close to Oslo Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 24 / 31

Length of dry spell Distribution of dry spell length in Oslo and Bergen Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 25 / 31

Spatial correlation - Temperature Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 26 / 31

Spatial correlation - Precipitation Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 27 / 31

Variance at dierent aggregation scales Too low variance at higher aggregation times Ideas to improve it: Include external forcing (NAO??) for longer time trends Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 28 / 31

Open source software Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 29 / 31

Open source software wxgen Downloadable from: https://github.com/metno/wxgen/wiki What it does: Generate scenarios wxgen sim -db db.nc -n 21 -t 365 -o sim.nc Generate truth scenario wxgen truth -db database.nc -o truth.nc Verify scenario characteristics wxgen verif -v 0 truth.nc -m variance Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 30 / 31

Conclusions and further work Flexible framework Applicable to areas with bad data coverage Does not need to explicitly model covariances (time, space and inter-variables) Preliminary results: The simulated series are smooth and respect the correlation between covariates There is both seasonal and annual variation The variance of long time aggregated variables is a bit too low Next step: Increase the database and work on a European scale Sara Martino Joint work with T. Nipen and C. Lussana Stockholm 7th Dec. 2017 31 / 31