JOINT COOPERATION PROGRAMME

Transcription

1 JOINT COOPERATION PROGRAMME Component D1: Droughts Early Warning System Document D1.2 PPPs training on Use of PCRaster, NetCDF and grib data BMKG, Jakarta 1-4 August 2011 Project: Client: Water Mondiaal Partners for Water Royal Netherlands Embassy in Jakarta Period: January 2011 March 2013

2 Table of Contents 1. Training programme 3 2. Schmidt Ferguson classification Schmidt Ferguson classification in PCRaster Working with NetCDF and grib data 22

3 Program 3 rd JCP-training July 2011 Bandung 1 4 August Jakarta Ronald Vernimmen ronald.vernimmen@deltares.nl Program Bandung: July: Introduction to PCRaster July: Hydrometeorological reporting accessible via a clickable map interface question: BMKG year reports? Jakarta: 3. 1 August: Application of PCRaster (climatic maps: Schmidt-Ferguson) questions: official publication, jpgs of maps, legenda colours? other maps currently in use? ideas or interest for other PCRaster applications? 4. 2 August: Implementation SPI for pilot area Pemali-Comal 5. 3 August: Working with NetCDF and grib data formats question: additional NetCDF files (Wido had question earlier) 6. 4 August: Catchment water balance as check for quality discharge data

4 Schedule 29 July Hydrometeorological reporting accessible via a clickable map interface Update of DEWMS Demonstration of prototype hydrometeorological reports and clickable map + explanation manual + evaluation form coffee break presentation Pak Irfan: hydrological database and webservices Exercises Lunch presentation Ade: SIG webservices Exercises coffee break Exercises Update configuration DEWMS 1. new metadata file (additional columns as well: WS, warning levels,etc.) 2. import PusAir telemetry data (not automatic yet, via Hec-DSS database) 3. updated the explorer (rivers + DEM 900 m for whole of Indonesia) + link to online wiki 4. corrected Oldeman script (5. implemented Schmidt-Ferguson classification) - EXERCISE 6. imported Pemali Comal daily P and calculate SPI (Python script) 7. implement automatic TRMM download (via Python script and Windows task scheduler) 8. implement prototype of hydrometeorological annual reports the <DEMWS> directory!

5 Schedule 1 August Application of PCRaster Feedback training Friday, collect evaluation forms, update manual (Provinces shape layers, closing log box, data viewer panel to remove redundant information) Questions: BMKG year reports? Presentation on downscaling? NetCDF example? Publication Schmidt-Ferguson, other maps? invite BMKG staff generating SPI maps (tomorrow afternoon) Introduction to Schmidt - Ferguson Using PCRaster create Schmidt Ferguson map for Indonesia Schedule 2 August Application of PCRaster Feedback Schmidt Ferguson training (gdal_translate options explained: + nominal vs scalar Questions: Evaluation forms course Friday BMKG year reports? Presentation on downscaling? NetCDF example? Publication Schmidt-Ferguson, other maps? invite BMKG staff generating SPI maps (tomorrow afternoon) new update software + configuration! Explain steps SPI today create timeseries (BMKG format to CSV format) import timeseries compare BMKG series with PusAir series use PusAir series to create SPI (Excel method) compare PusAir Excel method with python method other interpolation methods / filling missing values Discussion, how to expand to whole of Indonesia?

6 date format of csv file date format is: yyyy-mm-dd <space> hh:mm:ss Schedule 3 August Working with NetCDF and grib data formats Feedback Implementation SPI Pemali Comal (csv import; use for all further data import, or via Data Editor) SPI comparison Excel vs Python, Griddisplay Presentation on downscaling? NetCDF example? Use of software panoply to visualize NetCDF and grib data import NetCDF file 3B43 Exercise: compare 3B43 with 3B42RT product import grib file (ECMWF seasonal forecast, P only) resample data from 0.5 degree to 0.25 degree Exercise: compare ECMWF seasonal forecast (Dec 10 Jun 11) with 3B42RT Exercise: import different parameter ECMWF Exercise: import your own NetCDF???? downscaling presentation by Mamie and/or Leni

7 Results BMKG vs PusAir comparison 2 August Conclusions: (1) Data not always the same. (2) Date shift occur, typing errors, or complete different data altogether (data from different station, data from different month? (3) date format of the CSV file is very important!!! (yyyy-mm-dd hh:mm:ss) Results SPI comparison 2 August 2 results: Moga (SPI-03) and Bantar Bolang (SPI-01) Python Excel SPI-03: Moga (73) Jan-90 Jan-92 Dec-93 Dec-95 Dec-97 Dec-99 Dec-01 Dec-03 Dec-05 Dec-07 Dec-09 DEWMS method tends to generate slightly more extremes

8 Results SPI comparison 2 August 2 results: Moga (SPI-03) and Bantar Bolang (SPI-01) Python Excel SPI-01: Bantar Bolang Jan-90 Jan-92 Dec-93 Dec-95 Dec-97 Dec-99 Dec-01 Dec-03 Dec-05 Dec-07 Dec-09 DEWMS method tends to generate slightly more extremes SPI 03: April June for Pemali Comal Using kriging interpolation (linear variogram, slope = 1, aniso = 1,0) implemented using Surfer software Kriging interpolation will be implemented in DEWMS next (currently Thiessen used) source: PusAir

9 SPI 03: April June for whole Indonesia Questions: which interpolation method used in SPI Indonesia below >> inverse distance with standard settings (no range or number of stations to be included) Station list available (coordinates?) Monthly values can be included in DEWMS? source: Schedule 4 August Catchment water balance as check for quality discharge data Feedback NetCDF + grib data (discuss use of wget) automatic import TRMM (via Windows Task Scheduler) rating curves discharge m 3 /s to m 3 Exercise 1: compare discharge in mm to precipitation in mm Exercise 2: import another ECMWF parameter (3 August) Exercise 3: import parameter from NetCDF file Wido (3 August)???? downscaling presentation by Mamie and/or Leni further planning and closing

10 General notes 3 August variables / filenames of maps in PCRaster can not start with value, always with character! Difference Maps 3B43 3B42RT bias corr. Now also included in the DEWMS! Jan 2011 Feb 2011 Mar 2011 Apr 2011 Conclusion: Apart from March 2011 (surprisingly) not much difference between the 3B43 and bias corrected 3B42RT product! STILL NEED TO CHECK IF THIS IS ALSO THE CASE FOR OTHER YEARS AND FOR ALL SEASONS!

11 Station coverage GPCP Indonesia Distribution of stations GPCP (or GPCC?) monitoring product January 2008 used in deriving 3B43 product Difference Maps ECMWF 3B42RT bias corr. Now also included in the DEWMS! Jan 2011 Feb 2011 Mar 2011 Apr 2011 Conclusion: Skill for Sumatra and Java reasonably good for 2nd month (Dec not shown here yet). Forecast severly underestimates on Borneo and later in the forecast also in Sumatra and Java. Skill clearly deteriorates with longer lead times!

12 Announcements new updates of the DEWMS during training: 1. fixed the tooltip issue for meteorological stations in reports 2. fixed the heading of the precipitation timeseries chart in the reports 3. fixed typing error in Oldeman workflow 4. added SIG-SDA river layers + lakes + irrigation 5. added Kecamatan + Desa layers 6. fixed error in IdMapping SPI, SPI workflow did not run 7. added calculation of difference maps 3B43 and ECMWF with 3B42RT bias corr. 8. implemented rating curves and conversion of discharge m 3 /s to mm Planning until next visit (half September) Everybody in close coordination with Bayu and Mamie: 1. continue data collection Pemali Comal (P until recent), fill missing years, higher resolution data (hourly as well), climatic stations! (evaporation calculation!), water level data, rating curve data points, update the metadata file and check position of meteorological and hydrological stations within the catchments start data collection for following WS: 2. Jratunseluna (Kali Garang, FFWS) (hydrological data with Oky) 3. Bengawan Solo (IFAS-ICHARM project) (hydrological data with Oky) expansion of SPI calculation: 4. AWS stations BMKG for SPI implementation whole Indonesia (via server?)

13 Schmidt-Ferguson classification Ronald Vernimmen Schmidt-Ferguson classification (1951) Classification used by whereas Oldeman used by different definition of dry and wet month! Schmidt-Ferguson (1951) adopted definition of E.C.J. Mohr (1933): dry: < 60 mm (whereas Oldeman < 100 mm) wet: > 100 mm (whereas Oldeman > 200 mm) E.C.J. Mohr created a classification system for Indonesia based on pedology (soil) studies. Schmidt-Ferguson updated: climate type classification based on ratio dry months / wet months: Q = average number of dry months in a year? / average number of wet months in a year? Mohr, E.C.J., De bodem der tropen in het algemeen en die van Nederlandsch India in het bijzonder. Koninkl. Ver. Koloniaal Inst., Amsterdam, Mededeel, 31: deel 1. Schmidt, F.D., and J.H.A. Ferguson, Rainfall types based on wet and dry period ratios for Indonesia with west New Guinea. Direc. Meteorol. Geophys., Verhandelingen 42: 1-77.

14 Oldeman agroclimatic maps Classification based on number of consecutive wet and dry months in a year. Wet month = long term average > 200 mm Dry month = long term average < 100 mm WET months DRY months A >9 <2 B1 7-9 <2 B C1 5-6 <2 C C D1 3-4 <2 D D D4 3-4 >6 E1 <3 <2 E2 <3 2-3 E3 <3 4-6 E4 <3 >6 Schmidt-Ferguson classification (1951) Climate type classification based on ratio dry months / wet months: Q = average number of dry months in a year? / average number of wet months in a year? Climate type Q value (%) Condition of climate and vegetation A < 14.3 Very wet region, tropical rain forest B Wet region, tropical rain forest C Somewhat wet region, deciduous forest, in dry season D Moderately climate, seasonal forest E Somewhat dry climate, savanna forest F Dry climate, savanna forest G Very dry climate, grass H > Extremely dry climate, grass

15 PCRaster How would you start? Knowing that for each month ( ) you have TRMM 3B42RT bias corrected data in the form of PCRaster maps. 1. determine if month is dry or wet (1 map per month) 2. count the number of dry and wet months in a year (2 maps per year = 16 maps in total) 3. calculate the average number of dry and wet months in a year (2 maps) 4. calculate Q (1 map) 5. classification map (1 map) Steps in DEWMS The input TRMM_2003_01.map maps are maps which originate from our DEWMS. The maps are exported as ASCII files and converted into PCRaster map files using asc2map.exe (a PCRaster utility) from the DEWMS in ascii format Go first to the spatial display (click on the globe icon on the topleft of your screen or use CTRL-G). Go to the folder Precipitation (TRMM 3B42RT) [mm] directory select the TRMM 3B42RT (corrected month totals) and navigate to January 2003 With your mouse rightclick on the map and select D (Select frame as ASCII grid file) (see screenshot)

16 Steps in DEWMS Steps in DEWMS save the ASCII file in a convenient location for instance D:\TRMM_2003_01.asc it can be converted to PCRaster map format using FWTools You have already installed this program (for conversion shape to bna using ogr2ogr). try running the following command line where the TRMM_2003_01.asc is stored: gdal_translate -ot Float32 -of PCRaster -mo <space> PCRASTER_VALUESCALE=VS_SCALAR TRMM_2003_01.asc <space> TRMM_2003_01.map A new file TRMM_2003_01.map is created! More info on use of gdal_translate here:

17 Steps in DEWMS The ASCII files are automatically exported in DEWMS when running the workflow Import TRMM (T0: 1 Jan 2012). The ascii files are stored in the <DEWMS/Export> directory The filename is as follows: 3B42RT_CorrectedMonth0000.asc 3B42RT_CorrectedMonth0001.asc etc. To know which file belong to which year-month inspect the 3B42RT_CorrectedMonth.xml Steps in DEWMS <?xml version="1.0" encoding="utf-8"?> <MapStacks xmlns:xsi=" xmlns=" xsi:schemalocation=" version="1.2"> <geodatum>wgs 1984</geoDatum> <timezone>0.0</timezone> <mapstack> <locationid>indonesia</locationid> <parameterid>p.obs.trmm.3b42rt.corrected</parameterid> <timestep unit="nonequidistant"/> <startdate date=" " time="00:00:00"/> <enddate date=" " time="00:00:00"/> <file> <ascii file="3b42rt_correctedmonth????.asc"/> </file> <event date=" " time="00:00:00"/> <event date=" " time="00:00:00"/> ETC. first filename (3B42RT_CorrectedMonth0000.asc) is thus for timestep (December 2001) second filename (3B42RT_CorrectedMonth0001.asc) is for timestep etc. You can rename the files automatically using a batch file

18 Schmidt-Ferguson in PCRaster Daniel Tollenaar Contact: Contents 1. PCRaster scripting 2. Steps to be taken in the model script

19 PCRaster Scripting - introduction Running a script using the command: pcrcalc <space> -f <space> <scriptname>.mod pcrcalc will tell the system to use PCRaster. -f means you run a file with <scriptname> can be any name. By convention we use the extension.mod For your convenience you can put the command in a DOS batch-file with the extension.bat. A batch-file is a file which you can use to execute multiple commands. PCRaster scripting - batch files Batch and Mod files Content of Batch file See how we use the Batch and Mod files. Here we use two files with the same name, but that is not compulsory. The command pause you can use to be sure your screen does not close directly after pcrcalc -f DrainRajang.mod is executed.

20 PCRaster scripting - mod file A mod-script requires at least a binding section and a initial section: - In the binding section, data can be called as script variables. If data are not called in the binding section, the maps need to be directly adressed in the initial section. - In the initial section, scripts can be written similar as explained before. However, if variables are called in the binding section, you only have to refer to the variable. Notes: - ; every line - Use report to write output data to your model directory Schmidt-Ferguson in PCRaster The objective is to obtain the Schmidt Ferguson classification for every TRMM cell. Write a mod-file where you calculate for every TRIMM: 1. The average monthly precipitation over the period january 2003 till december Which cells are considered wet and which cells are considered dry according to the Schmidt and Ferguson classification for the maps you calculated at step Sum the dry and the wet months based on the maps obtained at step Calculate Q = 100 X # dry months / # wet months (%) based on the maps obtained at step Determine the Schmidt Ferguson classification based on the map obtained at step 4.

21 How to find your way around in PCRaster Please look at the manual how to use functions:

22 Working with NetCDF and grib data Ronald Vernimmen General In meteorology typically gridded forecast data is stored in the grib format. There are various kinds of grib formats (grib1 and grib2) which are well structured binary files following WMO conventions. The parameter names and units are fixed! Another standard which is widely used and not only by the meteorology community is NetCDF. Delft-OMS can import both file formats and you will be able to carry out your own imports after today. grib: ECMWF seasonal forecast data NetCDF: TRMM 3B43 product

23 First things first, reading the files NetCDF and grib imports grib example: ECMWF seasonal forecast (December 2010 June 2011) NetCDF example: TRMM 3B43 ECMWF seasonal forecast data (more info here: each files consist of 214 timesteps covering the whole world, 0.5 degree resolution each file has parameter name included, follows WMO convention see the website for the parameters: before we can import this data into DEWMS we need to map the parameters. parameter list for which we have data now is included in file: Contents of ECMWF grib files - RV xls both NetCDF and grib files can be visualized with panoply