ICAM conference 6 June 2013 Kranjska Gora (SLO) Objective forecast verification of WRF compared to ALARO and the derived INCA-FVG outputs

Transcription

1 ICAM conference 6 June 2013 Kranjska Gora (SLO) Objective forecast verification of WRF compared to ALARO and the derived INCA-FVG outputs Arturo Pucillo & Agostino Manzato OSMER ARPA FVG Visco (UD), Italy

2 Introduction - Outlines INCA and other models is an european project of the Central Europe area to apply and improve the INCA nowcasting software developed by ZAMG (A). Rain forecasting models: ALARO-5: LAM at about 4 km, provided directly by ZAMG (A). INCA-FVG: nowcasting software that downscales the initial ALARO-5 model output at 1 km and fits it with the surface stations and Fossalon radar observations. RR module runs every 15' on the INCA-FVG domain. - WRF ARW: LAM at about 3.5 km, initialized on ECMWF every 3h. Uses 3DVAR cold-run data assimilation to ingest about 170 surface stations on Northern-Italy and 5 high-resolution soundings, provided by the external expertise CETEMPS Università de L Aquila. - Eulerian Persistence: also a simple radar-frozen SRI method has been tried for some verification tests. Rain observations for verification: About 46 independent surface stations (different from those assimilated in WRF) have been used for the pointwise verification. SRI maps (500m res) of the Fossalon di Grado radar corrected with surface OSMER raingauges on the FVG is used for the spatial verification.

3 Domains The figure below shows in red the 170 stations assimilated by WRF (and INCA-FVG in its smaller domain) and in green the 60 stations used for an independent verification. In reality only about 46 green stations were placed inside the INCA-FVG domain. There are two domains: a bigger one used for the pointwise verification (that is the INCAFVG domain) and a smaller one (basically the Fossalon di Grado radar covered domain) used for the spatial verification.

4 Verification methods The main software tool used for the verification is MET (Model Evaluation Tool) made by the Developmental Testbed Center of NCAR (Boulder, CO, USA), while other specific programs were written in R. Time domain: the hourly rain was hincasted and verified only for the period 1/6/ /7/2011, for a total of about 1450 hourly cases: not so many! Note that there are two runs per day: 00 UTC and 12 UTC (for INCA it were used the 01 and 13 UTC runs, that use the 00 and 12 UTC ALARO). Data Preprocessing: GRIB files were regridded at about 3.8 km using the copygb utility using a nearest neighbour algorithm (Accadia et al., 2003). -> POINT verification (on the larger INCA-FVG domain): 1. pointwise verification on single stations data (nearest neighbour algorithm as model interpolation method), using continuous (MSE, R, BIAS) and categorical (Peirce Skill Score) metrics. -> SPATIAL verification (on the smaller FVG domain): 2. areal verification of nearest-neighbour using the Fractions Skill Score. 3. object-oriented spatial verification using cells attributes (interest metric).

5 Rain climatology in 60 verification stations Even small rainfall like 2 mm/h are relatively rare (during these two months p< 4%, so that there are only few cases, ~55). One event has a peak at 6 UTC, while other cases were concentrated on the afternoon/evening period. Rain > 1 mm/h Rain > 2 mm/h

6 1.1 Pointwise - continuous verification run 00 someway contrasting results... MSE R BIAS WRF: low MSE up tp +4h but R decreases until +7h, there is a dry BIAS; ALARO: similar to WRF, but lower MSE after +8h and better R after +5h; INCA: higher MSE in the first +4h, high variability in R and wet BIAS.

7 1.2 Pointwise - categorical verification contingency table and derived indices If Rain > Threshold then the categorical variable is 1, otherwise it is 0. The verification has been repeated on more rain thresholds: >=0, >= 1, >= 2, >= 3, >= 4, >= 5 mm/h a = correct hit b = false allarms c = missed events d = correct non-events Peirce Skill Score chosen as verification metric (see Manzato 2007 WAF): PSS=POD POFD, where a POD=, ( a+c ) POFD= b ( b+d )

8 Pointwise categorical run 00 PSS vs. lead time Rain > 1 mm/h WRF: WRF the small skill up to +4h seems a spin-up Rain > 2 mm/h ALARO seems to have more spin up up to +5h - After +5h ALARO seems to have higher skill, while before WRF is better only for Rain>2mm/h. - INCA seems better than LAMs in the first +3h (as expected) but EUL. PERS. seems even better?

9 2. Areal Verification nearest neighbour with Fractions Skill Score (FSS) N N 1 (P Pobs )2 N i=1 fcst FSS =1 N 1 N N 1 1 P + P N i=1 f cst N i =1 obs 2 2 FSS=1 1 N ( P fcst P obs )2 i= 1 N P fcst observed i= N N P obs 2 i=1 forecast FSS (Roberts and Lean 2008) compares spatialized observations and forecasts on scales larger and larger, varying also the rain intensity threshold (same for obs and for). The red box is the area under test and the number of gridboxes with rain > threshold is the only parameter considered (not the exact location), to avoid the double penalty problem. Tested with nearest neighbour method from 1 to 45 km windows width. Categorical thresholds: >=0, >= 1, >= 2, >= 3, >= 4, >= 5,... >= 12.

10 Fractions Skill Score run +1h WRF ALARO No skill for LAMs at +1h (spin up problem). Much better performance for INCA and even better for Eul. Persist. INCA Persist.Eul.

11 Fractions Skill Score run +3h WRF ALARO Still very low skill for the two LAMs. INCA and Eul. Persist. show similar performance: quite lower +1h lead time. Persist.Eul. INCA

12 Fractions Skill Score run +6h WRF ALARO ALARO shows the best performance, followed by INCA, INCA +6h should become very similar to its background model. INCA Persist.Eul.

13 Fractions Skill Score run +12h WRF ALARO Persist.Eul. INCA WRF seems better than ALARO, ALARO in particular at high rain thresholds. INCA seems even better than ALARO for the small rain thresholds (maybe due to the downscaling effect?). No skill for Eul. Persist..

14 Fractions Skill Score run +3h WRF ALARO Better +3h for the 12 UTC run than it was for the 00 UTC run. Probably because there were more rainy cases during the afternoon than during the night. That shows a dependence from the small sampling dataset. WRF seems the best in this case. Persist.Eul. INCA

15 3. Spatial Verification object-oriented attributes Identification Measure Attributes Matching: many objects are identified on the for and obs fields and their attributes (shape, area, centroides,...) are compared. Different attributes are summarized with the INTEREST index, that -of course- is very sensible to the parameters chosen. WRF OBS OBS INCA Merging Matching Comparison Summarize

16 INTEREST vs. lead time - run 00 for Rain>1mm/h INCA (run every 3h in this case) in the first +5h seems to have better description of the objects (because it starts from the radar SRI), but the simpler Eul. Persist. is also very good in the first +3h. WRF is the worst in the first +6h, but then improves after +10h.

17 INTEREST vs. lead time - run 00 for Rain>2mm/h In this case the two LAM perfromances are reversed: WRF performs better in the first +6h, while ALARO seems better after +10h. The results of this verification seems to vary a lot with the rain threshold.

18 Preliminary Conclusions - It is very difficult to make reliable conclusions given this small verification dataset (short period and small domain covered by radar). For this reason the statistical significance of these tests (computed in many cases but not shown) is very low; - In general it seems that INCA FVG has good performances in the nowcasting range (up to +3h), but these performances seem not much better than the simpler Eulerian Persistence; Persistence - In the short-term forecasting it seems that WRF seems better for the Rain>2 mm/h threshold, while ALARO is better for the Rain > 1 mm/h events. - Even WRF with data assimilation (cold-run 3D-VAR) of local stations and high-resolution soundings shows spin up problems at least in the first 3h.