Non catchment type instruments for snowfall measurement: General considerations and

Non catchment type instruments for snowfall measurement: General considerations and issues encountered during the WMO CIMO SPICE experiment, and derived recommendations Authors : Yves Alain Roulet (1), Audrey Reverdin (1), Samuel Buisan (2), Rodica Nitu (3) (1) MeteoSwiss, Payerne, Switzerland, yves alain.roulet@meteoswiss.ch (2) Spanish State Meteorology Agency (AEMET) Aragon Regional Office, Zaragoza, Spain (3) Environment and Climate Change Canada, Toronto, Canada 1 Introduction One objective of the WMO CIMO SPICE (Solid Precipitation Intercomparison Experiment) was to investigate the ability of emerging technologies to measure solid precipitation (accumulation and intensity) as an alternative to the traditional tipping bucket and weighing gauges, and to assess their operational capabilities under winter conditions. This category of instruments, also called noncatchment type instruments (among them disdrometers and present weather sensors) provides information on precipitation amount and intensity, and are also used for the discrimination of precipitation and/or weather type (SYNOP, METAR codes for real time applications such as airports). For disdrometers, these reported information are based on the measurement of hydrometeor size and fall speed velocity distributions, which can be retrieved as raw data from the sensor and be used for detailed event based analysis. The important number of non catchment type instruments tested along during the SPICE campaign under various winter climate conditions provides basics for the assessment of the ability for such instruments to measure and report snow, and increases the know how on this type of instrument. This paper will summarize the issues encountered with the operation of non catchment type instruments tested during the SPICE campaign, and will give preliminary statement on the ability of these instruments to report precipitation accumulation under winter conditions. 2 Methodology SPICE was a multisite experiment, ran over two winter seasons (2013/14 and 2014/15), involving 20 sites in 15 different countries (Figure 1). More than 30 different instrument types were provided by manufacturers and evaluated against reference measurements. 1

Figure 1: Location of sites participating to the SPICE field experiment. In order to guarantee the traceability and comparability of results across the sites, a common Field Working Reference System (FWRS) has been defined, based on the results and recommendations from the previous solid precipitation intercomparison (Goodison et al., 1998). It consists in an automatic weighing gauge (OTT Pluvio 2 or GEONOR T 200B3 gauge with 3 transducers) with a Single Alter shield and a DFIR (Double Fence Intercomparison Reference, Figure 2). A precipitation detector mounted on the inner fence of the DFIR was also addedd to the FWRS, in orderr to help discriminate real precipitation events from noise reported by the reference weighing gauge. An extensive description of the SPICE methodology and the different level of references (from bush gauges to automatic measurement within a DFIR) can be found in the final report of the experiment (WMO publication, coming soon). See also the SPICE keynote paper from TECO 2016 (Nitu et al., 2016). The data resolution was typically 1 min (down to 6 sec in some cases), but the analysis was conducted using 30 min events. Criteria for the 30 min event selection (i.e. precipitation event) for non catchment type instrumentss were as follow: Reference reporting precipitation ( Yes case, denoted Y below) : reference weighing gauge catching 0.25 mm or more AND precipitation detector recording 18 min or more of precipitation. Reference reporting no precipitation ( No case, denoted N below) : reference weighing gauge catching 0.1 mm or less AND precipitation detector recording 0 min of precipitation. 2

Instrument under test reporting precipitation ( Yes case, denoted Y below) : instrument reporting more than 0 mm of precipitation (accumulation). A lower threshold has been defined for non catchment instrument, since they have much lower sensitivity and noise level than a weighing gauge. Instrument under test reporting no precipitation ( No case, denoted N below) : instrument reporting 0 mm of precipitation. Figure 2: Field Working Reference System (FWRS), as configured on the Sodankylä SPICE site, Finland. The evaluation of the sensors under test (SUT) providedd by the manufacturers, addressed following questions: Reliability in detecting precipitation: Do both instruments report precipitation (any type) at the same time (YY cases) )? Performancee of SUT during no precipitation events: Do both instruments agree on period without precipitation (NN cases)? Performancee during precipitation events (YY cases): Quantitative assessment, using catch ratio, for rain, mixed and snow, respectively. Assessment of external factor influencing the measurement quality (wind speed, wind direction). Assessment of YN, NY events: When do both instruments disagree? Threshold selection: According to the analysis, what should be the threshold set for the instrument under test to allow reliable reporting of precipitation event (30 min interval)? For this purpose, plots were created, such as scatter plots (reference accumulation vs SUT accumulation), or catch ratio (CR) as a function of wind speed and precipitation type. Skills scores were also used, based on following contingency table (Table 1). 3

Reference SUT Precipitation No Precipitation Total Precipitation x (hits) z (false alarms) x + z No Precipitation y (misses) w (correct negatives) y + w Total x + y z + w N = x + y + z + w Table 1: Contingency table for precipitation detection. In this paper, the Probability of Detection (POD) and the False Alarm Rate (FAR) are used. They are defined as: % 100 The POD gives the fraction of events, out of all the precipitation events as indicated by the reference, which will also be reported as precipitation events by the SUT. In other words it gives the probability of the SUT agreeing on the occurrence of precipitation given that the reference detected precipitation. Ideally, POD would have a value of 100%. % 100 The FAR is the fraction of precipitation events as reported by the SUT which were judged by the reference as not meeting the precipitation event criteria. FAR gives an indication of how likely is that the sensor is not reliable when it reports the occurrence of precipitation. A larger percentage for FAR would imply that there is a high probability that the SUT fails to recognize precipitation events in a similar manner as judged by the reference. Ideally, the FAR would be zero. The Root Mean Square Error (RMSE) has also been calculated for each SUT and precipitation type. This value gives an indication of the variability of the SUT against the reference. It is defined as: 1/ Where Xai is the reference accumulation over the i th 30 min interval, Xbi the accumulation of the SUT over that same interval and n is the number of 30 min intervals over which the analysis was performed. The results, using metrics and plots described here above, are shown in Chapter 4. 4

3 Instruments under test The instruments submitted by manufacturers, and accepted as SUT within SPICE, are listed in Table 2, together with the corresponding SPICE host site. Instrument Model Measuring principle Host SPICE sites Thies Laser Precipitation Monitor LPM Disdrometer Marshall, Weissfluhjoch OTT Parsivel 2 disdrometer Disdrometer Sodankylä Campbell Scientific PWS100 Present Weather Sensor Haukeliseter, Marshall Vaisala FS11P (FS11/PWD32 combination) Present Weather Sensor Sodankylä Vaisala PWD 33 EPI Present Weather Sensor Sodankylä Vaisala PWD 52 Present Weather Sensor Sodankylä Yankee TPS3100 Hotplate Evaporative Plate Marshall, Haukeliseter, Sodankylä Table 2: Emerging technology instruments submitted by manufacturers (11 instruments in total). In total, 11 instruments, covering three different measuring principles (disdrometer, present weather sensor and evaporative plate), and allocated to four SPICE sites, representing various climate conditions, have been evaluated. They were installed according to manufacturer s requirements. It is to be noted that the Thies LPM was provided with a shield, both in Marshall and Weissfluhjoch. All instruments have been operated during two winter seasons (2013/14 and 2014/15), except the Hotplate in Haukeliseter, which was installed for the second season only. Operational considerations (e.g. installation, configuration, maintenance issues, etc.) have also been collected from the respective site managers, and were used for the assessment of the performance of each SUT. 4 Results A standardized Instrument Performance Report (IPR) has been produced for each of the seven different instrument type (according to Table 2), and will be available as an annex to the SPICE Final Report (soon to be published). It contains the evaluation of one SUT against the site reference. This chapter will give a summary of the main outcomes, instrument specific, and as an overall assessment for non catchment type instruments. 4.1 Reliability in detecting precipitation The reliability of SUT in detecting precipitation is assessed using skill scores, as defined in Chapter 2 above. A summary for all SUT is presented in Table 3. 5

POD [%] FAR [%] Thies LPM (Weissfluhjoch) 99.1 0.2 Thies LPM (Marshall) 100.0 43.2 Parsivel 2 (Sodankylä) 100.0 52.3 PWS100 (Haukeliseter) 87.2 0.0 PWS100 (Marshall) 97.3 4.1 FS11P (Sodankylä) 100.0 53.3 PWD52 (Sodankylä) 100.0 57.4 PWD33 (Sodankylä) 100.0 62.4 TPS Hotplate (Haukeliseter) 75.3 0.8 TPS Hotplate (Sodankylä) 100.0 82.9 TPS Hotplate (Marshall) 99.6 5.1 Table 3: Summary of skills scores for each SUT. POD: Probability Of Detection, FAR: False Alarm Rate. In bracket: SPICE host site. The POD ranges from 75 to 100%, which indicates fairly high reliability of the non catchment instruments in general to detect precipitation (independent from type and quantity). These instruments are usually more sensitive than traditional gauges (weighing and tipping bucket gauges), with lower detection threshold. The FAR varies from 0 (no false event reported) up to 82% (high probability that the SUT fails to recognize precipitation events according to the reference). The large differences in FAR among the SUT is to be underlined. It may be related to specific climate conditions from each site (e.g. all sensors tested in Sodankylä, except the TPS Hotplate, show the same order of FAR, around 50%), but the performance of the instrument has the most impact (e.g. the PWS100 show small FAR in both sites, Marshall and Haukeliseter). 4.2 Performance of SUT during no precipitation events The output signal of non catchment instruments during no precipitation events is usually a stable, noise free signal indicating 0 mm. This is an intrinsic feature of these instruments, where the output has already been processed internally. The consequence is that the threshold needed to be set to report precipitation adequately over an aggregated time step (typically 30 min) remains very low (0 to 0.1 mm/30 min if we want to reach the 3 STD). 4.3 Performance during precipitation events The assessment of the SUT in terms of reporting the correct accumulation during precipitation events can be summarized using RMSE calculation. The RMSE numbers for all SUT are presented in Table 4. 6

All [mm] Rain [mm] Mixed [mm] Snow [mm] Thies LPM (Weissfluhjoch) 0.483 0.248 0.505 0.486 Thies LPM (Marshall) 0.488 0.767 0.526 0.305 Parsivel 2 (Sodankylä) 0.208 0.075 0.192 0.241 PWS100 (Haukeliseter) 0.740 0.314 0.691 0.817 PWS100 (Marshall) 0.558 0.688 0.697 0.343 FS11P (Sodankylä) 0.146 0.137 0.157 0.133 PWD52 (Sodankylä) 0.138 0.133 0.149 0.124 PWD33 (Sodankylä) 0.363 0.176 0.485 0.143 TPS Hotplate (Haukeliseter) 0.333 0.409 0.360 0.306 TPS Hotplate (Sodankylä) 0.129 0.094 0.142 0.114 TPS Hotplate (Marshall) 0.232 0.344 0.283 0.121 Pluvio2 (all four sites) 0.1 0.45 0.0 0.2 0.05 0.4 0.1 0.5 Table 4: RMSE (Root Mean Square Error) in mm of precipitation related to the reference, for all noncatchment SUT, for all, rain, mixed, and snow events respectively. As a comparison, RMSE range for the Pluvio 2 weighing gauge from the four sites hosting non catchment type instruments (Haukeliseter, Marshall, Sodankylä and Weissfluhjoch) is indicated. The results show a large scatter across all instruments, and for each precipitation type, with no clear tendency. It was expected that the RMSE would generally be lower for rain than for snow, but some SUT show different behavior across different sites. As an example, the Thies LPM has a lower RMSE for snow than for rain at Marshall (0.305 mm and 0.767 mm, respectively), and the opposite is true in Weissfluhjoch (0.486 mm and 0.248 mm, respectively). The PWS100, the other SUT tested at two different sites, show the same pattern, with a higher RMSE for snow than for rain in Haukeliseter (0.817 mm and 0.314 mm, respectively) and the opposite in Marshall (0.343 mm and 0.688 mm, respectively). It is to be noticed that the number of rain events is generally low, and prevent for some sites to draw robust conclusions. Scatter in the 30 min events data may be a function of either site characteristics, or the SUT itself, or a combination of the two. Table 4 shows that all SUT located in Sodankylä (low wind conditions) have low RMSE, independently from the technology (three PWD Vaisala sensors, one Hotplate, and one Parsivel 2 ). An assessment of these sensors under high wind conditions, especially in terms of scatter, is necessary. The PWS100 and the Hotplate were both tested in Marshall and Haukeliseter. The RMSE ratio between these two sensors is of the same order for both sites, the Hotplate showing lower RMSE. This difference relates directly to the performance of the instrument. In order to fairly compare all RMSE, it should be calculated for wind speed up to 4 m/s (representing the wind maximum at Sodankylä). Higher RMSE for sites with higher winds is expected. This has to be taken into account when comparing RMSE from SUT located at different sites. 7

As a comparison, RMSE for unshielded and shielded (Single Alter) Pluvio 2 evaluated against the site reference (Pluvio 2 or Geonor) for the four sites hosting non catchment type instruments ranges from 0.0 to 0.2 mm for rain, 0.05 to 0.4 mm for mixed, and 0.1 to 0.5 mm for snow. As an example, the RMSE for Sodankylä, which hosted most of the non catchment type instruments, remains between 0 and 0.1 for all precipitation type. The catch ratio of SUT with respect to the reference has also been assessed as a function of wind speed. Unlike for weighing and tipping bucket gauges, where the catch ratio for snow and mixed precipitation decreases drastically with increasing wind speed, wind is expected not to have such a strong impact on non catchment type instruments. Some cases are presented below in Figure 3, as example, and a comprehensive results overview will be given during the oral presentation at TECO. 8

Figure 3: Boxplots based on 30 min YY events from the two seasons, representing the catch efficiency (CE) of SUT with respect to the corresponding site reference (SUT/Ref), against wind speed and discriminated by precipitation types for (top, left) the Thies LPM in Marshall, (top, right) the Parsivel 2 in Sodankylä, (center, left) the PWS100 in Haukeliseter, (center, right) the PWS100 in Marshall, (bottom, left) the PWD52 in Sodankylä, and (bottom, right) the Hotplate in Haukeliseter. The dashed black line at CE = 1 represents the ideal case. Note: X axis is identical for all 6 plots, Y axis vary with site. 9

Figure 3 shows that not all non catchment type instruments have similar behavior with respect to the influence of wind speed. The two box plots on the top represent the catch ratio as a function of wind speed for two disdrometers (Thies LPM in Marshall on the left, Parsivel 2 in Sodankylä on the right), showing opposite trends. The Thies LPM indicates a decrease of the catch ratio for snow and mixed precipitation with increasing wind speed. The mean catch ratio drops to 0.5 by winds at 4 m/s. The Parsivel 2 shows an increase of the catch ratio for snow and mixed precipitation with increasing wind speed (above 2 m/s), resulting in a clear overcatch from the SUT (mean CR around 2 by winds at 4 m/s). Note that the horizontal axis is not the same for both SUT, since Sodankylä has lower wind speed, with maximum around 4 m/s (10 m/s for Marshall). The two box plots in the center represent the catch ratio as a function of wind speed for the same instrument, PWS100, installed in Haukeliseter (left) and Marshall (right). A lot of snow and mixed events at Haukeliseter occurring under high winds (more than 6 m/s) resulted in a large overcatch, with CR for single events of 3 and more. For wind speed up to 6 m/s, the behavior of the SUT is similar for the two sites, showing a very large scatter below and above the ideal case of a CR equal to 1, with CR varying randomly between 0 and 2. There seems to be no relation with specific environmental conditions. Nevertheless, the mean CR is close to 1, indicating that this sensor seems to be a reliable instrument to account for the total accumulation over a longer period (e.g. one season). The two box plots on the bottom represent the catch ratio as a function of wind speed for two other non catchment type instruments, the Vaisala PWD52 in Sodankylä (left) and the TPS Hotplate in Haukeliseter (right). The mean catch ratio for snow events for the PWD52 is characterized by almost no trend with increasing wind speed (up to 4 m/s), staying around 1, and with a generally smaller scatter than for the other instruments above. The same trend is true for the Hotplate, up to wind speed at 14 m/s, but the scatter increasing at wind speed of 8 m/s and above. The scatter for wind speed up to 4 m/s is very similar for both instruments. 4.4 Assessment of YN, NY events The native resolution and sensitivity of present weather sensors and disdrometers are generally higher than for catchment instruments (traditional precipitation gauges). The non catchment type instruments are therefore suitable to detect light (or trace) precipitation events. As a result in the evaluation of the sensor, the number of YN cases are very low, i.e. cases where the non catchment type instrument would miss a precipitation event recorded as such by the reference. This is confirmed, with a number of YN cases for almost all the SUT ranging from 0 % (of the total Y cases from the reference) to 3 %. Only the PWS100 and the Hotplate, both in Haukelister, show higher percentage (12.8 % and 24.7 %, respectively). Haukeliseter being a windy site, these miss cases might be related to high wind speed conditions. The NY cases, i.e. when the reference is not reporting any precipitation and the SUT does (according to the thresholds defined in Chapter 2 above), vary from 0 % (of the total N cases from the reference) to 18 %, with most of the SUT being around 5 %. Due to the higher sensitivity of the non catchment type instruments already mentioned above, it might be possible that a certain number of NY cases are actually more a miss from the reference than a false alarm from the SUT. These cases need further in depth analysis. 10

5 Conclusion One objective of SPICE was to evaluate the ability for alternative technologies (i.e. other than traditional tipping bucket and weighing gauges) to be used operationally for snow measurement (accumulation). Several non catchment type instruments have been tested during two winter seasons within the SPICE field campaign. In total, 11 instruments from 7 different types were tested in 4 different SPICE sites. A standardized Instrument Performance Report (IPR) has been produced for each instrument, assessing its performance against the site reference. The ability of the SUT to detect precipitation according to the reference was assessed using metrics (contingency table, POD and FAR). The results showed a high POD for all SUT (100% for most of them), which confirms the generally higher sensitivity of disdrometers and present weather sensors than traditional precipitation gauges (weighing and tipping bucket gauges). The ability of the SUT to measure the correct amount of precipitation was also assessed, calculating the catch ratio of the SUT related to the site reference. The results, assessed with the RMSE, vary from one SUT to another, and from one site to another, making it difficult to give a general statement. Generally, RMSE tends to be higher for snow and mixed precipitation than for rain, but this is not always the case. The catch ratio was also calculated as function of wind speed, in order to understand the impact of winds on the quality of the SUT measurement. For weighing and tipping bucket gauges, a decrease of the catch ratio with increasing wind speed is expected. This relationship has not been fully analyzed for disdrometers or present weather sensors yet. The results showed all three tendencies, depending on the SUT, with decrease, increase or no changes in the catch ratio with increasing wind speed. This tends to show that the shape of the sensor (not identical for all SUT), but also their internal proprietary algorithm to convert the raw information into water quantity, is affecting this relationship in various manners. The generally large scatter showed when using the 30 min events tends to demonstrate that these sensors are usually not appropriate to measure snow accumulation over short interval (typically 30 min). But for some of them, the mean catch ratio was found to be acceptable (around 1), which indicates that these sensors might be used to measure precipitation accumulation over a longer period (e.g. one season). Further analysis is needed to better understand the behavior of these sensors, especially working with the raw data (drop size and fall speed distribution), and exploiting the full capacity of such sensors, which provide much more information than the precipitation accumulation (precipitation type, SYNOP and METAR code, etc.). Field tests on SPICE reference sites have been continued in that sense after the official end of the project, and will enhance the knowledge on the operational use of non catchment type instruments in winter conditions. Among others, data from disdrometers installed within a DFIR (the precipitation detectors that served as part of the FWRS, see Chapter 2) are being analyzed. Preliminary results have shown good agreement with the site reference (shielded weighing gauge in the DFIR) in terms of accumulated solid precipitation. This tends to confirm that the impact of wind speed on non catchment type instruments is relevant. 11

6 References Goodison B., Louie P.Y.T. and Yang D., 1998: WMO Solid Precipitation Measurement Intercomparison, Final Report, WMO IOM Report No. 67, WMO/TD No. 872. Nitu et al., 2016: WMO SPICE: Intercomparison of Instruments and methods for the measurement of Solid Precipitation and Snow on the Ground, Overall results and recommendations, Keynote 3A, TECO 2016, Madrid, Spain. 12