Community-based weather radar networking with BALTRAD

Transcription

1 Community-based weather radar networking with BALTRAD Daniel Michelson 1, Jan Szturc 2, Rashpal S. Gill 3, Markus Peura 4 1 Swedish Meteorological and Hydrological Institute, Norrköping, Sweden, daniel.michelson@smhi.se 2Institute of Meteorology and Water Management, Katowice, Poland, jan.szturc@imgw.pl 3 Danish Meteorological Institute, Copenhagen, Denmark, rsg@dmi.dk 4 Finnish Meteorological Institute, Helsinki, Finland, markus.peura@fmi.fi Daniel Michelson 1. Introduction Eight partners in seven countries are currently developing the latest-generation weather radar network. The project is called An advanced weather radar network for the Baltic Sea Region BALTRAD, and it is funded in part by the European Union from the Baltic Sea Region Programme (BSR). To our knowledge, BALTRAD is the first project funded with European money explicitly focussed on creating a real-time international weather radar network. There is no funding for radar hardware; instead the project is designed to integrate existing radars in the region to deliver an element of ICT regional infrastructure in support of transport and other sectors of society. In short, we are building a technical platform whereby polar data are exchanged in real time and they are processed locally in each country according to local needs using a commonly developed collection of algorithms. End-to-end treatment of quality information will be included. The technology we create will be freely available, which will allow the community to grow. This paper presents the project from a non-technical perspective. Other papers in this volume present the BALTRAD technical platform (Henja et al., 2010), an advanced compositing algorithm (Peura, 2010), dual polarization based hydrometeor classification (Gill et al., 2010), and treatment of data quality (Ośródka et al., 2010). 2. Heritage and boundary conditions NORDRAD was the first international network of Doppler weather radars. Its creation was financed by the Nordic Council of Ministers, and it got underway in the mid-1980 s. When complete, this network comprised radars in Norway, Sweden, and Finland. Data from radars in Denmark, Estonia, and Latvia were integrated later as the network grew. This network pioneered technology required to exchange non-trivial environmental datasets in real time. From the start, NORDRAD was developed as a decentralized network, where data were exchanged on equal terms and processed locally according to local needs. These same concepts continued when the second generation system was developed in the early 2000 s. In the mid-1990 s, the Baltic Sea Experiment (BALTEX) established a working group on radar, and this led to the establishment of a radar data centre where datasets were generated using data from NORDRAD and additional radars in Poland, Germany, and the Netherlands. The network of radars created for BALTEX is the original BALTRAD, and we have recycled this acronym for our current project. Research and development produced new and improved algorithms that were used there, and also integrated into the second generation NORDRAD system operationally. Meanwhile, a new group called the NORDMET Activity on Radar Applications was established to compliment the operational NORDRAD network with the development of new algorithms for operational deployment, e.g. beam blockage identification and correction. The heritage of international cooperation in weather radar is rich. However, the technology we have been using in NORDRAD is now obsolete. Cartesian products have been exchanged, and this has several disadvantages when creating composites and higher-order products based on them such as accumulated precipitation. Fortunately, the Baltic Sea Region Programme commenced in 2007, and we submitted a proposal during its first call in We updated the NORDRAD concept and adapted it to the BSR s priorities. By demonstrating that the NORDRAD cooperation has grown and developed into a durable and sustainable element of regional infrastructure, we have also shown the high potential to achieve the same with BALTRAD. Our proposal was approved in October BALTRAD s partnership and the project's boundary conditions are summarized in Tab. 1. Funding originates from two sources within the BSR, the European Regional Development Fund (EU countries), and the European Neighbourhood and Partnership Instrument (Belarus). Co-funding rates, i.e. the proportion of the project co-funded

2 from the BSR, are 75% for Sweden, Denmark and Finland, 85% for Estonia, Latvia, and Poland, and 90% for Belarus. Our original proposal included a partner from Russia, but unfortunately the EU and Russia failed to sign a cooperation agreement during 2008, and this means Russia was unable to participate as a partner in any BSR project. However, we are keen on working with colleagues in Russia, along with additional countries in the BSR that are not project partners: Germany, Norway, and Lithuania. Table 1. BALTRAD boundary conditions at a glance. Countries Sweden, Finland, Estonia, Latvia, Belarus, Poland, Denmark Partners Swedish Meteorological and Hydrological Institute (Lead Partner) Finnish Meteorological Institute Finnish Radiation and Nuclear Safety Authority Estonian Meteorological and Hydrological Institute Latvian Environment, Geology and Meteorology Centre Republican Hydrometeorogical Centre (Belarus) Institute of Meteorology and Water Management (Poland) Danish Meteorological Institute Start 1 February 2009 End 31 January 2012 Budget 2.1 Million for an ICT project (no money for new radars) Financed by Baltic Sea Region (European Union) Work 26.9 full-time equivalent years + external services The single most important condition in BALTRAD is that we are creating our new network ourselves, as a distributed development team. This means that knowhow distributed throughout the partnership is collected into a critical mass, prudent organization allowed the individual partners to contribute what they do best, knowhow is further built and kept within the partner organizations, and regional imbalances are mitigated within the partnership. Since we are building the system ourselves, the partnership stands to continue and even grow after the project's lifetime. 3. Project set-up and system description BALTRAD, like all BSR projects, is limited to seven work packages, where the first two are mandatory and deal with administration and communications. Work packages 3-6 are technical in nature, and work package 7 deals with our interaction with end users, the people who are supposed to benefit from our outputs. The general relation among these work packages is shown in Fig. 1. The realization of the technical platform is presented in Henja et al (2010). FIG. 1. General overview of BALTRAD work package and their tasks.

3 We are creating all the functionality required to create a complete network solution. The technology itself is rather generic, with a focus on applying marketable skills in our work. This strategy attempts to minimize the negative impact caused by the loss of personnel, since the same skills should be more readily available on the job market. We also believe this strategy is more attractive in general, because it means the development team works with up-to-date methods which gives results that should be more easily understood by more people, thereby demystifying the whole process. 3.1 Data exchange BALTRAD must exchange data, and we are doing so using the basic polar (spherical coordinates) data delivered by each radar. The software for doing so is a data exchange software component built in Java using the Spring Framework and the Tomcat server. Standard HTTP is used with regular POST and GET semantics. The BALTRAD messaging system has formulated a number of messages that are unique to the project. A BALTRAD message consists of an XML envelope and variable payload content depending on the message type. System components communicate using these messages coordinated by the messaging system. We have implemented a concept of adapters and routes in our messaging system. An adapter contains specific functionality, e.g. a communications protocol or a monitoring function. A route contains the criteria that need to be fulfilled in order to trigger a given adapter. Using these two generic concepts, new adapters and routes can be easily added. 3.2 Data catalogue All radar data exchanged in BALTRAD are in the HDF5 format, organized according to the OPERA Data Information Model (ODIM_H5) (Michelson et al., 2009) which is the European standard. The first step in the data flow is that data from each national radar network must be harmonized to ODIM_H5. Converter software has been created to facilitate this, since we recognize there are radars from several different manufacturers in the project and each partner's environment is unique. Once data have been converted, they are ingested into the BALTRAD system using the data exchange component. All data are managed by a component called the BALTRAD database. In practice, this is a combination of PostgreSQL server and a file system. Metadata attributes are read from ODIM_H5 files and stored in the database, while the ODIM_H5 files themselves are stored in the file system. This has two major advantages: 1. The database is not encumbered by data arrays unnecessarily. 2. Researchers and developers can access the data files independently of the rest of the system, for offline use. A C++ interface is written to access the database functionality, and this is in turn integrated with the other system components. The advantage with this is that application developers don't have to know anything about SQL or the inner workings of the database. They use the interface to interact with it at a high level of abstraction. 3.3 Product generation and management of quality information When we have system exchanging and managing data, we then need functionality for deriving value-added information from them: so-called products. For these purposes, we have a loosely integrated production framework that runs as a separate server. The BALTRAD node interacts with it through the messaging system with a unique production message, similarly to other system components. However, scheduling functionality remains within the messaging system. When products are generated, the results are transmitted back to the exchange component in a BALTRAD message where further action may be taken on them. An initial catalogue of algorithms available in the partnership has resulted in the following algorithms that are relevant for implementation during the project's lifetime: 1. Beam blockage diagnosis and correction 2. Anomaly detection and removal 3. Dual-polarization based attenuation correction 4. Dual-polarization based hydrometeor classification (Gill et al., 2010) 5. Networked VPR correction 6. Networked gauge adjustment

4 7. Corrected gauge observations, for gauge adjustment 8. Slow compositing of polar data, including management of quality indicators (reference algorithm) 9. Quality-based injective compositing of polar data (Peura, 2010) 10. Satellite-based identification and removal of non-precipitation 11. Sun monitoring of calibration levels 12. Super-observation Most of these algorithms have been documented and presented in the literature and at various events. Since this compilation of algorithms was conducted, two additional algorithms have been contributed (maximum detection range, and vertical profile product containing reflectivity and de-aliased winds, among other things). The starting end of our system is when polar data are ingested to it. The ending point is when products are made available to end users. We aim at including end-to-end management of quality information, with quality control and characterization procedures, such that the quality of any polar bin or Cartesian pixel will be characterized by one or more quality indicators. In turn, these will be used when generating higher-level products such as composites. The way this is being addressed is presented in Ośródka et al. (2010). We also intend to introduce functionality for visualizing the data quality that is output by our system. Doing so not only provides us with tools to monitor our system, but also the ability to communicate information on data quality to users who can benefit from such knowledge in their decision-making. 3.4 Deployment A so-called BALTRAD node will be operated in each country, exchanging data with the others and generating products for its purposes. To achieve this in a systematic and controlled manner, the software components must integrate into regular releases; this is achieved using central repository, continuous integration, and issue management facilities. This is a critical step in order to achieve an infrastructure that is durable and sustainable after the project s lifetime. 3.5 Licensing According to European Union policy, all software will be made available according to Open Source principles. In our case, we have decided to apply the Gnu Lesser General Public License. This means you are allowed to use our software for free, and preferably also contribute to it. This policy is another cornerstone to enable the radar networking community to outlive the project itself and expand organically. 4. Pilots and the Application Case Log The BALTRAD final aim is to provide end users of weather radar data with information in the form that is the most suitable for their needs and expectations. Since source radar data generated as 3D polar volumes are informative due to possibilities of generation very diversified 2D products, various requirements can be met. In the frame of BALTRAD, a survey was performed on the basis of a questionnaire and correspondence with potential end-users. As a result, a catalogue of the potential end-user groups has been compiled (Tab. 2). Planned BALTRAD enduser workshop (in October 2010) and further closer collaboration will be a great opportunity to find out how to generate radar-based product as useful as possible. The particular end-user groups express their interest in quite different kinds of products. For example, very specific and specialist products are expected by biologists and ecologists, like detection and tracing some biological objects such as bird flocks or insect swarms. However, the data can be very useful first of all in some branches of industry and economy. The biggest advantage of radar data is almost real-time delivery of precise (with high spatial resolution) information about weather phenomena, especially dangerous ones. In many cases available techniques of nowcasting (i.e. very short-term forecasting) constitute significant opportunity for variety of applications. Radarbased information is important e.g. for flood protection, but also for air traffic control, local rescue services (in the cases of fire, flood, radiation, etc.), power and hydropower industry, urban water management, agriculture, ecology etc. Moreover, graphical presentation of the data is useful for educational purposes and in public weather broadcasts.

5 Group of endusers Hydrologists (flood protection) Meteorologists (numerical weather prediction) Table 2. Identified groups of weather radar data end users and their expectations. Radar-based meteorological fields Surface precipitation Radar reflectivity Data format Numerical: as input to rainfall-runoff model Numerical: as input to NWP model Other required meteorological data Meteorological data from various sources Air traffic control All hazardous phenomena Graphical Lightning Yes Local rescue Surface precipitation and services (fire, flood, Graphical Yes wind etc.) Road and railway control and protection Surface precipitation (snow) Graphical Temperature Yes Radiation Precipitation and wind Graphical Yes Power industry Surface precipitation Graphical Lightning, temperature Yes Hydropower industry Urban water management Agriculture Ecology Surface precipitation Numerical Surface precipitation Numerical Yes Surface precipitation Surface precipitation and wind Graphical, numerical Graphical, numerical Temperature Meteorological data from various sources Research community All All All No Education All Graphical All No Mass media Surface precipitation and wind Graphical Clouds, temperature No Radar-based forecasts Yes No No (too small time-scale) No (too small time-scale) Pilot end users will access products generated by the BALTRAD system, and their feedback will be incorporated into the development cycle, as illustrated in Fig. 1. In this way, the sectors of society that stand the most to gain from BALTRAD will actively contribute to ensure that this happens. This is consistent with the goals of the BSR, that BALTRAD contributes an element of regional infrastructure that supports transport, hydrology, and other sectors of society. 5. Outlook Data exchange will be underway before the project's halfway mark, which is 31 July Starting in August 2010, the production chain will be introduced successively, and products will become available to project pilots for their feedback. More information can be found at the project s website, There is also a BALTRAD community on Facebook that you are welcome to join, along with the real BALTRAD networking community. Acknowledgment BALTRAD is financed in part by the European Union (European Regional Development Fund and European Neighbourhood and Partnership Instrument). References Gill R.S., Boevith T., Soerensen M.B., Overgaard S., and Michelson D.B., 2010: Development and evaluation of a BALTRAD dual polarization hydrometeor classifier. (this event). Henja A. Szewczykowski M., Ernes S., and Michelson D.B., 2010: The BALTRAD technical platform. Proc. ERAD 2010 (this event). No

6 Michelson D.B., Lewandowski R., Szewczykowski M., and Beekhuis H., 2009: EUMETNET OPERA weather radar information model for implementation with the HDF5 file format. OPERA Working Document WD_2008_03, Version 2.0. Peura M., 2010: The living composite. Proc. ERAD 2010 (this event). Ośródka K., Szturc J., Jurczyk A., Michelson D.B, Haase G., and Peura M., 2010: Data quality in the BALTRAD processing chain. Proc. ERAD 2010 (this event).