The Upper Air Database in the Research Data Archive at NCAR INTRODUCTION...1 UPPER AIR DATASET DESCRIPTIONS...3 REFORMATTING AND STANDARDIZATION...

Transcription

1 The Upper Air Database in the Research Data Archive at NCAR May 27, Contents INTRODUCTION...1 UPPER AIR DATASET DESCRIPTIONS...3 REFORMATTING AND STANDARDIZATION...7 COMPOSITING WMO STATIONS UADB COMPOSITED DATA COMBINING STATIONS WITH DIFFERENT WMO IDENTIFIERS UADB COMBINED DATA APPENDIX A MOISTURE CONVERSION APPENDIX B - ASCII FORMAT FOR THE NCAR UPPER AIR DATABASE BIBLIOGRAPHY Introduction For more than 40 years upper air sounding data have been collected and archived in the Research Data Archive (RDA) at NCAR. In the RDA there are nearly 50 different datasets that in total have comprehensive global coverage. The datasets are diverse, with various data formats, different measurement units and precision, varying amounts of metadata and documentation inconsistent quality assurance and control and much spatial and temporal overlap. These data characteristics impede users from easily accessing the valuable information contained in them. The Upper Air Database Project (UADB) was designed to improve user access by providing all the RDA sources from a convenient single access point with uniformly formatted and well- 1

2 documented upper air data, with duplicate soundings removed and station time series extended to their full length across source datasets. There exist many challenges in consolidating the upper air data within the RDA. In order to begin addressing the problem, we must first understand the characteristics and conditions necessary to maintain and access the database. The UADB is not a static archive. With time, data corrections, additions, and improvements will be necessary. Therefore, the data management system must be flexible and the data processing workflow repeatable. Relational analysis across and within datasets must be accurate, fast, and easy to perform. Users need rapid access to the data, either in full time series by station or through spatial and temporal sub-setting. User needs are diverse. It is important to provide a wide range of access options. This includes various ways to select subsets, apply data quality filtering, choose output formats, enable machine interoperable protocols, and perform simple file downloads. In order to satisfy these conditions, data is managed within a relational database (MySQL). In the UADB, there are several instantiations, also called versions, of the data. UADB Version 1 (UADB-V1) data holds the original data. UADB-V1 faithfully represents as much data as possible directly from the source datasets. UADB Version 2 (UADB-V2) is UADB-V1 transformed such that the data are uniformly formatted, the metadata are standardized, and the measurement units are consistent. Higher-level products, such as multi-source time series with duplicates eliminated, are derived from UADB-V2 and managed as UADB Version 3 (UADB-V3). For experienced users that wish to manipulate and explore the data with their own self-designed strategies, data output is extracted (upon request) from numerous points along the UADB data processing workflow. 2

3 Upper Air Dataset Descriptions Upper air data has arrived at NCAR from more than 50 different institutions, scientists, and countries. For the last 40 years, the RDA staff has done extensive stewardship and maintenance work on these collections. This work has resulted in the availability of over 85 million soundings that cumulatively have extensive global coverage. The temporal coverage is vast, with many station records beginning in 1948 and some records as far back as Modern date additions lag real-time by one or two months. Sounding data is crucial for many research topics. In particular, soundings are an essential resource in reanalysis projects. At NCAR, all soundings have been assimilated as a primary source of in situ upper air data for reanalyses including the NCEP-NCAR Reanalysis, ECMWF ERA-40 Reanalysis, and the NCEP North America Regional Reanalysis. Subsequently, they have also been used for reanalyses by JMA, NASA, and further reused by NCEP and ECMWF. The extensive UADB archive exhibits large variations between the source datasets. Differences can be found in time periods, spatial domains, and types of soundings. Time periods covered ranges from as short as a few months to more than 20 years. The spatial coverage of a dataset can be as fine as a single station or as large as all available stations for the globe. Parameters that are measured in a sounding profile can also vary. The measurements can be wind only, temperature with or without moisture, and a complete modern profile with wind, temperature, and moisture. Source datasets may contain any combination of these sounding types. Table 1 summarizes the UADB by source name, geographic region, period of record, station and sounding counts, and number of soundings segregated by type. The UADB has a significant amount of spatial and temporal overlap (exact or partial duplication) between the datasets, because the data originates from the same basic global upper air station network. In the period prior to 1960 most upper air data was archived at the National Climatic Data Center, either on magnetic tapes or punch cards. Note that the TD in several dataset names (Table 1) stands for tape deck and the number represents a unique tape set. Starting in the early 1960s, soundings were broadcast on the Global Telecommunication System (GTS). Data sharing was initially limited, because equipment to receive the satellite GTS broadcasts was generally only installed at government facilities. This problem persisted until the 1980 s at which time Internet access became prevalent. The problem was compounded, because computer storage media was expensive so many institutions only archived data for the specific region or time period. Due to the existence of multiple datasets from the same network of stations, many duplicate soundings exist in the cumulative collection. In the early 1960 s, a sounding for a given station and time may be found in as many as five or six different source datasets. After the 1960 s, the 3

4 number of source datasets decreases as does the number of duplicates. Of the 85 million soundings in the UADB, approximately 45 million are considered to be unique. It is also the case that in the early GTS period there are gaps in both spatial and chronological coverage. In fact, no one dataset has the full collection of soundings for every station. Over time, as the GTS and access to it improved, gaps became less frequent and are now rare. This is why in the later periods (1980 s and onward), fewer datasets are required for full temporal and spatial coverage. Some countries archive higher quality soundings for their regional stations than what are broadcast on the GTS. When data sharing is possible, these are included in the RDA and added to the UADB. Another challenge in the UADB is data quality. Several of the source datasets have data quality problems e.g. the MIT data contains imprecise moisture measurements, while the United States Air Force data (USAF) has personal communication metadata indicating there are several probable data quality issues, like incorrect values being archived and soundings being jumbled on transmission. Both of these datasets are important, but have been quarantined, at least until these problems can be researched and resolved. Nevertheless, these data sources are still included in Table 1, because they are relevant to global sounding history and will be included in future UADB releases. Though it is important that the UADB upper air datasets span large geographic regions and time periods, it is also necessary to include smaller and higher resolution field projects. Currently in the UADB, only a few specific field projects, like the Global Atmospheric Research Program (GARP) are included. As time permits more field project data will be added to the UADB. A good source for upper air data from field projects can be found at the Earth Observing Laboratory at NCAR ( Table 1 Upper air dataset sources from the NCAR Research Data Archive (RDA) included in the Upper Air Database (UADB). The data sources are described by name and UADB Version 2 Source Index, geographic region (if known), period of record, station and sounding count, and inclusive profile measured variables. The total sounding count is further segregated in three different types: 1) those that have measurements of temperature, moisture (nominally relative humidity), wind speed and wind direction, 2) those that have temperature and moisture, and 2) those that have only measurements of wind speed and direction. The USAF and MIT data sources, shown in grey, are not currently including in the UADB developed data products. Data Source Name UADB- V2 Source Index Geographic Region Period of Record Station Count Sounding Count Variables T + RH + WsWd Variables T + RH Only Variables WsWd Only NMC B3 1 Global USAF 2 Global C-CARDS 3 N. Hemisphere No U.S

5 TD390 4 Coastal Stations No U.S. SHERHAG 5 Europe, Middle East, Greenland TD53 6 U.S. and U.S Control TD54 7 Global Data from 8 Dominic Dominic Test NCAR key entered June GARP CARDS U.S U.S. Control 12 U.S. and U.S. Controlled Regions TD52 13 Global MIT 20 North Hemisphere Navy Kunia 21 Global Navy Spot 22 Global China 23 China Permanent 24 Atlantic Ships Pacific Permanent 25 Atlantic Ships Pacific Countries NCEP ADP 27 Global ,812, NCEP BUFR 28 Global ,478, Continuing NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck

6 NCDC Deck NCDC Deck NCDC Deck Canada 100 Canada French 110 France French 111 France South 120 S. Africa African South 121 S. Africa African Brazil 130 Brazil New 140 New Zealand Zealand Russia 150 U.S.S.R Russian Ships Australian 160 Australia United Kingdom Singapore 180 Singapore India 190 India Japan 200 Japan Argentina 210 Argentina

7 Reformatting and Standardization The collection of upper air datasets in the RDA presents a challenge because there are many datasets from numerous different data providers that have been received in a period spanning more than 40 years. Furthermore, modern data are added monthly and occasionally new historical datasets are discovered or prepared and must also be included. As expected this situation assures a diverse archive with varied formats, dissimilar measurement units, different variables sets, and non-standard descriptive metadata. This dataset configuration is not user friendly; so one primary goal for the UADB project is to transform the soundings from all the sources into an archive with consistent measurement units and variables, and standardized metadata. A MySQL database (DB) was chosen to manage this diversity and yet maintain flexibility and the potential for straightforward research and reprocessing. The first step was to ingest the original source datasets into the DB. This is referred to as UADB Version 1 (UADB-V1). In order to maximize reproducibility, all the data from each source were ingested into UADB-V1 without any conversions or metadata changes. Each dataset is stored in independent DB tables, as is necessary to accommodate the diversity between the sources. Thus, UADB-V1 represents an exact copy, to the highest degree possible, of the original data source. The second version (UADB-V2) of the sounding data is created in data processing steps that standardize the reported measurement units, variable types, and metadata into a consistent structure or format. UADB-V1 remains unchanged and each record is transformed into UADB- V2. Internal in the DB, the provenance of each record between versions is maintained, so if systematic data anomalies are discovered in UADB-V2 they can be easily investigated and reprocessed. Metadata Standardization A large diversity can be observed in the sounding metadata across the UADB-V1 sources. These metadata have been quality checked, consolidated into the vital elements, and additional metadata fields have been created. This provides concise uniform metadata for each sounding that includes: Station Identification Type defines Station Identification to be a WMO or WBAN number, or some other originators identification. Location Flag labels the station latitude and longitude source. Source Identification defines the original RDA source dataset. Sounding Type defines the sounding variable characteristics, e.g. a wind only profile or one that contains temperature, moisture and winds. 7

8 Level Type defines every level in a sounding as to whether it is a surface, mandatory, significant, or other type of level. Year, Month, Day, Hour Four digit year, two digit month, two digit day, and four digit hour where the first two digits are the hour and the last two digits are the minutes. Latitude, Longitude defines the station position in degrees to a maximum accuracy of hundredths, in a reference frame 0 to 360 Eastward and -90 South to 90 North. UADB-V2 and Sub-version Number defines the data first transformed from UADB-V1 to UADB-V2 and sub-version zero (i.e. 2.0) and any substantial changes in UADB-V2 incrementally as 2.n. Unique Sounding Identifier - defines a permanent unique identifier to every sounding. Station latitude, longitude, and elevation are important station metadata. In support of several global atmospheric reanalysis projects, extensive work was performed to correct and improve the station location and elevation libraries. The most current reanalysis library was constructed in 2002 and covers the full period of record up until that date. This library has been used to enhance the station metadata in the UADB. For later years, the station location and elevation are those assigned by NCEP. The sounding metadata are more fully described in the data format documentation (Appendix B). Unit and Variable Conversion In most UADB-V1 soundings, measurement units are fairly consistent. Geopotential height is measured in meters, temperature is measured in degrees Celsius, and wind direction is measured in whole degrees, on a 0 to 360 scale. The moisture data are less consistent and can be recorded as relative humidity, specific humidity, or dew point temperature. Wind speed are recorded in either knots or meters per second. Moisture Standardization Relative humidity (RH) is the most commonly recorded moisture variable in UADB-V1. Thus, specific humidity (q) and dew point temperature (Td) are converted to RH during the standardization process. These conversions are dependent on in situ atmospheric temperature and pressure, and rely on the algorithmic relationship between atmospheric water vapor and pressure. Appendix A gives the algorithmic details for these conversions. 8

9 Wind Speed Standardization Meters per second is the most commonly recorded measurement of wind speed in the UADB-V1, so wind speeds that were recorded in knots are transformed into units of meters per second using the relationship 1 knot is equivalent to ms -1. UADB Version 2 UADB-V2 is the collection of all data sources that have been homogenized into uniform structured database tables, or equivalently can be written into files with a consistent data format. The first data transformation from UADB-V1 to UADB-V2 is assigned a sub-version number of zero, i.e This first translation along with the addition of new metadata, unit and variable conversions, as described above, is generally sufficient for the homogenization process. Any further corrections or improvements that become necessary in the metadata are handled as new sub-versions. This technical implementation gives the UADB Project a flexible testing and evaluation environment, as well as the ability to reproduce various steps if problems arise or better algorithms become available. The UADB-V2 collection (at the highest sub-version status) is the starting point for building composites from multiple data sources, eliminating the duplicates from overlapping time periods, and creating longer time series at stations that have matching station identification, e.g. identical WMO station numbers. 9

10 Compositing WMO stations There are many RDA upper air sounding data sources that have spatial and temporal overlaps. This is due to data originating from the same basic global observing network. In the early era of recording upper air data, there was no well-coordinated global data collection center. Occasionally, scientific teams created collections for specific research, military organization made independent archives, and countries or groups of countries organized data for regional areas. Under these conditions it was inevitable that duplicate or near-duplicate soundings occur in more than one data source. For a given station as many as six duplicate soundings can be found across all the different original sources. Duplicates occur most frequently in the early 1960 s. This occurrence slowly decreases through the 1970 s and by the mid-1970 s the total number of overlapping sources reduces to a few. Thus, duplicate observations have also greatly decreased. Currently, there are few duplicates within any single data source. A primary goal for the UADB Project has been to provide access to an archive with long station time series where the duplicates have been eliminated. The data processing that brings together stations with matching identification (i.e. geographic location and elevation or station identifier) from different sources and removes duplicate soundings is called compositing. This process has also been called station time series merging by other data development teams. Removing duplicates is not a simple task. RDA data source analysis shows that many duplicates are not numerically exact. They can contain small precision differences in the recorded quantities (e.g. 0.1 to 0.2 C for temperature and 0.1 to 0.2 ms -1 for wind speed). In most data sources relative humidity is recorded in whole percentage increments and wind direction in whole compass degrees, thus fewer discrepancies exist for these quantities. These differences appear most often in the period from 1960 to 1970 and are related to ad hoc data management practices. During this period, atmospheric science was first introducing computing systems and digital storage as a practice. At times insufficient care was given to preserving numerical accuracy and precision. This left some observations inadvertently truncated. In addition, a standard measurement unit conversion did not exist, and various groups used separate algorithms that could give different results, especially at extremes in the data range. Another challenge stems from the vertical profiles. Duplicate soundings may differ by the amount of data reported in the vertical. This difference can be as small as one or two variables missing at a few levels, or as large as the exclusion of all the significant levels with the sounding only having the mandatory levels. 10

11 Poor historical data management practices have greatly contributed to the observed differences between the RDA data sources. This has become common knowledge at all data centers that attempt to construct useful composites from old atmospheric and oceanic observational datasets. Constructing a best set of data processing rules for eliminating duplicates between sources poses a serious challenge, and is an ongoing process at many institutions. With over 50 source datasets and 85 million soundings, the UADB archive is complex. As an example, in some data sources, there are stations with long time periods that have no data gaps while other similar stations have one or multiple gaps of over 6 months. Often, these gaps are not found in other data sources where the period of record may be shorter. Few singular data sources contain every existing sounding for a particular station, especially in the period before Further, the differences in measured values and extant data amounts do not follow any easily defined patterns, so it is difficult to exclude any specific sources. Excluding a particular source could also eliminate the only dataset in which some stations are available. Successful compositing is achieved by data processing rules that remove the duplicates and preserve as much extant data as possible. Further, since there may be numerical differences in the values between sources, care must be given to minimizing the total number of data segments integrated together from different sources when creating a times series for a single station. Additional research will be required to better understand the data quality impacts at the interfaces where compositing has joined different data sources. Effective compositing requires a balance between minimizing the number of changes in source datasets along the time series and maximizing the amount of data to be included. Several different methods were experimentally tested before one was chosen. Test method 1. In this method, station time series are created by successively adding data month-by-month from the original sources ranked by the total number of soundings for the month. Priority is given to the source with the highest number of soundings within the month. This method was rejected due to the number of source dataset changes along a station time series. This number was typically between 7-10 changes but could be as high as Test method 2. In this method, station time series are created using a process similar to method 1. However, an additional source ranking is included. This ranking is based on the recorded precision of temperature and wind speed. Analysis of the source datasets indicated that the precision of temperature and wind speed fell into three categories: 1) all measurements were recorded to a precision in tenths, 2) measurements were in tenths, but with unrealistic distribution, e.g. only n.5 or only n.1, n.3, n.5, etc., and 3) 11

12 measurements recorded only as whole units, no tenths. The numerical precision was consistent within a source. The compositing process was again based on a month-bymonth evaluation. First datasets with the highest precision (category 1) were considered. From that group the data source with the most soundings for the month was selected and added to the time series. If no category 1 data sources were available the same process was applied for category 2 sources and subsequently category 3, if necessary. In most cases, the temperature and wind recorded precision was the same, however, when they were not, temperature precision was used as the default. This method was rejected, because there was not a substantial improvement over method 1 and in fact, the amount of data in the final time series decreased for many stations. Test method 3. This is similar to method 2, but utilized an evaluation period of one year rather than one month. This reduced the number of source dataset changes along the time series, but also reduced the total number of soundings. As a result, this method was rejected, because between 100 and 1000 soundings were excluded from many station composited time series. Compositing method implemented. The method used for compositing long station times is based on what was learned in the test experiments and a time series gap filling procedure. The station time series was created by successively adding data year-byyear from the sources ranked by the total number of soundings for the year in question. Calendar months with no data were then filled with data from the remaining sources using the same ranking system based on the target month. With the mean number of source dataset changes between four and six per station, this procedure provides a reasonable balance between minimizing the number of changes in source datasets along the time series and maximizing the amount of data that is included. The long station time series dataset created from the compositing procedure is a data product directly derived from the UADB-V2 sources. We distinguish it as a UADB Version 3 (UADB-V3) product. Certainly, as the UADB-V3 is studied and evaluated by users much will be learned about the time series data characteristics. We expect and welcome users questions, feedback, and insights on the data quality and how the compositing procedure can be improved. The UADB system has a flexible design; new data sources can be quickly added to UADB-V1 and standardized into UADB-V2, there is easy accessibility to all UADB-V2 sources that makes further analysis of them straightforward, and finally improved compositing schemes can be rapidly tested and implemented to create new UADB-V3 datasets. Further, all through this data workflow the provenance of each sounding is preserved. Over time, this permits the global data collection, managed in the UADB, to progress toward higher quality and greater completeness. 12

13 UADB Composited Data Upper air stations launch soundings which provide measurements of temperature, moisture, wind, and pressure or simply wind-only. These measurements are pilot balloons (pibals) tracked optically by theodolite, or rawindsondes tracked by radio signal positioning. In the period before 1973 it was sometimes the practice to only archive the parts of the sounding that were needed. In some instances, the wind data or temperature and moisture data were separated into independent soundings, with one or the other excluded from the archive. The fragmented situation is made even more difficult, because the documentation about data practices is not available and there is no guarantee all the individual sounding pieces have been recovered. Optimally, all available pieces from each sounding would be recombined to create a complete sounding. The complexities in doing so are many and there is risk of introducing more uncertainty and errors. In order to minimize this error, composting is based on the number of soundings. However, due to the significant number of wind-only profiles and the nature of the compositing scheme, it is not possible to create the most data complete time series if all the profiles are treated equally. Thus, two products are created in order to maximize the amount of data accessible to users. The first consists of soundings that contain temperature or moisture data at any level, named UADB-TRH, and the second contains wind data at any level, named UADB-Wind. Compositing rules were adjusted to be product specific and the data processing was carried out independently. Note that all measured values from a qualified profile were carried forward into the product time series. This means soundings in the UADB-TRH product could also have wind measures, if wind observations were part of the archived sounding, and profiles in UADB-Wind will also occasionally have temperature and moisture values. Soundings with temperature, moisture, and wind (typical for modern data) will appear in both UADB-TRH and UADB-Wind products. The first UADB-TRH station time series begin in 1943 with approximately 5 stations (Figure 1). There is a steady increase in the annual station count through the mid-1980 s, where it reaches a maximum of approximately 1150, followed by a decrease to around 850 in the present days. The brief increase in station count from can be attributed to the NCEP BUFR data source. Other variations in station count are also the direct result of short-term irregular additions from the many source datasets. As expected the number of stations in the UADB-TRH product decreases as the time series length increases (Figure 2). There are nearly 1800 stations that report for at least 5 years, nearly 1100 report for 20 years or more, and 400 stations report data for at least 50 years. In addition, the number of UADB-TRH stations contributed by each source dataset varies greatly (Figure 3). Many data sources are country specific and provide 13

14 Figure 1. Annual station count for the NCAR composited Temperature-Moisture Database (UADB-TRH) product. The maximum number of stations, approximately 1,100, exist during the 1980's. Later the number of stations slowly decreases to present day levels of approximately 850 stations. Figure 2. Station period of record in the NCAR Composited Upper Air Temperature- Moisture Database (UADB-TRH) product. The station totals are cumulative starting at the longest period (70 years) through the shortest period (5 years). All stations in the 70- year interval are a subset and added into the shorter intervals. There are about 2,000 unique stations with almost 1,800 that reported for at least 5 years and about 400 that reported for 50 years and longer. fewer than 25 stations, like the Japanese (JAPNR), Argentinian (ARGNR), and French (FRCHR) datasets. The datasets with global coverage contribute the most stations, with NCEP BUFR (NBUFR) and NCEP ADP 14

15 (NCADP) providing more than 800 stations each, and NMC B3 (NMCB3) adding more than 500 stations. Overall, there are 34 source datasets that contribute to UADB-TRH product. The UADB-Wind product can be summarized in a similar way. The annual station counts begin Figure 3. Number of stations contributed by each source dataset in the NCAR Composited Upper Air Temperature-Moisture Database (UADB-TRH) product. Data sources are shown on both right and left vertical axes (see Table 1 for source details). The color bar scale interval is 25 stations in the range, and then 100 stations per interval up to 1,000. Most datasets contain fewer than 25 stations, while the datasets with global coverage, like NBUFR and NCADP provide over 800 stations and NMCB3 provides over 500 stations each. in 1922 with three stations and reaches 100 stations near 1950 (Figure 4). By 1970 the number of stations is more than 1600 station. Then the station count begins decreasing in the 1980 s until it reaches about 1000 stations in Unlike UADB-TRH, the number of UADB-Wind stations continues to steadily decline, because wind-only soundings less frequently used in the modern period.. As with UADB-TRH, irregularities in station count over time, such as the drop in are related to changes in the individual source datasets. Over 2500 UADB-Wind stations are at least 5 years long, nearly 1600 stations are least 20 years, and more than 500 stations have data of over 50 years (Figure 5). Again, the decrease in numbers of UADB-Wind stations with increasing time series length is very regular. The UADB-Wind product source datasets contribute to station time series as they did to UADB-TRH (Figure 6.). Many data sources that are country specific have fewer than 25 stations, while the global datasets 15

16 Figure 4. Annual station count for the NCAR composited Wind Database (UADB-Wind) product. The maximum is over 1600 stations in the late 1960 s and decreases to present day levels of approximately 1,000 stations. Figure 5. Station period of record in the NCAR Upper Air composited Wind Database (UADB-Wind) product. The station totals are cumulative starting at the longest period (75 years) through the shortest period (5 years). So, all stations in the 75-year interval are a subset and added into the shorter intervals. More than 2,500 stations are at least 5 years long and approximately 500 are 50 years or longer. 16

17 contribute over 700 stations. Soundings for the UADB-Wind product are drawn from 34 source datasets. Figure 6. Number of Stations contributed by each source dataset in the NCAR Composited Upper Air Wind Database (UADB-Wind) product. Data sources are shown on both right and left vertical axes (see Table 1 for source details). The color bar scale interval is 25 stations in the range, and then 100 stations per interval up to 1,000. Most datasets contain fewer than 25 stations, while the datasets with global coverage, like NBUFR, NCADP and NMCB3 provide over 700 stations. 17

18 Combining Stations With Different WMO Identifiers Occasionally, WMO identifiers have changed randomly at upper air stations and in extreme cases countries have systematically changed all the station identifiers (e.g. Canada in 1977). Further, WMO identifiers may or may not change when stations are moved small distances to new locations, which can also change the station reference altitude. In order to create the longest possible time series at a location, allowing for small station moves, it is necessary to combine stations with different WMO identifiers. Success in doing this relies heavily on histories of station metadata. These histories, themselves, are not always complete or reliable. When the upper air network was being developed, good station metadata libraries were not maintained, especially ones that tracked station moves. The central question when combining stations having different WMO identifiers is, what is the maximum tolerable station re-location over which the sounding data can be assumed to be nearly identical? The best way to approach this question is to analyze each station separately, and attempt to quantify a tolerable threshold based on the sounding measurements. Moreover, this approach is still insufficient, because the threshold can be dependent on the research question under consideration. For a long-term global network, like the UADB, this approach is not tractable, because highly automated data processing, re-processing in the future, and data reproducibility are requirements. In addition to using simple station separation as a condition for combining time series, historical station metadata libraries can be leveraged for guidance. One of the most promising station histories is the Global Historical Metadata File of Upper Air Observing Stations and Instruments (GHM-UA) that was created by the Validated Atmospheric Profiles for Operations and Research project (Schroeder, 2008). The GHM-UA includes stations as far back as the early 1900 s and up through Each station history is independent with no station cross-referenced to more than one location. The station combining procedures begin by setting the reference station positions to the most modern reported locations in the UADB-V3 Composited times series products. Then data from related WMO stations, as determined from the GHM-UA, are combined with the reference station to create a longer time series. In most cases, the GMH-UA stations are all within about 100km radius from the reference location. A more conservative approach is used here, as stations are only combined if they are 40km or less from the reference location. The data are added to reference station time series on a month-by-month basis working backward in time. If multiple composite stations are available the station with the highest number of soundings for the month is given selection priority. Gaps in calendar months of the reference station are similarly filled using soundings from related stations as specified in the GMH-UA. 18

19 This combining procedure was applied to both the UADB-TRH and UADB-Wind composited time series products. They are referred to as the combined times series products, i.e. UADB-TRHC and UADB-WindC products. These are the longest possible time series for each station. In the event that no station combining was possible the simple composited time series, identical to what is found in the UADB-TRH and UADB-Wind products, are replicated in the combined products. As with the UADB-TRH and UADB-Wind products, the flexibility of the UADB system will allow us to analyze and implement improved techniques for combining stations with different WMO and other identifiers (e.g. WBAN and CHUAN), including updates from the GHM-UA, and other station libraries as they become available. 19

20 UADB Combined Data The UADB-TRHC product has 181 stations where the time series have been extended with stations having different WMO identifiers. Over 100 station time series were extended by at least five years and about 50 stations have been extended over 25 years (Figure 7). Time series extension of various lengths occur across the globe (Figure 8.) The maximum impact is generally in Europe and eastern Asia with a number of systematic extensions in Canada. Figure 7. Number of years added by station in the NCAR Combined Upper Air Temperature-Moisture Database (UADB-TRHC) product. 181 stations have at least 1 month added to the period of record. About 90 stations have their period of record extended by 10 years while nearly 50 stations were extended more than 30 years. The UADB-WindC product has over 200 stations where the time series was extended by at least one month (Figure 9). Nearly 150 stations have been extended by at least 5 years and over 50 stations have been extended over 25 years. Time series extension of various lengths occur across the globe (Figure 10.), much like the UADB-TRHC product. The maximum impact is generally in Europe and eastern Asia with a number of systematic short period extensions in Canada. 20

21 Figure 8. Station locations where the period of record has been extended in the NCAR Combined Upper Air Temperature-Moisture Database (UADB_TRHC) product. The largest impacts are in Europe and eastern Asia with systematic changes in Canada. Figure 9. Number of years added by station in the NCAR Combined Upper Air Wind Database (UADB-WindC) product. A total of 218 stations have at least 1 month added to the period of record, with nearly 130 stations being extended by 10 years and about 50 stations more than 30 years. 21

22 Figure 10. Station locations where the period of record has been extended in the NCAR Combined Upper Air Wind Database (UADB-WindC) product. Impacts are greatest in Europe and eastern Asia, with very systematic short period augmentations in Canada. 22

23 APPENDIX A Moisture Conversion Across the original UADB data sources the moisture content has been recorded in several ways, including relative humidity (RH), specific humidity (q), and dew point temperature (Td). In order to combine data profiles from the various sources a single measurement of moisture is necessary. For simplicity, RH was chosen, and conversions of q and Td to RH were computed using published algorithms (Saucier, 1955). This approach utilizes the fewest number of conversions. Td to RH To compute RH beginning with Td, first vapor pressure is calculated as follows: e = Td A Td+B (1) where A = 7.5 for water and 9.5 for ice, and B = for water and for ice. In the UADB, A=7.5 for all cases. Next, saturated vapor pressure is calculated. e s = T A T+B (2) where T is temperature, and A and B are defined the same as in Equation (1). Finally, from the calculations of e and es, RH can be computed as follows: RH = 100 e e s (3) q to RH To compute RH beginning with q, first note that the formula for specific humidity is as follows: q = e (P e) (4) where P is the pressure in millibars and e is the vapor pressure in Equation (1). After rearranging and solving for e the following is obtained: 23

24 = e q P P (5) Let C be a constant where 1 = C. Then, plugging in the known values yields: e C = q P P (6) So, e = 1 C and es can be computed from Equation (2). Similarly, Equation (3) is finally used to calculate RH. 24

25 Appendix B - ASCII Format for the NCAR Upper Air Database As described in the previous sections, the Composited UADB products (UADB-TRH and UADB- Wind) and Combined (UADB-TRHC and UADB-WindC) products have been output in an ASCII format. The output format contains one descriptive header record (Table B-1) for each sounding, followed by all data records (Table B-2) for that sounding, with one level per line. Note the fields in both the header and data records are delimited by a space. Within the descriptive header record, all metadata regarding the sounding and station can be found. This information is necessary in order to properly describe each sounding. Tables B-3 through B-9 describe all of the codes and abbreviations that are found in the descriptive header record. Table B-1. Format of the descriptive header record. Each sounding has a header record that is formatted as in the table. Following the header record are the data records. Character Range Type Parameter Name Description Character A character, H, used to identify the header record Space Field separator Integer Unique Sounding Identifier A UADB number, which is used to identify each unique sounding Space Field separator Character Station ID ID used to identify the station type. These IDs can be a WMO, WBAN, CALL SIGN or OTHER Space Field separator Integer Station ID Flag A number used to identify the type of station ID in the previous field. See Table B-3 for values. Note that in most cases, this will primarily be a 5-digit WMO or WBAN integer ID Space Field separator. 25

26 26-28 Integer Source Dataset A number used to identify the original RDA dataset where the sounding originated. See Table B-6 for dataset descriptions Space Field separator Float Version of Source Version number the source dataset, from version 2.n of the UADB. This is used to track changes in the source datasets Space Field separator Integer Date Flag A number used to describe the accuracy of the date and time of the observations. Table B-4 describes the Space Field separator Integer Year Year of the sounding, formatted in four digits Space Field separator Integer Month Month of the sounding, formatted in two digits Space Field separator Integer Day Day of the sounding, formatted in two digits Space Field separator Integer Hour Hour of the sounding, formatted in four digits. The first two digits describe the hour while the last two describe the minutes Space Field separator Integer Location Flag Space Field separator. A number used to indicate the source of the latitude, longitude, and elevation. The number describes if the location comes from a local station library or the original report. See Table B-5 for descriptions of these numbers. 26

27 58-67 Real Number Latitude Latitude of the station, measure in degrees and thousandths 1. A negative indicates a location in the Southern Hemisphere, while a positive indicates a location in the Northern Hemisphere Space Field separator Real Number Longitude Longitude of the station, measured in degrees and hundredths. Locations are written from 0 to 360 East Space Field separator Real Number Elevation Elevation of the station, measured in whole meters (m) Space Field separator Integer Sounding Type A number used to indicate whether a sounding contains temperature or moisture and/or wind. Table B-7 describes the various types Space Field separator Integer Number of Levels A number that indicates the total number of levels within the sounding Space Field separator Character Product Version A character indicator of the product version. The format is xx.yy.zz, where xx is the major product identifier, yy is the subproduct identifier, and zz is the actual version of the xx.yy combination. For example, indicates that this is Version 3 of the UADB (composited version), the sub-product is 1 (TRH) and the actual version number of the composited TRH data is 0. See Table C-8 for list of product versions. 1 Latitude and Longitude have a precision of degrees and hundredths, but have been written as degrees and ten-thousandths to accommodate higher precision. 27

28 Table B-2. Format of the data records. Data records immediately follow the header record. The number of levels field in the header indicates how many data records follow. Character Range Type Description 1-4 Integer A number used to indicate the level type (i.e. mandatory, significant, tropopause, etc.). See Table B-9 for a description of all level types. 5-5 Space Field separator Real Number Pressure of the observation measured in millibars (mb). A missing value is coded , Precision is written to the tenth mb Space Field separator Real Number Geopotential height of the observation, measured in meters. A missing value is coded Precision is written to the tenth of a meter Space Field separator Real Number Temperature of the observation, measured in degrees Centigrade. A missing value is coded Precision is written to the tenth of a degree Space Field separator Real Relative humidity of the observation. A missing value is coded Precision is written to the tenth RH Space Field separator Integer Wind Direction (0-360), missing value is -999, written to whole degrees Space Field separator Real Number Wind speed of the observation, in meters per second (m/s). A missing value is coded Precision is written to tenth of a m/s. 28

29 Codes for Flags Used in the Upper Air DataBase (UADB) Project Table B-3. Station ID flags used to indicate what type of identifier describes the station. ID Type Code Description 1 WMO (five digit) 2 WBAN 3 ASCII call sign 4 COOP Station 5 Platform name 6 Other 7 WMO with zero for 6th digit 8 Other WNO number with non-zero sixth digit Table B-4. Time flags used to describe the precision of the time the sounding was launched. Code Description 0 Indicates an unknown time. 1 2 Time of actual observation, in UTC. This is recorded as four digits for the year, two digits for the month, two digits for the day, and four digits for the hour. Precision is written to the nearest hundredth. Time of the observation to nearest 1-3 hours, in UTC. Values are recorded as described for Code 1. 29

30 Code 3 Description Time of the observation to the nearest hour, in UTC. For these observations, the launch minute is missing. Values are recorded as described for Code 1 with minutes set to Local time of the observation. Values are recorded as described for Code 1. 2 Occasionally 51 has been used to indicate a missing value. When time is recorded as yyyymmddhhhh, where hhhh = hour and hundredths of minutes, there is no place indicator for a missing a value. When the possible range of minute values is converted to hundredths, 51 is not impossible (30 minutes = 50%, 31 minutes rounds to 52%). Thus, this value was selected in order to minimize the error of incorrect use. Statistically speaking, choosing a value near the middle of all possible values minimizes the amount of time you might be off. In addition, this number is unique which also serves to mitigate error. Table B-5. Description of the station library flags for the UADB Project. Each number indicates the source of the latitude, longitude, and elevation. Code Description 0 Indicates an unknown station library flag. 1 Latitude, longitude, and elevation are from original report; this is a fixed station. 2 Latitude, longitude, and elevation are from original report; this is a moving platform station. 3 Latitude, longitude, and elevation are from original report. Type of station unknown. 4 Latitude, longitude, and elevation are from a local station dictionary/library. Table B-6. Source datasets for the UADB Project. Each dataset has a unique number that indicates where the source of the data can be traced to. Source Code 1 NCEP B3 Description 2 USAF DATSAV 30

31 3 C-CARDS 4 TD390 5 SHERHAG 6 TD53 7 TD54 8 Data from Dominic tests 9 NCAR Key Entered 10 June 1970 GARP 11 CARDS U.S. Control 13 TD52 20 MIT 21 Navy Kunia 22 Navy Spot 23 China 24 Permanent Ships from NCDC 25 Permanent Ships from Canada 26 NCAR Time-series Raobs - NCDC 27 NCEP ADP 28 NCEP BUFR 50 NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck NCDC Deck

32 100 Canada 110 French 111 French 120 South Africa 121 South Africa 130 Brazil 140 New Zealand 150 Russian 151 Russian Ships 160 Australian 170 United Kingdom 180 Singapore 190 India 200 Japan 210 Argentina raobs 32

33 Table B-7 Sounding type code for the UADB data. This number is an indicator that describes whether the sounding has temperature, moisture or wind data available. Obs. Type Description 1 Sounding includes temperature and/or moisture data only. 2 Sounding includes winds data only. 3 Sounding includes temperature or moisture, and wind data. 33