Performance and Application of Observation Sensitivity to Global Forecasts on the KMA Cray XE6

Transcription

1 Performance and Application of Observation Sensitivity to Global Forecasts on the KMA Cray XE6 Sangwon Joo, Yoonjae Kim, Hyuncheol Shin, Eunhee Lee, Eunjung Kim (Korea Meteorological Administration) Tae-Hun Kim, Beom-Soo Kim (Cray Center of Excellence for Earth System Modeling) July 25, 2012

2 Table of Contents Introduction... 3 About KMA s National Meteorological Supercomputer Center... 3 Sensitivity Experiment... 4 Computational Performance... 6 ODB - Observational DataBase... 6 OPS - Observation Processing System DVAR 4-Dimensional Variational Data Assimilation... 8 Conclusion... 9 About the Cray Center of Excellence for Earth System Modeling and the KMA-Cray Earth System Research Center List of Figures Figure 1: KMA's Cray XE6 Systems... 3 Figure 2: Total impact of each observation (J/kg) at 00 UTC 10 August Figure 3: Horizontal distribution of the total impact of (a) TEMP, (b) BOGUS, and (c) ATOVS at 00 UTC 10 August 2010 (J/kg) Figure 4: Vertical total impact of (a) TEMP, (b) ATOVS, and (c) Aircraft at 00 UTC 10 August 2010 (J/kg)6 Figure 5: Elapsed time of ODB for each observation data type (core count labels are found on each vertical bar)... 7 Figure 6: Elapsed time of OPS for (a) Satwind, (b) ATOVS, (c) AIRS, and (d) IASI... 8 Figure 7: Scalability of 4DVAR (1 thread on the left and 3 threads on the right)... 9 ESRC-WP Page 2 of 10

3 Introduction Data assimilation plays a key role in reducing forecast error in the numerical weather prediction (NWP) process. With the transition to the UK Met Office NWP suite and the introduction of the new high performance computing facility in 2009, the Korea Meteorological Administration (KMA) was able to introduce an advanced 4-dimensional variational data assimilation (4DVAR) scheme into operations. KMA s current 4DVAR cycle is composed of ODB 32R.405_K03, OPS vn26.1, 4DVAR vn26.1, SURF 18.3 and UM vn7.7. With the exception of ODB, these components began operational usage on the KMA Cray XE6 in May ODB is developed by the European Center for Medium Range Weather Forecasting (ECMWF). OPS, 4DVAR, and UM (Unified Model) are developed by the UK Met Office. A further upgrade of these components will take place later in 2012 to OPS vn27.2, VAR vn27.2, and UM vn7.7ps28. VAR vn27.2 includes an observation sensitivity tool which is very useful to evaluate the impact of the specific observation and provides an objective approach to investigate the cause of the forecast error. The objective of the experiment described in this paper was to perform an initial analysis of the observation sensitivity for the UM data assimilation and to improve the computational performance of the UM data assimilation processes. Typhoon Dianmou, which made landfall on the Korea peninsula in August 2010 was used as the test case in this experiment. About KMA s National Meteorological Supercomputer Center KMA s National Meteorological Supercomputer Center houses a high performance computing (HPC) facility that supports NWP operations. The HPC facility is composed of two Cray XE6 supercomputers (for a combined total of 758 teraflops/s peak performance) with a multi-tier, multi-petabyte storage, archive and back-up system. System Overview: Two 20-cabinet Cray XE6 supercomputers configured for operational redundancy 45,120 cores per system 60.2 TB memory per system Figure 1: KMA's Cray XE6 Systems ESRC-WP Page 3 of 10

4 Sensitivity Experiment KMA runs an observation sensitivity program after every 4DVAR, which is run four times per day. The examination of observation sensitivity through the investigation of hind cases is very useful to evaluate the impact of observations and provides an objective approach to investigate the cause of the forecast error. Observation data can be extracted from the 4DVAR analysis data into various image files. The impact and sensitivity of all observations in horizontal and vertical aspects can then be visualized. The impact and sensitivity by observed channel, variable, time window, and surface type can also be examined. Various hypotheses on the relation between the forecast error and observation can then be investigated. An experiment using the case of typhoon Dianmou which made landfall on the Korean peninsula in August 2010 was performed. A target domain of 120~140 E and 25~45 N was defined to analyze the impact of observations over the area of the typhoon. In the results of the experiment, TEMP, Aircraft and AMDAR data were shown to positively impact the forecast near the typhoon area as shown in figures 2 and 3 (a negative value means the positive effect on reducing the forecast errors). On the other hand, the BOGUS data causes negative effects on forecast in this case. It is also shown that the impact of ATOVS is small despite a large number of observations being assimilated. The result is that positive and negative effects are vertically cancelled out as shown in figure 4. However, TEMP and Aircraft data produce generally positive effects in the vertical levels. These results are preliminary and represent the impact of each observation through the analysis of only a single test case. A more complete analysis over a longer period and with further test cases is required to determine which observations and observation areas are important for forecast accuracy. Figure 2: Total impact of each observation (J/kg) at 00 UTC 10 August 2010 ESRC-WP Page 4 of 10

5 (a) (b) (c) Figure 3: Horizontal distribution of the total impact of (a) TEMP, (b) BOGUS, and (c) ATOVS at 00 UTC 10 August 2010 (J/kg). (a) (b) ESRC-WP Page 5 of 10

6 (c) Figure 4: Vertical total impact of (a) TEMP, (b) ATOVS, and (c) Aircraft at 00 UTC 10 August 2010 (J/kg) Computational Performance The execution of the observation sensitivity process, and subsequent decision on which observational data to include, must be completed within the limited window of time preceding the forecast model. The total elapsed time to complete the observation sensitivity testing is driven in part by the number of test cases that are analyzed for each type of observation data. In addition, ODB, OPS, and 4DVAR must be executed sequentially as the input of one is fed into the next; with OPS and 4DVAR consuming the most of the elapsed time. In order to complete sensitivity testing within the allowable operational time window, a two time speed-up of the entire UM data assimilation process from 2,040 seconds to 1,020 seconds was required. The performance characteristics of each application were investigated to optimize the entire data assimilation process using the CrayPAT performance analysis tool. A summary of the optimization method and result for each model is described in the following sections. Table 1: Configuration of ODB, OPS and 4DVAR ODB OPS 4DVAR Version ODB 32R.405_K03 vn26.1 vn26.1 Horizontal Resolution Vertical levels Input Binary Universal Form for data Representation (BUFR) format data Observational data from ODB N144(~90km in midlatitudes) 70 (lid ~80km) varobs and cx from OPS ODB - Observational DataBase The ODB database software is developed by ECMWF to manage very large observational data volumes in an efficient manner for use by the Integrated Forecasting System (IFS)/4DVAR system and to enable flexible post-processing of this data. In addition, ODB simplifies introduction of new observation types, ESRC-WP Page 6 of 10

7 manages and controls observation processing data flow through data queries, provides full control over the desired features via source code, and replaces the old Central-Memory Array (CMA)-file format. KMA s infrastructure is composed of pre- and post-processing Linux clusters that act as auxiliary servers to the Cray XE6s. Due to its modest computational requirements, ODB has been running on a Linux cluster since operation of the new global and regional 4DVAR cycles began in It was also set up on the Cray XE6 since its library is needed to compile OPS. The primary performance issue of ODB was the unexpected absence of any core-level parallelism in spite of being parallelized model using both MPI and OpenMP mode. A straight-forward modification to the run scripts enabled parallel execution using MPI. For ATOVS, the optimized run is about four times faster than the original run. Elapsed time and the number of cores used by ODB for each observation data type (SATWIND, IASI, AIRS, ATOVS, Aircraft, SSMIS, and SS) are found in Figure 5. Figure 5: Elapsed time of ODB for each observation data type (core count labels are found on each vertical bar) OPS - Observation Processing System OPS can handle several different sources of observation (ODB, Obstore, GRIB, Radar, and ASCII format observational data). It performs quality control and thinning of observations and cycles through each observation type (i.e. Scatwind, ATOVS, IASI, Aircraft, Sonde, and etc.) and produces a reduced best set of observations. Quality-controlled observations from OPS are passed to the 4DVAR assimilation system to produce the forecast analysis. Optimization was required to reduce the elapsed time. For example, as the observational data was increased in the IASI case, OPS took around eight minutes using twenty MPI processes. OpenMP parallelism was implemented to improve the performance and resulted in a decrease in elapsed time to approximately four minutes using twenty MPI processes and six OpenMP threads. The focus of optimization was to reduce the elapsed time for IASI, AIRS, and ATOVS. SATWIND observation types as these required the largest amount of time. ESRC-WP Page 7 of 10

8 Wallcclock time(s) Wallclock time(s) Wallclock time(s) Figure 6 summarizes the performance for each observation type as the number of threads increases. The Cray XE6 with Magny-Cours twelve core (MC-12) was used and each MC-12 CPU has two NUMA (non-uniform memory access) nodes each consisting of 6 cores. So using 1, 2, 3, 6, 12, and 24 threads are the preferred threads count for OpenMP application. In this case 1, 2, 3, and 6 threads were used. (a) (b) SATWIND ATOVS Number of Threads Number of Threads (c) (d) AIRS Number of Threads Figure 6: Elapsed time of OPS for (a) Satwind, (b) ATOVS, (c) AIRS, and (d) IASI 4DVAR 4-Dimensional Variational Data Assimilation 4DVAR calculates a forecast model trajectory that best fits the available observations to within the observational error over a period of time. The observational error includes allowance for the finite resolution of the model. It uses the finite resolution of the model because fitting a model trajectory to observations requires very intensive computation. The original code had a number of performance issues including limited scalability. Scaling stopped at 2,016 cores with an elapsed time of approximately twenty three minutes. KMA s timing target for 4DVAR was under ten minutes. Two performance bottlenecks were determined by using the CrayPAT analysis tool: performance of the tridiagonal matrix solver (mpp_tri_solve_exe) and the model output scheme. ESRC-WP Page 8 of 10

9 wall clock time(s) wall clock time(s) A new version of mpp_tri_solve_exe was implemented to increase parallelization and reduce communication. The new code parallelized a serial portion of code that implemented the tridiagonal matrix solver on boundary points. In order to minimize data movement, an optimal number of processors were chosen per each East-West row; the chosen processors were stridden across the rows to minimize network contention. The major reason of 4DVAR s performance saturation was the slowness of writing model output. When 4DVAR wrote results to disk, it was found that there is a bottleneck on the host rank that simultaneously receives packed data from all the other ranks and sometimes even caused the 4DVAR model to crash. (There are no results for 1,176 and 1,344 MPI ranks as shown in figure 7). A solution was to insert a synchronize function that enforce the host rank to receive data in order and the writing performance was improved. Figure 7 shows the scalability results of the original and optimized 4DVAR. Following these two optimizations, performance and scalability were improved and resulted in an overall speed-up of over 2.5 times. The elapsed time using 3,024 cores was approximately nine minutes (compared to the original best elapsed time of approximately twenty three minutes using 2,016 cores). This optimized elapsed time satisfied the KMA s requirement of ten minutes OPT ORG OPT ORG # of cores # of cores Figure 7: Scalability of 4DVAR (1 thread on the left and 3 threads on the right) Conclusion In the initial analysis of observation sensitivity in the UM data assimilation using the case of Typhoon Dianmou, TEMP, Aircraft, and AMDAR observational data positively impact the forecast near the typhoon area. On the other hand, the BOGUS data causes negative effects on the forecast in this case. In addition, the impact of ATOVS is small despite a large number of assimilated observations. A more complete analysis over a longer period with further test cases is required to determine the types of observations and observation areas that are important for improving forecast accuracy. The execution of the observation sensitivity process and subsequent decision on which observational data to include must be completed within the limited window of time preceding the forecast model. Following a performance investigation and optimization, the entire UM data assimilation process requires 920 seconds and falls within the maximum. The performance of each application was investigated to maximize the performance of the entire UM data assimilation processes resulting in an improvement of 4, 2 and 2.5 times for ODB, OPS and VAR, respectively. ESRC-WP Page 9 of 10

10 About the Cray Center of Excellence for Earth System Modeling and the KMA-Cray Earth System Research Center The performance optimization work was performed by the Cray Centre of Excellence for Earth System Modeling (CoE-ESM) team located in Seoul, Korea. The CoE-ESM is a component of the KMA-Cray Earth System Research Center (ESRC). The ESRC is a cooperative venture established in 2005 to advance the science of earth system modeling over the East-Asia Pacific region and promote the use of advanced high performance computing facilities. Cray CoE-ESM team members work closely with KMA scientists to meet the operational and research objectives of KMA. ESRC-WP Page 10 of 10