Silicon Cube: A Supercomputer Specially Designed for Meteorological Applications

Transcription

1 Silicon Cube: A Supercomputer Specially Designed for Meteorological Applications Chaoqun SHA School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing , P.R.China Pingzhong Yan LAPC,Institute of Atmospheric Physics, Chinese Academy of Sciences Beijing, # Hua Yan Li, Qi JIa Huo Zi, Chao Yang District, P. O. Box 9804,Beijing , P. R. China Dongming Qin The Key Laboratory of Embedded System and Service Computing, Ministry of Education; Tongji University, Shanghai , China.. Zhennan CAO National Research Center for High-Performance Computing Engineering Technology, Sugon #8 Dongbeiwang West Road, Haidian District, Beijing Qing JI* National Research Center for High-Performance Computing Engineering Technology, Sugon #8 Dongbeiwang West Road, Haidian District, Beijing jiqing@sugon.com * Corresponding author Abstract Supercomputers have been demonstrated for almost all short-range, medium-range, and long-rang weather prediction researches, besides the climate analysis. These meteorological applications, in contrast, cast the greatest challenges to supercomputer itself due to its comprehensive physical models, harsh time-to-results demands, and huge data volume etc. Therefore, the innovations on supercomputer for meteorological applications shall benefit not only the meteorological studies, but also the development of supercomputer itself. The Silicon Cube is a supercomputer specially designed for meteorological applications by Sugon. It was ranked 95th on Top500 (Jun. 2015). The supercomputer adopted the advanced Liquid Cooling Blade System, which can lower the Power Usage Effectiveness (PUE) < 1.2. It is also equipped with the 3D-Torus network and the related job scheduling software. The network provides the better communication for near nodes, and then more suitable for the neighboring data exchange in the meteorological applications. In the supercomputer, the Check-point and Fault- Tolerance parallel algorithm have guaranteed the continuous running jobs. As illustrated by its name, Silicon-Lattice-Style nodes arrangement adds extra flexible extensibility for supercomputer updating. Those popular meteorological applications, such as WRF, NAQPMS, IAP-CoLM-LICOMR, and CAS-ESM Land-Dynamic Vegetation Coupling Integration, 3D Eulerian chemical transport model, have successfully tested on the Silicon Cube supercomputer. Keywords - HPC ; Supercomputer;WRF; NAQPMS; IAP- CoLM-LICOMR; Simulation I. INTRODUCTION As the appearing and developing of computer, a new era for science and technology has undoubtedly arrived, whether you like it or not. This is especially true for meteorological research, which is also a traditional application of high performance computing (HPC or supercomputer). In fact, the world's first electronic computer ENIAC had been used for weather forecast.[1] Since then, almost every country and region has used computer or supercomputer for weather forecasting, ocean forecasting, climate modelling, and pollution monitoring & control etc.. [2, 3] More important, meteorology is definitely not an isolated-island. It is comprehensively coupled with water, biosphere, ice & snow, and lithosphere and so on. [4] That is why the meteorological simulation, more or less, overlapped with earth simulation. In 2015, a realistically simulating the forces inside the Earth that drive plate tectonics was conducted by Lawrence Livermore National Laboratory. The simulation reached an unprecedented 97 percent parallel efficiency in scaling the

2 solver to 1.6 million cores and then successfully claimed the Gordon Bell Prize. [5] There are many supercomputers focus on meteorological research. The supercomputer Yellowstone is hosted by the National Center for Atmospheric Research (NCAR).[6] It features 72,576 Intel Sandy Bridge processors and TB of memory, reached 1.5-petaflops peak performance. The supercomputer has served many applications, including CDAT - Climate Data Analysis Tools, GEMPAK - GEneral Meteorology PAcKage, NCAR Graphics, CESM - Community Earth System Model, DART - Data Assimilation Research Testbed, ESMF - Earth System Modeling Framework, MM5 - PSU/NCAR Mesoscale Model, MOZART - Model for OZone and Related chemical Tracers, NRCM - Nested Regional Climate Model, WRF - Weather Research & Forecasting etc.. The supercomputer Earth Simulator, developed by the Japanese government's initiative "Earth Simulator Project", was a highly parallel vector supercomputer system for running global climate models to evaluate the effects of global warming and problems in solid earth geophysics. Later, the supercomputer was replaced and upgraded to 11.8Tflops. Another Japanese supercomputer K computer is currently installed at the RIKEN Advanced Institute for Computational Science campus in Kobe.[7] The K computer is based on a distributed memory architecture with over 80,000 computer nodes. It is used for a variety of applications, including climate research, disaster prevention and medical research. In 12 th five-year plan of China, 16 major scientific and technological infrastructures obtained special attention and priority, one of which relates Earth Simulator supercomputer. [8] The goals of the project lay on several aspects: to design and construct a supercomputer, to explore the understanding of the Earth's environment complex system, to simulate the change of the earth system spheres and long-term climate, and to accurately describe and forecast the earth physical, chemical and biological processes. The supercomputer Silicon Cube serves as the prototype system for the project (full name: Prototype System of Earth System Numerical Simulator). It is also China s first supercomputer specially designed for Earth and Meteorology research. The supercomputer Silicon Cube kicked off the construction on 2015 May, and officially announced on September at the year. The stage I of Silicon Cube has been finished with a series of benchmark and application tests. While the stage II will be deployed in the first half year of This paper reports the detailed hardware architecture, software stack, and related system administration design for Silicon Cube stage I. We also present the popular benchmark tests and application tests to solid demonstrate the advantages of Silicon Cube. II. SYSTEM OVERVIEW A. Sugon and Supercomputer Sugon, the major developer of Silicon Cube, rooted from the Institute of Computing Technology of the Chinese Academy of Sciences (ICT), and was the first (and now largest) supercomputer local vendor in China. Since 1990, Sugon has been working on supercomputing, producing seven generations of HPC systems, such as Dawning I and Dawning 1000 to It has successfully supported more than 10,000 HPC projects. In 2014, Sugon was successfully listed on the Shanghai Stock Exchange (Stock code: ). Besides the Silicon Cube, Sugon has successfully developed the Sugon 4000A (#10, Top ) and Nebulae(#2, Top ). It has continuously dominated the China HPC Top100 for the past 7 years and hit the 3rd of Top500 (Nov, 2015) per vendor market share. Figure 1. Sugon and Supercompter. B. Silicon Cube Supercomputer Slightly different from Nebulae, Silicon Cube is largely redesigned and managed to make a careful balance between the need for technological advancement of proprietary components and the availability of existing components. The goals of the design are to provide high computational performance under acceptable total of cost, power budgets, capabilities of supporting reliability, availability, and serviceability (RAS), complexity of application efficiency. Silicon Cube was designed with two principles: application mode and system architecture. Meteorological applications, in general, characterize as multi-mode, complex data, high solution, and multiple physical process coupling application. These characters lead to unbalanced workloads, data locality, communication locality, and intensive data movement in computing. To solve these challenges, we adopted 3D-Torus interconnection to match the grid spacing in simulation. The physical structure of Silicon Cube fits the layered models in meteorological applications. Multiple partitions in the supercomputer meet the computing demands of different patterns. Customized special computing components were selected for particular coupler computing. Co-design

3 processors promote the coupling of the hardware and applications. Silicon Cube is composed with Intel Xeon E v3 12 cores processor, DDR4 2133MHz ECC memory, 10Krpm SAS hard disk, 56Gb FDR InfiniBand interconnection, and so on, delivering a peak performance of 1 PFLOPS. The computer room (primary machine room with auxiliary compartments) covers an area of 208 square meters in which there are more than 1,000 Sugon liquid cooling servers TC4600E-LP in 128 chassises. The total power is kW and the power density is over 3kW per square meter. The System thoroughly reforms the design schemes of refrigerating system. Its design begins with the heat sources of servers. Both air cooling and liquid cooling are used here. Sugon liquid cooling servers TC4600E-LP are employed in the system and a cold plate indirect liquid cooling technology is used to take away all heat from CPU. Heat from other parts and components is taken away by air cooling methods. The whole liquid cooling system dissipates heat through high-temperature water and no compressor is used, which makes natural cooling possible all the year around and provides a brand new, more efficient and reasonable cooling mode for the data center. Results showed that the average PUE value is 1.17, the lowest in China as of today. two-way nodes, or four one-way nodes, or two types of oneway nodes through several steps setting. In Silicon Cube, each computing node has the processor and accelerator. Eight computing nodes comprise a supernode. Six supernodes form a Silicon Node. Silicon Nodes can be deployed in a silicon-lattice like arrangement and then produce the Silicon Cube supercomputer. This is the details of the multilevel design for the system, as shown in figure 3. Figure 3. Multilevel design Petascale and then exascale supercomputers require and will require hundreds of thousands of cores to efficiently work together. That is why the interconnection is one of the top challenges in supercomputer technologies. As a switchless way to connect nodes in a supercomputer system, the 3D torus network topology is a good answer to both problems of speed and scalability. It can assure very low latency and linear system scalability. Silicon Cube adopts 3D torus network as well. This is the first 3D torus network developed and deployed in China. Figure 2. Applications on Silicon Cube III. HIGHLIGHTS Take the advantages of the popular supercomputer architectures: MPP (massive parallel-processing computing platform), which means one machine with thousands of CPUs tightly interconnected, and cluster, which means each machine is largely independent of the others in terms of memory, disk and then interconnected by certain network, the Silicon Cube adopts HCAPP (Hyper Converged Adaptive Parallel Processing) architecture. It is also believed to be a promising solution for the Exascale Computing. HCAPP bases on the industry standards and can stay in good compatibility for processor, accelerator, network, storage, system software and so on. HCAPP fuses with multiple computing resources, including GPU, FPGA, SPU, and DCU. This makes Silicon Cube possesses optimized power consumption, memory, and adaptive architecture. Its nodes embrace elastic design, can be dynamically configured to two Figure 4. Liquid cooling in Silicon Cube As shared by insidehpc.com, the liquid cooling is about 3,500 times better at storing and transferring heat than air. Direct contact liquid cooling (DCLC) uses the exceptional thermal conductivity of liquid to provide dense, concentrated cooling

4 to targeted small surface areas. By using DCLC, the dependence on fans and expensive air conditioning and air handling systems is drastically reduced. Reducing internal heat is essential to avoid temperature-related system damage and downtime. Silicon Cube uses the cold plate liquid cooling, TC4600E-LP, to increase the cooling efficiency of the system. The system s power usage effectiveness (PUE) is lower than 1.2. Compared to traditional air cooling, TC4600E-LP can successfully lower CPU temperature by 20 and then boost performance by 5%. Note that the noise of the system is underneath 45dB. Meteorological researches tightly relate to Big Data issue. According to the news reports, China metrological data already greater than ~5PB with the annual increment 1PB. In most cases, all those data need to be kept almost forever. In the view of this situations, Sugon introduced the ParaStor 200 parallel storage system, its excellent performance and high expandability suitably meets the demands of meteorological researches. Meanwhile, it is able to confidently deal with the future data flow caused by larger area and higher resolution in the research. The aggregation performance of the ParaStor200 parallel storage system increase linearly with the increase of the quantity of ostor data controller nodes. Based on the actual measurement results, each ostor data controller node is inserted with 2 double-port gigabit Ethernet cards, provides 4 data transmission channels, a single node is able to provide a writing bandwidth up to 150MB/s and a reading bandwidth up to 360MB/s. For the ParaStore 200 parallel storage system deployed at Shenzhen Cloud Computing Center, its total system capacity is up to 16PB and it is able to provide an aggregation bandwidth up to 100 GB/s. Gridview HPC Suite is an integrated monitoring, management and job scheduling software platform for HPC and has been installed in Silicon Cube. Gridview is designed with several pluggable function modules. It could dynamically monitor the overall and detailed status of both computing center and cluster, provide comprehensive cluster management, real-time and historical alerting, and powerful job scheduling. The suite provides Computing Center Visualization, Cluster Monitoring, Performance Analysis, Asset Management, Cluster Management, and Alert Management. Gridview offers powerful and flexible scheduling policies, fault tolerance, easy-to-use application Web Portals and clear accounting system. All these features greatly improving management efficiency and utilization of high performance computers. According to communication pattern of parallel job, Gridview choose 1D, 2D or 3D-adjacent nodes and obey adjacent node allocation policy for meteorology application in general. More recently, Gridview introduced a new partner, EasyOP, which can provide system operation through smart phone, data center/ computing center management through cloud, and resource exchange platform. Figure 5. Job scheduling in Sicon Cube through Gridview In order to fulfill the extra harsh time-to-results demands in numerical meteorological research, we deployed Check-point & Fault-Tolerance parallel algorithm in system level and user level. By minimizing the time for system restore, system breakdown check and check-point record, system check-point (Clusnap), Silicon Cube can obviously minimize the time consumption related and then efficiently improve the system s availability. IV. BENCHMARK & APPLICATION TESTS Silicon Cube is installed with Red Hat Enterprise Linux Server release 6.6, with the inner-core version el6.x86_64. Intel Compiler XE and OpenMPI are both installed as well. We have conducted tests of LINPACK, CAS-ESM, and NAQPMS. The three tests study the supercomputer performance, Ocean-Land-Air Coupling, and pollution diffusion process, respectively. The LINPACK Benchmarks are a measure of a supercomputer system's floating point computing power. Introduced by Jack Dongarra, they measure how fast a computer solves a dense n by n system of linear equations Ax = b, which is a common task in engineering. [9] The latest version of these benchmarks is used to build the TOP500 list, ranking the world's most powerful supercomputers. We have tested Linpack : HPL 2.1 with OpenMPI 1.6.5, Intel compiler. The tested peak performance is 738 Tflops with ~75% efficiency for all 696 common nodes,which is much better than the average for those large supercomputer systems 70%. The data show that Silicon Cube can run stably in largescale, big pressure computing for quite a long time. It also implies that the integral design of the supercomputer is reasonable. Moreover, the test illustrates the Infiniband network scheme based on 3D-Torus can work well and demonstrates the liquid cooling solution can be safe and stable operation for a useful calculating time. As a part of Earth Simulator project in 12th five-year plan of China, Chinese Academy of Science Earth System Model (CAS-ESM) is required to be developed. The CAS-ESM contains whole climate system and ecological environment components. It integrates Atmospheric Model, Ocean Model, Seaice Model, Land Model, Atmospheric Chemistry and Aerosol Model, DGVM, OBM and WRF.[10] In addition,

5 Space Weather Model (SWM) will be added to the system in the future. Also, the coupling between different pairs of component models will be realized. At that time, the simulation research of Atmosphere, Current, Land Surface Procedures and Ecology will be more accurate. In this work, we tested the control test workload of air, land, sea and sea ice (full coupled). The test used 512 cores. Each node has 16 cores, so in total 32 nodes. Computing speed of the test is about 6 years/day. The results are shown in figure 6 and 7. It indicates that CAS-ESM has quickly and correctly run on the Silicon Cube and the computing speed can match the demand of those computing scientists. NO x, CO and other trace gases and dust in air. It can simulate carbon aerosols, sulfate, nitrate, ammonium salt aerosol composition etc. as well. NAQPMS is composed of four main parts: meteorological treatment, emission source, air quality model, and the model output. We have simulated the nationwide high resolution air quality simulation and forecast with NAQPMS on Silicon Cube. The simulation used one nested layer for the entire nation area. The simulation resolution is 5 km and the grid is 1000 x1070. Lambert projection, suitable for the mid-latitude region, is adopted in the test. In vertical direction, the test used terrain following coordinate system, and was divided into 20 vertical layers unevenly. The lowest 10 layers located 2.5 km above the ground, while that of the highest layer is 20 km. The test simulated for 4 days, from the 12:00 October 20, 2015 to 12:00 24 October, Model input and output frequency was 1 hour. Time step was 300 s. With 480 cores, simulating one day only needs 0.89 hours. The test showed that the NAQPMS can run smoothly with 5KM resolution on Silicon Cube. Figure 6. (a) CAS - ESM simulation and (b) the observation of the winter average sea level pressure field Figure 7. Simulation of global annual average sea surface temperature The Nested Air Quality Prediction Modeling System (NAQPMS) is independently developed by IAP/CAS and aims at reliable air quality forecast for China, especially for the PM 2.5 and Ozone. [11] The model focuses on the atmospheric compositions, significant for environment and/or meteorology, at city level or region level with three-dimensional eulerian air quality numerical model. The model can simulate O 3, SO 2, Figure 8. NAQPMS Simulation of the daily average concentration distribution for SO 2, NO x, and PM2.5

6 CONCLUSION Silicon Cube is a supercomputer specially design and optimized for numerical calculation in meteorological research and Earth science. Its peak performance is 1.08 Petaflops. Memory capacity reaches 80 TB. Storage capacity achieves 5.7 PB. Its interconnection network adopted the advanced 3D Torus network, of which the one-way bandwidth exceed 120 GBPS. This network matches the grid spacing in common meteorological simulation. The physical structure of Silicon Cube fits the layered models in meteorological applications as well. Adopted liquid cooling makes the system s PUE smaller than 1.2 and noise lower than 45dB. ParaStor 200 and Gridview are installed for the meteorological Big Data storage and job scheduling, respectively. We have tested LINPACK, CAS-ESM, and NAQPMS on Silicon Cube. The LINPACK test showed 75% efficiency of the supercomputer and demonstrated the reasonability of integral design and the serviceability of the 3D Torus network. The CAS-ESM tests have quickly and correctly run on the Silicon Cube with the computing speed 6 years/day, which matches the demand of those computing scientists. With 480 cores, NAQPMS simulates one day only needs 0.89 hours. This indicates that the NAQPMS can run smoothly with 5KM resolution on Silicon Cube % speedup can be achieved in those application tests ran on Silicon Cube. REFERENCE [1] G. W. Platzman, The ENIAC Computations of 1950-Gateway to Numerical Weather Prediction, Bulletin of the American Meteorological Society, vol. 60, no. 4, pp , [2] G. Gutman, and A. Ignatov, The derivation of the green vegetation fraction from NOAA/AVHRR data for use in numerical weather prediction models, International Journal of Remote Sensing, vol. 19, no. 8, pp , [3] A. C. Lorenc, Analysis methods for numerical weather prediction, Quarterly Journal of the Royal Meteorological Society, vol. 112, no. 474, pp , [4] J. W. Deardorff, Efficient prediction of ground surface temperature and moisture, with inclusion of a layer of vegetation, Journal of Geophysical Research Atmospheres, vol. 83, no. NC4, pp , [5] J. R. A. M. T. Isaac, etc, An Extreme-Scale Implicit Solver for Complex PDEs: Highly Heterogeneous Flow in Earth s Mantle, SC15, proceedings, [6] E. Kalnay, M. Kanamitsu, R. Kistler et al., The NMC/NCAR 40-year reanalysis project, Bulletin of the American Meteorological Society, vol. 77, no. 3, [7] M. Yokokawa, F. Shoji, A. Uno et al., "The K computer: Japanese next-generation supercomputer development project." pp [8] K. China, China's 12th Five-Year Plan, Kpmg Insight, [9] J. J. Dongarra, J. D. Croz, S. Hammarling et al., A set of level 3 basic linear algebra subprograms, Acm Transactions on Mathematical Software, vol. 16, no. 1, pp. 1-17, [10] J. He, M. Zhang, W. Lin et al., The WRF nested within the CESM: Simulations of a midlatitude cyclone over the Southern Great Plains, Journal of Advances in Modeling Earth Systems, vol. 5, no. 3, pp , [11] W. U. Qizhong, Z. Wang, X. U. Wenshuai et al., Multi-model simulation of PM_(10) during the 2008 Beijing Olympic Games:Effectiveness of emission restrictions, Huanjing Kexue Xuebao, vol. 30, no. 9, pp , 2010.