Piz Daint & Piz Kesch : from general purpose supercomputing to an appliance for weather forecasting Thomas C. Schulthess 1
Cray XC30 with 5272 hybrid, GPU accelerated compute nodes Piz Daint Compute node: > Host: Intel Xeon E5 2670 (SandyBridge 8c) > Accelerator: NVIDIA K20X GPU (GK110) 2
September 15, 2015 Today s Outlook: GPU-accelerated Weather Forecasting John Russell Piz Kesch 3
Swiss High-Performance Computing & Networking Initiative (HPCN) High-risk & high-impact projects (www.hp2c.ch) Application driven co-design of pre-exascale supercomputing ecosystem Phase II Monte Rosa Cray XT5 14 762 cores Hex-core upgrade 22 128 cores Three pronged approach of the HPCN Initiative 1. New, flexible, and efficient building 2. Efficient supercomputers 2016 3. Efficient applications Upgrade to Phase II 2015 Cray XE6 K20X based hybrid 2014 47,200 coresphase I 2013 Development & Aries network & multi-core procurement of 2012 petaflop/s scale 2011 supercomputer(s) 2017 Pascal based hybrid Upgrade 2009 2010 Begin construction of new building New building complete 4
Platform for Advanced Scientific Computing Climate Materials simulations Life Sciences Structuring project of the Swiss University Conference (swissuniversities) 5 domain science networks > distributed application support >20 projects see: www.pasc-ch.org Physics Solid Earth Dynamics 1.ANSWERS 2.Angiogenesis 3.AV-FLOPW 4.CodeWave 5.Coupled Cardiac Simulations 6.DIAPHANE 7.Direct GPU to GPU com. 8.Electronic Structure Calc. 9.ENVIRON 10.Genomic Data Processing 11.GeoPC 12.GeoScale 13.Grid Tools 14.Heterogen. Compiler Platform 15.HPC-ABGEM 16.MD-based drug design 17.Multiscale applications 18.Multiscale economical data 19.Particles and fields 20.Snowball sampling 5
6
Leutwyler, D., O. Fuhrer, X. Lapillone, D. Lüthi, C. Schär, 2015: Continental-Scale Climate Simulation at Kilometer resolution. ETH Zurich Online Resource, DOI: http://dx.doi.org/10.3929/ethz-a-010483656, online video: http://vimeo.com/136588806 7
Meteo Swiss production suite until March 30, 2016 COSMO-7 3x per day 72h forecast 6.6 km lateral grid, 60 layers ECMWF 2x per day 16 km lateral grid, 91 layers COSMO-2 8x per day 24h forecast 2.2 km lateral grid, 60 layers Some of the products generate from these simulations: Daily weather forecast on TV / radio Forecasting for air traffic control (Sky Guide) Safety management in event of nuclear incidents 8
Albis & Lema : CSCS production systems for Meteo Swiss until March 2016 Cray XE6 procured in spring 2012 based on 12-core AMD Opteron multi-core processors 9
Improving simulation quality requires higher performance what exactly and by how much? Resource determining factors for Meteo Swiss simulations Current model running through spring 2016 New model starting operation on in spring 2016 COSMO-2: 24h forecast running in 30 min. 8x per day COSMO-1: 24h forecast running in 30 min. 8x per day (~10x COSMO-2) COSMO-2E: 21-member ensemble,120h forecast in 150 min., 2x per day (~26x COSMO-2) KENDA: 40-member ensemble,1h forecast in 15 min., 24x per day (~5x COSMO-2) New production system must deliver ~40x the simulations performance of Albis and Lema 10
State of the art implementation of new system for Meteo Swiss Albis & Lema: 3 cabinets Cray XE6 installed Q2/2012 New system needs to be installed Q2-3/2015 Assuming 2x improvement in per-socket performance: ~20x more X86 sockets would require 30 Cray XC cabinets New system for Meteo Swiss if we build it like the German Weather Service (DWD) did theirs, or UK Met Office, or ECMWF (30 racks XC) Current Cray XC30/XC40 platform (space for 40 racks XC) CSCS machine room Thinking inside the box is not a good option! 11
COSMO: old and new (refactored) code main (current / Fortran) main (new / Fortran) dynamics (C++) physics (Fortran) dynamics (Fortran) physics (Fortran) with OpenMP / OpenACC stencil library X86 GPU Shared Infrastructure boundary conditions & halo exchg. Generic Comm. Library MPI MPI or whatever system system Used by most weather services (incl. MeteoSwiss until 3/2016) as well as most HPC centres HP2C/PASC development in production on Piz Daint since 01/2014 and for Meteo Meteo Swiss since 04/206 12
Piz Kesch / Piz Escha: appliance for meteorology Water cooled rack (48U) 12 compute nodes with 2 Intel Xeon E5-2690v3 12 cores @ 2.6 GHz256 GB 2133 MHz DDR4 memory 8 NVIDIA Tesla K80 GPU 3 login nodes 5 post-processing nodes Mellanox FDR InfiniBand Cray CLFS Luster Storage Cray Programming Environment 13
Origin of factor 40 performance improvement Performance of COSMO running on new Piz Kesch compared to (in Sept. 2015) (1) previous production system Cray XE6 with AMD Barcelona (2) Piz Dora Cray XE40 with Intel Haswell (E5-2690v3) Current production system installed in 2012 New Piz Kesch/Escha installed in 2015 Processor performance Improved system utilisation General software performance Port to GPU architecture Increase in number of processors Total performance improvement Bonus: simulation running on GPU is 3x more energy efficient compared to conventional state of the art CPU 2.8x 2.8x 1.7x 2.3x 1.3x ~40x Moore s Law Software refactoring 14
A factor 40 improvement with the same footprint Current production system: Albis & Lema New system: Kesch & Escha 15
GPU - accelerated hybrid Xeon Phi (accelerated) 2017+ Summit Tsuname-3.0 2016 MeteoSwiss Aurora Multi-core post-k U. Tokyo 2015 Both architecture have heterogeneous memory! 2014 2013 2012 2011 DARPA HPCS 16