Climate Computing: Computational, data and scientific scalability

Transcription

1 Climate Computing: Computational, data and scientific scalability HPCS 2010 Toronto CANADA V. Balaji Princeton University 7 June 2010 Balaji (Princeton University) Climate computing 7 June / 34

2 Outline 1 Climate computing: a computational profile Climate modeling: a science driver for HPC "On exactitude in science" 2 Science at weak scaling Climate modeling at high resolution Uncertainty sampling Uncertainties in representation Globally coordinated modeling experiments 3 Scientific scalability: enabling "downstream" science Examples of downstream science Climate research and climate impacts research 4 The software basis for the global climate modeling enterprise Making the heroic routine Challenges for HPC Balaji (Princeton University) Climate computing 7 June / 34

3 Outline 1 Climate computing: a computational profile Climate modeling: a science driver for HPC "On exactitude in science" 2 Science at weak scaling Climate modeling at high resolution Uncertainty sampling Uncertainties in representation Globally coordinated modeling experiments 3 Scientific scalability: enabling "downstream" science Examples of downstream science Climate research and climate impacts research 4 The software basis for the global climate modeling enterprise Making the heroic routine Challenges for HPC Balaji (Princeton University) Climate computing 7 June / 34

4 Societal requirements drive climate computing Need for regional policy-scale information from global-scale research. (Figure 4 from 2007 IPCC SPM). Balaji (Princeton University) Climate computing 7 June / 34

5 Climate modeling: a science driver for HPC Climate modeling, in particular the tantalizing possibility of making projections of climate risks that have predictive skill on timescales of many years, is a principal science driver for high-end computing and data. It will stretch the boundaries of computing along various axes: resolution, where computing costs scale with the 4th power of problem size along each dimension (archive scales as 3rd power) complexity, as new subsystems and feedbacks are added to comprehensive earth system models capacity, as we build ensembles of simulations to sample uncertainty, both in our knowledge and representation, and of that inherent in the chaotic system. In particular, we are interested in characterizing the "tail" of the PDF (weather extremes) where a lot of climate risk resides. A petaflop climate computer has recently been announced in the US, the NOAA Climate Modeling and Research System. Balaji (Princeton University) Climate computing 7 June / 34

6 Borges: On Exactitude in Science...In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography. Quoted, with pleasure but without permission, from Borges, Collected Fictions, p.325. Balaji (Princeton University) Climate computing 7 June / 34

7 Climate modeling, a computational profile Intrinsic variability at all timescales from minutes to millennia; distinguishing natural from forced variability is a key challenge. coupled multi-scale multi-physics modeling; physics components have predictable data dependencies associated with grids; Adding processes and components improves scientific understanding; New physics and higher process fidelity at higher resolution; algorithms generally possess weak scalability. In sum, climate modeling requires long-term integrations of weakly-scaling I/O and memory-bound models. Balaji (Princeton University) Climate computing 7 June / 34

8 Outline 1 Climate computing: a computational profile Climate modeling: a science driver for HPC "On exactitude in science" 2 Science at weak scaling Climate modeling at high resolution Uncertainty sampling Uncertainties in representation Globally coordinated modeling experiments 3 Scientific scalability: enabling "downstream" science Examples of downstream science Climate research and climate impacts research 4 The software basis for the global climate modeling enterprise Making the heroic routine Challenges for HPC Balaji (Princeton University) Climate computing 7 June / 34

9 Regional scales are better represented There is a dramatic improvement in our ability to model regional scale climate response as we go to high (i.e beyond the IPCC AR4 norm) resolution. (Figure courtesy Isaac Held). Balaji (Princeton University) Climate computing 7 June / 34

10 Amazon precipitation improves with resolution From Zhao et al (2009), J. Climate, courtesy Ming Zhao, NOAA/GFDL. Balaji (Princeton University) Climate computing 7 June / 34

11 Interannual variability of hurricane frequency number of hurricanes M1 (mean= 5.92; std=2.68; cor=0.66) M2 (mean= 6.04; std=3.19; cor=0.73) M3 (mean= 5.32; std=2.59; cor=0.65) M4 (mean= 5.44; std=2.58; cor=0.61) EN (mean= 5.68; std=2.21; cor=0.83) OBS (mean= 6.20; std=3.06) WA year Interannual variability of W. Atlantic hurricane number from in the C180 runs. (Figure courtesy Ming Zhao and Isaac Held, NOAA/GFDL). Balaji (Princeton University) Climate computing 7 June / 34

12 A simple predictor of hurricane counts? Difference between Atlantic surface temperature T A and mid-tropospheric global temperature T G dtermines hurricane generation rate. From Zhao et al (2009). Balaji (Princeton University) Climate computing 7 June / 34

13 Nested models for hurricanes and climate change From Bender et al, Science, Balaji (Princeton University) Climate computing 7 June / 34

14 CHiMES: Coupled high-resolution modeling CM2.4 couples a 25 km resolution ocean ( eddy permitting ) model to a 100 km resolution atmosphere ( tropical cyclone permitting ) model. (Figure courtesy Tony Rosati and Tom Delworth, NOAA/GFDL). Balaji (Princeton University) Climate computing 7 June / 34

15 ENSO is modulated over millennial timescales Can we sample these using ensembles to parallelize in time? Figure from Wittenberg (2009), GRL. Balaji (Princeton University) Climate computing 7 June / 34

16 Long-term natural variability Maybe very long runs are needed! Wittenberg (2009) GRL. Balaji (Princeton University) Climate computing 7 June / 34

17 Multi-model ensembles to overcome structural uncertainty Reichler at al (2006) compare improvement of models ability to simulate 20th century climate, over 3 generations of models. Models are getting better over time. The ensemble average is better than any individual model. Improvements in understanding percolate quickly across the community. Balaji (Princeton University) Climate computing 7 June / 34

18 The IPCC-AR4 data archive: a global resource The IPCC data archive at PCMDI is a truly remarkable resource for the comparative study of models. Archives results from 20 models, used in 300 papers... Graphics such as this from Held and Soden (2006) are so routinely produced from the IPCC AR4 database that we ve ceased to marvel at it. This is a composite of output from 20 models worldwide, run with minimal coordination. It is worthwhile noting that the ensemble has greater skill by some measure than any individual model (see e.g Reichler et al 2006). Balaji (Princeton University) Climate computing 7 June / 34

19 Data delivery from multi-model ensembles increased reliance of federated database and petabyte-scale distributed archives. Critically depends on software, metadata, and data standards. Balaji (Princeton University) Climate computing 7 June / 34

20 Metadata standards: an unsung hero The unglamorous and mostly unfunded activity of building metadata standards proceeds under the guidance of informal grassroots activities, all recently acknowledged as central by WMO working groups WGCM and WGNE: CF Conventions: GO-ESSP Consortium: METAFOR: Models and experiments e.g IPCC AR4. Variable names e.g Temperature with units kelvin. Model grids time and space and planetary geometry. Balaji (Princeton University) Climate computing 7 June / 34

21 Borges again, on metadata and classification... a certain Chinese Encyclopedia, the Celestial Emporium of Benevolent Knowledge, in which it is written that animals are divided into: those that belong to the Emperor, embalmed ones, those that are trained, suckling pigs, mermaids, fabulous ones, stray dogs, those included in the present classification, those that tremble as if they were mad, innumerable ones, those drawn with a very fine camelhair brush, others, those that have just broken a flower vase, those that from a long way off look like flies. Quoted, with pleasure and without permission, from Borges, The Analytical Language of John Wilkins. Balaji (Princeton University) Climate computing 7 June / 34

22 Outline 1 Climate computing: a computational profile Climate modeling: a science driver for HPC "On exactitude in science" 2 Science at weak scaling Climate modeling at high resolution Uncertainty sampling Uncertainties in representation Globally coordinated modeling experiments 3 Scientific scalability: enabling "downstream" science Examples of downstream science Climate research and climate impacts research 4 The software basis for the global climate modeling enterprise Making the heroic routine Challenges for HPC Balaji (Princeton University) Climate computing 7 June / 34

23 Downstream science: statistical downscaling Cayan et al (2008): downscaled AR4 model output used to drive hydrology model (VIC). Required rerunning model at GFDL with modified outputs (dailies saved for from A2 run). Potential service activity if the service has access to computing and archival resources. Balaji (Princeton University) Climate computing 7 June / 34

24 Downstream science: energy policy Keith et al, PNAS, 2005: The influence of large-scale wind power on global climate. Feedback on atmospheric timescales: but does not require model to be retuned. Balaji (Princeton University) Climate computing 7 June / 34

25 The "scientific scalability" challenge More consumers than producers of climate projection data. (Figure courtesy EPA). Balaji (Princeton University) Climate computing 7 June / 34

26 Outline 1 Climate computing: a computational profile Climate modeling: a science driver for HPC "On exactitude in science" 2 Science at weak scaling Climate modeling at high resolution Uncertainty sampling Uncertainties in representation Globally coordinated modeling experiments 3 Scientific scalability: enabling "downstream" science Examples of downstream science Climate research and climate impacts research 4 The software basis for the global climate modeling enterprise Making the heroic routine Challenges for HPC Balaji (Princeton University) Climate computing 7 June / 34

27 The routine use of Earth System models in research and operations Earth system models are now increasingly accepted as tools to be run routinely for purposes beyond basic science. Climate prediction demand for model-based prediction on long timescales; Climate impacts downstream users of our models with their own requirements; Decision support models and scenarios routinely run for climate policy, energy strategy, risk pricing. Fundamental research the use of models to develop a predictive understanding of the earth system and to provide a sound underpinning for all applications above. This will require a radical shift in the way we do modeling: an infrastructure for moving the building, running and analysis of models and model output data from the heroic mode to the routine mode. Balaji (Princeton University) Climate computing 7 June / 34

28 From heroic to routine in other fields The polymerase chain reaction was awarded a Nobel prize not long ago. Later, you could get a PhD for developing PCR in different contexts. Now you order online and receive samples through the mail... What will the transition from heroic to routine look like in our field? Balaji (Princeton University) Climate computing 7 June / 34

29 Modeling frameworks The construction of complex Earth system models out of components is now commonplace in the design of modeling software (FMS, ESMF, PRISM). Components are embedded in the framework sandwich. FMS Superstructure Component code FMS Infrastructure Earth System Model Atmosphere Land Ice Ocean AtmDyn AtmPhy LandBio Hydro OcnBio OcnClr Rad H 2 O PBL Balaji (Princeton University) Climate computing 7 June / 34

30 Curators: archives coupled to scientific workflow Example: FRE, operational since 2003, designed to provide an environment for integrated testing and production. Rigorous standardized test procedure for evaluating new code and new model assemblies and configurations. Integrated existing post-processing structure. Captures complete configuration from source assembly to compilation to running, post-processing and analysis. Simulation database provides retrieval of model output, model analysis, and now model state and configuration information. The XML-based FRE workflow is again influential in community, with curators being prototyped at various sites. Balaji (Princeton University) Climate computing 7 June / 34

31 How to get to exascale If individual arithmetic processors are going to remain at 1 GHz (10 9 ) how do we get to exascale (10 18 )? We need billion-way concurrency! Components of a coupled system will execute on O(10 5 ) processors (driver-kernel programming model) There will be O(10) concurrent components coupled by a framework (FMS, ESMF, PRISM) We will reduce uncertainty by running O(10-100) ensemble members. We will use a task-parallel workflow of O(10-100) to execute, process and analyze these experiments (FRE). Exascale software and programming models are expected by 2013, hardware by Balaji (Princeton University) Climate computing 7 June / 34

32 Hardware and software challenges We still haven t solved the I/O problem. (Useful data point: our IPCC-class climate models have a data rate of 0.08 GB/cp-h). Integrated systems assembled from multiple manufacturers: chips, compilers, network, filesystems, storage, might all come from different vendors. Many points of failure. Multi-core chips: many processing units on a single board. Since our codes are already memory-bound, we do not expect to scale out well on multi-core. New programming models may be needed, but are immature: Co-Array Fortran and other PGAS languages, OpenCL. Reproducibility as we now understand it is increasingly at risk: GPU for instance does not appear to have a formal execution consistency model for threads. And we might need to embrace irreproducibility: see Advocating noise as an agent in ultra-low-energy computing : probabilistic CMOS. Balaji (Princeton University) Climate computing 7 June / 34

33 Climate science: HPC challenges Adopt high-level programming models (frameworks) to take advantages of new approaches to parallelism should they become operational. Component-level parallelism via framework superstructure. Approach models as experimental biological systems: single organism or cell line not exactly reproducible; only the ensemble is. There is more downstream science than there are climate scientists: a scientific scalability challenge. Use curator technology to produce canned model configurations that can be run as services on a cloud. Balaji (Princeton University) Climate computing 7 June / 34

34 Thank you! Questions? Balaji (Princeton University) Climate computing 7 June / 34