Around the world with Underworld - distributed development and collaboration in computational geodynamics

Transcription

1 School of Geosciences Around the world with Underworld - distributed development and collaboration in computational geodynamics Louis Moresi Katie Cooper Auscope SAM Wendy Mason 1

2 What is AuScope / AuScope S.A.M. AuScope a national geoscience infrastructure programme funded under NCRIS with support for Geospatial research, Earth Imaging, a Core Library, an access scheme for Geochemical gadgets, Simulation software, and a Data Grid. AuScope S.A.M. (Simulation, analysis, modelling) The AuScope Simulator is a toolkit of simulation, modelling, inversion and data mining tools, underpinned by parameters provided through the AuScope Earth Composition and Evolution component. The AuScope simulator makes extensive use of software, techniques and expertise developed through the Australian Computational Earth Systems Simulator (ACcESS) Major National Research Facility. 2

3 What is CIG A coordinated effort to develop reusable, well-documented and open-source geodynamics software The basic building blocks an infrastructure layer of software by which state-of-the-art modeling codes can be quickly assembled Extension of existing software frameworks to interlink multiple codes and data through a superstructure layer 3

4 The Simulation components of AuScopeS.A.M. Applications Natural Hazards Resources and Mining Sustainable Energy Geodynamics Auscope interoperability framework ACT pplates ESyS_Crustal ESyS_Particle QLD VIC StGFEM glucifer pmd*crc Reactive Transport WA Escript StGermain PICellerator ESysGeodyn Finley SPModel Underworld Gplates Basic science models Coupled Processes Earthquake Cycles Rock Mechanics Mantle Dynamics Plate- Mantle System Continental Deformation Surface Processes NSW Geophysical Fluid Dynamics A suite of specialized software packages designed specifically for Earth Science Simulation problems and a targeted set of workflows which point these codes at end users in specific application areas. 4

5 Applications Geodynamics Leaders: Dietmar Müller (Sydney), Louis Moresi (Monash), Gordon Lister (ANU) Goals: A workflow for linking rigid plate kinematic models with mantle convection and plate deformation combined with a library of subduction process simulations. Demonstrator project: Tectonics and geodynamics of South East Asia. The Java oil field is potentially very large. However we do not know the regional tectonics well enough to allow efficient extraction. The region is also rich in gold deposits but we lack a clear understanding of the tectonic controls on mineralization. Further this is one of the most potent regions for large earthquakes and volcanic eruptions. We are building workflows to allow simulation, analysis, and modelling using different S.A.M. functionalities to enable a better understanding of the relationship between sea-floor age distributions, oceanic plateaus, slab tears and windows and the resulting dynamic evolution of the region, including the effects of expulsion of mass southwards, eastwards and westwards from the Tibetan Plateau. The workflow brings together Pplates, Underworld, GPlates and CitcomS. 5

6 Applications Natural Hazards Leaders: Paul Tregoning (ANU), Hans Mühlhaus (UQ), Huilin Xing (UQ), Gordon Lister (ANU). Goals: A better understanding of megathrust earthquakes and tsunami generation Demonstrator project: Tsunamigenic megathrusts appear to form in specific geodynamic environments, typified by the Sumatran megathrust. The demonstrator integrates the geometry obtained from the Virtual Earth, with reconstruction obtained using the Tectonics and Geodynamics demonstrator, and forward modelling using forward modelling (Underworld and escript). The Natural Hazards SAM demonstrator seeks to bridge between research enabled by SAM and two other AuScope components: Geospatial and Earth Imaging. 6

7 Applications Energy Leaders: Steve Quenette (VPAC), Huilin Xing (UQ), Klaus Regenauer-Lieb (UWA, CSIRO), Lutz Gross (UQ) Goals: Thermal, mechanical, fluid transport modeling for geothermal energy, petroleum exploration and sequestration. Demonstrator project: A collection of geothermal modelling applications which support geothermal exploration: including heat flow models at the regional scale, crustal heat transport by fluid flow at low temperature, a hot-dry rock simulator, and a geothermal, multiphase-fluidtransport engineering model. The collection also includes a suite of benchmarks and verification models. Related project: Basin formation and fill with sediment production and transport Linkage-supported co-investment to study feedbacks between tectonics and surface processes 7

8 Overview underworld & friends Modelling capabilities Basin modeling Continental deformation Plate and mantle dynamics Long term dynamics of the lithosphere Melting & melt transport Software packages SPModel Underworld GALE MAGMA Framework components StGermain StGDomain StGFEM glucifer PIC External collaboration Interoperability layer Auscope CIG 8

9 Overview underworld & friends Victoria: specialized 3D Particle-in-cell / Finite Element Code and surface process models time active "fault" Coupled deformation of viscoelastic fluids and solids Free surfaces subject to external modification (e.g. erosion, eruption) Deforming rocks have memory (like biomaterials, foods, slurries) Magma generation, faults & localization, plate-boundaries... 9

10 Overview underworld & friends Victoria: specialized 3D Particle-in-cell / Finite Element Code and surface process models time active "fault" Material interfaces, material damage and stresses are carried / rotated by the broad scale flow and deformed by the local flow gradients. Coupled deformation of viscoelastic fluids and solids Mantle dynamics Free surfaces subject to external modification (e.g. erosion, eruption) Lithospheric instabilities Deforming rocks have memory (like Basin biomaterials, evolution foods, slurries) Magma generation, faults & localization, Subduction plate-boundaries models

11 Equations τ ij,j p,i = ρ(t, C,...)g i f t,i u i,i = 0 Momentum and Mass conservation τ ij µ + τ ij η + αλ ijklτ kl = u i x j + u j x i τ Viscoelastic stress prediction Plastic correction to yield surface D Constitutive rule T,t + u i T,i = (κt,i ),i + Q Energy conservation C,t + u i C,i = 0 Material tracking Failure models η = η(t, p, τ, C, τ, γ P ) Viscosity 10

12 Material point method Fixed mesh with moving particles Regular Eulerian mesh for momentum equation (efficient solvers) Lagrangian reference frame for: Compositional tracking Stress-history tensor Plastic strain history (scalar / tensor) Finite element formulation robust, versatile very simple to go back and forth between particle and mesh K E = K E = p Ω E B T (x)c(x)b(x)dω w p B T p (x p )C p (x p )B p (x p ) Lagrangian integration point FEM - integration points are material points; weights must be computed for each configuration in each element 11

13 Integration schemes η η Ω e ξ δω e m map to/from master element ξ Provided particles represent an approximately spherical region of fluid, then the Voronoi diagram seems like an ideal way to construct the weights. Construct the stiffness matrix at the centroids of the cells for better accuracy 12

14 Underworld Mantle convection / Continents Periodic boundary conditions, viscosity contrast in rigid block 13

15 Underworld Mantle convection / Continents Farrington, Stegman, Sandiford... PEPI 2009 Analysis of the influence of continental blocks on heat-flow transients after continental breakup 14

16 Modeling capabilities Slab evolution The time-evolution of a slab with an over-riding plate and a tear 15

17 Modeling capabilities Slab evolution The time-evolution of a slab with an over-riding plate and a tear 15

18 Modeling capabilities Basin evolution The pattern of shear bands in these simple systems is a function of relative strength even in the face of perturbations. 16

19 Modeling capabilities Tectonics + Sediment transport 17

20 Underworld-driven Collaboration Communicating precise information on software needs between collaborators / developers Discussing results (images, movies, graphs) Tracking who is running what and where (and why!) Sketching out papers, writing up equations and algorithms hopefully reusing the notes we made on the way! American Geophysical Union: Session on Web 2.0 in teaching and research 18

21 Underworld-driven Collaboration American Geophysical Union: Session on Web 2.0 in teaching and research 19

22 Aims Our collaboration takes places across such a lot of time zones that we are rarely available to communicate with each other in real time. We need ways to each keep track of what the other is doing: storing many different files (plots, documents, images, drawings) and keeping them in sync organizing a discussion around those files coordinating and reporting on HPC calculations which may queue and run over several days sharing the writing of sections of technical documents around the shared files sharing with other collaborators Our solution needs to be very simple and should also be available offline. We don't want to spend time setting up servers and maintaining them. We don't expect that many editing conflicts will occur because we are usually not working at the same time, but if we do create conflicting versions of files, we would like to know and be able to fix them straight away. 20

23 What works for us Tiddlywiki + jsmath Very simple way to build a personal lab notebook with mathematics, backups, revision history... robust distributes easily publishable (www) sufficiently nerdy to be satisfying (customizable) works offline Not scalable, but that is actually the point! 21

24 What works for us Dropbox Just works!! shared space observes but doesnʼt try to resolve conflicts publishable / shareable at file / folder level works surprisingly well offline cross platform Itʼs a pain to do for multiple projects which need to be kept apart... but it can be done. Not for raw data, just results. 22

25 What works for us & Chat! Youʼre using it anyway... why worry about a fancy conflict-managing tool when you can just ask are you still editing that file? (plus itʼs more friendly) Skype / Video Chat Sometimes that arm-waving is actually conveying something important Phone / SMS c AGU? 23

26 What doesn t work for us Technology that gets in the way! Googlewave (at the moment) is a perfect example it does lots of things in one place but all of them are slower and clunkier than the existing solution. Fun to play with, but not a disappearing technology. Facebook? Has its place in starting collaboration, but not in running it! 24

27 Underworld on the Grid Stable releases of Underworld can be accessed through the ARCS grid. The Underworld release module installation on the Monash Sun Grid (MSG) can now be accessed on the ARCS Grid using Grisu. BUT... half the point of underworld is that ability to plug in your own modules / toolboxes at run time and we havenʼt quite figured the best way to do that on the grid. AND... how much do you need to know about the guts of the machines? 25

28 Underworld performance tuning Performance is quite sensitive to underlying architecture particularly the implicit solve of the momentum equation. Strong, emergent variations in material properties means performance can be very sensitive to problem specification too. Someone with experience usually has to go onto each machine in advance Sometimes there are surprises! We are building a test-suite of typical problems to automate the first pass of this process Examples... Tako Persephone Ranger 26

29 persephone.geol.wsu.edu SiCortex (RIP) CPU Type - Proprietary, MIPS based CPU Speed GHz Cores per Node - 6 Number of Nodes Total Processors Memory per CPU GB Total Distributed Memory GB Persephone Lots of cores available, not that much memory. Problems are potentially over-decomposed. Observation: this is a very balanced machine. It is effective to use (almost) all the cores on each CPU, and aggressive decomposition of a problem is not a huge problem. 27

30 tako.maths.monash.edu.au Xenon CPU Type - AMD CPU Speed GHz Cores per CPU - 2 Number of Nodes - 8 Total Processors - 16 Memory per CPU - 16 GB Total Memory GB Tako Lots of memory per core available, single image machine, onboard interconnect (but not uniform) Observation: this is an easy machine to set up and keep going but non-uniformity in interconnect leads to inefficiencies in solver code. You cannot use this machine blind! 28

31 ranger.tacc.teragrid.org SiCortex (RIP) CPU Type - Opteron quad core CPU Speed GHz Cores per Node - 16 Number of Nodes Total Cores Memory per Processor - 2 GB Total Distributed Memory GB Ranger Huge system with noticeable granularity at the 16 core level. Plenty of memory. Observation: Not as balanced / elegant as persephone, but makes up for it through brute force. Easy to use without much knowledge of underlying architecture (except the 16wayness). 29