FLASH and It s Research Communities

Transcription

1 FLASH and It s Research Communities D. Q. Lamb University of Chicago NSF Community Code Workshop Chicago, IL 6 September 2012

2 Outline of Talk Role of community codes Properties community codes need to have Related needs FLASH basics FLASH capabilities Code contribution policy and major contributions FLASH research communities: Astrophysics and cosmology (examples: thermonuclear-powered supernovae and galaxy cluster mergers) High energy density physics (example: shockgenerated magnetic fields) Computational fluid dynamics

3 Role of community codes Community codes serve their scientific communities by providing: Extensive physics capabilities that would otherwise not be available and that can be used to address cutting edge research problems in science and engineering. A framework with existing capabilities into which new open or proprietary solvers can be imported. A means of maintaining solvers over time and migrating them to new platforms, which is particularly important given the radical new architectures expected in the next generation of platforms. An opportunity to support the education and training of the applied mathematicians, computer scientists, and computational scientists needed to develop the algorithms and codes needed to run at scale on these architectures.

4 Properties community codes need to have The extensive experience the Flash Center has had with designing, implementing, disseminating, and maintaining FLASH has shown the importance of a community code having certain properties in order to successfully serve a large research community: It must be fully modular, extensible, and open only if this is the case can it support creative approaches to algorithms and solvers, facilitate prototyping, and enable individual researchers to use its capabilities while adding their own solvers, whether the researchers make them open or retain proprietary control of them for shorter or longer periods of time. It must be professionally written and documented this is essential to maintaining the code over time, and making it possible for members of the user community to change it. It must be subject to continuous, rigorous verification, including unit and regression testing this is essential if it is to be robust.

5 Related needs Having more than one community code for a research community is usually desirable one size/ shape usually does not fit all. Development of algorithms, solvers, and codes by individuals and small groups also needs to be supported this encourages the development of innovative new algorithms and solvers, many of which will end up enhancing the capabilities of the code.

6 FLASH code contributors Current Flash Center Contributors : John Bachan, Sean Couch, Chris Daley, Milad Fatenejad, Norbert Flocke, Carlo Graziani, Cal Jordan, Dongwook Lee, Min Long, Prateeti Mohapatra, Anthony Scopatz, Petros Tzeferacos, Klaus Weide Current External Contributors: Paul Ricker, John Zuhone, Marcos Vanella, Mats Holmstrom, Seyit Hocuk, Chris Orban, Igor Golovkin, Tommaso Vinci, William Gray, Cristoph Federrath, Richard Wunsch Past Major Contributors: Katie Antypas, Alan Calder, Jonathan Dursi, Robert Fisher, Bruce Fryxell, Murali Ganapathy, Shravan Gopal, Nathan Hearn, Timur Linde, Bronson Messer, Kevin Olson, Tomek Plewa, Lynn Reid, Katherine Riley, Andrew Siegel, Dan Sheeler, Frank Timmes,, Dean Townsley, Natalia Vladimirova, Greg Weirs, Mike Zingale

7 FLASH basics FLASH is a multi-physics finite-volume Eulerian code and framework implemented on a block-structured mesh with adaptive mesh refinement (AMR) FLASH scales to well over a hundred thousand processors; it uses a variety of parallelization techniques including domain decomposition, mesh replication, and threading to best utilize hardware resources FLASH is extremely portable and can run on a variety of platforms from laptops to supercomputing systems such as the IBM BG/P and BG/Q

8 FLASH basics (continued) FLASH is composed of interoperable units/modules; particular modules are combined to run individual simulations, so only the code relevant to a particular problem is included when FLASH is compiled, allowing important compile-time optimizations. FLASH is professionally managed software with daily, automated regression testing on a variety of platforms, version control, coding standards, extensive documentation, user support, and integration of code contributions from external users. FLASH has been downloaded more than 3,000 times; more than 1,000 scientists around the world have now used FLASH, and more than 700 papers have been published that directly use it.

9 A Snapshot of FLASH Simulations Shortly: Relativistic accretion onto NS Gravitationally confined detonation Turbulent Nuclear Burning Wave breaking on white dwarfs Shock-generated B-field experiment Intracluster interactions Nova outbursts on white dwarfs Laser-driven shock instabilities Rayleigh-Taylor instability Magnetic Rayleigh-Taylor Cellular detonation Helium burning on neutron stars Orzag/Tang MHD vortex Richtmyer-Meshkov instability

10 A Snapshot of FLASH Simulations FLASH capabilities include: Adaptive mesh refinement (AMR) on a block-structured mesh Multiple state-of-the-art hydrodynamic solvers (1T and 3T) State-of-the-art magnetohydrodynamics (currently 1T) Implicit solvers for diffusion using the HYPRE library Turbulent (currently Nuclear Burning Gravitationally confined being used to model thermal conduction, detonation radiation diffusion, Shock-generated and viscosity) B-field experiment Many physics modules relevant to astrophysics and cosmology, including gravity and nuclear burning Many physics modules relevant to high energy density physics, Shortly: Relativistic accretion onto NS Wave breaking on white dwarfs Magnetic Rayleigh-Taylor including laser drive, 3T, electron conduction, multi-group Nova outbursts white dwarfs Laser-driven shock instabilities Rayleigh-Taylor instability radiation diffusion, tabular EOS and opacities Incompressible Navier-Stokes solver Generic, highly scalable parallel particles framework (currently used for PIC simulations, laser ray tracing, dark matter, Lagrangian tracer particles, fluid-structure interactions) Cellular detonation Helium burning on neutron stars Orzag/Tang MHD vortex Intracluster interactions Richtmyer-Meshkov instability

11 Flash Center code contribution policy The code contribution policy is designed with both the intellectual property rights of the contributors and the benefits to other researchers in mind. Often researchers want to add some functionality to the code that they may not wish to share in a public release. The Flash Center's policy is to allow individual contributors a prenegotiated time when their code can exist in the main FLASH source tree without being included in the public release. During this period the testing and maintenance of the code is carried out as an integral part of the code without any cost to the contributor, as long as the contributor provides supporting documentation and test problems for verification. At the end of this period, the contributor agrees to release the code with the public distribution. In return, the Flash Center assumes the responsibility for maintaining the contributed code and migrating it to new platforms.

12 Major code contributions so far Already in the code: 2D staggered mesh unsplit MHD solver contributed by Dongwook Lee before he joined the Center Huang Greengard method-based multigrid solver contributed by Paul Ricker after he left the Center Ionization Salvatore Orlando Radiation transfer using hybrid characteristics Eric Jan Rijkhorst Several direct solvers for uniform grid Marcos Vanella Hybrid particle-in-cell capability Mats Holmstrom In the next release (FLASH 4.0 in mid-september): Sink particles Christoph Federrath, Robi Banerjee, Martin Schron, Barnes-Hut tree Richard Wunsch Primordial chemistry William Gray

13 FLASH research communities Astrophysics and cosmology (examples: thermonuclear-powered supernovae and mergers of galaxy clusters) High energy density physics (example: shock-generated magnetic fields) Computational fluid dynamics

14 Pulsationally assisted gravitationally confined detonation model of thermonuclear-powered supernova

15

16 Merger of two galaxy clusters

17

18 FLASH 2D hydrodynamics simulation of the Vulcan foil experiment 2D FLASH hydro simulation of a laser-driven jet breaking out of a 50 micron-thick Carbon target.

19 Vulcan foil experiment Al Slab 100 μm thick 25 mm wide t = 400 ns Jet, v~35 km/s Induction Coil 3 cm Schlieren shadowgraphy image 1 Bdot probe traces 1 1 Images courtesy Jena Meinecke and Gianluca Gregori.

20 MHD simulation of the Vulcan foil experiment MHD simulation of the Vulcan foil experiment in which a magnetic field is generated by the Biermann battery effect with no magnetic dissipation in a 2D cylindrical (R-Z) computational domain. A 10 mm x 0.1 mm carbon foil inside an Argon-filled chamber is illuminated from below by a single laser beam of 448 Joules for 1 nsec. A breakout shock propagates through the foil, producing a jet-like flow and generating a field of the order of a few kg. The image shows the magnitude of the B-field superposed on the density, and reveals that magnetic field generation is stronger immediately downstream of the shock, in its wake.

21 Summary Community codes contribute to research communities in many important ways Community codes need to have certain properties FLASH is a highly capable, fully modular and extensible code with a broad user base Users have made major contributions to it FLASH primarily serves three research communities: Astrophysics and cosmology (examples: thermo-nuclearpowered supernovae and galaxy clusters) High energy density physics (example: shock-generated magnetic fields) Computational fluid dynamics