Numerical Relativity for everyone: The Einstein Toolkit

Transcription

1 Numerical Relativity for everyone: The Einstein Toolkit Frank Löffler Center for Computation and Technology Louisiana State University, Baton Rouge, LA Supported by NFS grants / / /

2 What is the Einstein Toolkit?

3 What is the Einstein Toolkit? Not quite so fast

4 Challenges in Computational Astrophysics

5 Challenges

6 Challenges Astrophysical Challenges Computational Challenges

7 Challenges Astrophysiıcal Challenges Computatiıonal Challenges

8 Astrophysics Challenges 天体物理的难题 Black Holes and Neutron Stars 黑洞和中子星 Supernovae 超新星 Cosmology 宇宙学 Gravitational Wave Data Analysis 引力波数据分析 Frank Lo ffler The Einstein Toolkit

9 Gravitational Wave Physics

10 Models, Equations Einstein equations: Theoretical form in 4-dimensional spacetime... G µν = 8πT µν When reformulated for 3-dimensional space with time evolution fully 2nd order PDE ( 偏微分方程 ) evolution equations 4 coupled constraint equations 4 gauge conditions GRMHD hydrodynamics ( 广义相对论磁流体 ) Radiation transport ( 辐射输运 ) Fully numerical 3D models needed

11 Challenges Astrophysical Challenges Computational Challenges

12 Challenges Astrophysiıcal Challenges Computatiıonal Challenges

13

14

15

16 More and more diverse hardware

17

18 Computational Challenges 计算难题 Simulate cutting edge science Use latest numerical methods Make use of latest hardware Cache Vector SMP parallelism Scale to many nodes

19 Challenges Astrophysical Challenges Computational Challenges

20 Challenges Astrophysiıcal Computatiıonal Challenges Collaboratiıve Challenges Challenges

21 group group? computation group group

22 group group community group group

23 group group community group group

24

25 group group! problems group group

26 group group! standards group group

27 C/C++ C! tools Perl/C Fortran

28 group group! credit group group

29 What is the Einstein Toolkit?

30 What is the Einstein Toolkit? Set of tools for numerical relativity

31 Community Effort! 社区做起

32 Einstein Toolkit (爱因斯坦工具箱) Supported by NSF (美国国家科学基金会) grants ( / / / ) Set of tools for numerical relativity, open source Over 50 contributors over the past decade, both physics and CS Currently 95 members from 50 sites worldwide 9 maintainers from 6 sites > 200 publications, > 30 theses building on these components Regular, tested releases User support through various channels Frank Lo ffler The Einstein Toolkit 爱 因 斯 坦 工 具 箱

33

34 Guiding Principles Open, community-driven software development Separation of physics software and computational infrastructure Completely modular Portable Stable interfaces, allowing extensions Simplify usage where possible, to be able to concentrate on physics

35 Modules C, C++, Fortran 77, Fortran 90 Encapsulating some functionality Rarely use MPI directly Communication/Configuration via defined APIs grid setup and memory allocation (driver) input/output interpolation initial data boundary conditions evolution systems equations of state remote steering (e.g. https server)

36 Basis Module Overview Basis for scalable algorithm development Most used: finite differences on structured meshes Parallel driver components Simple Unigrid Carpet: Multipatch, Mesh- refinement Method of lines Interfaces to external Libraries/Tools Interface to elliptic solvers (e.g. PETSc, Lorene) Input/Output: HDF5 Visualization: VisIt, OpenDX, Vish Other: PAPI, Hypre, Saga, Flickr, Twitter

37 Module dependencies

38 Basic Formalism: ADM Choose initial spacelike hypersurface Provide initial data (3-metric, extrinsic curvature) Choose next hypersurface: gauge(lapse, shift) Evolve 3-metric, extrinsic curvature

39 Modularizing on the equation-level

40 The Einstein Equations

41 spacetime curvature

42 spacetime curvature constants

43 spacetime curvature constants matter

44 spacetime curvature constants hydrodynamics matter el.-magnetism particle radiation

45

46 ADMBase

47 ADMBase TmunuBase

48 ADMBase TmunuBase ML_BSSN

49 ADMBase HydroBase TmunuBase ML_BSSN

50 ADMBase HydroBase TmunuBase ML_BSSN GRHydro

51 Initial Data / Analysis ADMBase HydroBase TmunuBase ML_BSSN GRHydro

52 ID / Analysis ID / Analysis ID / Analysis ADMBase HydroBase TmunuBase ML_BSSN G-evol G-evol GRHydro T-evol T-evol

53 Scaling Unigrid scales to hundreds of thousands of cores Production runs use 10 levels of mesh refinement, nested grids of size 60x60x60 Current mesh refinement runs scale up to 15k cores Runtime from weeks to few months

54 Programming Languages within the Toolkit ~/ET_2013_05$ cloc --skip-uniqueness lib src doc arrangements Makefile utils simfactory manifest repos text files unique files files ignored. v 1.56 T=116.0 s (29.0 files/s, lines/s) Language files blank comment code C C Fortran C/C++ Header Fortran Perl Bourne Shell Objective C MATLAB HTML make Python m Bourne Again Shell Pascal Lisp D awk Objective C vim script SUM:

55 Convenience Tools GetComponents 获得组件 Simfactory 模拟工厂 Formaline 福尔马林

56 Tools: GetComponents Task: Collect software from various repositories at different sites Example simulation assembly: Cactus Flesh and Toolkit (svn.cactuscode.org) Core Einstein Toolkit (svn.einsteintoolkit.org) Carpet AMR (carpetcode.org, git) Tools, Parameter Files and Data (svn.einsteintoolkit.org) Group Modules (x.groupthorns.org) Individual Modules (x.mythorns.org) x: cvs, svn, darcs, git, hg, http

57 Tools: Simulation Factory Task: Provide support for common, repetitive steps: Access remote systems, synchronize source code trees Configure and build on different systems semi-automatically Provide maintained list of supercomputer configurations Manage simulations (follow best practices, avoid human errors)

58 Tools: Formaline Task: Ensure that simulations are and remain repeatable, remember exactly how they were performed Take snapshots of source code, system configuration; store it in executable and/or git repository Tag all output files

59 Credits, Citations In academics: citations, citations, citations! For Einstein Toolkit: Open and free source No requirement to cite anything However: requested to cite a few publications Which publications: One, maybe two for the Toolkit itself Some components list a few as well List published on website and manage through publication database

60 Vision Roughly from near-term to long-term: Inclusion of more physics, e.g. more general MHD, more equations of state Improve scaling of existing mesh methods Radiation (neutrino, EM) Chemical reactions Fundamental extension/change of numerical methods Requirement-based scheduling Completely data-driven programming?

61 Summary Numerical relativity community: Now generally comfortable with sharing software Didn t happen overnight Some fundamental issues resolved first (e.g., BH-BH) Einstein Toolkit: Set of tools for numerical relativity Free software, developed by large number of groups Where to start: New users tutorial: for New Users Users mailing list: users@einsteintoolkit.org

62 Questions?