COSMOLOGICAL SIMULATIONS OF THE UNIVERSE AND THE COMPUTATIONAL CHALLENGES. Gustavo Yepes Universidad Autónoma de Madrid

Transcription

1 COSMOLOGICAL SIMULATIONS OF THE UNIVERSE AND THE COMPUTATIONAL CHALLENGES Gustavo Yepes Universidad Autónoma de Madrid

2 Numerical simulations in Cosmology: what are they useful for? They help to understand how galaxies and other structures formed in the Universe

3 The Ingredients of the Universe

4 Physical Processes in Cosmological Simulations

5 Computational Resources Gravity: N-body method (O(N 2 )) New algorithms (tree, pm, etc) can scale as O (N log N) MPI implementations of N-body algorithms using domain decomposition Cold Dark matter fluctuations have contribution at all scales: Need to resolve many scale lenghts simultaneously, N > partícles RAM Memory > 10 Tbytes Large number of processors.> 5,000 Gas dynamics: AMR and SPH codes More demanding than pure N-body due to Courant Condition Very large dynamical range and strong shocks show up in the fluid. Data Storage and Visualization Huge datasets generated (4 dimensional problem) Need of access to large storage systems. Parallel codes for visualization of billions of computational elements.

6 Moore s Law for Cosmological N-body Simulations Moore`s Law: Capacity of professors double every 18 months Millenium Run N.Body simulations double the number of particles every 16.4 months Hubbel Volume Extrapolating: partículas in 2008 but it was done in 2004 Springel et al 2005

7 IBM Regatta p690+ cluster 512 Power4 processors 1 Tbyte RAM 500 wall clock hours 300,000 HOURS CPU

8 Grand Challenges in Computational Cosmology Most of the information from the Universe comes from the tiny fraction of normal matter: baryons Galaxies detected from light coming from the stars inside. Gas in galaxy clusters is detected by X-ray emission. Intergalactic gas is measured from features in light from QSO Realistic simulations would require inclusion of dissipantional component: gas Gasdynamical simulations in dark matter dominated models need more than an order of magnitude larger computational resources than pure N-body

9 Grand Challenges in Computational Cosmology Galaxy formation does not involve only gravity and gasdynamics : but more complex physics: Cooling, heating, star formation, supernova explosions,feedbacks,. Extreme Computing intensive simulations Larger scale range in density, space and time Strong time constrains due to cooling and star formation. Huge Problems with scaling and load/balancing in MPI. E.g. GADGET scales well only for few dozens of processors. OpenMP or MPI+OpenMP codes can perform better More inforrmation to store and postprocess.

10 Grand Challenges in Computational Cosmology Multi-billion particle simulations with N-body and gasdynamics in large volumes X-ray Galaxy clusters distributions The problem of missing baryons. Galaxy formation simulations: First objects formed in early times Distribution of primeaval galaxies The smallest visible galaxies and the small scale structure problem Tool to understand future observations with next generation telescopes

11 The MareNostrum Numerical Cosmology Project (MNCP) Goal:Take advantage of the uprecedented computing power of MareNostrum to create grand challenge cosmological simulations of the Universe. Different scales and physics: Large Scale Structure (adiabatic physics) X-ray Clusters. SZ effect, baryon distribution at Large Scales Galaxy formation: (including star formation) High redshift objects Faint objects in different environments Local Universe Constrained simulations

12 What is the Mare Nostrum? IBM JS20 Blade Center: 4812 PPC processors 9.6 Tbytes of Memory Myrinet interconection. 233 Tbytes disk

13 What is the Mare Nostrum? IBM JS20 Blade Center: 4812 PPC processors 9.6 Tbytes of Memory Myrinet interconection. 233 Tbytes disk Its now 3rd most powerfull supercomputer in Europe. It is located at the BSC in Barcelona

14 What is the Mare Nostrum? IBM JS20 Blade Center: 4812 PPC processors 9.6 Tbytes of Memory Myrinet interconection. 233 Tbytes disk After major upgrading in september (doubling cpu and memory) it will be again the most powerfull supercomputer in Europe. A perfect place to create a Universe, not only because of its power

15 Collaborators: S. Gottlober (AIP Germany) Andrey Kravtsov (U. Chicago) A. Klypin (MNSU) Y. Hoffman (Racah Institute, Israel) R. Sevilla (UAM, Spain) L. A. Martínez (UAM Spain) M. G. Rivero (UAM, Spain) M. Hoeft (Bremen, Germany) Massimo Meneghetti (ZAH, Heildelberg) Christian Wagner (AIP Germany) Arman Khalatyan (AIP, Germany) V. Turchaninov (IAM Moscow) A. Faltenbacher (USC, California) M. Plionis, S. Basilakos (NOA, Greece) F. Atrio (USAL, Spain) A. Doroshkevich (Moscow)

16 The MareNostrum Universe TREEPM+SPH simulation LCDM model (WMAP1) 500/h Mpc 3 volume GADGET 2 code (Springel 2005) Adiabatic SPH+TREEPM Nbody FFT for the PM force. 15 kpc force resolution. 2x dark and sph particles partículas 8x10 9 M dark matter 10 9 M for gas particles 1 million dark halos bigger than a typical galaxy (10 12 Mo) Simulation done at MareNostrum 512 processors (1/8 total power) 1Tbyte ram 500 wallclock hrs (29 cpu years) Output: 8600 Gbytes of data. Same computing power than the Millenium Run.

17 Moore s Law for Cosmological N-body Simulations Moore`s Law: Capacity of professors double every 18 months Millenium Run MareNostrum Universe (SPH) Hubbel Volume N.Body simulations double the number of particles every 16.4 months Extrapolating: partículas in 2008 but it was done in 2004 Springel et al 2005

18 Moore s Law for Cosmological N-body Simulations Theoretical N-body limit at MN Moore`s Law: Capacity of professors double every 18 months Millenium Run MareNostrum Universe (SPH) Hubbel Volume N.Body simulations double the number of particles every 16.4 months Extrapolating: partículas in 2008 but it was done in 2004 In fact it is possible to do in 2006 An order of magnitude over less than 2 years Springel et al 2005

19 Halo Mass function 4063 clusters with M > h -1 Msun groups+clusters with M >10 13 h -1 Msun objects with M > h -1 Msun More than 1 million objects with more than 100 particles

20 Shapes of halos Shape of the dark matter distribution at virial overdensity: halos with more than 500 particles

21 Shape of halos Shape of dark matter halos at Virial overdensity Shape of the corresponding gas distribution in same halos

22 Baryon phase space diagram

23 Baryon phase space diagram

24 Baryon phase space diagram

25 Baryon phase space diagram HOT Warm-Hot COLD

26 Evolution of baryon phases (T > 10 7 K) (10 5 K < T < 10 7 K) (T < 10 5 K)

27 Evolution of baryon phases All baryons WHIM baryons

28 Clusters of galaxies Most massive cluster:

29 X-ray Temperature function

30 Scaling relations: Resolution effects Comparison of Lx for two versions of the same simulation: Colored points: MareNostrum run Black points: low-res version with 2x512 3 particles: (Basilakos et al 2005 Ascasibar et al

31 Scaling relations: Resolution effects Comparison of Lx for two versions of the same simulation: Colored points: MareNostrum run Black points: low-res version with 2x512 3 particles: (Basilakos et al 2005 Ascasibar et al ) Green Points are data..

32 The MareNostrum SIMULATION OF GALAXY FORMATION Gasdynamics and N-body with 2 billion particles + Detailed modeling of baryonic physics.

33 BARYONIC PHYSICS: To study in detail the galaxy formation process we need to account at least for Radiative and Compton cooling UV-photoionization Multiphase ISM. Star Formation. Star-Gas backreactions. Use Springel-Hernquist (2003) implementation of multiphase SPH modeling in GADGET-2.

34 MareNostrum galaxy formation simulation z = field of view Box 50/h Mpc 2x 10 9 gas+dark LCDM model Mgas=1.4 x10 6 Msun Mdark=10 7 Msun. Mhalos > 10 9 Msun Gas density

35 MareNostrum galaxy formation simulation Box 50/h Mpc 2x 10 9 gas+dark LCDM model Mgas=1.4 x10 6 Msun Mdark=10 7 Msun. Mhalos > 10 9 Msun Mass Function z = 11.2 z = 7.3 z = 5.6

36 MareNostrum galaxy formation simulation z = 5.6 Box 50/h Mpc 2x 10 9 gas+dark LCDM model Mgas=1.4 x10 6 Msun Mdark=8x10 6 Msun. Mhalos > 10 9 Msun Star density Fraction stars = 0.74% Comoving Luminosity density: r U = 7.6x10 7 Lsun/Mpc 3 r B = 1.3x10 8 r R = 1.0x10 8 r I = 7.2x10 7 r K = 2x10 7

37 CODE COMPARISON MNCP GADGET (SPH) 800 processors of MN Resolution: 500 pc. 126 YEARS CPU HORIZON -RAMSES (AMR) More than 2000 processors Resolution: 2 kpc 32 YEARS CPU

38 GRID COMPUTING IN COSMOLOGICAL SIMULATIONS Difficult to use standard grid computing to perform N-body simulations Strong coupling of particles due to gravity large communication among processors Grid technology can help in the analyses of large simulated datasets: e.g Insbruck Numerical Simulation Group use of Austrian Grid Mutimass simulations can also benefit from grid supercomputing : Select objects from low resolution simulation and resimulate many of them independently using grid technology to distribute the runs among many multiprocessor computers. An example of a grid useful in computational astrophysics is DEISA

39 DISTRIBUTE EUROPEAN INITIATIVE FOR SUPERCOMPUTER APPLICATIONS 10 supercomputer centres interconnected in GRID. France, Finland, Netherland, UK, Italy, Spain, Germany DEISA Network: Heterogeneous supercomputer grid Fundamental Concept Global parallel filesystem 10% of computing time is allocated to DECI projects: : 29 projects approved. 8 Computational Astrophysics 3 Computational Biology 4 Material science 4 Earth Science 4 Fluid dynamics 2 Computational chemistry 2 Particle-Nuclear Physics 1 Quantum computing 1 Combustion

40 DECI PROJECTS 2006: Computational Astrophysics projects: SIMU-LU (N-body, G. Yepes, Spain) LFI-sim (Planck mission, F. Pasian,Italy) HORIZON (Galaxy formation, Devrient, France)) STARS (Stellar MHD, Brun, France) FEARLESS (Star formation, Niemeyer, Germnay) GIMIC (Galaxy formation, S. White, Germany) SUPERNOVA (Type IA supernova simulations Hilebrandt, Germany) REIONIZATION (Radiative transfer simulations, Mellema, Netherlands)

41 SUMMARY Cosmological Numerical simulations of galaxy and structure formation are the most useful tools to investigate how the unobserved universe will look like: Universe at early epoch Faintest objects in different environments. One the biggest challenges for HIGH PERFORMANCE COMPUTING. The MareNostrum Numerical Cosmology Project is intended to make state-of-the art numerical simulations with multi-billion particles that will resolve the first objects formed at high redshifts and the smallest objects that survive at present in different structures. These simulations will be useful, among many other things, as a complementary tool to prepare observations with the next generation detectors that will be able to explore the unknown objects in the Universe. GRID Computing not very common in Numerical Cosmology yet. Useful tool for analyzing big numerical simulations. DEISA and similar initiatives of grid supercomputing are very important for the development of this field

42 Thank You!