Hideyuki Usui 1,3, M. Nunami 2,3, Y. Yagi 1,3, T. Moritaka 1,3, and JST/CREST multi-scale PIC simulation team 1 Kobe Univ., Japan, 2 NIFS,Japan, 3 JST/CREST,
Outline Multi-scale interaction between weak dipole field and solar wind A new PIC code using adaptive mesh refinement and its parallelization Summary
Small scale dipole field interaction Mag-sail Converts solar wind momentum to the thrust through interaction with dipole field Spacecraft with a superconducting coil Earth s magnetosphere This system is based on the solar wind interaction with a weak dipole field Magnetopause located around 10 R e (10s kilometers) Ion inertia length in solar wind (S i ): 100 km Magnetosphere is much larger than S i Magnetic field structure is smaller than S i
Magnetosphere formation (smaller than ion inertia length) In a case of weak dipole field, almost no interaction will take place because ions are very little affected by the dipole field. However, Jz Ion inertia length -10 10 0 x/s e Electron scale magnetosphere is created Cross-field current at the boundary to sustain the field structure
When S i is comparable to or larger than L We think ions do not stagnate at the front side and tend to pass through the dipole field because of large Larmor radius. ions s/c coil Plasma flow L (determined by pressure balance) S i (ion inertia length)
Nevertheless, magnetic fields are compressed at the distance of L. Magnetized electrons are decelerated in the dayside region. Then charge separation takes place between electrons and ions and strong electrostatic field is generated. Ions dynamics are modified by the electrostatic force They are indirectly affected by the magnetic field even in a weak dipole field ions Compression region s/c coil Plasma flow ions L (determined by pressure balance) S i (ion inertia length)
The location of the compression is determined mainly by ion dynamic pressure. However, the thickness of the magnetic compression and current layer is comparable to the local electron Larmor radius S e (electron Lamor radius) Compression region s/c coil Plasma flow L (determined by pressure balance) S i (ion inertia length)
Multi-scale plasma particle simulation Multi-scale characteristics Electron scale : ~ 1km Dipole field: ~ 1-10 km Ion inertia length: 100 km Multi-scale analysis including plasma kinetic effects Conventional 3D EM-PIC simulation codes hire uniform and fixed spatial grid system needs huge resources. For the analysis of non-uniform multi-scale phenomena it is better that the resolution is adaptively adjusted during a simulation run, depending on local plasma phenomena. Combination of AMR(Adaptive Mesh Refinement) and PIC
Application of AMR to PIC Example of AMR grid system Solar wind FTT (Fully Threaded Tree) data structure Hierarchical system of grids is maintained by pointers. Each cell is treated as an independent unit organized in a refinement tree structure rather than conventional element of arrays, Level L cell cell cell cell particle neighbor Level L+1 parent child cell cell cell cell Ion density profile for the magnetospheresolar wind interaction FTT structure using pointers enables us to create or destroy the hierarchical structure of grids very easily particle particle
Cells are defined by structure in Fortran90 which consists of pointers to neighboring cells as well as parent and child cells, Level of hierarchical system, some flags, Spatial position & Physical data (Fields, Current density ).
A test simulation with AMR scheme Expansion of laser produced plasma in B-field. Fine grids are adaptively created at the region where the plasma density exceeds a certain value Current version is relatively slow. We have been parallelizing the code. The results agree with those with uniform fine grids Electron density Mesh Refinement 11
Thread parallelization using one node Fujitsu SPARC Enterprise M9000 (kyoto Univ.) (SPARC64 VII, Peak performance 40GFLOPS) Close to 10 % of the theoretical peak performance Usage of Cell structure using pointers is not bad at all for the parallelization. 12
Benchmark : weak scaling Plasma simulator @ NIFS Increase the number of fixed sub-domains and Data transfer of boundary values with MPI The performance increases almost linearly (parallel efficiency > 90% for 2048 cores) and has 7-8% of theoretical peak. High computational performance is achieved even with FTT (fully threaded tree) data structure using pointers and structures in Fortran 90 13
Load balance in Domain decomposition parallelization In AMR, the number of cells in the whole system dynamically changes in time. A simple domain decomposition with the fixed regular sub-domains cannot guarantee the load balance between processors because the number of cells assigned to each sub-domain is different. Example Distribution of plasma cluster AMR Density (A.U.) Domain decomposition Homogenous separation of area Load balance is not achieved!
Dynamic Domain Decomposition (DDD) scheme To keep the load balance between processes during a simulation run, we are developing dynamic domain decomposition (DDD) scheme. 1. Use a space-filling curve with the Morton ordering method. Number all the base cells (Level-0) in such a manner that neighboring cells are closely ordered in a series 1 2 5 6 3 4 7 8 9 10 13 14 11 12 15 16
2. Calculate work weight. Since cost of particle calculation is dominant in PIC simulation, we consider the number of particle calculation loops for each cell and obtain a work weight. N particle : the number of particle in each cell N process : the number of process in parallelization L: the hierarchical level of the corresponding cell 3. Divide the Morton ordering curve into the number of process Decompose the domain so that the work weight becomes the same in each sub-domain. Each process has approximately the same amount of load.
Summary We started developing a new plasma particle code for multi-scale simulation by incorporating AMR. To achieve the load balancing in the domaindecomposition parallelization, we consider the load of the particle loop by using the Morton ordering. We have been investigating interaction between a small magnetosphere and the solar wind. As a future work, we will complete our AMR-PIC code and apply to the multi-scale particle simulations.