An Efficient Implementation of Multiscale Simulation Software PNP-cDFT

Transcription

1 Mater. Res. Soc. Symp. Proc. Vol Materials Research Society DOI: /opl An Efficient Implementation of Multiscale Simulation Software PNP-cDFT Da Meng, Guang Lin* and Maria L. Sushko Pacific Northwest National Laboratory, Richland, WA 99352, U.S.A. ABSTRACT An efficient implementation of PNP-cDFT, a multiscale method for computing the chemical potentials of charged species is designed and evaluated. Spatial decomposition of the multi particle system is employed in the parallelization of classical density functional theory (cdft) algorithm. Furthermore, a truncation strategy is used to reduce the computational complexity of cdft algorithm. The simulation results show that the parallel implementation has close to linear scalability in parallel computing environments. It also shows that the truncated versions of cdft improve the efficiency of the methods substantially. INTRODUCTION A novel hierarchical multiscale model has been proposed to study the mechanism of ion and electron transport in the material at the nano- to micrometer scales [1-4]. The method couples classical density functional theory (cdft) with the Possion-Nernst-Plank (PNP) formalism, where cdft is used for mesoscale study of the interactions of particles and the PNP is used for dynamic effects. The application of the method to complex system, however, is limited by its computing complexity. In this paper, an efficient implementation of cdft is introduced. THEORETICAL MODEL Diffusion channel (Figure 1) [3] is represented as a uniform medium. Li + migrates in the channel from one interstitial lattice site to another. In a 1D model, these equilibrium lattice sites are represented as a one-dimensional array of stationary points, i.e. only sites I 0 are considered and Li + migrates is through hopping between I 0. In a 3D model, sites II 0 and II* are also considered, the motion of Li + may follow different paths. The flux of charged particles in stationary conditions is calculated within the Possion- Nernst-Plank (PNP) formalism [1-3]: = () " + 1 " 1 " = 0 " + " " + " " ()() " = In these equations, J i are the fluxes for Li + and electrons, D i (z) and ρ i (z) are their diffusion *Corresponding author. address: guang.lin@pnnl.gov

2 coefficients and densities along the channel (z axis) in a 1D case, respectively, A(z) is the cross section of the channel, ϕ is the electrostatic potential, µ0 and µex are the ideal and excess chemical potential of Li+ and electrons, respectively, kt is the thermal energy, and e is the electron charge. In this system of equations, the first describes the flux, the second is the stationary condition, and the third is Poisson s equation for the calculation of the electrostatic potential. Figure 1. Diffusion channel of Li+ in LiPON. Blue spheres are interstitial equilibriums (I0); Yellow spheres (II0) and gray spheres (II*) are metastable sites. a, 2b and 2c are the size along three crystallographic directions. The cdft is used to evaluate the chemical potentials of charged species. In this model, the total free energy is divided into two parts, the ideal part (Fid), which includes the contributions from the configurational entropy of the non-interacting species, and bonding enthalpy, if any, and the excess free energy (Fex), which has contributions from all interactions in the system. In the case of charged species in the channel, these include the free energies of Coulomb interactions (C), electrostatic correlations (el), hard sphere repulsion (hs), and shortrange interactions (sh) with the stationary points [1-4] (see [4] for more details): " " " = " + "" + + " = " " 1, "" = "" "#$ " 2 " " "#$, " " ( "#$ )( "#$ ),, " = " " = " 1 = 2 2,,, Φ [ ()] ",,

3 Approximate Truncated Computation Among the PNP-cDFT model, both PNP and cdft are time consuming among where the complexity of cdft is O(n 2 ) where is the number of particles in the system since the computation involves the interaction of all particles in the system. To reduce the high computational complexity, an approximation will be used for computing the potentials of charged species. This is based on the observation that the potential values for a particle are highly related to the particles around it while only weakly related to particles which are far from itself. Figure 2 shows W 0, which is one of interacting coefficients between particles, decreases exponentially when the distance of two particles increases. This suggests that we can truncate the computation without comprising the accuracy of computation too much. W 0 Distance between particles (in grid unit) Figure 2. Exponential decrease of W 0 with the distance between particles. Parallelization The parallelization of PNP-cDFT was implemented based on spatial decomposition. The whole system is partitioned into equally-sized subsystems. For 1D system with length, the system is partitioned along direction as shown in Figure 3. For 3D system with lengths,, and along,, and direction, each subsystem is represented by a rectangle cube. The computation of different subsystem will be assigned to different processors. Since the truncated strategy is used, most of the computation needs only data local to the current processer and there is no massive data communication involved. In practice, load balance is important for the performance of parallelization [5]. Imperfect load balancing leads to increased execution time. In our implementation, the task granularities are reduced, i.e., each subsystem is decomposed to generate smaller tasks. The assignment of these tasks is managed by a task manager. When a process finishes current subtask it acquires a new subtask from task manager. The cdft stops when all subtasks are completed.

4 Figure 3. Space decomposition of cdft. The model was implemented in a Fortran code, which was used in all calculations reported here. The message passing interface (MPI) is used for parallel implantation of PNP-cDFT. The PNP equations were solved numerically using Newton s method for a 1D system and using successive overrelaxation (SOR) for the 3D model [6]. Scaling test was carried on PNNL Institutional Computing (PIC) cluster environment. RESULTS AND DISCUSSION We have applied the proposed method to both a 1D system and a 3D system. The settings of 1D system is as followings: channel width is nm, channel length is 30c (8.877 nm), Li + ions were spherical particles with charge q+ = 1 and diameter 0.06 nm. Electrons with the parameters q- = -1 and diameter σ- = nm. The concentration of Li + ions and electrons in the channel was 0.01 M. A uniform grid of points separated by the distance of σ + /10 is used. The convergence criterion was a 10-6 decrease in the difference between the next solution of the system of equations and the previous one. Scaling results of parallelization (Figure 4) show that near-linear scaling performance with the number of processors can be obtained for the cdft computation. The complexity of the algorithm is also tested by increasing the number of stationary sites in the model. As shown in Table I, the complexity has been greatly reduced Time(minutes) Number of processors Figure 4. The scaling of 1D model with repect to number of processors used in the computation.

5 Table I. Number of stationary sites vs cdft running time. Number of stationary sites Running time (s) CONCLUSIONS In conclusion, we have developed an efficient implementation based on truncated cdft and simulations show that the implementation has near to linear scalability with respect to the number of processors used in parallel environments. There are some remaining issues that need to be addressed later. To begin with, the dimensions of particles are assumed to be the same in the system and inclusions of the dimensions of particles may improve the simulation results. Another work is to implement an efficient way, e.g. multi-grid methods, for solving the PNP equations. ACKNOWLEDGMENTS The development of the PNP-cDFT software is supported by the Laboratory-Directed Research and Development Program at Pacific Northwest National Laboratory (PNNL). The study of surfactant self-assembly at surfaces is supported by the U.S. Department of Energy, Office of Basic EnergySciences, Division of Materials Sciences and Engineering under Award KC FWP PNNL is a multiprogram national laboratory operated for DOE by Battelle under contract DE-AC05-76RL REFERENCES 1. Maria L. Sushko, Kevin M. Rosso, and Jun Liu, Mechanism of Li+/Electron Conductivity in Rutile and Anatase TiO2 Nanoparticles, J. Phys. Chem. C, 114, (2010) 2. Maria L. Sushko, Kevin M. Rosso, and Jun Liu, Size Effects on Liþ/Electron Conductivity in TiO2 Nanoparticles, Chem. Phys. Lett. 2010, Maria L. Sushko, Kevin M. Rosso, Ji-Guang (Jason) Zhang, and Jun Liu, Multiscale Simulations of Li Ion Conductivity in Solid Electrolyte, Chem. Phys. Lett. 2011,2, Maria L. Sushko and Jun Liu, Structural Rearrangements in Self-Assembled Surfactant Layers at Surfaces, J. Phys. Chem. B, 114, (2010) 5. Willebeek-LeMair, M.H. and A.P. Reeves, Strategies for dynamic load balancing on highly parallel computers, Parallel and Distributed Systems, IEEE Transactions on, vol.4, no.9, pp , Sep M G Kurnikova, R D Coalson, P Graf, A Nitzan, A lattice relaxation algorithm for threedimensional Poisson-Nernst-Planck theory with application to ion transport through the gramicidin A channel, Biophys J February; 76(2):