One-sided Communication Implementation in FMO Method

Transcription

1 One-sided Communication mplementation in FMO Method J. Maki, Y. nadomi, T. Takami, R. Susukita, H. Honda, J. Ooba, T. obayashi, R. Nogita,. noue and M. Aoyagi Computing and Communications Center, yushu University Fukuoka ndustry, Science & Technology Foundation

2 Overview of FMO method (1) FMO(Fragment Molecular Orbital method) has been developed by itaura (AST) and co-workers to calculate electronic states of a macromolecule such as a protein and nucleotide The target molecule is divided into fragments (monomers) ab initio molecular orbital (MO) calculation is carried out on each fragment and fragment pair (dimer). Usually executed with high parallel efficiency (>90%) The method can execute all-electron calculation on a protein molecule with 10,000 atoms.

3 Example of fragmentation

4 Overview of FMO method (2) monomer calculation Hamiltonian environmental electrostatic potential ρ J ( ri) [ ] J i H h V = + + i J i> j ( r) 1 r r (potential from other monomers) electron density of monomer J V J A monomer calculation depends on other monomers electron density A i j i, j ( r ) J Z ρ J A = + r R r r dr iterated until all electron densities unchanged (SCC procedure)

5 Overview of FMO method (3) Schrødinger equation H Ψ = E Ψ RHF (Restricted Hartree-Fock) SCF method Molecular Orbital (MO) { C ij} MO coefficients basis functions { χ } j electron density D = 2 CC jk i ji ki ϕi ( r) = χ ( r) C j j ρ = 2 χ χ CC = χ χ D ji j k ji ki j jk jk i jk density matrix elements N N b b N b number of basis functions

6 Overview of FMO method (4) dimer calculation Hamiltonian environmental electrostatic potential ( ri) [ ] i J H h V = + + i J, J i> j J V ( r) A A 1 r r i i, j j ( r ) Z ρ A = + r R r r dr Schrødinger equation Total energy H Ψ = E Ψ J J J J J E = E ( N 2) E > J not all dimers are calculated by SCF

7 dimer-es approximation For distant dimers, SCF calculation are not carried out Energy is calculated by electrostatic approximation ES (non-scf) dimer monomer SCF dimer ρ J ( r ) ρ ( r ) J J 1 2 E E + E dr1dr2 r1 r2

8 Flow chart of FMO calculation initial density calculation { D 0 } electronic structure calculation for monomer H not yet converged Ψ = E Ψ convergence check of SCC procedure already converged electronic structure calculation for SCF-dimer J J J J H Ψ = E Ψ SCC procedure ES dimer calculation J ρ ( r ) ρ ( r ) J J 1 2 E E + E dr1dr2 r1 r2 total energy calculation J E = E ( N 2) E > J For all monomers For all SCF-dimers For all separated dimers

9 Density requirements and update in FMO calculation initial density calculation { D 0 } electronic structure calculation for monomer H not yet converged convergence check of SCC procedure already converged electronic structure calculation for SCF-dimer J J J J H ES-dimer calculation J ρ ( r ) ρ ( r ) total energy calculation Ψ = E Ψ Ψ = E Ψ J J 1 2 E E + E dr1dr2 r1 r2 J E = E ( N 2) E > J SCC procedure update monomer density matrices refer monomer matrices needed in electronic structure calculation update monomer density matrices refer monomer matrices needed in electronic structure calculation refer two monomer matrices needed in dimer-es calculation For all monomers For all SCF-dimers For all separated dimers Update all monomer density matrices Refer some monomer density matrices

10 matrix element of χ V = χ r V r r dr ij i j A ( r ) ρ = χ r χ r r + χ r χ r dr dr kl, Z A i j d i j r RA i, j r r involves 4 center tow electron integrals ρ ( r ) χi ( r) dr χ j ( r) dr r r = V ij ( r) ( r) ( r ) ( r ) χ χ χ χ i j k l dr dr Dkl r r order of calculation costs Approximation of V J calculation (1) J V r (needed in SCF calculation) b N b O N b 4-center two electron integral calculations for all environment monomers are prohibitive number of basis functions

11 Approximation of V J J calculation (2) esp-aoc approximation ( r) i r i r D S ρ χ χ S i ( D S ) ii ( r ) χ ( r ) χ V ( r) + dr D S r R r r Z A i i A overlapmatrix of fragment ( S = χ r χ r dr ) Mulliken AO population of A i ii ij i j χi ( r) ii

12 Approximation of V J J calculation (3) esp-ptc approximation ρ ( r) δ ( r R ) Q Q A A A Q = D S = D S A A ii ij ji i A i A, j point charge approx. of Mulliken atomic charge of the nucleus A V A A ( r) Z Q + r R r R A A A A N4C number of environment monomers for which 4-center two electron integral calculations are carried out

13 Hypothetical petascale computing environment Number of CPUs : 1PFlops = 100,000 CPU (current one CPU peak performance ~10GFlops) Number of fragments N f : 10, ,000 (the current largest N f ~1,000)

14 Memory requirement in FMO Number of fragments The size of one density matrix D 350B nterfragment distance data N 2 f / 2 N f R J N f D R J 1, MB 4MB 5, GB 95MB 10, GB 380MB 50,000 17GB 9.3GB 100,000 34GB 37GB Memory per one CPU core > 10 GB t is difficult for each process to store all necessary data

15 OSC implementation in OpenFMO (1) n its execution, only the process of rank 0 is used as a server process for dynamic load balancing. All the other processes are worker processes and are divided into groups and the two-level parallelization is used as the other implementations. Group00 Group01 Group02 node00 node01 node02 node03 node

16 OSC implementation in OpenFMO (2) Group00 Group01 Group02 node00 node01 node02 node03 node MP_Get memory window created by MP_Win density matrix data

17 Estimation of communication cost (1) Assumptions 1. The sizes of all density matrices are the same.all groups have the same number of worker processes. 2. The time to put or get one density matrix is equal to the average point-to-point communication time. 3. MP_Bcast implements a binomial tree algorithm so that the time to broadcast one density matrix over N processes is obtained by t log N. p2p 2 4. And for OSC scheme, dynamic load balancing works well and the delay due to competing put or get requests is ignorable. Therefore all groups execute the same number of monomer and dimer jobs and this means that all groups have the same Nproc / Ngroup

18 Estimation of communication cost (2) N proc N f Nproc / N group 10, ,000 96, NSCC mon N4C dim N4C dim N SCF tp2p SCC iterations neighboring monomers (average) neighboring monomers (average) monomers with which a monomer forms a SCF dimer SCF dimers N N f dim /2 SCF point-to-point communication time to transfer one density matrix

19 Estimation of communication cost (3) OSC scheme number of monomer get requests total number of dimers number of SCF dimers number of SCFdimer get requests N N N N N mon mon get = SCC f 4C + 1 f N N f N 1/2 f dim /2 SCF dim dim dim get = f SCF 4C + 2 /2 N N N N number of ES dimers number of ES dimer get requests N N 1/2 N N /2 f f f dim SCF es dim Nget = N f N f 1/2 N fnscf /2 2 total number of get requests N = N + N + N mon dim es get get get get = N N ( N + 1) SCC f mon 4C + N N N /2+ N N 1 dim dim f SCF 4C f f

20 Estimation of communication cost (4) OSC scheme cost of MP_Bcast cost for one get requests tp2p log2 N t ( 1+ log N / N ) p2p 2 proc group total cost of put requests total cost per one group T = N N t / N OSC SCC f p2p group N N t / N SCC f p2p group + t 1+ log N / N N / N p2p 2 proc group get group Bcast scheme total cost per one group = T N N t log N Bcast SCC f p2p 2 proc

21 Estimation of communication cost (5)

22 Simulation using skelton code (1) FMO calculation was simulated using skelton code quantum calculation parts are removed Skelton code accumulates estimated time of : molecular integral calculation Fock matrix builging, SCF procedure Time of each communication are estimated and accumulated MP_Send, MP_recv, MP_get, MP_put estimated from measurements MP_Bcast, MP_Allreduce estimated from point to point communication time by assuming binomial tree, butterfly algorithm

23 Simulation using skelton code (2) integral calculation time estimation formula integrals required for environmental potential 1 electron f N = AN + BN A= E-05 B= E-08 f N = AN + BN 2 electron (3center) A= E-05 B= E-07 Fock builging time estimation env 2 1e b b b env 2 2e3C b b b 2 electron (4center) f N = AN + BN A= B= E-06 integrals required for SCF f N = AN + BN A= E-05 B= E-07 A= E-05 B= E-06 env 2 2e4C b b b 2 3 2e b b b 2 3 = + f N AN BN Fock b b b

24 simulation using skelton codes (3) Aquaporin protein PDBD 2F2B, N f =492, 14,492 atoms 6-31G* basis set Nproc / N group =16 N group =4, 8, 16, 32, 64 OSC cost < Bcast cost with Nproc > 512 ( Ngroup > 32) Linux cluster in REN Super Combined Cluster (RSCC) was used

25 Summary OSC of MP-2 standard has been implemented in a new FMO code to reduce the memory requirement per process. OSC implementation also makes the exchange of by broadcast unnecessary. Evaluation of communication costs shows that OSC scheme has an advantage over the Bcast scheme for N f =10, ,000, Nproc =96,000. Simulations using skelton code also show that OSC scheme has an advantage in communication cost with large Ngroup. These results show OSC scheme is prospective for the petascale computing environment. D