A scalable parallel algorithm for large-scale reactive force-field molecular dynamics simulations

Transcription

1 Computer Physcs Communcatons A scalable parallel algorthm for large-scale reactve force-feld molecular dynamcs smulatons Ken-ch Nomura Rav K. Kala Achro Nakano Prya Vashshta Collaboratory for Advanced Computng and Smulatons Department of Computer Scence Department of Physcs & Astronomy Department of Chemcal Engneerng & Materals Scence Unversty of Southern Calforna Los Angeles CA USA Receved 18 June 2007; receved n revsed form 16 August 2007; accepted 17 August 2007 Avalable onlne 15 September 2007 Abstract A scalable parallel algorthm has been desgned to perform multmllon-atom molecular dynamcs MD smulatons n whch frst prncplesbased reactve force felds ReaxFF descrbe chemcal reactons. Envronment-dependent bond orders assocated wth atomc pars and ther dervatves are reused extensvely wth the ad of lnked-lst cells to mnmze the computaton assocated wth atomc n-tuple nteractons n 4 explctly and 6 due to chan-rule dfferentaton. These n-tuple computatons are made modular so that they can be reconfgured effectvely wth a multple tme-step ntegrator to further reduce the computaton tme. Atomc charges are updated dynamcally wth an electronegatvty equalzaton method by teratvely mnmzng the electrostatc energy wth the charge-neutralty constrant. The ReaxFF-MD smulaton algorthm has been mplemented on parallel computers based on a spatal decomposton scheme combned wth dstrbuted n-tuple data structures. The measured parallel effcency of the parallel ReaxFF-MD algorthm s on IBM BlueGene/L processors for a 1.01 bllon-atom RDX system Elsever B.V. All rghts reserved. PACS: c; Ns; Db Keywords: Molecular dynamcs; Reactve force feld; Parallel computng 1. Introducton Molecular dynamcs MD s an atomstc smulaton method for studyng a wde class of materals such as metals ceramcs and bomolecules under ambent as well as extreme condtons. Contnuous efforts are beng made to ncorporate chemcal reactons nto MD smulatons so that broad materal processes such as catalyss corroson and detonaton can be studed. Currently a number of software packages are avalable for frst-prncples MD smulatons based on quantum mechancal QM calculatons to descrbe reactve events such as bond breakng and formaton [1]. One of the longstandng problems n materals scence s mechancally nduced chemcal reacton such as stress corroson crackng and shock-nduced detonaton [23] whch may drastcally change materal propertes. A study of chemcal reactons nduced by mechancal loadng s a challengng task because they are often nseparable from geometrcal heterogenety nvolvng mllons of atoms. Ther understandng thus requres the combnaton of vast spatal extent and accurate chemstry. Because of the ntensve computatonal requrement and poor scalablty however the system sze n QM smulatons s usually lmted to 10 2 atoms. One approach to tackle ths problem s the hybrdzaton of dfferent methodologes such as QM/molecular mechancs MM schemes [4 8]. However they requre adaptve selecton of reactve stes and sophstcated treatment of QM/MM boundares whch are nontrval for complex mechanochemcal processes. An alternatve approach s to ft essental chemcal reactons by sem-emprcal force felds [9 11] n whch parameterzed nteratomc potentals are traned to reproduce datasets obtaned wth accurate QM calculatons. Recent advances n reactve force * Correspondng author. E-mal address: anakano@usc.edu A. Nakano /$ see front matter 2007 Elsever B.V. All rghts reserved. do: /.cpc

2 74 K.-. Nomura et al. / Computer Physcs Communcatons felds ReaxFF [12 14] have opened up a possblty to study reactve processes n a wde range of materals such as hydrocarbons [12] slca [13] alumna[15] ntramnes [14] catalysts [16] and polymers [17]. We have recently developed a scalable mplementaton of ReaxFF MD on massvely parallel computers whch provdes us wth the requste couplng of quantummechancally accurate descrpton of chemcal reactons and large length-scale mechancal processes to study mechanochemcal processes under extreme condtons. For example our parallel ReaxFF algorthm has enabled mllon-to-bllon atom MD smulatons on a large number 10 5 of processors to study shock-nduced ntaton of energetc materals wth focuses on the effects of mcrostructures e.g. vods and gran boundares and nanofludcs e.g. nanoets on chemcal reactons [1819]. In ths paper we present the desgn and mplementaton of our parallel ReaxFF algorthm along wth ts scalablty tests on varous parallel computers. Secton 2 descrbes the ReaxFF MD algorthm and ts parallel mplementaton s dscussed n Secton 3. Results of benchmark tests are presented n Secton 4 and Secton 5 contans summary. 2. Reactve force feld molecular dynamcs algorthm In the reactve force feld ReaxFF nteratomc nteractons between N atoms comprse valence nteractons descrbed wth bond order BO as well as noncovalent nteractons [16]: E ReaxFF {r } {r k } {r kl } {q } {BO } = E bond + E lp + E over + E under + E val + E pen + E coa + E tors + E con + E hbond + E vdwaals + E Coulomb 1 n whch the valence nteractons nclude the bondng energy E bond lone-par energy E lp overcoordnaton energy E over undercoordnaton energy E under valence-angle energy E val penalty energy E pen 3-body conugaton energy E coa torson-angle energy E tors 4-body conugaton energy E con and hydrogen bondng energy E hbond.ineq.1 the total energy s a functon of relatve postons of atomc pars r trplets r k and quadruplets r kl as well as atomc charges q and bond orders BO between atomc pars. The noncovalent nteractons comprse van der Waals energy E vdwaals and Coulomb energy E Coulomb whch are screened by a taper functon [12]. ReaxFF also ncorporates a charge transfer mechansm to descrbe realstc chemcal reacton pathways. The envronment-dependent charge dstrbuton s descrbed usng the electronegatvty equalzaton method EEM [20] n whch ndvdual atomc charges vary n tme. The analytcal formulas for these energy terms are gven n the followng subsectons Bond-order calculaton An essental buldng block of ReaxFF s the bond order that descrbes the nature of covalent bonds n terms of three exponental functons correspondng to σπ and double-π bonds. The bond order BO s an attrbute of atomc par where = 1...N wth N beng the number of atoms. The BO calculaton n ReaxFF nvolves two steps. The frst step computes raw BOs BO that are smply functons of the nteratomc dstances of all pars of atoms wthn sngle covalent-bond cutoff dstance r cb typcally 3 4 Å. In the second step each raw BO s corrected as a functon of the valences and of the atoms comprsng the par to obtan the full bond order BO. Many-body effects are mplctly ncluded through the valency terms. We frst compute the raw bond order BO as a sum of σπ and double-π bond contrbutons [ pbo2 ] [ pbo4 ] [ BO = BO σ + BO π ππ r r + BO = exp p bo1 ro σ + exp p bo3 ro π + exp p bo5 r r ππ o pbo6 ] where r = r = r r s the nteratomc dstance between th and th atoms r s the poston of the th atom. These raw bond orders are subsequently corrected to produce correct atomc valences as 2 BO = BO σ + BOπ + BOππ BO σ = BO σ f 1 f4 BO f5 BO BO π = BO π f 1 f1 f4 BO f5 BO BO ππ = BO ππ f 1 f1 f4 BO f5 BO = Val + f 1 = 1 2 n BO Val + f 2 Val + f 2 + f 3 + Val + f 2 Val + f 2 + f 3 f 2 = exp p boc1 + exp p boc1 f 3 = 1 { 1 [ ln exp pboc2 p boc2 2 + exp p boc2 ]} 3a 3b 3c 3d 3e 3f 3g 3h

3 K.-. Nomura et al. / Computer Physcs Communcatons Table 1 Bond-order calculaton algorthm Algorthm BO_calc {r = 1...N}=set of atomc postons where N s the number of atoms {n}=set of neghbor lsts where n s the lst of neghbor atoms of the th atom {BO }=set of full bond orders { }=set of full valences {BO }=set of raw bond orders { }=set of raw valences 1 compute raw bond orders {BO } and valences { } for each atom r for each atom r n BO BO σ r + BO π r + BO ππ r // see Eq. 2 Val // Val s atomc valency n Eq. 3e for each atom r for each atom r n + BO // Eq. 3e 2 compute full bond orders {BO } and valences { } for each atom r for each atom r n BO BO f 1 f 4 f 5 // f 1 f 4 and f 5 are BO correcton functons n Eq. 3 Val for each atom r for each atom r n + BO // Eq. 4 f 4 BO = exp p boc3 p boc4 BO BO boc f 5 BO = exp p boc3 p boc4 BO BO boc boc = Val boc + BO n 3 + p boc5 3 + p boc5 3k where n s the set of neghbor atoms that are wthn sngle covalent-bond cutoff dstance r cb from the th atom and the parameters n Eqs. 2 and 3 are lsted n Table A.1 n Appendx A. After the bond orders BO are computed for all pars the valency of the th atom s computed as = Val + BO. 4 n Table 1 shows the BO calculaton algorthm BO_calc. The algorthm also computes full valences as a functon of the corrected bond orders BO Eq. 4. To avod repeated computaton of BO dervatve values n subsequent energy and force calculatons we compute the dervatve of BOs wth respect to the raw BO and valency here and keep them untl the atomc coordnates are updated Energy and force calculatons The varous nteracton functons n Eq. 1 represent a varety of chemstres whch can be grouped accordng to 1 valence or noncovalent type 2 number of explctly nvolved atoms and 3 degree of the neghbor-atom effect. Here the number of explctly nvolved atoms s the nestng level of summatons that consttute each energy term e.g. 3 for the trply nested summaton n E val n Secton below. The degree of the neghbor-atom effect s defned as the number of mplctly nvolved loops through valency terms n the energy functons. For example the corrected valency term s of second degree snce ts evaluaton ncludes doubly nested sums: BO BO = BO BO BO m BO n. 5 m n

4 76 K.-. Nomura et al. / Computer Physcs Communcatons Table 2 Classfcaton of nteracton functons Number of explct atoms Energy terms 1 E lp 2 E over 3 E under 3 2 E bond 1 E vdwaals 0 a E Coulomb 0 a 3 E val 2 E pen 2 E coa 2 E hbond 2 4 E tors 2 E con 2 a Noncovalent nteractons. The numbers n the parentheses are the greatest degrees of the neghbor atom effect. Fg. 1. Schematc of atomc confguratons n energy terms: a 1-body b 2-body c 3-body d 4-body e hydrogen bondng and f noncovalent nteractons respectvely. A gray sphere represents the poston of the prmary atom n each energy term. Open bars represent covalent bonds whle dotted lnes are noncovalent bonds. In ths subsecton we frst explan the nature of nteracton functons whch s essental to map them onto a parallel computer. We then present actual algorthms. Table 2 classfes the energy terms accordng to the number of explctly nvolved atoms. The noncovalent nteractons are parwse 2-body functons wth two explct atoms whle valence terms nvolve up to four explct atoms. Each explct atom mplctly nvolves effects from surroundng atoms through BO and valency terms. The mplct atoms requre addtonal computatons and ncur ncreased cutoff dstances. For example n force calculaton the corrected valency term causes a doubly nested loop over a neghbor lst of the th atom wth the assocated computatonal cost OM 2 where M s the average number of covalent bonds per atom. The nteracton cutoff length s accordngly ncreased to 2r cb. The level of neghbor lst s thus regarded as the degree of envronment effect. The numbers n the parentheses n Table 2 shows the greatest degree of neghbor-atom effect among explct atoms. Fg. 1 shows the confguratons of atoms nvolved n the computaton of 1-body 2-body 3-body 4-body hydrogen bondng and noncovalent energy functons respectvely. In the fgure a gray sphere represents the poston of a prmary atom scanned by the outermost loop of each energy-term calculaton. The prmary atoms play a pvotal role on our parallel mplementaton n Secton 3. Note that the energy terms n ReaxFF share common computatonal structures. In addton to algorthms to compute ndvdual energy terms we defne two algorthms force_bo and force_ whch consttute the kernel of energy and force calculatons. The force_bo and force_ algorthms are gven n Tables 3 and 4 respectvely. The algorthm force_bo n Table 3 computes force components nvolved n sngle BO term where the A 1 A 2 and A 3 terms are the pre-computed dervatves of BO wth respect to BO and respectvely. The algorthm force_ n Table 4 computes the force contrbutons arsng from the dervatve wth respect to whch s smply a wrapper functon of force_bo loopng over the neghbor lst of the th atom. We now defne sx algorthms 5 valence and 1 noncovalent for force calculatons each of whch contans energy functons sharng the same computatonal characterstcs n Sectons Combned wth a multple tme-step ntegrator ths groupng of the force components facltates effcent tme ntegraton whch wll be dscussed n Secton body energes Among the sx force-calculaton algorthms force_1body s responsble for the 1-body energy calculatons E lp E over and E under whch contan one explct atom see Eqs Here the valency term n the lone-par functon e the number of lone par electrons n lp the coordnaton effect lp and the correcton term lpcorr due to lone-par electrons n the over/under coordnaton energy functon E over and E under are defned n Eq. 9. The parameters n Eqs. 6 9 are lsted n Table A.2 n Appendx A. E lp = p lp2 lp 1 + exp 75 lp 6

5 K.-. Nomura et al. / Computer Physcs Communcatons Table 3 Force calculaton arsng from the dervatve of BO Algorthm force_bo {r }=set of atomc postons n n = lsts of neghbor atoms for the th and th atoms C BO = E/ BO {f }=set of updated atomc forces f f + C BO A 1 r // A 1 = BO / BO BO / r f f + C BO A 1 r for each atom r k n f f + C BO A 2 r k // A 2 = BO / / r k f k f k + C BO A 2 r k for each atom r k n f f + C BO A 3 r k // A 3 = BO / / r k f k f k + C BO A 3 r k E over = n p ovun1de σ BO [ lpcorr lpcorr + Val E under = p ovun5 1 expp ovun6 lpcorr 1 + exp p ovun2 lpcorr e = Vale + BO n e n lp = + exp 2 lp = n lp-opt n lp lpcorr = p lp1 Table 4 Force-calculaton arsng from the dervatve of Algorthm force_ {r }=set of atomc postons n = lst of neghbor atoms for the th atom C = E/ {f }=set of updated atomc forces for each atom r n call force_bor r C 2 + e 2 e expp ovun2 lpcorr lp ] p ovun7 exp[p ovun8 n lp BOπ + BOππ p ovun3 expp ovun4 n lp BOπ + BOππ. 7 8 ] 9a Due to the trply nested neghbor-lst loops n the summaton of the valency see Table 5 force_1body requres the traversal of a deep tree structure emanatng from each prmary atom n force computaton wth a relatvely large nteracton cutoff dstance. In Table 5 E 1body denotes one of E lp E over and E under terms Bondng 2-body energy E bond n Eq. 10 descrbes the covalent bond energy of atomc pars whch ncludes energy contrbutons from the σπ and double-π bond terms. The parameters n the 2-body nteractons are lsted n Table A.3 n Appendx A. E bond = D σ e BO σ exp[ p be1 1 BO σ pbe2 ] D π e BO π Dππ e BO ππ Algorthm force_2body n Table 6 computes E bond. Though Table 6 does not dstngush the σπ and double-π sub-terms for smplcty they are handled smply by nputtng dfferent dervatves n algorthm force_bo. InTable 6 ID denotes a sequental ndex of the th atom. Our parallel MD algorthm mantans global sequental ndces over all atoms across processors.. 9b 9c 9d 10

6 78 K.-. Nomura et al. / Computer Physcs Communcatons Table 5 1-body force calculaton Algorthm force_1body {r }=set of atomc postons {n}=set of neghbor lsts where n s the lst of neghbor atoms of the th atom {f }=set of atomc forces C 1 = E 1body / BO C 2 = E 1body / for each atom r compute C 1 and C 2 for each atom r n call force_bor r C 1 call force_r C 2 Table 6 2-body force calculaton Algorthm force_2body {r }=set of atomc postons {n}=set of neghbor lsts where n s the lst of neghbor atoms of th atom {f }=set of atomc forces C 3 = E bond / BO for each atom r for each atom r n f ID > ID then compute C 3 call force_bor r C body energes The 3-body nteracton functons E val E pen and E coa n Eqs descrbe the stablty of the valence angle Θ k and covalent bonds between and k atoms. In E val the functon f 7 n Eq. 11a guarantees that the energy term goes to zero as the two covalent bonds from the center atom dssocate. Eqs. 11c 11f compute the equlbrum valence angle takng nto account π double-π and lone par electron effects. E pen s the penalty energy functon n atomc trplets. E coa s the conugaton energy among trplet atoms for example an NO 2 group n ntramnes. The parameters n Eqs are lsted n Table A.4 n Appendx A. E val = f 7 BO f 7 BO k f 8 { p val1 p val1 exp [ 2 ]} p val2 Θ0 SBO Θ k 11 k f 7 BO = 1 exp p val3 BO p val4 11a 2 + expp val6 angle f 8 = p val5 p val5 1 11b 1 + expp val6 angle + exp p val7 angle SBO = BO π m + BO ππ m + [1 exp BO 8 ] angle m p val8 n lp 11c angle m n = Val angle + m n BO m m n 0 SBO 0 SBO SBO2 = p val9 0 < SBO SBO p val9 1 < SBO 2 2 SBO > 2 Θ 0 SBO = π Θ 00 { 1 exp [ pval10 2 SBO2 ]} 11d 11e 11f

7 K.-. Nomura et al. / Computer Physcs Communcatons Table 7 3-body force calculaton Algorthm force_3body {r }=set of atomc postons {n}=set of neghbor lsts where n s the lst of neghbor atoms of th atom {f }=set of atomc forces C 4 = E 3body / BO C 4k = E 3body / BO k C 5 = E 3body / C 5 = E 3body / C 5k = E 3body / k for each atom r for each atom r n for each atom r k n f ID > IDk then compute C 4 and C 5 call force_bor r C 4 call force_bor r k C 4k call force_r C 5 call force_r C 5 call force_r k C 5k compute force components from 3-body angle dervatves E pen = p pen1 f 9 exp [ p pen2 BO 2 2] exp [ p pen2 BO k 2 2] k 12 f 9 = 2 + exp p pen3 1 + exp p pen3 + expp pen4 12a E coa = 1 p coa1 [ p 1 + expp k coa2 val exp coa3 BO + 2 ] BO m m n exp [ p coa3 BO k + 2 ] BO km exp [ p coa4 BO 1.5 2] exp [ p coa4 BO k 1.5 2]. m nk Algorthm force_3body n Table 7 computes E val E pen and E coa energes whch have three explct atoms Eqs In force_3body and also n force_4body descrbed below the force arsng from angular dervatves needs to be computed separately from BO dervatves as s descrbed elsewhere [21]. InTable 7 E 3body denotes one of E val E pen and E coa body energes and parameters 4-body dhedral angle energy E tors and 4-body conugaton energy E con are descrbed by Eqs. 14 and 15. The angles Θ k and Θ kl are computed from atom trplets kand kl. The dhedral angle ω kl s the angle between the two planes defned by the two atom trplets. The parameters n the 4-body nteractons are lsted n Table A.5 n Appendx A. E tors = 1 f 10 BO BO k BO kl sn Θ k sn Θ kl 2 k l [ V cos ω kl + V 2 exp { p tor1 2 BO π k f 11 k 2} 1 cos 2ω kl + V cos 3ω kl ] f 10 BO BO k BO kl = [ 1 exp p tor2 BO ][ 1 exp p tor2 BO k ][ 1 exp p tor2 BO kl ] f 11 k = 1 + exp[ p tor3 angle 2 + exp[ p tor3 angle + angle k ] + angle k ]+exp[p tor4 angle + angle k ] E con = p cot1 f 12 BO BO k BO kl [ 1 + cos 2 ω kl 1 ] sn Θ k sn Θ kl k l f 12 BO BO k BO kl = exp [ p cot2 BO 1.5 2] exp [ p cot2 BO k 1.5 2] exp [ p cot2 BO kl 1.5 2]. 14a 14b 15 15a

8 80 K.-. Nomura et al. / Computer Physcs Communcatons Table 8 4-body force calculaton Algorthm force_4body {r }=set of atomc postons {n}=set of neghbor lsts where n s the lst of neghbor atoms of th atom {f }=set of atomc forces C 6 = E 4body / BO C 6k = E 4body / BO k C 6kl = E 4body / BO kl C 7 = E 4body / C 7k = E 4body / k for each atom r for each atom r k n f ID > IDk then for each atom r n f ID IDk then for each atom r l nk f ID IDk and ID IDl and ID IDl then compute C 6 and C 7 call force_bor r C 6 call force_bor r k C 6k call force_bor k r l C 6kl call force_r C 7 call force_r k C 7k compute force components from 4-body angle dervatves Algorthm force_4body n Table 8 computes E tors and E con energes that nvolve four explct atoms Eqs. 14 and 15. force_4body consumes large amount of computaton tme to handle atomc quadruplets at every MD step whch can be reduced by the multple tme-step scheme descrbed n Secton 2.4. InTable 8 E 4body denotes ether E con or E tors Hydrogen bondng energy and parameters The hydrogen bondng energy functon E hbond n Eq. 16 takes a 3-body nteracton form.e. as a functon of the and k bonds as well as the angle Θ k.thee hbond term thus conssts of three explct atoms where the th atom s hydrogen the th atom s chosen from atom s neghbor lst and the kth atom s selected wthn the noncovalent nteracton cutoff radus r cnb 10 Å from atom. The parameters n the hydrogen bondng nteractons are lsted n Table A.6 n Appendx A. E hbond = [ p hb1 1 expphb2 BO ] [ r o exp p hb hb3 + r ] k r k rhb o 2 k sn 8 Θ k 2 To handle the unque form of E hbond we defne a separate algorthm force_hbond n Table 9. Because of ts asymmetrc structure consstng of valence and noncovalent bonds t s necessary that atomc trplets k and k are dealt dfferently. Forces arsng from the dervatve wth respect to noncovalent atoms k are thus computed separately n Table Table 9 Hydrogen bondng force calculaton Algorthm force_hbond {r }=set of atomc postons {n}=set of neghbor lsts where n s the lst of neghbor atoms of th atom {f }=set of atomc forces C 8 = E hbond / BO for each atom r for each atom r n for each atom r k {r k r k r cnb } compute C 8 call force_bor r C 8 compute force components from 3-body angle dervatves compute force components from noncovalent term

9 K.-. Nomura et al. / Computer Physcs Communcatons Table 10 Noncovalent force calculaton Algorthm force_noncov {r }=set of atomc postons {f }=set of atomc forces C 9 = E noncov / r for each atom r for each atom r {r r r cnb } f ID > ID then compute C 9 compute energy and force components Noncovalent energes and parameters The noncovalent nteractons consst of E vdwaals Eq. 17 and E Coulomb Eq. 18 terms. The functons are screened by a taper functon wth a cutoff length r cnb n Eq. 17b where the coeffcents of the polynomal are Tap 7 = 20 Tap 6 = 70 Tap 5 = 84 Tap 4 = 35 Tap 3 = 0 Tap 2 = 0 Tap 1 = 0 and Tap 0 = 1. Noncovalent energy functons n ReaxFF are all-atom nteractons whch do not requre any excluson rule [22] as n some other force felds. E vdwaals = { D Tapr exp [α 1 f 13r r vdw [ f 13 r = r p vdw1 1 pvdw1 ] 1/pvdW1 + γ w 7 α r Tapr = Tap α r cnb α=0 E Coulomb {r } {q } = χ q + 1 q Hr q 2 Tapr Hr = J δ + [r 3 + 1/γ 3 ] 1/3 1 δ { 1 = δ = 0 ] 2exp [ 1 2 α 1 f ]} 13r r vdw where χ s the electronegatvty J s the self Coulomb repulson coeffcent and γ s a parameter for the smeared Coulombc functon. Algorthm force_noncov n Table 10 computes van der Waals energy E vdwaals and Coulomb energy E Coulomb whch are functons of nteratomc dstance wth the cutoff dstance r cnb Eqs. 17 and 18. We use potental tables to compute the noncovalent nteractons n order to reduce the number of floatng-pont operatons. In Table 10 E noncov denotes ether E vdwaals or E Coulomb. Snce the noncovalent nteractons are functons of nteratomc dstance precomputng the functon values along wth a proper nterpolaton scheme sgnfcantly reduce the computaton tme Electronegatvty-equalzaton scheme In ReaxFF atomc charges q are varables that change dynamcally n tme. Whenever after atom coordnates are updated the EEM subroutne updates the charge dstrbuton {q } by mnmzng the Coulomb energy E Coulomb under the charge-neutralty constrant q = 0. Wth the Lagrange-multpler method the constraned energy mnmzaton s equvalent to solvng the electronegatvty equalzaton problem [ ] g E Coulomb = μ 19 q where μ s the electronegatvty. We solve the constraned mnmzaton problem usng the conugate-gradent method [2627] n Table 11 n whch the gradent vector g s orthogonalzed to the unform shft vector e = 1 = 1...N. Because of the electronegatvty equalzaton condton Eq. 19 the force contrbuton from E Coulomb {r } {q } E Coulomb r N k=1 E Coulomb q k q k = E Coulomb μ r r r N k=1 q k = E Coulomb r 17 17a 17b 18 18a 20

10 82 K.-. Nomura et al. / Computer Physcs Communcatons Table 11 Electronegatvty-equalzaton computaton Algorthm EEM {r }=set of atomc postons {q }=set of atomc charges {g }={ E Coulomb / q }=gradent vector {h }=conugate-gradent vector compute the ntal gradent vector {g 0} n 0 do f n = 1 then h 0 = g0 = 1toN else h n gn 1 q n q n 1 g n gn 1 g n 1 g n 2 h n 1 = 1toN + gn 1 g n 1 h n H h n h n = 1toN + gn 1 g n 1 h n H h n H h n = 1toN + gn 1 g n 2 n n + 1 untl the energy s converged does not contan chan-rule terms assocated wth varables q k and thus the noncovalent force calculaton descrbed n Secton remans vald the last equalty n Eq. 20 results from the charge-neutralty condton k q k = 0. It s also noteworthy that the Hessan Hr n Eq. 18 s a constant untl atom coordnates are updated. Thus reusng pre-computed Hr durng the EEM loop n Table 11 sgnfcantly reduces the total computaton tme Tme ntegrator Our parallel ReaxFF algorthm employs the rrespa tme ntegrator [28 30] whch s a tme-reversble algorthm based on symmetrc Trotter decomposton of the tme-propagaton operator. It allows decomposng the energy functons nto groups wth dstnct characterstc tme scales. Accordngly the forces f are decomposed nto stff force components f stff whch need to be updated wth hgher frequency and soft and slowly varyng force components f soft : f = f soft + f stff. Table 12 shows the rrespa algorthm comprsng of a doubly nested tme ntegraton loop. The compute ntensve force calculaton.e. f soft s performed only once per outer loop wth a large tme step t typcally t 1 fs. The atomc veloctes are updated accordng to f soft at the begnnng and the end of the outer loop. The nner loop updates the atomc postons and veloctes wth a fner tme resoluton δt = t/n mts usng the stff force components f stff typcally N mts 5. It s often the case that the soft force components f soft are computatonally ntensve [28]. Thus proper force decomposton sgnfcantly reduces the total computaton tme. For example n dense covalently bonded materals the 4-body force calculaton s most tme consumng. In test calculatons on a damond the 4-body force calculaton accounts for nearly half of the total computaton tme. We have reduced the total smulaton tme by 30% wth the number of multple tme step N mts = 5 see Table 12 by separatng out the 4-body force terms nto f soft.e. by performng the tme-consumng 4-body force calculaton only once n 5 MD steps. 3. Parallelzaton To desgn a scalable ReaxFF-MD smulaton algorthm on massvely parallel computers we employ spatal decomposton [31] combned wth lnked-lst cell and Verlet neghbor-lst methods [21]. In spatal decomposton the smulaton space wth lengths L x L y and L z n the x y and z drectons respectvely s decomposed nto P = P x P y P z rectangular sub-domans Ω 0 Ω P 1 of equal volume whch are then mapped to P processors n a parallel computer. Specfcally atom at poston r = r x r y r z s assgned to processor p [0P 1] where 21 { p = px P y P z + p y P z + p z p α = r α P α /L α α = xyz. 22

11 K.-. Nomura et al. / Computer Physcs Communcatons Table 12 Multple tme-step algorthm Algorthm multple_tme_step {r t}=set of atomc postons at tme t {v t}=set of atomc veloctes at tme t N step = number of total MD steps N mts = number of multple tme steps t = N mts δt = tme dscretzaton unt {r t + N step δt}=set of atomc coordnates at t + N step δt {v t + N step δt}=set of atomc veloctes at t + N step δt {f }=set of atomc forces compute soft force components {f soft } for outer =1toN step /N mts v v + t/2m f soft // m s the mass of the th atom compute stff force components {f stff } for nner = 1toN mts v v + δt/2m f stff r r + δtv compute stff force components {f stff } v v + δt/2m f stff compute soft force components {f soft } v v + t/2m f soft Each processor stores the coordnates r veloctes v and types σ e.g. hydrogen carbon ntrogen or oxygen of all atoms that are assgned to t and s also n charge of updatng the coordnates and veloctes by numercally ntegratng the Newton s equaton of moton. In the lnked-lst cell method each doman s further decomposed nto smaller rectangular cells. The atom nformaton r v σ n each cell s organzed usng a lnked lst whch reduces the computatonal cost of neghbor-lst calculaton to ON/P where N s the total number of atoms. We defne the dmenson of a lnked-lst cell L c to be no less than the sngle covalent-bond cutoff r cb.ther cb s determned as the nteratomc dstance at whch any type of BO becomes neglgble. We use a crteron BO r cb<10 4. At the doman boundares t s necessary to copy suffcent atom nformaton from neghbor domans to compute forces. In 3D smulaton space each doman has 26 adacent domans from whch atom nformaton needs to be cached nto buffer layers that surround the doman shown as a shaded area n Fg. 2. The requste thckness of the buffer layer L b s determned by the greatest cutoff dstance among all nteracton functons. Specfcally L b = maxr cnb l max r cb where l max s the deepest level of covalently bonded trees wth the prmary atom as a root see Fg. 2B. The cachng operaton s mplemented as the exchange of Fg. 2. A Schematc of the spatal decomposton conssts of sub-domans Ω 0 Ω 3. On the sub-doman Ω 0 atom nformaton from neghbor domans Ω 1 Ω 2 and Ω 3 are stored n the buffer layer wth the thckness L b shown as shaded area surroundng Ω 0. The prmary atoms n each cluster gray spheres must be n Ω 0 whle other atoms whte spheres may be coped from neghbor domans. B An example of 4-body atom confguraton on a doman boundary. The explct atoms are shown wth sold-lne spheres and bonds k andl and so are mplct atoms wth dotted-lne spheres and bonds m 1 m 5. Here th atom s the prmary of the cluster. In ths example the deepest level from the prmary atom s l max = 3.

12 84 K.-. Nomura et al. / Computer Physcs Communcatons messages between processors where the messages contan the nformaton of atoms that le wthn L b from the doman boundary. After cachng processor p has the nformaton of a set of atoms S = S resdent S cached ={ r Ω p } { r / Ω p r Ω p <L b } where Ω p s the outer-boundary surface of doman Ω p and r Ω p s the shortest dstance between atom and Ω p.ths allows each processor to compute the forces on ts resdent atoms S resdent wthout tme-consumng nterprocessor communcatons. Our parallel ReaxFF algorthm attempts to mnmze the total computaton tme by avodng duplcated energy/force calculatons among processors. For ths purpose care needs to be taken for the cached atoms stored n the buffer layer. Here we llustrate the energy/force calculatons n the parallel MD framework. Fg. 2B shows a sample case of 4-body calculaton that nvolves four explct atoms denoted as k and l and fve mplct atoms m 1 m 5. At the begnnng of the energy/force calculaton on doman Ω 0 atomslm 1 and m 2 are cached from doman Ω 1 to Ω 0 va communcaton network. We requre that the prmary atom s a resdent atom of doman Ω 0. The computed potental energy of the atomc cluster s stored n the doman of the prmary atom. At the end of energy/force calculaton the reacton forces on the cached atoms are transferred back to the orgnal atoms n doman Ω 1. Subsequently atom coordnates are updated accordng to the tme ntegraton algorthm n Table 12. Resdent atoms mgrate to new domans f they have moved out of the current doman. All nter-doman communcatons follow a 6-way communcaton method [32]. The amount of communcaton s proportonal to the number of atoms on doman surface whch s ON/P 2/3. The parallel ReaxFF program s wrtten n the Fortran 90 language wth the message passng nterface MPI lbrary [33] for communcatons. 4. Performance test The scalablty of parallel ReaxFF program has been tested on varous hgh-end supercomputers ncludng the processor IBM BlueGene/L at the Lawrence Lvermore Natonal Laboratory LLNL the processor SGI Altx 3000 at the NASA Ames Research Center and an AMD dual-core Opteron cluster at the Hgh Performance Computng Center HPCC of the Unversty of Southern Calforna USC. The parallel ReaxFF algorthm maxmally utlzes data localty regardless of actual physcal archtecture and accordngly no performance degradaton has been observed on any of these archtectures. Descrpton of each platform and computatonal resources used for each benchmark are gven below. BlueGene/L The BlueGene/L system at the LLNL comprses of computatonal node CN chps each of whch has two PowerPC 400 processors processors n total wth 700 MHz clock speed. On sngle CN the two processors share 512 MB memory. Each processor has a 32 KB nstructon/data cache a 2 MB L2 cache and a 4 MB L3 cache. The theoretcal peak performance s 2.8 Gflops per processor. Two types of nterconnecton 3-D torus and tree topologes are desgned for dstnct purposes. The 3-D torus network s used mostly for common e.g. pont-to-pont communcatons whle the tree network s optmzed for collectve communcatons. The nterconnecton bandwdths are 175 MB/s and 350 MB/s per lnk respectvely. Altx 3000 The SGI Altx 3000 system named Columba at NASA-Ames conssts of 20 of SGI Altx model 3700 boxes each consstng of 512 Intel 1.5 GHz Itanum2 processors. Each processor has 128 floatng-pont regsters a 32 KB L1 cache a 256 KB L2 cache and a 6 MB L3 cache and ts theoretcal peak performance s 6 Gflops. The SGI NUMAlnk4 nterconnecton provdes 1TB memory globally shared among cluster nodes wthn an Altx box. We have used up to four Altx boxes up to 1920 processors for our benchmark. Opteron USC-HPCC operates 1824-node Lnux cluster wth 15.8 Tflops Lnpack performance. We have used up to 512 nodes of dual-cpu dual-core AMD 2 GHz Opteron processors 2048 processors n total. Each node has 4 GB memory and Myrnet nterconnecton provdes 256 MB/s bandwdth. Sngle Opteron core has 64 KB nstructon/data caches and 1 MB L2 cache. Fg. 3 shows the parallel effcency of the parallel ReaxFF algorthm on the three archtectures. Here the sogranular parallel effcency s defned as the total executon tme dvded by that on P = 1 where the gran sze.e. the number of atoms per processor s kept constant N/P = on the BlueGene/L on Columba and on the Opteron cluster. In all performance tests we smulate energetc molecular crystal RDX 135-trntro-135-trazne C 3 N 6 O 6 H 6 [14]. Overall the parallel ReaxFF algorthm acheves nearly perfect 1 effcency. On BlueGene/L processors the parallel effcency 23

13 K.-. Nomura et al. / Computer Physcs Communcatons Fg. 3. Isogranular parallel effcency of the parallel ReaxFF algorthm as a functon of the number of processors on A IBM Blue Gene/L wth the number of atoms per processor N/P = B SGI Altx 3000 wth N/P = and C AMD dual-core Opteron cluster wth N/P = Fg. 4. Executon tme of the parallel ReaxFF program as a functon of the number of atoms on processor IBM Blue Gene/L 1920-processor SGI Altx 3000 and 2048-core AMD Opteron. s Only excepton s the sudden drop n the parallel effcency on the Opteron cluster whch s lkely due to the non-dedcated testng envronment n whch other users processes share the same network. Fg. 4 shows the executon tme as a functon of the number of atoms whle the total number of processors s fxed. We use IBM BlueGene/L processors 1920 Itenum2 processors of SGI Altx 3000 and 2048 AMD Opteron cores. The largest number of atoms s on the BlueGene/L. The parallel ReaxFF algorthm exhbts perfect lnear scalablty as a functon of the problem sze. 5. Summary For chemcally-reactve molecular dynamcs smulatons we have desgned a scalable parallel algorthm that maxmally exposes data localty. The parallel ReaxFF algorthm ncorporates spatal decomposton and multple tme-steppng for effcent valence/noncovalent force calculatons. Envronment-dependent varable charges are ncorporated wth an electronegatvty equalzaton method. Benchmark tests have exhbted hgh scalablty parallel effcency s on BlueGene/L processors as well as good performance portablty. The parallel ReaxFF algorthm enables unprecedented scales of chemcally reactve smulatons whch pave a way to study longstandng scentfc problems such as mechancally nduced chemcal reactons. Acknowledgements Ths work was partally supported by ARO MURI DOE DTRA and NSF. Performance tests were performed at the Unversty of Southern Calforna usng the 5384-processor Lnux cluster at the Research Computng Faclty and the 2048-processor Lnux

14 86 K.-. Nomura et al. / Computer Physcs Communcatons cluster at the Collaboratory for Advanced Computng and Smulatons. We thank Dr. A.C.T. van Dun and Prof. W.A. Goddard III for valuable dscussons. Appendx A. ReaxFF parameters Appendx A contans Tables A.1 A.7. Table A.1 The parameters n the bond-order functons p bo1 p bo6 ro σ rπ o rππ o Val Val boc p boc1 p boc2 p boc3 p boc5 bond parameters bond radus parameters valences of atom overcoordnaton parameters 1 3 bond order correctons Table A.2 The parameters n the 1-body energy Val e n lp-opt p lp1 p lp2 p ovun1 p ovun4 p ovun5 p ovun6 p ovun8 valency of atom the number of lone par electrons at normal condton lone par parameter lone par energy parameter overcoordnaton parameters undercoordnaton energy parameter undercoordnaton parameters Table A.3 The parameters n the E bond De σ Dπ e Dππ e p be1 p be2 σπ and double π bond energy parameters bondng energy parameters Table A.4 The parameters n the 3-body energy p val1 p val2 p val7 p val8 p val9 p val10 Θ 00 Val angle p pen1 p pen2 p pen3 p pen4 p coa1 p coa2 p coa4 valence angle energy parameter valence angle parameters valency/lone par parameter valence angle parameters equlbrum valence angle atom valency penalty energy parameter double bond/angle parameter double bond/angle parameters for overcoordnaton 3-body conugaton energy parameter valency angle conugaton parameters Table A.5 The parameters n the 4-body energy V 1 V 3 p tor1 p tor2 p tor3 p tor4 p cot1 p cot2 torson energy parameters torson parameter torson/bo parameter torson overcoordnaton parameters 4-body conugaton energy parameter 4-body conugaton parameter Table A.6 The parameters n the hydrogen bondng energy p hb1 p hb2 p hb3 rhb o hydrogen bond energy parameter hydrogen bond parameters hydrogen bond radus

15 K.-. Nomura et al. / Computer Physcs Communcatons Table A.7 The parameters n the noncovalent energy D α r vdw p vdw1 γ w γ van der Waals energy parameter van der Waals parameter van der Waals parameter van der Waals sheldng van der Waals parameter Coulomb parameter References [1] D.G. Truhlar V. McKoy Computng n Scence & Engneerng [2] J.J. Glman Scence [3] J.J. Glman Materals Scence and Technology [4] S. Ogata E. Ldorks F. Shmoo A. Nakano P. Vashshta R.K. Kala Computer Physcs Communcatons [5] S. Dapprch I. Komárom K.S. Byun K. Morokuma M.J. Frsch Journal of Molecular Structures Theochem [6] M.J. Feld P.A. Bash M. Karplus Journal of Computatonal Chemstry [7] U.C. Sngh P.A. Kollman Journal of Computatonal Chemstry [8] A. Warshel M. Levtt Journal of Molecular Bology [9] E.E. Santso K.E. Gubbns Molecular Smulaton [10] J. Tersoff Physcal Revew B [11] D.W. Brenner Physca Status Sold b [12] A.C.T. van Dun S. Dasgupta F. Lorant W.A. Goddard Journal of Physcal Chemstry A [13] A.C.T. van Dun A. Strachan S. Stewman Q.S. Zhang X. Xu W.A. Goddard Journal of Physcal Chemstry A [14] A. Strachan A.C.T. van Dun D. Chakraborty S. Dasgupta W.A. Goddard Physcal Revew Letters [15] Q. Zhang T. Cagn A.C.T. van Dun W.A. Goddard Y. Q L.G. Hector Physcal Revew B [16] K.D. Nelson A.C.T. van Dun J. Oxgaard W.Q. Deng W.A. Goddard Journal of Physcal Chemstry A [17] K. Chenoweth S. Cheung A.C.T. van Dun W.A. Goddard E.M. Kober Journal of the Amercan Chemcal Socety [18] K. Nomura R.K. Kala A. Nakano P. Vashshta A.C.T. van Dun W.A. Goddard Physcal Revew Letters n press. [19] P. Vashshta R.K. Kala A. Nakano B.E. Homan K.L. McNesby Journal of Propulson and Power [20] W.J. Morter S.K. Ghosh S. Shankar Journal of the Amercan Chemcal Socety [21] M.P. Allen D.J. Tldesley Computer Smulaton of Lquds Oxford Unversty Press [22] W.D. Cornell P. Ceplak C.I. Bayly I.R. Gould K.M. Merz D.M. Ferguson D.C. Spellmeyer T. Fox J.W. Caldwell P.A. Kollman Journal of the Amercan Chemcal Socety [23] A.K. Rappe W.A. Goddard Journal of Physcal Chemstry [24] F.H. Stretz J.W. Mntmre Physcal Revew B [25] T.J. Campbell R.K. Kala A. Nakano P. Vashshta S. Ogata S. Rodgers Physcal Revew Letters [26] A. Nakano Computer Physcs Communcatons [27] B. Bayraktar T. Bernas J.P. Robnson B. Rawa Pattern Recognton [28] G.J. Martyna M.E. Tuckerman D.J. Tobas M.L. Klen Molecular Physcs [29] M.E. Tuckerman B.J. Berne G.J. Martyna Journal of Chemcal Physcs [30] M.E. Tuckerman D.A. Yarne S.O. Samuelson A.L. Hughes G.J. Martyna Computer Physcs Communcatons [31] A. Nakano R.K. Kala P. Vashshta T.J. Campbell S. Ogata F. Shmoo S. San Scentfc Programmng [32] A. Nakano R.K. Kala K. Nomura A. Sharma P. Vashshta F. Shmoo A.C.T. van Dun W.A. Goddard R. Bswas D. Srvastava Computatonal Materals Scence [33] W. Gropp E. Lusk A. Skellum Usng MPI second ed. The MIT Press 1999.