DFT with Planewaves pseudopotential accuracy (LDA, PBE) Fast time to solution 1 step in minutes (not hours!!!) to be useful for MD

Transcription

1 LLNL-PRES Ths work was performed under the auspces of the U.S. Department of Energy by under contract DE-AC52-07NA Lawrence Lvermore Natonal Securty, LLC

2 Sequoa, IBM BGQ, 1,572,864 cores O(N) scalable algorthm For nsulators, semconductors Large problems > 5,000 atoms DFT wth Planewaves pseudopotental accuracy (LDA, PBE) Fast tme to soluton 1 step n mnutes (not hours!!!) to be useful for MD 2

3 Unlke classcal physcs models, n Quantum models the number of physcal varables (electrons) grows wth system sze O(N 2 ) degrees of freedom and O(N 3 ) operatons n DFT Reducng computatonal complexty to O(N) typcally mples Introducton of controllable approxmatons / truncate fast decayng terms More complcated data structures sparse vs. full matrces tolerance S -1 3

4 Example: C 2 H 4 Strctly localzed, non-orthogonal, Not centered on atoms (adaptve) Auxllary bass set (orthogonal) Maxmally Localzed Wanner functons Mnmze the sum of the spread of all the functons N [Marzar and Vanderblt, PRB 1997] Xˆ 1 Xˆ 2 4

5 Energy mnmzaton for general non-orthogonal orbtals [Gall and Parrnello, PRL 1992] E KS ( r) N N N 1 1 S r r F S rv r 1 1 S r r Assume fnte gap N, 1, 1, 1 Assumng functons are lnearly ndependent No need for any egenvalue computaton! N N 1 ext To take nto account non-orthogonalty S r r dr 5

6 Real-space (fnte dfference) dscretzaton Norm-conservng pseudopotentals Parallel doman decomposton Confne functons to fnte sphercal regons Each Φ lves on Fnte Dfference mesh, n a localzaton regon of center R and radus Rc O(1) d.o.f. for each orbtal Iteratve solver: drect mnmzaton of energy functonal follow precondtoned steepest descent drectons + block Anderson extrapolaton scheme [JLF, J. Comp Phys 2010] Truncate tral soluton at each step [JLF and Bernholc, PRB 2000, JLF and F. Gyg, Comp Phys Comm 2004, PRB 2006] 6

7 7 Not even expensve, but requrng a lot of communcatons O(N 3 ) solver becomes a bottleneck beyond 10,000 atoms and/or 10,000 MPI tasks Smaller sze than n Tght-Bndng models or LCAO methods Global couplng Need to calculate selected elements of the nverse of Gram matrx S We essentally need the elements S -1 s.t. S 0 r r S r r V r S F r r S E N ext N N N KS 1, 1 1, 1 1, 1 1 ) (

8 S r r dr S c R c 0 c c poston of 2R local functon radus c S s sparse, Symmetrc Postve Defnte Condton number s ndependent of problem sze!! Inverse In prncple full matrx But off-dagonal elements decay exponentally fast [Demko et al., Math. Comp. 1984] [Benz & Razouk, ETNA 2007] Assumpton: spectrum of S bounded away from 0, ndependently of N 8

9 Polymers, 1888 atoms How to make effcent use of t on large parallel computers? 9

10 Based on the approxmate nverse strategy Solve : arg M mn 1 SM I M S, I F N N dentty matrx Sparsty pattern of M s predetermned by geometrc dstance c, defne J k c c R and set M k L 0k J, R s determnes accuracy of selected elements of the nverse for some dstance k s R s 10

11 Include rows and columns of S correspondng to closest local functons (dstance between centers) Solve for column k usng ILU0 precondtoned GMRES Note: S not close to Identty matrx!!! (unlke n Tght-Bndng or LCAO approach where no precondtoner s needed [Stechel et al. PRB 1994]) 11

12 Example: polymer For nternal use only 12

13 Subdoman assocated wth an MPI task Localzed orbtal are dstrbuted across processors Each MPI task owns peces of several functons Each MPI task computes partal contrbuton to the global matrces (overlap, ) 13

14 Energy can be wrtten as: E ks Tr( S 1 H ) F( ), where H T H S -1 s approxmated, sparse and has complete but dstrbuted entres H ϕ s sparse and dstrbuted (ncomplete entres) Each PE only needs entres correspondng to locally centered functons Need to consoldate partal contrbutons of H ϕ Effcent data communcaton and assemblng algorthm s needed 14

15 Each parallel task compute partal contrbutons to some matrx elements Need to assemble local prncpal submatrx matrx Sum up partal dot products computed on varous processors We use a short range communcaton pattern where data s passed down to nearest neghbor only, one drecton at a tme, for as many steps as needed Overlap communcaton and computaton Accumulate receved data n sparse data structure whle sendng data for next step Need to scatter results to adacent processors that need column of S -1 15

16 Fnte dfference Mesh spacng error O(h 4 ) 2 parameters to control O(N) truncaton Localzaton of functons Cutoff for S -1 (R c =9 Bohr) 16

17 Lqud water 1536 atoms (512 molecules) 2048 orbtals Replcate 2x2x2 3x3x3 4x4x4 17

18 IBM BGQ 18

19 Dstrbutng atomc postons Lqud water on IBM/BGQ 19

20 W. Kohn s nearsghtedness prncple [PRL 1996] Nearsghtedness n computatonal algorthm leads to O(N) and parallel scalng beyond 100,000 MPI tasks Practcal accuracy acheved wth short range communcatons / no global communcatons for nsulators 20

21 Research supported by LLNL LDRD program Recent Publcatons D. Ose-Kuffuor and JLF, PRL 2014 D. Ose-Kuffuor and JLF, SIAM J. Sc. Comput Future Speed-up tme-to-soluton (threadng) Applcatons Dstrbuton of ons n dlute soluton: K + Cl n water Bology Extenson to metals 21