VASP: running on HPC resources. University of Vienna, Faculty of Physics and Center for Computational Materials Science, Vienna, Austria

Transcription

1 VASP: running on HPC resources University of Vienna, Faculty of Physics and Center for Computational Materials Science, Vienna, Austria

2 The Many-Body Schrödinger equation 1 2 X i i + X i Ĥ (r 1,...,r N )=E (r 1,...,r N ) 1 V (r i )+ X 1 A (r 1,...,r N )=E (r 1,...,r N ) r i r j i6=j For instance, many-body WF storage demands are prohibitive: (r 1,...,r N ) (#grid points) N 5 electrons on a grid ~ 10 PetaBytes! A solution: map onto one-electron theory: (r 1,...,r N )! { 1 (r), 2 (r),..., N (r)}

3 Hohenberg-Kohn-Sham DFT Map onto one-electron theory: (r 1,...,r N )! { 1 (r), 2 (r),..., N (r)} (r 1,...,r N )= NY i i(r i ) Total energy is a functional of the density: E[ ] =T s [{ i [ ]}]+E H [ ]+E xc [ ] + E Z [ ]+U[Z] The density is computed using the one-electron orbitals: 1 2 (r) = NX i (r) 2 The one-electron orbitals are the solutions of the Kohn-Sham equation: + V Z (r)+v H [ ](r)+v xc [ ](r) i i(r) = i i (r) BUT: E xc [ ] =??? V xc [ ](r) =???

4 Exchange-Correlation E xc [ ] =??? V xc [ ](r) =??? Exchange-Correlation functionals are modeled on the uniform-electron-gas (UEG): The correlation energy (and potential) has been calculated by means of Monte- Carlo methods for a wide range of densities, and has been parametrized to yield a density functional. LDA: we simply pretend that an inhomogeneous electronic density locally behaves like a homogeneous electron gas. Many, many, many different functionals available: LDA, GGA, meta-gga, van-der-waals functionals, etc etc

5 An N-electron system: N = O(10 23 ) Hohenberg-Kohn-Sham DFT takes us a long way: (r 1,...,r N )! { 1 (r), 2 (r),..., N (r)} (#grid points) N N (#grid points) Nice for atoms and molecules, but in a realistic piece of solid state material N= O(10 23 )!

6 Translational invariance: Periodic Boundary Conditions Translational invariance implies: and nk(r + R) = nk (r)e ikr nk(r) =u nk (r)e ikr u nk (r + R) =u nk (r) All states are labeled by Bloch vector k and the band index n: The Bloch vector k is usually constrained to lie within the first Brillouin zone of the reciprocal space lattice. The band index n is of the order if the number of electrons per unit cell.

7 Reciprocal space & the first Brillouin zone L z 4pi/L z z b 2 b 1 b 2 b 1 x a 2 a 1 a 3 y x y x y b 3 b 3 A B C b 1 = 2 a 2 a 3 b 2 = 2 a 3 a 1 b 3 = 2 a 1 a 2 = a 1 a 2 a 3 a i b j =2 ij

8 Sampling the 1 st BZ The evaluation of many key quantities involves an integral over the 1 st BZ. For instance the charge density: (r) = 1 BZ X n Z BZ f nk nk (r) 2 dk We exploit the fact that the orbitals at Bloch vectors k that are close together are almost identical and approximate the integral over the 1 st BZ by a weighted sum over a discrete set of k-points: (r) = X nk w k f nk nk (r) 2 dk, Fazit: the intractable task of determining (r with N= ,...,r N ), has been reduced to calculating nk(r) at a discrete set of k-points in the 1 st BZ, for a number of bands that is of the order if the number of electrons in the unit cell.

9 The total energy E[, {R,Z}] =T s [{ nk [ ]}]+E H [, {R,Z}]+E xc [ ]+U({R,Z}) The kinetic energy The Hartree energy where T s [{ nk [ ]}] = X nk E H [, {R,Z}] = 1 2 w k f nk h nk 1 2 nki ZZ ez (r) ez (r 0 ) r r 0 dr 0 dr ez (r) = (r)+ X i Z i (r R i ) (r) = X nk w k f nk nk (r) 2 dk The Kohn-Sham equations V H [ ez ](r)+v xc [ ](r) nk(r) = nk nk (r) The Hartree potential V H [ ez ](r) = Z ez (r 0 ) r r 0 dr0

10 A plane wave basis set nk(r) =u nk (r)e ikr u nk (r + R) =u nk (r) All cell-periodic functions are expanded in plane waves (Fourier analysis): u nk (r) = 1 1/2 X G C Gnk e igr nk(r) = 1 1/2 X G C Gnk e i(g+k)r (r) = X G G e igr V (r) = X G V G e igr The basis set includes all plane waves for which C rnk = X G 1 2 G + k 2 <E cuto Transformation by means of FFT between real space and reciprocal space: C Gnk e igr FFT! CGnk = 1 N FFT X r C rnk e igr

11 real space FFT reciprocal space G cut τ N 1 0 b τ 1 x = n / N 1 1 C rnk = X G τ g = n 2 π / τ b 1 N/2+1 C Gnk e igr FFT! CGnk = 1 X C rnk e igr N FFT r N/2

12 The charge density G cut FFT ψ G 2 G cut ψ r ψ r FFT ρ G ρ r

13 The Self-Consistency-Cycle (cont.) Two sub-problems: Optimization of { n } Iterative Diagonalization e.g. RMM-DIIS or Blocked Davidson in Construction of Density Mixing e.g. Broyden mixer

14 The self-consistency-cycle A naïve algorithm: express the Hamilton matrix in a plane wave basis and diagonalize it: Self-consistency-cycle: H = hg Ĥ[ ] G0 i!diagonalize H! { i, i } i =1,..,N FFT 0! H 0! 0! 1 = f( 0, 0 )! H 1!... Iterate until: = 0 BUT: we do not need N FFT eigenvectors of the Hamiltonian (at a cost of O(N FFT3 )). Actually we only the N b lowest eigenstates of H, where N b is of the order of the number of electrons per unit cell (N b << N FFT ). Solution: use iterative matrix diagonalization techniques to find the N b lowest Eigenvector of the Hamiltonian: RMM-DIIS, blocked-davidson, etc.

15 Key ingredients: Subspace diagonalization and the Residual Rayleigh-Ritz: diagonalization of the N b x N b subspace with X m H nm B mk = X m app k S nm B mk H nm = h n Ĥ mi Snm = h n Ŝ mi yields N b eigenvectors with eigenvalues ε app. ki = X m B mk m i These eigenstates are the best approximation to the exact N b lowest eigenstates of H within the subspace spanned by the current orbitals. The Residual: R( n )i =(Ĥ appŝ) ni app = h n Ĥ ni h n Ŝ ni (its norm is measure for the error in the eigenvector)

16 Blocked-Davidson Take a subset of all bands: { n n =1,..,N} ) { 1 k k =1,..,n 1 } Extend this subset by adding the (preconditioned) residual vectors to the presently considered subspace: { 1 k/g 1 k = K(H app S) 1 k k =1,..,n 1 } Rayleigh-Ritz optimization ( sub-space rotation ) in the 2n 1 dimensional subspace to determine the n 1 lowest eigenvectors: diag{ 1 k/g 1 k} Extend subspace with the residuals of { 2 k} { 2 k k =1,..,n 1 } { 1 k/g 1 k/g 2 k = K(H app S) 2 k k =1,..,n 1 } Rayleigh-Ritz optimization ) { 3 k k =1,..,n 1 } Etc The optimized set replaces the original subset: { m k k =1,..,n 1 } { n n =1,..,n 1 } Continue with next subset: { k k 1 = n 1 +1,..,n 2 }, etc, After treating all bands: Rayleigh-Ritz optimization of { n n =1,..,N}

17 The action of the Hamiltonian The action H nk i Using the convention V (r) nk(r) hr G + ki = 1 1/2 ei(g+k)r! h G + k nk i = C Gnk Kinetic energy: hg + k hg + k V nk i = 1 N FFT X 1 2 nki = 1 2 G + k 2 C Gnk Local potential: Exchange-correlation: easily obtained in real space FFT to reciprocal space {V xc,r }! {V xc,g } Hartree potential: solve Poisson eq. in reciprocal space Add all contributions V G = V H,G + V xc,g + V ext,g FFT to real space {V G }! {V r } The action r V r C rnk e igr N NPLW V = V H [ ]+V xc [ ]+V ext V xc,r = V xc[ r ] V H,G = 4 G 2 G N FFT log N FFT

18 Solving the KS equations V ext (r)+v H (r)+v xc (r) nk(r) = nk nk (r) FFTs extensively used to evaluate H n i We actually use a mixed basis set (Projector-Augmented-Waves): n i = e ni + X i ( ii e ii)hep i e ni which involves projection of the pseudo-wave functions on local projection operators (DGEMM). One needs to keep the solutions (bands) at each k-point orthonormal: essentially done by a Choleski decomposition (LU) of the overlap matrix, followed by an inversion of U (LAPACK/scaLAPACK) and a transformation between the wave function (ZGEMM). Diagonalization of the Hamiltonian in the subspace of the current wave functions (LAPACK/scaLAPACK or ELPA), followed by a unitary transformation between the wave function (ZGEMM).

19 A typical workload Action: H ni FFT( n) N ln N 8 n N 2 ln N fftw V (r) n(r) N 8 n N 2 h p i ni N 8 i, n N 3 const. N 2 (real sp.) BLAS3 (DGEMM) LREAL= A Subspace rotation: H nm = h n H mi 8 n, m N 3 BLAS3 (ZGEMM) diag(h) N 3 (sca)lapack 0 ni = X m U nm mi 8 n N 3 BLAS3 (ZGEMM)

20 Scaling with system size (N) Self-Consistency Cycle (SCC): RMM-DIIS running on 8 Intel X5550 quadcore procs. (total: 32 x 2.66 GHz cores)

21 Distribution of work and data Distribute work and data over-orbitals Default NCORE = 1 (or equivalently: NPAR = #-of-mpi-ranks) KPAR = 1 The Kohn-Sham equation: V ext (r)+v H (r)+v xc (r) nk(r) = nk nk (r) Orbital index n

22 Distribution of work and data Distribute work and data over-orbitals Default NCORE = 1 (or equivalently: NPAR = #-of-mpi-ranks) KPAR = 1 Distribution work and data over-plane-waves NCORE = #-of-mpi-ranks (or equivalently: NPAR = 1) KPAR = 1

23 Distribution of work and data Combinations of over-orbitals and over-plane-wave distributions are allowed as well

24 Distribution of work and data Additionally work may be distributed over-k-points KPAR = n (n>1) m = (#-of-mpi-ranks / n) must be an integer Work is distributed in a round-robin fashion over groups of m MPI-ranks Data is duplicated! 1 + V ext (r)+v H (r)+v xc (r) nk(r) = nk nk (r) 2 Orbital index n, k-point index k

25 All-2-All communication

26 Parallel FFT: Ball-2-Cube x z y 4G 4G G FFT G 2G FFT 4G FFT 4G (A) (B) (C) (D) In-house parallel Ball-2-Cube FFT: Less 1D-FFTs (reduction: 1.76 ) BUT: communication from (B) (C) and (C) (D) For small to medium sized FFT grids a highly optimized 3D-FFT (e.g. from Intel s mkl) is equally fast

27 Hardware considerations Typical configuration: N interconnected nodes 2 packages/node M cores/package Distribution over-plane-waves : MPI-ranks that share an orbital should reside on the same node (better even on the same package). NCORE = n 2M (2M / n) should be an integer Typically: n = M or n = M/2 Distribution over-k-points : KPAR = n #-of-k-points (NKPTS) (#-of-mpi-ranks / n) should be an integer If memory allows: KPAR = NKPTS Default placement of MPI-ranks on the nodes/packages/cores depends on the particulars of the MPI implementation and its configuration!

28 For instance: 2 interconnected nodes 2 packages/node 4 cores/package Default placement of MPI-ranks on the nodes/packages/cores depends on the particulars of the MPI implementation and its configuration!

29 For instance: 2 interconnected nodes 2 packages/node 4 cores/package Good: Place subsequent MPI-ranks close together, i.e., first on subsequent cores of the same package, then moving on the second package of the same node, before starting to fill the next node.

30 For instance: 2 interconnected nodes 2 packages/node 4 cores/package Bad: Distribute subsequent MPI-ranks in a round-robin fashion over the packages.

31 For instance: 2 interconnected nodes 2 packages/node 4 cores/package Worse: Distribute subsequent MPI-ranks in a round-robin fashion over the nodes.

32 NCORE, KPAR, and the #-of-mpi-ranks N interconnected nodes 2 packages/node M cores/package 2NM MPI-ranks Distribution over-plane-waves : MPI-ranks that share an orbital should reside on the same node (better even on the same package). NCORE = n 2M (2M / n) should be an integer Typically: n = M or n = M/2 Distribution over-k-points : KPAR = n #-of-k-points (NKPTS) (#-of-mpi-ranks / n) should be an integer If memory allows: KPAR = NKPTS Distribution over-orbitals : The number of orbitals (NBANDS) is such that NBANDS / (2NM / NCORE / KPAR) is an integer number Increasing #-of-mpi-ranks may lead to an unnecessarily large NBANDS (i.e., adding empty orbitals) Some algorithms converge faster when each MPI rank owns (part of) a few orbitals (e.g. blocked-davidson) Generally speaking: having lots of MPI ranks and very little work/data per rank is never a good idea since (all-2-all) communication becomes unreasonably expensive.

33 Strong/Weak scaling (Si N )

34 Scaling under MPI (on a Cray XC-40) 5713 e- 2 k-points RMM-DIIS PBE KPAR=2 Courtesy of P. Saxe, Materials Design Inc. (and Cray).

35 Scaling under MPI (on a Cray XC-40) 864 e- 8 k-points HSE06 Davidson KPAR=8 Courtesy of P. Saxe, Materials Design Inc. (and Cray).

36 Large systems Use real-space PAW projectors (instead of reciprocal space projectors) h p i ni N 8 i, n N 3 const. N 2 (real sp.) BLAS3 (DGEMM) LREAL= A If you can limit k-point sampling to the Gamma (Γ k = 0) point: use the gamma-only version of VASP In the gamma-only version of VASP the orbitals are stored as real quantities in real-space: real-2-complex FFTs DGEMMs instead of ZGEMMs

37 Compilation Fast FFTs: the FFTs from Intel s mkl-library seem to be unbeatably fast scalapackfor large systems A compiler that effectively generates AVX2 instructions and libraries (e.g. BLAS) that are optimized for AVX2 (up to 20% performance gain)

38 The End Thank you!