ELECTRONIC STRUCTURE CALCULATIONS FOR THE SOLID STATE PHYSICS

Transcription

1 FROM RESEARCH TO INDUSTRY 32 ème forum ORAP 10 octobre 2013 Maison de la Simulation, Saclay, France ELECTRONIC STRUCTURE CALCULATIONS FOR THE SOLID STATE PHYSICS APPLICATION ON HPC, BLOCKING POINTS, Marc Torrent CEA, DAM, DIF F Arpajon, France PAGE 1

2 OUTLINE Computing properties of electrons in matter Density Functional Theory (DFT) : a brief introduction DFT in practice ABINIT Overview of the project Examples Parallelization scheme DFT parallelism: how to go further Performance analysis Possibles solutions Conclusion

3 COMPUTING PROPERTIES OF ELECTRONS IN MATTER : DENSITY-FUNCTIONAL THEORY

4 DENSITY FUNCTIONAL THEORY A system made of interacting particles: Atomic nuclei Electrons Eventual external field To be solved within the «Born-Oppenheimer approximation» Electrons relaxation and nuclei motion are decoupled. We consider here the electrons as the interacting particle in the field due to the nuclei. The system made of the electrons is a Many-Body system.

5 DENSITY FUNCTIONAL THEORY To be solved: the Schrödinger equation (for the electrons and time independent) Fundamental postulate of Quantum Mechanics Hamiltonian H ψ = εψ Wave function Hohenberg et Kohn Theorem, 1964 The ground-state properties of a many-electron system are uniquely determined by the electron density ρ This density minimizes the total energy H [ ρ ] ψ = εψ E totale [ ρ] = Emin

6 KOHN-SHAM DENSITY FUNCTIONAL THEORY Kohn and Sham DFT, 1965 Replace the Many-Body system by a fictitious system of non-interacting electrons that generate the same density A local effective potential A set of equations H KS DFT [ ρ ] ψn = ε nψn ρ ( r) = n f n ψ ( r) n 2 Express the effective potential acting on electrons as the sum of: - the external potential (nuclei) - the electrostatic potential - the exchange-correlation potential which is approximated

7 DFT IN PRACTICE

8 DFT: SOLVE A SELF-CONSISTENT SYSTEM Solve a self-consistent set of eigenvalue equations: To solve these Kohn-Sham equations, need An exchange-correlation functional Vxc A basis set for expressing the eigenvectors ψ n An (iterative) algorithm for finding the eigenvectors

9 DFT: SOLVE A SELF-CONSISTENT SYSTEM Density Hamiltonian Local Non-local Kinetic Eigenvalue problem Iterative algorithm Convergence check? Wave functions Diagonalization in eigenvectors subspace Nuclei motion Structure relaxation Molecular Dynamics Energy, Forces

10 DFT: SOLVE A SELF-CONSISTENT SYSTEM Solve a self-consistent set of eigenvalue equations To solve these Kohn-Sham equations, need An exchange-correlation functional Vxc A basis set for expressing the wave-functions ψ n An (iterative) algorithm for finding the eigenvectors

11 DFT: EIGENSOLVERS H KS-DFT Need to solve this eigen problem without knowing the Hamiltonian matrix Hamiltonian application is a very expensive task Need only the «occupied» states, i.e. the ones associated with the lowest eigenvalues Use of an iterative eigensolver

12 DFT: EIGENSOLVERS Most used iterative eigensolvers in DFT H ε n ψ ~ Goal : minimize the residual ( ) n Conjugate gradient (CG) Requires explicit orthogonalizations «Conjugate the new search direction to the previous ones» Residual minimization method (RMM-DIIS) Direct inversion in iterative subspace Band-by-band algorithm «Minimize the norm of the residual vector» Blocked-Davidson like algorithms «Blocked» algorithm, iterative subspace «Append a block of vectors to the subspace» ABINIT: Locally Optimal Blocked Conjugate Gradient (LOBPCG) Knyazev, SIAM Journal of Scientific Computing 23, 517 (2001) Bottin, Leroux, Knyazev, Zerah, Comp. Mat. Sc. 42, 329 (2008)

13 DFT: EIGENSOLVERS Iterative eigensolvers : parallelism issues Orthogonalization Need communications Diagonalization (in subspace) Medium-sized problem not optimized on HPC Congugate-Gradient: A full orthogonalization No explicit diagonalization RMM-DIIS: Easy to parallelize (electron by electron) Medium-sized diagonalization Very sensitive to the initial guess Blocked-Davidson: Robust Can be parallelized by block Medium-sized diagonalization

14 DFT: SOME CODES Codes differ in their basis choice, functionalities, approximations, license, VASP (commercial) Quantum Espresso (GPL) ABINIT (GPL) CASTEP (commercial) SIESTA (free for academic) CP2K (GPL) BigDFT (GPL) GPAW (GPL) Most of them are written in FORTRAN

15 ABINIT: OVERVIEW OF THE PROJECT

16 ABINIT OVERVIEW Open-source package (GNU-GPL), freely distributed on the web Initiated in 1997 in Louvain-la-Neuve (Belgium) Density-Functional Theory Plane-wave basis (native) or wavelet basis (BigDFT library) for the eigenvectors ~1300 registered users at forum.abinit.org ~80 names in contributors file ABINIT v7 ~50 developer branches (~25 active)

17 ABINIT A DISTRIBUTED DEVELOPMENT A collaborative development Most active developer groups Louvain-la-Neuve university (Belgium) Liège university (Belgium) CEA, Bruyères-le-Châtel (France) CEA, Grenoble (France) CEA, Saclay (France) Dalhousie university, Halifax, (Canada) Montréal university (Canada) Rutgers university, New-Jersey (USA) San-Sebastian university (Spain) Cinvestav, Quérétaro (Mexico) Bogota university (Colombia)

18 ABINIT ORGANIZING THE DEVELOPMENT Continuous Integration (CI) development workflow - Extensive test suites (1000+ automatic tests) - Daily reviewing of contributions (automation) - Computer farm management Concurrent Version System : bazaar (bzr) ( Automatic Nightly Builds New published changes automatically tested on 20 architectures Managed by buildbot (

19 ABINIT: EXAMPLES Melting of Aluminum 2-phase method 512 atoms, CPU cores Bouchet, Bottin, Jomard, Zerah, Phys. Rev. B. 80, (2009)

20 ABINIT: EXAMPLES Hydrogen diffusion in electrolyte for fuel cells Computation of energy barriers between various sites for the proton Coupling with a higher scale(multiscale modelling) 40 atoms, CPU cores Hermet, Torrent, Bottin, Dejzanneau, Geneste, Phys. Rev. B. 87, (2013)

21 ABINIT: PARALLELIZATION SCHEMES

22 ABINIT: PARALLELIZATON LEVELS Electronic density formula: r ρ r r ( ) i = ( ) ( k + g ) r C g e r r v ( ) A r dk ρ ( ) n, k + σ spins n Bands r k Reciprocal space r g Plane waves 2 A atoms compensation Parallelization levels: Spins Bands k vectors Plane waves Atoms Parallelization over k points: easy; interesting for metals Parallelization over plane waves: requires a parallel 3-dim FFT Parallelization over bands: requires a blocked eigen-solver Parallelization over cells: not far from embarrassingly parallel

23 ABINIT PARALLELISM : OVERVIEW Distributed memory parallelism High level Message Passing Interface (MPI) Computational load distribution Over k-points, spins, atomic sites, bands and plane waves Each level has its own parallel efficiency Distribution of data Over k-points, spins, atomic sites, bands and plane waves Only collective communications used Currently no computation/communication overlap optimization

24 ABINIT PARALLELISM : OVERVIEW Shared memory parallelism Low level OpenMP (v3) compiler directives OpenACC or Intel compiler directives in preparation All loops over plane waves Linear and Matrix Algebra FFT Accelerators Some expensive parts have been ported on Nvidia GPU Speed-up : x3 x5

25 ABINIT PARALLELISM: EXAMPLE Gold, 108 atoms, 1200 electrons, TERA Supercomputer (CEA) «plane-waves, bands» parallelisation «k-points, plane-waves, bands» parallelisation By distributing the g-vector grid over two levels of processors rather than one, we improve the efficiency.

26 IMPROVE DFT SCALABILITY? DFT scalability can be improved if we Make some assumption Example : the electronic wave function is localized around atoms order-n algorithms can be applied But this is not a valid approximation for metals Put more physics in the theory Example : generalized DFT : replace the model Exchange- Correlation by a more sophisticated one (Hartree-Fock) Perfectly scalable, suitable for GPU, But the size of the simulated system does not increase but this is not useable for standard ABINIT calculations

27 ABINIT PARALLELIZATION: HOW TO GO FURTHER?

28 ABINIT PERFORMANCE ANALYSIS Repartition of time in in a ground-state calculation varying the number of band CPU cores (strong scaling) TEST CASE A vacancy in a 108 atoms cell (gold) TGCC-Curie (Intel Westmere) Iterative eigensolver: Consuming parts: eigensolver + comm Hamiltonian application: ~ linear scaling Analog results for weak scaling

29 CONSUMING PARTS AND BOTTLENECKS Fast Fourier Transforms used to apply local Hamiltonian Application of non-local Hamiltonian Linear algebra in iterative algorithm used to solve the eigenvalue problem Matrix algebra to get wave functions diagonalisation and orthogonalisation

30 CONSUMING PARTS WITH EFFICIENT PARALLELISM Fast Fourier Transforms used to apply local Hamiltonian Application of non-local Hamiltonian Linear algebra in iterative algorithm used to solve the eigenvalue problem Matrix algebra to get wave functions diagonalisation and orthogonalisation

31 BOTTLENECKS Fast Fourier Transforms used to apply local Hamiltonian Application of non-local Hamiltonian Linear algebra in iterative algorithm used to solve the eigenvalue problem Matrix algebra to get wave functions diagonalisation and orthogonalisation

32 CONSUMING PARTS AND BOOTLENECKS EFFICIENT Density Hamiltonian Local Non-local Kinetic Eigenvalue problem Iterative algorithm BOTTLENECK Convergence check? Wave functions Diagonalization in eigenvectors subspace Energy and forces

33 ABINIT BOTTLENECK ANALYSIS The efficiency of the blocked-davidson eigensolver algorithm depends strongly on the size of the blocks : Competition : orthogonalization vs diagonalization The size of the matrixes manipulated here is small (medium). The improvement of the eigen-solver on medium-sized dense hermitian matrix has not been addressed by mathematicians

34 PARALLEL ABINIT : HOW TO GO FURTHER? Optimize existing algorithm, trying to make the diagonalization (orthogonalization?) disappear Let s try to optimized matrix algebra library ELPA library, PLASMA library Let s try accelerators MAGMA libray Change the algorithm ; replace blocked-davidson by???... Currently under study : FEAST (contour integration technique) Polizzi, 2009

35 CONCLUSION

36 CONCLUSION Density Functional Theory on HPC, what can be done Many core architectures still evolve Implement an openacc version of ABINIT Davidson-like algorithms clearly show their limitations on petascale architectures; Find alternatives (?) On manycore architectures, the data distribution (governing memory access) has to be carefully set up A deep investigation of memory access has to be done for ABINIT

37 Commissariat à l énergie atomique et aux énergies alternatives Etablissement public à caractère industriel et commercial RCS Paris B