Large Scale Electronic Structure Calculations

Transcription

1 Large Scale Electronic Structure Calculations Jürg Hutter University of Zurich 8. September, 2008 / Speedup08

2 CP2K Program System GNU General Public License Community Developers Platform on "Berlios" (cp2k.berlios.de) Open CVS for download Written in Fortran95, lines of code MPI and OpenMP parallelization Quality control: automatic regression and memory leak testing Force Methods: Kohn Sham DFT, Classical Force Fields, QM/MM Sampling Methods: Molecular Dynamics, Meta-Dynamics, Monte Carlo

3 CP2K/Quickstep Gaussian basis sets Plane waves auxiliary basis for Coulomb integrals Sparse matrices, efficient screening linear scaling KS matrix computation Fast/robust direct wavefunction optimizer (OT) Memory scaling : MN CPU time scaling : MN 2 M number of basis functions, N: number of states For given number of electrons, O(N) in basis set size All-electron calculations with projector augmented-waves

4 Ru-dye in Acetonitrile Solution Box:(21.43 Å) 3, 620 atoms, 6000 bsf, 850 occupied orbitals, 1 min/ MD step / 128 CPUs

5 TiO 2 / Acetonitrile Interface 101 anatase surface, slab calculation, system size: Å (TiO 2 ) 72 (NCCH 3 ) 68, 624 atoms, 5632 electrons, bsf

6 Rubredoxin in Water Solution Solvated Rubredoxin (FeS4); 2800 atoms; bsf 117 s / SCF on 1024 CPUs (XT3), 80% parallel efficiency

7 Ab Initio Molecular Dynamics Chemistry: Charged particles (electrons, nuclei) Coulomb interactions Time-dependent Schrödinger equation Ψ(x; t) i = HΨ(x; t) t H Hamiltonian operator Ψ(x; t) wavefunction

8 Born Oppenheimer Approximation Separation of movement of heavy particles (nuclei) and light particles (electrons). Electrons always assume a stationary state for all nuclear positions. i Ψ N(R; t) t = H N Ψ N (R; t) H N = A 1 2M A δ A + V(R) H e Ψ N (x; R) = V(R)Ψ N (x; R)

9 Newton equation of motion Approximation: Time evolution of nuclei can be described by classical mechanics. M A RA = V(R) R A H e Ψ N (x; R) = V(R)Ψ N (x; R) Newton EOM : Molecular dynamics Time-independent Schrödinger equation : Quantum chemistry

10 Electronic structure theory Kohn Sham density functional theory V(R) = Min (E KS ({Φ}; R)) E KS ({Φ}; R) = 1 Φ i δφ i dr + i n(r)n(r ) + r r drdr + E xc [n] n(r) = Φ i (r) 2 Φ i Φ j dr = δ ij i V ext (r) n(r) dr N (number of electrons) coupled, single-particle, non-linear Schrödinger equations Constrained minimization problem

11 Basis: Atom-centered Gaussians Advantage Localized, analytic integrals, biased, product theorem Disadvantage Non-orthogonal, biased Other choices used: Exponential functions, plane waves, wavelets, etc. Φ i (r) = M C αi ϕ α (r) New variational parameters: M N matrix C. α

12 Kohn Sham Method Minimize E KS (C) Constraints: C T SC = 1 S: Overlap matrix, metric Optimization using gradient methods: Gradient H KS C H KS is the Kohn Sham matrix

13 Kohn Sham Matrix Kohn Sham matrix is sparse (in localized basis functions) allows for a linear scaling calculation of H KS HIV-1 Protease-DMP323 complex in solution (3200 atoms)

14 Coulomb energy and potential E H = 1 2 V H (r) = n(r)n(r ) r r dr dr = 2π Ω n(r ) r r dr = 4π Ω G G n (G)n(G) G 2 n(g) eig r G 2 Plane wave expansion of charge density n(r) = G n(g)e ig r

15 Real Space Grid Finite cutoff and computational box define a real space grid {R}

16 Coulomb Potential P = CC T n(r) FFT n(g) V H (G) = n(g) FFT V }{{ G 2 H (R) V } O(n log n) n(r) = µν P µν ϕ µ (R)ϕ ν (R) = µν P µν ϕ µν (R) V µν = R V(R)ϕ µ (R)ϕ ν (R) = R V(R) ϕ µν (R) Efficient screening of sums using ϕ µν (R).

17 Scaling Kohn-Sham Matrix Time [s] Number of Basis Functions

18 Parallelization Conflicting needs for optimal distributions Data layout for sparse operator matrices. Data layout for real/fourier space grids. Parallel 3d-FFT (typical grid sizes ). Typical support of basis function is grid points. Large overlapping regions. Optimize a complex, empirical work function using Monte Carlo techniques.

19 Parallel efficiency 100 Grid Routines 3d-FFT Efficiency [%] Number of Cores Test system: 64 water molecules, 192 atoms, 256 orbitals, 2560 basis functions, 4s/molecular dynamics step on 256 cores Cray XT-3, dual core, CSCS Manno

20 Optimization Standard (reference) method : Fix point iteration with diagonalization H(C) Build KS matrix Extrapolate HC=SCE Solve KS equations Convergence

21 Orbital transformation method Direct optimization technique. Can use algorithms for unconstrained optimizations. Memory MN M Number of basis functions Scaling MN 2 N Number of occupied orbitals J. VandeVondele, JH, J. Chem. Phys (2003) J. VandeVondele et al. Comp. Phys. Comm (2005)

22 Orbital transformation method Set of reference occupied orbitals: C 0 (M N Matrix) New variables X (M N Matrix) C(X) = C 0 cos U + XU 1 sin U ( ) 1/2 U = X T SX Linear constraint X T SC 0 = 0 Analytic gradient for arbitrary X Standard optimization with line search and pre-conditioning

23 Avoiding Matrix Functions cos(u) = K i=0 If C 0 C opt X 0 ( 1) i K (2i)! U2i, U 1 ( 1) i sin(u) = (2i + 1)! U2i i=0 U = (X T SX) 1/2 64 water molecules on 32 CPUs IBM SP4 (seconds) DZVP TZVP TZV2P QZV2P QZV3P Basis OT ScaLapack

24 Simplification of minimization method Transform constrained minimization problem { } C = arg min E[C] C T SC = 1, C into an unconstrained minimization of where C = arg min C { E[f(C)]}, E[f(C)] = E[f n (C)] + O(E[δC n ]). with C = C 0 + δc and C T 0 SC 0 = 1. V. Weber, JH, J. Phys. Comm (2008)

25 Orbital Refinement Function z = z 0 + δz and z T 0 Sz 0 = 1 f 2 (z) = 1 z (3 Y), 2 f 3 (z) = 1 8 z (15 10Y + 3Y 2), f 4 (z) = 1 16 z (35 35Y + 21Y 2 5Y 3), and f 5 (z) = z ( Y + 378Y 2 180Y Y 4), f T n (z)sf n (z) 1 = O(δz n ).

26 Optimization Nonlinear conjugate gradients with line search Includes only matrix additions and multiplications Simplifies parallelization and use of sparsity Reduces prefactor but is less stable than orbital transformations

27 Example: Liquid Water (H 2 O) 512 (H 2 O) 1024 (H 2 O) 2048 basis functions CPUs (XT4) n iter SCF f ks f other f mini t tot [min]

28 Linear scaling optimization To reach linear scaling, matrix C has to be sparse. Solution is invariant to unitary transformations of C We only need a representation of the space spanned by C. Sparse representations of C are called localized orbitals. Standard techniques in quantum chemistry based on physical properties. Efficient algorithms need parallel sparse matrix-matrix multiply

29 L1-Norm Localization Smooth sparseness function based on L1 norm { y ε1 /2 y ε φ(y) = 1 y 2 /2ε 1 y < ε 1, Parameterization of orbitals: C(U) = CU, U is a M M unitary matrix U(X) = exp X where X is an antisymmetric matrix. Optimization of U = arg max φ(u(x)). X

30 Example:Water Occupation l1-norm 512 DM 512 Boys-Foster 1024 l1-norm 1024 DM 1024 Boys-Foster 2048 l1-norm 2048 DM 2048 Boys-Foster log10(threshold)

31 Combine Optimization and Localization Molecular dynamics, 1024 water molecules MO 1E-5/1E-6/1E-7 DM 1E-5/1E-6/1E-7 Occupation Total SCF iterations Open question: How does sparseness threshold affect MD stability?

32 Multi-level parallelization Embarassingly parallel level (MPI) String methods Property calculations Path integral algorithms Electronic structure level (MPI) Fine-grain parallelisation (OpenMP) Matrix muliply Multiple 1d-FFT Loop level parallelization

33 Electronic structure calculations Applications of CP2K in Chemistry Biochemistry Solid state physics Material science Nanotechnology

34 Acknowledgment Joost VandeVondele Valery Weber Teodoro Laino Marcella Iannuzzi Florian Schiffmann Manuel Guidon University of Zurich, Swiss National Science Foundation