Compression Algorithms for Electronic Structure Computations

Transcription

1 Compression Algorithms for Electronic Structure Computations François Gygi University of California, Davis IPAM Workshop, Computation meets Big Data, Feb 2, 2017 Supported by DOE BES DE-SC and MICCoM 1

2 Context: First-Principles Molecular Dynamics Monte Carlo Molecular Dynamics Statistical Mechanics FPMD Electronic Structure Theory Quantum Chemistry Density Functional Theory 2

3 Context: First-Principles Molecular Dynamics Molecular Dynamics Density Functional Theory 2 d m R = dt F i 2 i i FPMD ( ) ρ Δ + V ϕ ( x) = εϕ ( x) ( x) eff n = ϕi ( x) i= 1 i i i 2 Newton equations Kohn-Sham equations R. Car and M. Parrinello (1985) 3

4 Electronic Structure and dynamical properties of complex structures Complex structures Nanoparticles Assemblies of nanoparticles Embedded nanoparticles Liquid/solid interfaces 4

5 Nanoparticles Multiplicity of locally stable structures requires extensive sampling Electronic structure requires accurate methods (beyond DFT) Finite temperature properties require first-principles molecular dynamics Cd 34 Se 34 5

6 Liquids and Liquid-Solid Interfaces Embedded nanoparticles, assemblies of nanoparticles Annealing of structures requires MD simulations Calculation of band gaps and band alignments requires accurate electronic structure (beyond DFT) Si/ZnS S. Wippermann, M. Vörös, A. Gali, F. Gygi, G. Zimanyi, G.Galli, Phys. Rev. Lett. 112, (2014).

7 Liquids and Liquid-Solid Interfaces H 2 O/Si(100)H Liquids require finite temperature simulations ab initio MD multiple samples/replica exchange simulations Electronic structure band alignment Spectroscopy requires calculation of IR and Raman spectra 7

8 Challenges Multiple length and time scales e.g. nanoparticle assembly process, non-equilibrium processes Finite temperature: MD/MC simulations ab initio MD for large systems (>1000 atoms) Need for accurate electronic structure hybrid DFT and/or GW/BSE level 8

9 Qbox code: DFT and hybrid DFT first-principles molecular dynamics massively parallel first-principles MD C++/MPI/OpenMP DFT and hybrid DFT MD GPL license All algorithms discussed here are available in the Qbox code 9

10 Computation meets Big Data: Why data compression is necessary Storage of restart files in simulations contain full information about wave functions size is O(N 2 ) for N atoms, reaches 100 GB-1 TB for large problems Acceleration of DFT and/or hybrid DFT calculations Reduce cost (e.g. from O(N 4 ) to O(N 3 )) Exploit locality of data on modern computer architectures 10

11 Desirable properties of compression schemes Accuracy control ideally a single parameter controlling the accuracy gradual reduction of the error to zero Efficiency tradeoff between space efficiency and cost of compression algorithm 11

12 Three compression approaches Reduced resolution Easy: in the Fourier basis: Reduce energy cutoff Efficient: requires only 1 FFT per orbital Problem: only moderate reduction is possible Compute Wannier functions Efficient (if using the right algorithm) Truncation procedure is ill-defined (no error control) Recursive Subspace Bisection (this work) Efficient (as fast as Wannier function calculation) Controllable error and systematic truncation scheme 12

13 Wannier functions Existence of exponentially localized Wannier functions Kohn (1959), des Cloiseaux (1964), Nenciu (1983), Helffer et al (1989), Brouder et al (2007), Panati (2007, 2013) Computation of Wannier functions: find an orthogonal transformation among orbitals that minimizes the spread σ 2 X = x x ( ) 2 Optimization problem (with local minima) Marzari, Vanderbilt (1997) Use conjugate gradients, etc. Approximate simultaneous diagonalization problem F.G, Fattebert, Schwegler (2003) F.G., J.L.Fattebert, E.Schwegler, Comput.. Phys. Comm. 155, 1 (2003) J.F.Cardoso and A. Souloumiac, SIAM J. Mat. Anal. Appl. 17, 161 (1996). 13

14 Spread Functionals Spread of an operator  (single orbital) σ 2 (  φ) = φ (  φ  φ ) 2 φ = φ Â2 φ φ  φ 2 Spread of a set of orbitals ({ }) ( ) ˆ σ φ = σ φi 2 2 Aˆ i A i 14

15 Spread Functionals The spread is not invariant under orthogonal transformations among orbitals ψ i = j x ij φ j X! n n orthogonal 2 σ Â ({ ψ }) 2 i σ Â ({ φ }) i There exists a matrix X that minimizes the spread 15

16 Spread Functionals Let A,B! n n a ij = i  j b ij = i Â2 j 2 σ  ({ ψ }) i = tr X T BX Minimize the spread = maximize = diagonalize A n i=1 ( ) ( X T AX ) 2 ii n i=1 ( X T AX ) 2 ii 16

17 Spread Functionals Case of multiple operators operators matrices 2 σ Â Â (k ) k =1,,m A (k ) k =1,,m ({ ψ }) 2 i = σ k ) Â( i k ( ψ ) i Minimize the spread = maximize n i=1 k = joint approximate diagonalization of the matrices A (k) ( X T A (k ) X ) 2 ii 17

18 Spread Functionals Example of multiple operators ( ϕ) ˆ (1) = ˆ ˆ (,, ) ϕ(,, ) A X X x y z x x y z ( ϕ) ˆ (2) = ˆ ˆ (,, ) ϕ(,, ) A Y Y x y z y x y z ( ϕ) ˆ (3) = ˆ ˆ (,, ) ϕ(,, ) A Z Z x y z z x y z The matrices A (k) do not necessarily commute, even if the operators  (k) do commute 18

19 Calculation of Wannier functions In periodic systems  (1) =  (3) = Ĉ y  (5) = Ĉx cos 2π L x ˆx  (2) = cos 2π L y ŷ  (4) = Ŝ y Ĉz cos 2π L z ẑ  (6) = Ŝx sin 2π L x sin 2π L y ˆx 2π Ŝz sin ẑ L z ŷ 19

20 Calculation of Wannier functions The spread is minimized by simultaneous diagonalization of the matrices C x,s x,c y,s y,c z,s z Positions of the center of mass of the localized solutions ( Wannier centers ) Spreads x θi Lx 2π y θi x τi = Ly θi arctan 2π = z θi Lz 2π 2 2 ( σ i ) = Lx 1 ( Cx) ( Sx) x ( 2 2 ) ii ii ( Sx ) ( C ) x ii ii F.G., J.L.Fattebert, E.Schwegler, Comput.. Phys. Comm. 155, 1 (2003) J.F.Cardoso and A. Souloumiac, SIAM J. Mat. Anal. Appl. 17, 161 (1996). 20

21 Maximally localized Wannier functions Extended orbital Wannier function 21

22 Truncation of Wannier functions Wannier functions (WFs) can be truncated in real space truncate orbital to zero below a given threshold truncate orbital to zero outside of a given radius WFs can have variable localization properties 22

23 Maximally localized Wannier functions Extended orbital Wannier function 23

24 Truncation of Wannier functions Wannier functions (WFs) can be truncated in real space truncate orbital to zero below a given threshold truncate orbital to zero outside of a given radius WFs can have variable localization properties Are there other ways to localize orbitals? 24

25 Recursive subspace bisection Localize some orbitals on domains of decreasing size divide the simulation domain into two subdomains Ω localize orbitals on 1) 2) or 3) keep extended on Ω L Ω L ΩR R apply recursively to smaller domains V ΩL ΩR ΩL ΩR F.G. Phys. Rev. Lett. 102, (2009) 25

26 Subspace Bisection Y = [φ 1 φ n ] ΩL ΩR Y YV 26

27 The CS decomposition A matrix Y having orthogonal columns, can be T decomposed as Y1 U1Σ1V Y = T Y = 2 U2Σ2V where U 1, U 2, V are orthogonal matrices,! Σ 1 = # C " 0 $! & Σ 2 = # % " C = diag(c 1,,c n ) S = diag(s 1,,s n ) c i 2 + s i 2 =1 S 0 $ & % Stewart (1982) 27

28 CS Decomposition ΩL ΩR Y YV 28

29 CS Decomposition norm = c 2 i ΩL ΩR Y norm = si = 1 c YV 2 2 i 29

30 CS Decomposition ΩL localized in Ω L!" # $# 2 ci < ε ΩR Y 2 si < ε!" # $#!" # $# extended states localized in Ω R 30

31 Subspace Bisection Algorithm 1. Choose the acceptable 2-norm error 2. Perform a CS decomposition of the matrix Y 3. For each vector of YV: if c i 2 < ε orbital localized in Ω R (truncate in Ω L ) else if s i 2 < ε orbital localized in Ω L (truncate in Ω R ) else orbital is extended Ideal limit: 2-fold data reduction Cost: The CS decomposition can be achieved by diagonalization of the matrix Y 1T Y 1 ε 31

32 Recursive Subspace Bisection Bisection is applied simultaneously in 3 directions Implementation: simultaneous (approximate) diagonalization of symmetric matrices F.G, J.-L.Fattebert, E.Schwegler, Comp. Phys. Comm. 155, 1 (2003). J.F.Cardoso and A. Souloumiac, SIAM J. Mat. Anal. Appl. 17, 161 (1996). 32

33 Recursive subspace bisection Use multiple bisecting planes in each direction W j (x) W 1 (x) W 3 (x) W 6 (x) W 12 (x) Optimal placement of bisecting planes is provided by Walsh functions W j (x) j=1,3,6,12, x 33

34 Recursive subspace bisection Phys. Rev. Lett. 102, (2009). 34

35 (H 2 O) 512 orbitals after bisection localized extended 35

36 Compression ratio (H 2 O) 512 ε = 10 Data size reduction: N % N 1/ % N 1/ % N 1/ % (19x0) Carbon nanotube (304 atoms) ε = 2048 orbitals 10 3 Data size reduction: 2.72 N % N 1/ % N 1/ % N 1/ % 608 orbitals 36

37 Compression ratio using recursive bisection recursion on l max levels, l max =1,2,3 1 4x compression ratio 0.1 l max = 1 l max = 2 r = x 36x l max = N mol r = truncated MLWFs 1/(N logn) slope F.G., Phys. Rev. Lett. 102, (2009) 37

38 Localization of orbitals in inhomogeneous systems W. Dawson and F.G, JCTC 11, 4655 (2015) 38

39 Free electron gas: Distribution of CS singular values 2 c i Y matrix = F n p (section of the Fourier matrix, i.e. eigenvectors of the Laplacian) 2 c i Y matrix = random orthogonal matrix A. Edelman et al. SIAM J. Sci. Comput. 20, 1094, (1999) 39

40 Hybrid density functionals Conventional density functionals are often insufficient to describe weak bonds (e.g. hydrogen bonds) or optical properties (band gap) Hybrid density functionals include a fraction of the Hartree-Fock exchange energy The Hartree-Fock exchange energy involves N(N-1)/2 exchange integrals (for all e - pairs) E x = 1 2 N i, j ϕ ( i r 1 )ϕ ( i r 2 )ϕ ( j r 1 )ϕ ( j r ) 2 dr 1 dr 2 r 1 r 2 Cost: O(N 3 log N) (with large prefactor) for plane waves For atom-centered basis sets: O(N) (Strain, Scuseria (1996), Burant, Scuseria, Frisch (1996), Schwegler, Challacombe, Head- Gordon (1997)) 40

41 Acceleration of Hartree-Fock and hybrid DFT calculations N(N-1)/2 exchange integrals (all e - pairs) E x = 1 2 N i, j ϕ ( i r 1 )ϕ ( i r 2 )ϕ ( j r 1 )ϕ ( j r ) 2 dr r 1 r 1 dr 2 2 using bisection: non-overlapping pairs can be skipped in the sum The error is positive F.G., I. Duchemin, JCTC 9, (2013). 41

42 Speedup of hybrid-dft calculations vs truncation threshold (H 2 O) 32, (H 2 O) 64 (H 2 O) 256 truncation threshold (%) ε time (s) speedup 0.0 (ref) % % % % % %

43 Truncation error due to bisection (H 2 O) 63 Cl Energy error E time( )/time(0) E (a.u.) time( )/time(0)

44 Energy error per orbital vs threshold 44

45 Force error vs threshold 45

46 Error in ionic forces for MD applications "fhist-0.01.dat" Error caused by bisection with 0.01 threshold "fhist-0-100ry.dat" Error caused by changing Ecut from 85Ry to 100Ry Force error (a.u.) Recursive bisection affects forces in a controlled way 46

47 Error in ionic forces for MD applications "fhist dat" Error caused by bisection with threshold "fhist-0-100ry.dat" Error caused by changing Ecut from 85Ry to 100Ry Force error (a.u.) Recursive bisection affects forces in a controlled way 47

48 HOMO-LUMO gap, band gaps Hybrid DFTs lead to large improvements in band gaps (Henderson, Paier, Scuseria, PhysStatSol 2011) Bisection algorithm: occupied and empty orbitals must not be mixed when localizing orbitals (H 2 O) 63 Cl - A (k ) = 0 threshold Egap (ev)

49 Localization of empty orbitals W. Dawson and F.G, JCTC 11, 4655 (2015) 49

50 Hybrid-DFT electronic structure of Si nanoparticles embedded in ZnS LDA PBE0 Si 66 Zn 228 S 218 DOS Si 66 Zn 228 S electrons 200 empty orbitals PBE0 DOS (arb. units) 1 scf step, 5 iterations BG/Q, 16k cores, 772 s E (ev) S. Wippermann, M. Vörös, A. Gali, F. Gygi, G. Zimanyi, G.Galli, Phys. Rev. Lett. 112, (2014).

51 Hybrid-DFT electronic structure of bulk SiC (4096 atoms) 4096 atoms electrons PBE0 electronic structure (hybrid) recursive subspace bisection ANL Mira (BG/Q) 64k cores 357s/self-consistent iteration hybrid DFT electronic structure for 4096 atoms

52 Where is the (Big) Data? Making data accessible is critical for verification and validation of simulation software Agreeing on data formats has proved difficult.. XML schemas for electronic structure data repository of reference simulations 52

53 Summary Simulation of complex materials Truncation of Maximally Localized Wannier functions Recursive subspace bisection Controlled error in inhomogeneous systems Acceleration of hybrid DFT simulations Supported by DOE BES DE-SC and the MICCoM DOE center 53

54 Acknowledgements Giulia Galli (UChicago) Marco Govoni (UChicago) Stefan Wippermann (MPI) Eric Schwegler (LLNL) Funding DOE BES computer time DOE INCITE/ANL-ALCF NSF XSEDE NERSC William Dawson (CS, UCDavis) Martin Schlipf (CS, UCDavis) Cui Zhang (UCDavis) Alex Gaiduk (UChicago) Marton Vörös (UCDavis, ANL) Quan Wan (UChicago) T.-Anh Pham (LLNL) Supported by DOE BES DE-SC and the DOE MICCoM center