Materials Science with WIEN2k on vsc1 and vsc2

Transcription

1 Materials Science with WIEN2k on vsc1 and vsc2 P. Blaha Institute of Materials Chemistry TU Wien

2 Computational Materials Science describe materials by quantummechanical simulations ab initio (parameter free) simulate: infinite ( perfect ) bulk solids impurities, vacancies in solids surfaces nanostructures atoms + molecules

3 Computational Materials Science atomic structure geometry stability phase transitions mechanical properties

4 Computational Materials Science electronic structure bandstructure + density of states magnetism chemical bonding,. spectroscopies (IR, Raman, XPS, XAS, XES, EELS,Mössbauer, NMR, STM)

5 Method: DFT calculations numerical solution of a Schrödinger-like equation (Kohn-Sham) 2 potential V: 1 2 classical Coulomb potentials between nuclei and electrons quantummechanical electron-electron interaction density-functional-theory (DFT Walter Kohn) in principle: exact in practice: various approximations LDA GGA Hybrid-DFT dynamical mean field theory V ( r) k i k i k i

6 Walter Kohn, Nobel Prize 1998 Chemistry i are illegitimate children of DFT time-dependent DFT, GW, BSE, DMFT

7 Method: DFT calculations numerical solution of a Schrödinger-like equation (Kohn-Sham) k k k V ( r) i i i 2 expansion into augmented plane waves basisfunctions Kn : variational principle 1 2 k = C K n k n k < E n n APWs / atom < H > < E >= < > C k n > = 0 generalized eigenvalue problem H C=E S C Setup and diagonalization of (real or complex) matrices of size to (up to 200 Gb memory, only 10% of i )

8 Loops: loop 1: different structures (atomic positions) loop 2: scf-cycle (solve [-½ 2 +V] = E new V iterate) loop 3: k-loop (solve H =E for different k-points) loop 4: setup + diagonalization of HC= E SC largest effort, highest optimization, best parallelization, scaling of time and memory processors F90, mpi, Scalapack, blas loop over APWs in parallel (via scripts, slow network, common NFS) sequential (efficient multi-secant BROYDEN-mixing; L.Marks PRB 78, ) coarse grain parallel (different jobs) or sequential (forces new positions)

9 WIEN2k software package An Augmented Plane Wave Plus Local Orbital Program for Calculating Crystal Properties Peter Blaha Karlheinz Schwarz Georg Madsen Dieter Kvasnicka Joachim Luitz November 2001 Vienna, AUSTRIA Vienna University of Technology developed over more than 25 years 2000 registered groups 1800 mailinglist users Europe: A, B, CH, CZ, D, DK, ES, F, FIN, GR, H, I, IL, IRE, N, NL, PL, RO, S, SK, SL, SI, UK, ETH Zürich, MPI Stuttgart, FHI Berlin, DESY, TH Aachen, ESRF, Prague, IJS Ljubjlana, Paris, Chalmers, Cambridge, Oxford America: ARG, BZ, CDN, MX, USA (MIT, NIST, Berkeley, Princeton, Harvard, Argonne NL, Los Alamos NL, Oak Ridge NL, Penn State, Purdue, Georgia Tech, Lehigh, John Hopkins, Chicago, Stony Brook, SUNY, UC St.Barbara, UCLA) far east: AUS, China, India, JPN, Korea, Pakistan, Singapore,Taiwan (Beijing, Tokyo, Osaka, Kyoto, Sendai, Tsukuba, Hong Kong) 55 industries (Canon, Eastman, Exxon, Fuji, Hitachi, IBM, Idemitsu Petrochem., Kansai, Komatsu, Konica-Minolta, A.D.Little, Mitsubishi, Mitsui Mining, Motorola, NEC, Nippon Steel, Norsk Hydro, Osram, Panasonic, Samsung, Seiko Epson, Siemens, Sony, Sumitomo,TDK,Toyota).

10 setup and iter. diagonalization of HC=ESC ifort: vsc1 2x faster mkl: (dgemm) vsc1 1.5 x faster diag with scalapack: (blocked davidson) vsc1 ~ 1.2 x faster (256 cores) vsc2 does not scale for more than 768 cores vsc2 (even with 2048 cores) cannot beat 768 vsc1-cores (matrix size: , real*8)

11 speedup: speedup of own mpi-code close to optimal little communication sequential parts < 0.1 (1% seq: 50 x speedup on 100 cores max 100 x on 1000 cores ) iter. diag using scalapack: VSC-2 scaling limited to ~768 cores (even for VERY large problem) pdsymm (m-m mult) does not scale well some smaller parts show constant time time for diag of small sub-space (10000) problem increases 2x and dominates needs some additional code optimizations

12 calculate density from eigenvectors: 2 way parallelization: N-core = N-atoms * N-energies (less cpu) (less comm +I/O) cores: VSC-1 NE= VSC-2 NE= NE= NE= read 5GB + distribute to N-atom cores (mpi_bcast, mpi_scatter) VSC-1 NE= VSC-2 NE= NE= NE= I/O or infiniband bottlenecks on vsc2?

13 I/O or infiniband problems of vsc2: $HOME (NFS): jobs hang frequently during longer calc. $GLOBAL: jobs hang much less ls -als of large dir: 7s ( 0s on vsc1) loading of 500MB file in vi: 7s (<3s on vsc1)

14 starting 16 single-core jobs on one node: time command of such a parallel run: irregular run time with up to 50% overhead r01n01(1) u 0.348s 1: % r01n01(1) u 0.264s 1: % r01n01(1) u 0.197s 1: % r01n01(1) u 0.197s 1: % r01n01(1) u 0.199s 1: % r01n01(1) u 0.245s 1: % r01n01(1) u 0.215s 1: % r01n01(1) u 0.235s 1: % r01n01(1) u 0.217s % r01n01(1) u 0.226s 1: % r01n01(1) u 0.240s 1: % r01n01(1) u 0.227s 1: % r01n01(1) u 0.209s % r01n01(1) u 0.250s 1: % problem has been fixed using taskset command. All jobs finish after ~60 sec (within 2-3 sec).

15 wish-list vsc-3: don t look at Top-500 performance, but ensure sustained performance (mpi, scalapack - scaling) faster cpus (instead of more cores)

16 Thank you for your attention!