Introduction to Parallelism in CASTEP

Transcription

1 to ism in CASTEP Stewart Clark Band University of Durham 21 September 2012

2 Solve for all the bands/electrons (Band-) Band CASTEP solves the Kohn-Sham equations for electrons in a periodic array of nuclei: Ĥ[ρ]ψ b = E b ψ b where particle b has the b th solution ( band ), and Ĥ[ρ] = 2 2m 2 + ˆV HXC [ρ] + ˆV ext.

3 Bloch s Theorem ( ) Band where L is any lattice vector. V ext (r + L) = V ext (r) Bloch s theorem: the density has the same periodicity; the possible wavefunctions are all quasi-periodic : ρ(r + L) = ρ(r) ψ bk (r) = e ik.r u bk (r), where u k (r + L) = u k (r); e ik.r is an arbitrary phase factor. ψ bk (r + L) = e ik.(r+l) u bk (r + L) = e ik.l ψ bk (r)

4 Plane-waves basis set ( ) Band Since u bk (r) is periodic, we express it as a 3D Fourier series u bk (r) = G c Gbk e ig.r where c Gbk are complex Fourier coefficients, and the sum is over all the reciprocal lattice vectors, or s. Putting all this together we have: ψ bk (r) = e ik.r G c Gbk e ig.r = G c Gbk e i(g+k).r

5 The Wavefunction Band

6 sampling The bands for different s are independent of each other, so we get a different set of Kohn-Sham equations at each: Ĥ k [ρ]ψ bk = E bk ψ nk where ρ(r) = bk ψ bk (r) 2 Band So to find ψ bk we need Ĥk, which depends on ρ, which depends on ψ bk... We have to solve the equations iteratively.

7 Where does CASTEP spend its time? Band Applying Ĥk The kinetic energy is applied in Fourier space, the local potential is applied in real-space we need to Fourier transform between the two. Orthogonalisation We need to ensure our trial bands are orthogonal to each other. We compute the overlap matrix between all pairs of bands, and invert it.

8 Fourier transforms Band A 3D Fourier transform can be performed as 3 separate 1D transformations one in each direction (x, y and z). to transform ψ bk (G) ψ bk (r) scales as N G log(n G ). Every band at every has to be transformed, so total time is N G N b N k log(n G ).

9 Orthogonalisation Band We construct the band-overlap matrix for each S nmk = ψ nk ψ mk. Total time scales as N G N 2 b N k. Invert S k to find an orthogonalising transformation at each. Total time scales as N 3 b N k. Apply transformation to get orthogonal bands. Total time scales as N G N 2 b N k.

10 Large s Band As we simulate larger and larger systems, N G and N b increase and N k decreases. for Fourier transforms scales as (N G log N G )N b N k. for orthogonalisation scales as N G N 2 b N k. Orthogonalisation dominates in large s.

11 Band As the simulation system gets bigger and bigger, the orthogonalisation time dominates. We want to be able to use more computer cores in our to speed it up how can we do this?

12 ism Bands at different s are almost entirely independent of each other give each core a subset of the s. each core solves a subset of Kohn-Sham equations Cores only communicate when constructing the density Band ρ(r) = bk ψ bk (r) 2

13 TiN Benchmark Band The TiN simulation is a small standard benchmark 33 atoms 8 s 164 bands 10,972 s

14 ism in action Band

15 s and large systems Band ism is almost perfect. As simulations get bigger, N k gets smaller. the bigger the simulation, the fewer the cores we can use!

16 ism S nmk = ψ nk ψ mk = G c Gnk c Gmk Band give each core a subset of s. Contributions to S are summed over cores. N G is large so can use lots of cores. As simulation size increases, N G also increases.

17 ism in action Band

18 ism Band Each core only has some of the s. Each core only has some of the real-space r-vectors. Fourier transform: all s contribute at all points in real-space.

19 ism Band 3D transform can be performed as 3 1D transforms Give each core all s in a column in z Each core does transform in z All cores swap data so they have y-column data Each core does transform in y All cores swap data so they have x-column data Each core does transform in x Each core ends up with real-space data in x

20 ism Band Start: s inside cut-off sphere put on grid.

21 ism Band Now perform FFT in z-direction...

22 ism Band Transpose (swap) data into y-columns.

23 ism Band Now perform FFT in y-direction...

24 ism Band Transpose data into x-columns.

25 ism Band Now perform FFT in x-direction...

26 ism Band Now have real-space data in x-columns.

27 ism Band Actual transforms distribute well Transpositions are a problem Every core has to communicate with every other core! time scales as N 2 core. as N core increases, Fourier transform will dominate. when the communication time is comparable to the compute time, there s no point using more cores (it might even make CASTEP slower). There are ways to optimise the FFT, but the basic problem remains the same.

28 ism in action Band

29 scaling limit Band

30 k and G ism Band and ism is independent Can combine both to improve scaling E.g. if N k = 2, N G = 9, 000 and N core = 6: Data 1 2 s 1-3,000 core 1 core 4 s 3,001-6,001 core 2 core 5 s 6,001-9,000 core 3 core 6 For any, the data is split across 3 cores this is 3-way ism. For any subset of s the data is split between 2 cores this is 2-way ism

31 k+g ism in action Band

32 Optimal performance Band It is always worth exploiting ism when you can, but not all computers let you run on any number of cores. If you can t use N core = N k then try to have a high common factor between them. E.g. if N k = 35, N core = 35 will give an excellent speed-up, but N core = 5 or 7 will also be very efficient. N core = 20 or 21 would use some ism, but also give good efficiency. (Note that 2-way ism is not very quick, so N core = 10 or 14 might not be the best choices.)

33 Optimal performance Band Remember that as you increase the ism, the communication time increases. Eventually your will scale poorly, and if you keep increasing N core it will even start to run slower. CASTEP always defaults to using as much ism as it can, and then uses ism across any other cores.

34 Very large s and the Γ-point Band For isolated or very large simulation systems you only need 1. For well-isolated or extremely large simulations this can be the special point k=(0,0,0), called the Γ-point. Why do we care?

35 Γ-point s Band Bands at Γ are real in real-space, not complex. the Fourier coefficients for -G are the complex conjugate of those at G don t bother with -G; only need half the s Inner products are real, not complex we don t need to bother computing the imaginary parts. Bands take up only half as much memory. FFT about 2x faster, orthogonalisation 8x faster. CASTEP detects if you re only using the Γ-point and uses these automatically.

36 Γ-point s Band No ism (N k = 1). Orthogonalisation speed-up better than FFT one Can show poorer scaling. Worth using if this sampling is sufficiently accurate. Occasionally worth using a bigger simulation cell if it allows accurate Γ-point sampling.

37 Band Calculations Band Hamiltonian is the same for all bands at the same Fourier transforms of different bands are independent perfect scaling with band-ism when applying the Hamiltonian

38 However...we need to orthogonalise Band Need to construct overlap matrix S at each S nm = ψ n ψ m Inner product is between all pairs of bands Need all-to-all communications as band-ism increases, communication dominates

39 Band distribution Band We distribute the bands in a round-robin fashion, e.g. if N b = 11 and N core = 3: Core Bands 1 1,4,7,10 2 2,5,8,11 3 3,6,9 For the FFTs each core just transforms its own bands.

40 Band ism performance Band FFTs scale perfectly Orthogonalisation requires all-to-all amongst the cores communication time scales as N 2 core communications dominate as N core increases

41 Band example Band

42 Band with non-local XC Band

43 k, G and B ism Band, and band-ism are independent Can combine all three to improve scaling Define: kp-group: group of cores with same s and bands, but different s gv-group: group of cores with same s and bands, but different s bnd-group: group of cores with same s and s, but different bands

44 CASTEP performance Band Everything scales well with ism As the number of cores in the gv-group increases, the communication time in the FFT dominates As the number of cores in the band-group increases, the communication time in the orthogonalisation dominates need to find the right balance between gv- and bnd-ism.

45 Using band-ism Band New functionality Will compute ground-state energies, forces and stresses Accessed via a devel_code string, e.g.: devel_code : bandpar=2 in your.param file would use 2-way band-ism.

46 Why a devel_code? Band Band-ism and ism use the network differently Low-latency networks good ism High-bandwidth networks good band-ism Difficult for CASTEP to automatically decide best isation strategy Some limitations to current implementation

47 Band- Limitations Band Not optimised heavily yet Only for density mixing Only for some s, e.g. task : singlepoint These limitations are temporary!

48 The future of band-ism Band Already integrated with main source code Limitations on band- s will be lifted CASTEP will do more automatically (as we gain experience)

49 Multi-core architectures Band Common to have multi-core processors These have shared memory (RAM) No need to go through the inter-processor interconnect Access via param file num_proc_in_smp : <integer>

50 Performance Band

51 Band ism is very efficient, but you eventually run out of s ism is good but becomes worse as you use more and more cores. Combining these two allows CASTEP to scale well to many cores. Careful choice of the number of cores can improve CASTEP performance considerably. (Didn t include task-farming isation, available for path integral MD in CASTEP)