Multigrid Methods for Linear Systems with Stochastic Entries Arising in Lattice QCD. Andreas Frommer

Transcription

1 Methods for Linear Systems with Stochastic Entries Arising in Lattice QCD Andreas Frommer

2 Collaborators The Dirac operator James Brannick, Penn State University Björn Leder, Humboldt Universität Berlin Stephan Krieg, Jülich Research Centre Karsten Kahl, Matthias Rottmann, Artur Strebel, Universität Wuppertal A. Frommer, in Lattice QCD 1/24

3 What we learned so far Setting: Statistical mechanics: Observables are expected values of operators depending on random variables Lattice QCD: 4d lattice L in space-time We need configurations U = {U µ (x) : x L, µ = 1,..., 4} of gauge links These are obtained from fixed point distribution via Markov chain MC Computationally: importance sampling via HMC Connection to Michael Günther s talk QCD and Lattice QCD Dirac and Dirac-Wilson U ν(x + ˆµ) U µ(x ˆµ + ˆν) U µ(x + ˆν) x + ˆν U ν(x) U ν(x + ˆµ) U µ(x ˆµ) U µ(x) U µ(x + ˆµ) x x + ˆµ U ν(x ˆν) U ν(x + ˆµ + ˆν) U µ(x + ˆµ + ˆν) x + ˆµ + ˆν U ν(x + ˆµ ˆν) A. Frommer, in Lattice QCD 2/24

4 QCD is The Dirac operator the standard model of the strong interaction between quarks as the constituents of matter a quantum field theory described by the Dirac operator depending on a background field Lattice QCD is a discretization of QCD requiring a discretized Dirac operator and a discrete version of the background field: a configuration of gauge links Connection to Michael Günther s talk QCD and Lattice QCD Dirac and Dirac-Wilson A. Frommer, in Lattice QCD 3/24

5 The Dirac operator Connection to Michael Günther s talk QCD and Lattice QCD Dirac and Dirac-Wilson (D + mi)ψ = η, where ψ = ψ(x) and η = η(x) represent quark fields x = (x 0, x 1, x 2, x 3 ) point in space-time m (implicitly) sets the quark mass gluons are represented in the Dirac operator D 3 D = γ µ ( µ + A µ ) µ=0 A. Frommer, in Lattice QCD 4/24

6 Connection to Michael Günther s talk QCD and Lattice QCD Dirac and Dirac-Wilson D = 3 γ µ ( µ + A µ ) µ=0 µ = / x µ A is the gluon (background) gauge field, A µ (x) su(3) (3 3, skew-hermitian and traceless) γ µ generators of Clifford algebra γ µ γ ν + γ ν γ µ = 2 δ µν for µ, ν = 0, 1, 2, 3. Consequences: ψ(x) C 12 C 3 4, 3 colors, 4 spins γ 5 = γ 0 γ 1 γ 2 γ 3 satisfies γ 5 γ µ = γ µ γ 5, µ = 0, 1, 2, 3. A. Frommer, in Lattice QCD 5/24

7 Connection to Michael Günther s talk QCD and Lattice QCD Dirac and Dirac-Wilson The Dirac-Wilson matrix D W Idea: Approximate via covariate centralized finite differences and regularize with 2nd order term. ψ = ψ(x) with x L discrete quark field U µ (x) SU(3) (unitary, det(u µ (x)) = 1) (D W ψ)(x) = m ψ(x) 1 a 2a 1 2a 3 ((I 4 γ µ ) U µ (x)) ψ(x + ˆµ) µ=0 3 ( (I4 + γ µ ) Uµ H (x ˆµ) ) ψ(x ˆµ), µ=0 m 0 sets the quark mass A. Frommer, in Lattice QCD 6/24

8 Connection to Michael Günther s talk QCD and Lattice QCD Dirac and Dirac-Wilson Properties of the Dirac-Wilson matrix D W 4 D W is non-normal spectral gaps m 0 is set s.t. spectrum approaches the imaginary axis from the right γ 5 D W is Hermitian (λ, x) right eigenpair (λ, γ 5 x) left eigenpair L of size 64 4 : D W C 192M 192M A. Frommer, in Lattice QCD 7/24

9 in a Nutshell Challenges Domain decomposition and aggregation History Smooth The V cycle Finest Grid Prolongation Restriction Fewer Dofs First Coarse Grid A. Frommer, in Lattice QCD 8/24

10 Challenges for Wilson-Dirac I The Wilson-Dirac operator is special: Challenges Domain decomposition and aggregation History We only know the gauge links U µ (x) for a given lattice These are random variables, not just perturbed entries D W has a non-trivial symmetry: γ 5 D W = D W γ 5 Eigenvectors to small eigenvalues are not geometrically smooth A. Frommer, in Lattice QCD 9/24

11 Challenges for Wilson-Dirac II Challenges Domain decomposition and aggregation History Consequences: Smoothing (= reducing large eigenmodes) is easy (Jacobi, GMRES,... ) Coarse grid system DW c must be constructed algebraically Using a Galerkin ansatz: DW c = P D W P with a prolongation operator P, we aim at preserving γ5 symmetry preserving locality range(p ) should approximate eigenvectors to small eigenvalues There is hope: Local coherence of small eigenmodes [Lüscher 2008] A. Frommer, in Lattice QCD 10/24

12 Challenges Domain decomposition and aggregation History Two-grid error propagator for ν steps of pre-smoothing E (ν) 2g = (I P Dc 1 P D) (I MD) ν, D }{{}}{{} c := P DP }{{} coarse grid correction smoother coarse operator low accuracy for Dc 1 and M is sufficient introduce recursive construction for D c multigrid To Do: Define interpolation P and smoother M DD-αAMG M: Schwarz Alternating Procedure (SAP) P : Aggregation Based Interpolation A. Frommer, in Lattice QCD 11/24

13 Challenges Domain decomposition and aggregation History Preview: The multigrid principle for Dirac-Wilson Smoother: I MD Effective on large eigenvectors small eigenvectors remain SAP (1 MD) vi / vi number i of eigenvalue λ i Dv i = λ i v i with λ 1... λ 3072 A. Frommer, in Lattice QCD 12/24

14 Challenges Domain decomposition and aggregation History Preview: The multigrid principle for Dirac-Wilson Coarse-grid correction: I P Dc 1 P D small eigenvectors built into interpolation P Effective on small eigenvectors SAP Aggregation + Low Modes (1 MD) vi / vi (1 P D c 1 P D) vi / vi number i of eigenvalue λ i number i of eigenvalue λ i Dv i = λ i v i with λ 1... λ 3072 A. Frommer, in Lattice QCD 12/24

15 Challenges Domain decomposition and aggregation History Preview: The multigrid principle for Dirac-Wilson Two-grid method: E 2g = (I P D 1 c P (I MD) ν D) Complementarity of smoother and coarse-grid correction Effective on all eigenvectors! DD-αAMG 1 E2g vi / vi number i of eigenvalue λ i Dv i = λ i v i with λ 1... λ 3072 A. Frommer, in Lattice QCD 12/24

16 Challenges Domain decomposition and aggregation History SAP: Schwarz alternating procedure Two color decomposition of L L 1 L 4 L 2 L 3 canonical injections I Li : L i L block restrictions D Li = I DI L i Li block inverses B Li = I Li D 1 I L i L i 1: in: ψ, η, ν out: ψ 2: for k = 1 to ν do 3: r η Dψ 4: for all green L i do 5: ψ ψ + B Li r 6: end for 7: r η Dψ 8: for all white L i do 9: ψ ψ + B Li r 10: end for 11: end for A. Frommer, in Lattice QCD 13/24

17 Aggregation based interpolation Construction: Challenges Domain decomposition and aggregation History Define aggregates: domain decomposition A 1,..., A s A 1 A 3 A 2 A 4 P P Calculate test vectors w 1,..., w N Decompose test vectors over aggregates A 1,..., A s A 1 A 1 (v (1),..., v (k) ) = = A2 P = A 2 A s A s A. Frommer, in Lattice QCD 14/24

18 Setup: how to obtain test vectors Challenges Domain decomposition and aggregation History Bootstrapping process start with random test vectors {v 1,..., v n } smooth all v i η times set all v i = ψ i / ψ i build P and D c apply method itself to all systems Dψ i = v i no satisfied? stop yes A. Frommer, in Lattice QCD 15/24

19 Some history The Dirac operator Challenges Domain decomposition and aggregation History Adaptive algebraic multigrid αamg: Brezina, Falgout, Manteuffel, MacLachlan, McCormick, Ruge 2004 Inexact deflation method: Lüscher Solves D(I P Dc 1 P D)ψ = η using SAP as a preconditioner. αamg for lattice QCD: Babich, Brannick, Brower, Clark, Manteuffel, McCormick, Osborn, Rebbi DD-αAMG: F., Kahl, Leder, Krieg, Rottmann : Alexandrou, Bacchio, Finkenrath, F., Kahl, Rottmann: extension to twisted mass Dirac operator currently: further improvemet for setup A. Frommer, in Lattice QCD 16/24

20 Parameters The Dirac operator Parameters and configurations Snapshots on performance parameter default setup number of iterations n inv 6 number of test vectors N 20 size of lattice-blocks for aggregates on level size of lattice-blocks for aggregates on level l > coarse system relative residual tolerance (stopping criterion for the coarse system) ( ) ɛ solver restart length of FGMRES n kv 10 relative residual tolerance (stopping criterion) tol number of post-smoothing steps ( ) ν 5 size of lattice-blocks in SAP ( ) 2 4 number of Minimal Residual (MR) iterations to solve the local systems in SAP ( ) 3 K-cycle maximal length ( ) 5 maximal restarts ( ) 2 relative residual tolerance (stopping criterion) ( ) 10 1 ( ) : same in solver and setup A. Frommer, in Lattice QCD 17/24

21 Configurations The Dirac operator Parameters and configurations Snapshots on performance id lattice size pion mass CGNR shift clover provided by N t Ns 3 m π [MeV] iterations m 0 term c sw , BMW-c , BMW-c , BMW-c , BMW-c , BMW-c , CLS , CLS Table: Ensembles used A. Frommer, in Lattice QCD 18/24

22 Parameters and configurations Snapshots on performance Snapshots on performance: setup time vs solve time number of average average lowest highest average average setup setup iteration iteration iteration solver total steps n inv timing count count count timing timing Table: Evaluation of DD-αAMG-setup(n inv, 2), 48 4 lattice, configuration id 4), 2,592 cores, averaged over 20 runs. A. Frommer, in Lattice QCD 19/24

23 Conf. 1 slight speed up by using 3rd level A. Frommer, Conf. in Lattice 2, 3, QCD4 significant speed up by using 3rd level 20/24 The Dirac operator Parameters and configurations Snapshots on performance Comparison of 2, 3 & 4 Level DD-αAMG configuration lattice size pion mass m π 135 MeV 135 MeV 270 MeV 190 MeV two levels setup time 316s 736s 630s 701s solve time 48.6s 130s 113s 141s three levels setup time 374s 744s 719s 948s solve time 42.6s 75.2s 74.0s 79.0s four levels setup time 806s 755s 1,004s solve time 79.8s 75.7s 79.1s processes local lattice level level level level

24 Parameters and configurations Snapshots on performance Snapshots on performance: oe-bicgstab vs DD-αAMG BiCGStab DD-αAMG speed-up factor coarse grid setup time 22.9s solve iter 13, ,716 ( ) solve time 91.2s 3.15s s total time 91.2s 26.1s 3.50 Table: BiCGStab vs. DD-αAMG with default parameters, configuration id 5, 8,192 cores, ( ) : coarse grid iterations summed up over all iterations on the fine grid. A. Frommer, in Lattice QCD 21/24

25 Parameters and configurations Snapshots on performance Snapshots on performance: mass scaling and levels time to solution (in seconds) m crit m u m ud m d 0.05 two-level DD-αAMG three-level DD-αAMG four-level DD-αAMG mp oe BiCGStab m 0 Figure: Mass scaling of 2, 3 and 4 level DD-αAMG, 64 4 lattice, configuration id 5, restart length n kv = 10, 128 cores A. Frommer, in Lattice QCD 22/24

26 Parameters and configurations Snapshots on performance Snapshots on performance: mass scaling and levels time to solution (in seconds) m ud inexact deflation inexact deflation with inexact projection two-level DD-αAMG three-level DD-αAMG 30 m crit m u m d m 0 Figure: Mass scaling of 2, 3 and 4 level DD-αAMG, 64 4 lattice, configuration id 5, restart length n kv = 10, 128 cores A. Frommer, in Lattice QCD 23/24

27 Conclusions The Dirac operator Parameters and configurations Snapshots on performance Stochastic entries require algebraic multigrid Adaptive setup is mandatory Setup is relatively costly Aggregation based coarsening justified because of local coherence Other adaptive setups are possible: bootstrap AMG [Brandt, Brannick, Kahl, Livshits 2011] Underlying assumption: a small set of small eigenmodes locally represents all small eigenmodes, at least approximately A. Frommer, in Lattice QCD 24/24