Lattice QCD on Blue Waters

Transcription

1 Lattice QCD on Blue Waters PI: Robert Sugar (UCSB) Presenter: Steven Gottlieb (Indiana) (USQCD) NCSA Blue Waters Symposium for Petascale Science and Beyond Sunriver Resort June 12-15, 2016

2 Collaborators Alexei Bazavov (Iowa Indiana) Nuno Cardoso (NCSA Lisbon) Mike Clark (NVIDIA) Carleton DeTar (Utah) Daping Du (Illinois Syracuse) Robert Edwards, Bálint Joó, David Richards, Frank Winter (Jefferson Lab) Kostas Orginos (William & Mary) Thomas Primer, Doug Toussaint (Arizona) Mathias Wagner (Indiana NVIDIA) 2

3 Key Challenges Calculations of QCD must support large experimental programs in high energy and nuclear physics QCD is a strongly coupled, nonlinear quantum field theory Lattice QCD is a first principles calculational tool that requires large scale computer power Using the highly improved staggered quark (HISQ) action, we study fundamental parameters of the standard model of elementary particle physics quark masses, CKM mixing matrix elements Using Wilson/Clover action, we study masses & decays of excited and exotic states of QCD 3

4 Involvement with experimental program 4 12 GeV science case Second phase of GlueX program with BaBar DIRC-s (approved) JLab CLAS12 expt (approved) Hybrid baryons CLAS12 expt (approved) Searching for the Rules that Govern Hadron Construction J. Dudek, R. Mitchell, M. Shepherd Expt/Theory Review for Nature (in press) USQCD All-Hand s Meeting April 29, 2016

5 ρ resonance at different pion masses BW couplings nearly constant in pion mass (will come back to this later ) m p = 236 MeV m R = 790 ± 2 MeV g = 5.69 ± 0.07 m p = 391 MeV m R = 855 ± 1 MeV g = 5.70 ± USQCD All-Hand s Meeting April 29, 2016 PRD

6 Why Blue Waters? Lattice field theory calculations proceed in two stages: Generate gauge configurations, i.e., snapshots of quantum fields Compute physical observables on the stored configurations First stage is done in a few streams When computing observables on stored configurations, order 1000 jobs may be run in parallel We can use Blue Waters GPUs for some production running in our projects, e.g., Wilson Clover gauge generation runs well on GPUs Decay constant calculations also using GPUs We need large partitions to generate configurations We can run many smaller parallel jobs for 2nd stage 6

7 Why Blue Waters... It is very expensive to use up and down quark masses as light as in Nature, i.e., the physical value This has required using heavier quarks and extrapolating to the physical masses using chiral perturbation theory For the first time, Blue Waters is allowing us to create gauge configurations with small lattice spacing and quarks masses at the physical value This allows us to produce results with unprecedented precision We estimate that Blue Waters accelerates the progress of our nuclear physics calculation by approximately a factor of ten, compared to other available resources 7

8 Topology Awareness Top: configuration generation on grid Bottom: spectrum analysis Lower values are better Black points: grid order tool only; 32 procs/node Blue points: topology aware scheduling; 16 procs/node Red points: topology aware scheduling; 24 procs/node 8

9 JIT Performance Improvement QDP-JIT (F. Winter) improves Chroma performance on GPUs QUDA used for linear solver Gauge generation speed 4 times better using XK GPUs than XE CPUs See Winter, Clark, Edwards & Joó, IPDPS 14 proceedings Trajectory Time (sec) V=40 3 x256 sites, flavors of Anisotropic Clover, m π ~ 230 MeV, τ=0.2, 2:3:3 Nested Omelyan QDP-JIT + QUDA (GCR) CPU + QUDA (GCR) CPU only (XE Nodes) 4X Nodes 9

10 QDP-JIT Developments Initially, produced PTX code for NVIDIA GPU assembler Latest version of QDP++ library for Chroma QDP-JIT produces internal representation (IR) code for LLVM compiler. LLVM can be used to produce code for NVIDIA GPUs, x86 architecture, including Knights Landing (Xeon Phi), PowerPC, including BlueGene QPX. Important part of the performance portability effort for Chroma. 10

11 PAID Program Working with Bill Gropp s I/O IME group Allowed me to hire Alexei Bazavov at Indiana University Two objectives: Increase I/O speed Enhance performance of other parts of code, starting with gauge force Started with analysis of Darshan logs by Huong Luu (NCSA postdoc) Immediate benefit: NCSA Lustre obeys POSIX standard so our parallel write code should and does work. Found a low performance run with striping not set After Luu left, we ve been working with Adams and Karrels 11

12 PAID I/O Test case make available to IME group MILC has a number of I/O options including one file per I/O node fast, but not very convenient for archiving not as portable as single file mostly used for checkpoints Gained a better understanding of blocking required to reduce time spent on metadata looking at MPI-IO asynchronous writing ways to increase block size and alignment 12

13 PAID Gauge Force Operations in QCD mainly involve 3 3 complex matrices or 3- component complex vectors Matrix-matrix multiplies have a higher arithmetic intensity than matrix-vector multiplies M M: 198 flops; 144 bytes input, 72 bytes output M V: 66 flops; 96 bytes input, 24 bytes output Almost a decade ago I started wondering why gauge force routine (matrix-matrix multiplies) is slower than quark propagator solver (matrix-vector multiplies) Combination of cache unfriendliness and frequent nonoverlapping communication of gauge force routine S. Basak recoded gauge force outside of MILC style. 13

14 Gauge Force New code first fetches all matrices that will be needed in the entire computation, rather than each neighbor as needed. MILC style would loop over all the sites of grid for each operation. New code makes the loop over sites the outer loop and loops over all paths to neighbor sites in the inner loop. However, the code was not put into production. A. Bazavov is integrating with latest version of MILC and looking to generalize to other routines such as fermion force. 14

15 Gauge Force Performance grid strong scaling test note reverse order single precision speed per rank blue (super) is new code red is normal MILC code green is a different optimization that is part of QOPQDP by J. Osborn. Uses more memory, but fewer flops Mflops/s/rank MILC v.7.8 MD MILC v.7.8+scidac MD MILC v.7.8 MD super log 2 (local volume) 15

16 Shared Data Configurations are made available through the International Lattice Data Grid. Other groups use these configurations for may additional physics projects. Fermilab Lattice/MILC will be using them for several years to investigate a variety of weak decays of heavy-light mesons In the past, a number of other groups have also used MILC configurations for a wide variety of projects Some of the quark propagators are saved for other physics projects. Clover propagators stored at JLab for study of meson decays, baryon spectrum and exotic spectrum. 16

17 Why It Matters The standard model of elementary particle physics contains three of the four known forces: strong, weak and electromagnetic gravity is not included Standard model explains a wealth of experimental data However, there are many parameters that can only be determined with experimental input There are theoretical reasons that argue for the fact that the standard model is incomplete Many of the most interesting aspects of the strong force require better calculations of a strongly coupled theory 17

18 Calculating QCD We need lattice QCD to carry out first principles calculations of many effects of the strong force This requires large scale numerical calculation A central goal of nuclear physics is to predict new bound states of quarks, properties of glueballs and exotic states that are not predicted by quark model The CKM matrix describes how quarks mix under weak interactions Kobayashi and Maskawa received the 2008 Nobel Prize our calculations are necessary to determine elements of matrix If different decays give different results for the same matrix element, that requires new physical interactions (prize worthy!) 18

19 Kobayashi & Maskawa Won 2008 Nobel prize for realization that with three (or more) generations can have CP violation, which might explain baryon asymmetry of Universe. KEK photo from nobelprize.org 19

20 CKM Matrix Some relevant processes listed under each element 20

21 First Row: Light Quarks Processes involving only light quarks test first row unitarity leptonic semileptonic 21

22 Test of First Row Unitarity Magenta diagonal band from fk/fπ (this work) Yellow vertical band from nuclear β decay. Black diagonal is unitary condition Hatched yellow band from semileptonic decay also on Blue Waters (El Khadra) Some tension in latter result V us V ud 2 22

23 High Precision Required Without high precision calculations of QCD, we cannot accurately determine CKM matrix elements from expensive (many hundreds of megadollars), high precision experiments New interactions outside the standard model are expected to be weak, so their effects are small Understanding QCD is important for a deeper understanding of the fundamental laws of physics Precision Higgs boson studies at Large Hadron Collider require higher precision values for quark masses and strong coupling constant Muon g-2 theory error dominated by QCD effects 23

24 Lattice QCD for Nuclear Physics Over $300 million has been spent to upgrade JLab to look for new QCD bound states Focus of GlueX experiment at Hall D and CLAS12 at Hall B We want predictions prior to the experiment to maximize impact and synergy Lattice QCD input is needed to meet several key Nuclear Science Advisory Committee milestones Results are relevant to other experiments such as COMPASS (CERN), BES III (Beijing),... 24

25 Accomplishments Blue Waters has allowed us to produce the most realistic gauge configurations to date These are the most challenging calculations we have ever undertaken ( , physical light quarks, a=0.042 fm; , ml/ms=0.2, a=0.042 fm; , ml/ms=0.2, a=0.03 fm) HISQ configurations have allowed us to make the most precise calculations of a number of meson decays 2 Physical Review Letters (PRL), 2 Physical Review D (PRD) One PRL was designated an Editors Suggestion The Clover quark propagators produced on Blue Waters play a major role in the spectrum calculations described before configurations completed, > PRL, 2 PRD, Physics Letters 11 conference proceedings 25

26 Conclusions Blue Waters has accelerated our scientific achievements by a large factor We have generated gauge configurations that will be useful to the broad USQCD physics program and are also shared internationally We have also carried out important physics analyses directly on Blue Waters Many additional quantities are studied with the Blue Waters configurations at other supercomputer centers and on USQCD computers However, much more work remains to provide the theoretical input required to interpret a large number of experiments Grateful that PI Mackenzie heads a new PRAC for USQCD high energy physics; hoping for nuclear physics 26