GPU Accelerated Reweighting Calculations for Neutrino Oscillation Analyses with the T2K Experiment

Transcription

1 GPU Accelerated Reweighting Calculations for Neutrino Oscillation Analyses with the T2K Experiment Oxford Many Core Seminar - 19th February Richard Calland rcalland@hep.ph.liv.ac.uk

2 Outline of Talk Neutrino Physics The T2K experiment Neutrino Oscillation analysis Statistical & Computing techniques How GPUs can help Feasibility of analysis Improve sensitivity of results Financially Conclusions 2

3 Neutrino Physics A branch of High Energy Physics (CERN etc) What are neutrinos? Why are they interesting? How do we study them? 3

4 What Neutrinos are the lightest particles we know of, but they are some of the most abundant They have tiny cross-sections Flooded by neutrinos from the sun, supernova explosions, the earths core and even produced in the upper atmosphere 100 billion solar neutrinos passing through your thumbnail every second! They pass straight through the earth! They have no charge (like a neutron) and only interact via the weak interaction (responsible for nuclear decays) There are 3 flavours (types), electron neutrino, muon neutrino and tau neutrino Analogues to the electron, muon and tau particles and are deeply related 4

5 5

6 Why Until recently, neutrinos were thought to have no mass (like light particles i.e. photons) The standard model of particle physics predicts massless neutrinos, and cannot explain the neutrino masses Don't let those CERN guys fool you into thinking the Higgs boson was the last piece of the puzzle Neutrinos are the ONLY particles we know that show signs of physics beyond the standard model 6

7 Neutrino Oscillations The reason we know neutrinos have mass is because we have observed a phenomenon called neutrino oscillation Simply put, neutrinos of one flavour (e,μ,τ) can transform into another Breaking a conservation law! Hint at physics beyond the standard model? This is explained in a quantum mechanical way 7

8 Neutrino Oscillations Neutrinos have 3 flavours (e,μ,τ) and 3 different masses m1,2,3 But they are more complicated than that Actually exist as a quantum superposition of all 3 Long story short, a neutrino created (e.g. in a radioactive decay inside the sun) as an electron neutrino could be observed later on as a muon neutrino 8

9 After ~400 km, non-zero probability for the electron neutrinos to be detected as muon neutrinos A macroscopic manifestation of a purely quantum mechanical phenomenon! 9

10 Oscillation Probability Accelerator Nuclear Reactor Solar 10

11 Neutrino Oscillations in Matter The interactions with ambient electrons in matter cause the neutrinos to feel an extra potential These so-called matter effects must be modelled Addition of an extra potential term to calculations creates a different oscillation probability compared to vacuum 2 neutrino approximation 11

12 Arbitrary state vector in flavour space Initial state transition amplitude Time evolution of the state Has 3 independent solutions for row vector ψj, assemble into matrix X Probability = A 2 12 Barger et al. Phys. Rev. D22(1980) 2718

13 How to study neutrino oscillations 13

14 The Tokai-to-Kamioka Experiment Measure the beam here Fire a beam of neutrinos from here 14

15 Over 500 members 11 Countries UK institutions one of the largest contributors (including Oxford) 15

16 How to measure neutrino oscillations At T2K we want to measure θ23 and θ13. We have sensitivity to both through νμ and νe samples at the far detector (SuperK) νμ = 20% νμ = 99% νe = 10% νe = 1% Can't see ντ = 0% νμ beam ντ = 70% 295 km Electron-like sample Electron neutrinos interact to produce an electron. Observe an excess of electrons compared to prediction muon-like sample Muon neutrinos interact to produce a muon. Observe a deficit of muons compared to prediction 16

17 17

18 Cerenkov Radiation 18

19 19

20 March 2011 Earthquake 20

21 Neutrino Oscillation Analysis 21

22 Basic Idea Simulate the neutrino beam, and how it looks in at Super-K, i.e. what kind of energy spectra it produces Build a probability distribution function (PDF) empirically from the detector Monte Carlo We then change the energy spectra to represent changing oscillation parameters Wobble these oscillation parameters (along with ~200 systematic uncertainty parameters) until a good fit has been found I.e. construct a likelihood function and minimize 22

23 Changing The MC to Simulate Neutrino Oscillation Apply a weight to each event which is equal to the probability of neutrino oscillation + matter effects + matter effects 23

24 Reweight method Samples of Monte Carlo events used to create probability distribution function Construct histogram, lose information but gain speed Treat every event separately, retain all information, lose speed Of course, I chose the latter 24

25 Binned Reweighting Fill once Reweight each bin using bin center 1,000,000 Monte Carlo Events Pro: Only have to do <Nbin> calculations per iteration No longer have to store all Monte Carlo in memory Con: All information for each event is lost 25

26 Event-by-event Reweighting 1,000,000 Monte Carlo Events For each event, calculate weight, fill histogram Pro: Retain all information about individual events Calculated weight for each event is more accurate Con: Have to store all Monte Carlo in memory Have to do 1,000,000 calculations per iteration 26

27 Flaws with binned reweighting Large binning to ensure adequate statistics per bin. If using binned reweighting, an event lying on the edge of a wide bin would not be represented fairly by the bin center value. We find differences ~1% 27

28 Flaws with binned reweighting (2) Monte Carlo events are broken down into sub categories If using the binned method, all the information about events is lost Can't model the effects of events migrating between different interaction modes / samples properly Proper modelling would give better sensitivity to final result! 28

29 3 Neutrino Oscillations with Matter Effects Luckily, someone already wrote a library to do this Unfortunately, rather slow ~ 2-3 seconds to reweight all 1,000,000 MC entries Need to do this ~20 million times for one analysis 2,000,000,000,000 calculations needed! (2 x 1012) ~2 years on a single machine with no GPU! Ported some functions to GPU using CUDA Vacuum calculations no speedup Full Matter Effects (constant density) speedup! 29

30 Benchmark Code compared with original Original test yielded x300, turns out wasn't compiled with -O2... Code also compared with quick OpenMP implementation In general, CUDA sees ~75-130x OpenMP sees ~2-4x speed improvement Double Precision Machine dependent 30

31 Benchmark Code Snippet clock.start(); CPU for (int i = 0; i < N; ++i) { bnu >SetMNS( nominal[0], nominal[2], nominal[1], nominal[3], nominal[4], nominal[5], 100.0, true ); bnu >propagatelinear( 2, 295, 2.6 ); sample_weights[i] = bnu >GetProb(2, 2); } clock.stop(); clock.start(); OpenMP #pragma omp parallel for num_threads(4) for (int i = 0; i < N; ++i) { bnu >SetMNS( nominal[0], nominal[2], nominal[1], nominal[3], nominal[4], nominal[5], 100.0, true ); bnu >propagatelinear( 2, 295, 2.6 ); sample_weights[i] = bnu >GetProb(2, 2); } clock.stop(); clock.start(); CUDA setmns(nominal[0], nominal[2], nominal[1], nominal[3], nominal[4], nominal[5], true); GetProb(2, 2, 295, 2.6, energy, N, sample_weights); clock.stop(); 31

32 extern "C" host void GetProb(int Alpha, int Beta, double Path, double Density, double *Energy, int n, double *oscw) { size_t dmsize = 3*3*sizeof(double); typedef double dmarray[3]; dmarray *d = (dmarray*)malloc(dmsize); memcpy(d, &dm, dmsize); dmarray *dm_device; Copy mixing matrix and mass cudamalloc((void **) &dm_device, dmsize); matrix (matter effects) to cudamemcpy(dm_device, dm, dmsize, cudamemcpyhosttodevice); size_t mixsize = 3*3*2*sizeof(double); typedef double mixarray[3][2]; mixarray *m = (mixarray*)malloc(mixsize); memcpy(m, &mix, mixsize); mixarray *mix_device; cudamalloc((void **) &mix_device,mixsize); cudamemcpy(mix_device, mix, mixsize, cudamemcpyhosttodevice); size_t size = n * sizeof(double); double *energy_device = NULL; cudamalloc((void **) &energy_device, size); cudamemcpy(energy_device, Energy, size, cudamemcpyhosttodevice); device Could copy to constant / texture memory Copy Monte Carlo event energies to device double *osc_weights; cudamalloc((void **) &osc_weights, size); dim3 block_size; block_size.x = 512; dim3 grid_size; grid_size.x = (n / block_size.x) + 1; Execute kernel propagatelinear<<<grid_size, block_size>>>(alpha, Beta, Path, Density, mix_device, dm_device, energy_device, osc_weights,n); } cudamemcpy(oscw, osc_weights, size, cudamemcpydevicetohost); clean_up(); // cudafree everything Copy oscillation weights32 back to host

33 Benchmark Strange features Xeon 2.67 GHz All threads occupied here Binned coarse reweighting around here Using all information with event33 by-event reweighting

34 Speedup Factor Xeon 2.67 GHz 34

35 Validation 10 million random comparisons between CPU and GPU calculations Agreement to precision, more than good enough for this application Difference attributed to extended ALU of CPU and different hardware implementations of non-associative calculations 35

36 Publication In the process of submission to Journal of Instrumentation (JINST) Preprint:

37 Non-linear Parameter Response For many parameters, the way they change is not gaussian Our models are too computing intensive to add into the analysis We create splines for each variation and use that instead Interaction mode C Interaction mode B Interaction mode A Reconstructed energy True energy 37

38 Splines A spline is a series of polynomials connected by points that have the same differential at the node Ensures a smooth changing function Want to sample each spline at various values 38

39 However As shown 2 pages back, there are a lot of splines (~4,000,000) A large chunk of memory (~Gbs) The next significant bottleneck of our code Off-loading work onto the GPU As we know, GPUs are compute- rather than data-intensive 39

40 Reformatting HEP experiments use a C++ library named ROOT Our splines are formatted as TSpline3 cubic spline objects Lots of bloat: 4M instantiations of a C++ class Try to reformat to perform better on GPU Instead of a class, format as an array and use a kernel function to evaluate 40

41 Convert TSpline3 Into an array 41

42 The Spline Monolith Convert TSpline3 objects into a monolithic array Eval(x) becomes Eval(i,x) i is the spline ID Already 2x faster on CPU! Smaller, takes up about 1.2 Gb RAM 42

43 GPU Implementation Copy all 1.2 Gb single array onto GPU RAM at initialization, keep it there Every iteration, evaluate all splines with a CUDA kernel and push the weight from each spline back onto CPU RAM Crude GPU implementation yields ~20x speed-up for the evaluation of 4,000,000 splines Issues with branching that are unavoidable Not using any shared memory which could be useful Would like to revisit this, but for now it helps 43

44 Bringing Everything Together Since neutrinos interact very rarely, neutrino experimental datasets are relatively small (~100 events) Need to make the most of what we have Event-by-event reweighting takes care of this Need a way to treat systematic uncertainties properly Must be able to handle large number of parameters Must be able to scale across a cluster 44

45 Markov Chain Monte Carlo To evaluate the posterior distribution, need to integrate over high-dimensions MCMC provides an efficient way to perform the ~200-dimensional integral MCMC performs a semi-random walk through parameter space, following the path of the likelihood function Can run multiple chains on a cluster and combine output 45

46 Method 1 Throw 200 proposed new values of parameters Evaluate likelihood function 2 Complete cycle ~20 million times 3 Is new state better than current? Accept or reject performed on GPU Write to disk Time per step ~0.3 seconds ~ 20x speed-up 46

47 Emerald Cluster MC samples, Splines etc MCMC chain MCMC chain job_script.py MCMC chain Combine output MCMC chain Data file (real or fake) MCMC chain MCMC chain Chains are considered independent samples and can simply be added together. 47

48 Final Result Best fit points and errors extracted from the posterior density of the combined markov chains A lot of information can be extracted from this high dimensional distribution (covariance, correlations, model comparisons etc) 48

49 Financial & Environmental Benefit Like to compare total dissipated power (TDP) and total costs for a full analysis M2090 = 225 W X5850 = 95 W kwh = 0.15 Assuming running at max TDP CPU version 95 W * 100 jobs * (11*20) hours = MW = 9.5 kwh GPU version 225 W + 95 W * 100 jobs * 11 hours = MW = 32 kwh ~ 6x less power / less expensive 49

50 Conclusions Wanted to write an analysis framework that makes full use of the information in our detector Monte Carlo Off-loaded bottlenecking algorithms to GPU to make this a possibility Combination of neutrino oscillation weight calculations and spline evaluations done on GPU ~20x faster than CPU only Full analysis takes 12 hours on Emerald Would take 10 days on an equivalent CPU cluster! Needed to produce 100 toy experiments for validation, again, without Emerald this would have taken 3 years on a CPU cluster! ~2 months with Emerald 50

51 The Future Currently writing my thesis High energy physics taking GPUs very seriously to solve big data problems they will face in the future CERN are investing in computing, they cannot handle data from the LHC upgrade presently! This analysis only off-loads a few components, with a rewrite, could make much better use of GPUs for reweighting Using shared memory CUDA + MPI 51

52 Thanks for listening! 52