Hydra. A library for data analysis in massively parallel platforms. A. Augusto Alves Jr and Michael D. Sokoloff

Transcription

1 Hydra A library for data analysis in massively parallel platforms A. Augusto Alves Jr and Michael D. Sokoloff University of Cincinnati aalvesju@cern.ch Presented at NVIDIA s GPU Technology Conference, May 8-11, Silicon Valley, US A. Augusto Alves Jr. Hydra May 7, / 23

2 Outline Design and goals of Hydra Basic functionalities and main algorithms Performance Multidimensional numerical integration Phase-space Monte Carlo generation Interface to ROOT::Minuit2 and fitting Summary A. Augusto Alves Jr. Hydra May 7, / 23

3 Motivation The Large Hadron Collider (LHC) and other facilities acquire 10 s petabytes of data anually. The collective effort to analyze this amount data requires state-of-the-art software tools that: Scale efficiently to face the increasing statistics from the experiments. Meet the high precision requirements typically necessary to address High Energy Physics (HEP) problems. Are efficient and flexible enough to face the different conditions of specific HEP experiments. Are portable, scalable, compatible with existing software and hardware standards. A. Augusto Alves Jr. Hydra May 7, / 23

4 Hydra Hydra is a header only templated C++ library designed to perform common HEP data analyses on massively parallel platforms. It is implemented on top of the C++11 Standard Library and a variadic version of the Thrust library. Hydra is designed to run on Linux systems and to use OpenMP, CUDA and TBB enabled devices. It is focused on portability, usability, performance and precision. A. Augusto Alves Jr. Hydra May 7, / 23

5 Design and features The main design features are: The library is structured using static polymorphism. There is absolutely no need to write explicit back-end oriented code. Clean and concise semantics. Interfaces are easy to use correctly and hard to use incorrectly. The same source files written using Hydra and standard C++ compile for GPU or CPU, just exchanging the extension from.cu to.cpp and one or two compiler flags. A. Augusto Alves Jr. Hydra May 7, / 23

6 Features Generation of Phase-space Monte Carlo samples. Sampling of multidimensional probability density functions. Data fitting using binned and unbinned multidimensional datasets. Evaluation of multidimensional functions over heterogeneous data sets. Numerical integration of multidimensional functions. A. Augusto Alves Jr. Hydra May 7, / 23

7 Functors Hydra adds features and type information to generic functors using the CRTP idiom. A generic functor with N parameters is represented like this: s t r u c t MyFunctor : p u b l i c hydra : : BaseFunctor <MyFunctor, double, N> { // MyFunctor c o n s t r u c t o r and o t h e r i m p l e m e n t a t i o n d e t a i l s... // User a l w a y s need to implement t h e E v a l u a t e ( ) method template<typename T> host device i n l i n e double E v a l u a t e (T x ) { // a c t u a l c a l c u l a t i o n } } ; All functors deriving from hydra::basefunctor<func,returntype,npars> can be cached, used to perform fits and to compose more complex mathematical expressions. A. Augusto Alves Jr. Hydra May 7, / 23

8 Arithmetic operations and composition with functors All the basic arithmetic operators are overloaded. Composition is also possible. If A, B and C are Hydra functors, the code below is completely legal.... // b a s i c a r i t h m e t i c o p e r a t i o n s auto A_plus_B = A + B ; auto A_minus_B = A B ; auto A_times_B = A B ; auto A_per_B = A/B ; // any c o m p o s i t i o n o f b a s i c o p e r a t i o n s auto any_functor = (A B) (A + B) (A/C ) ; // C(A,B) i s r e p r e s e n t e d by : auto compose_ functor = hydra : : compose (C, A, B)... The functors resulting from arithmetic operations and composition can be cached as well. No intrinsic limit on the number of functors participating on arithmetic or composition mathematical expressions. A. Augusto Alves Jr. Hydra May 7, / 23

9 Support for C++11 lambdas Lambda functions are fully supported in Hydra. The user can define a C++11 lambda function and convert it into a Hydra functor using hydra::wrap_lambda():... double two = 2. 0 ; // d e f i n e a s i m p l e lambda and c a p t u r e "two" auto my_lambda = [ ] host device ( double x ) { r e t u r n two s i n ( x [ 0 ] ) ; } ; // c o n v e r t i s i n t o a Hydra f u n c t o r auto my_lamba_wrapped = hydra : : wrap_lambda ( my_lambda ) ;... CUDA 8.0 supports lambda functions in device and host code. A. Augusto Alves Jr. Hydra May 7, / 23

10 Data containers hydra::point represents multidimensional data points including its coordinates, value and errors. hydra::pointvector Looks like an array of structs, but data is stored in structure of arrays. // two d i m e n s i o n a l p o i n t typedef hydra : : Point <GReal_t, 2> point_ t ; // two d i m e n s i o n a l data s e t on t h e d e v i c e hydra : : P o i n t V e c t o r <point_t, d e v i c e > data_d (1 e6 ) ;... // g e t data from d e v i c e hydra : : P o i n t V e c t o r <point_t, host > data_h ( data_d ) ; // f i l l a ROOT 2D h i s t o g r a m TH2D h i s t ( " h i s t ", "my h i s t o g r a m ", 100, min, max ) ; f o r ( auto row : data_h ){ auto p o i n t ( row ) ; h i s t. F i l l ( p o i n t. G e t C o o r d i n a t e ( 0 ), p o i n t. G e t C o o r d i n a t e ( 1 ) ) ; } A. Augusto Alves Jr. Hydra May 7, / 23

11 Functionalities Data fitting and Monte Carlo generation Interface to ROOT::Minuit2 minimization package. Phase-space generator. Multidimensional p.d.f. sampling. Parallel function evaluation over multidimensional datasets Numerical integration Flat Monte Carlo sampling. Vegas-like self-adaptive importance sampling (Monte Carlo). Gauss-Kronrod one-dimensional quadrature. Genz-Malik multidimesional quadrature. A. Augusto Alves Jr. Hydra May 7, / 23

12 Vegas-like multidimensional numerical integration The VEGAS algorithm is based on importance sampling. It samples the integrand and adapts itself, so that the points are concentrated in the regions that make the largest contribution to the integral. Hydra implementation follows the corresponding GSL algorithm. No limit in the number of dimensions. // V e g a s S t a t e h o l d r e s o u r c e s and c o n f i g u r a t i o n s VegasState <N, d e v i c e > State_d ( _min, _max ) ; State_d. S e t I t e r a t i o n s ( i t e r a t i o n s ) ; State_d. SetMaxError ( max_error ) ; State_d. S e t C a l l s ( c a l l s ) ; State_d. S e t T r a i n i n g C a l l s ( t c a l l s ) ; State_d. S e t T r a i n i n g I t e r a t i o n s ( 1 ) ; // Vegas i n t e g r a t o r o b j e c t Vegas<N, d e v i c e > Vegas_d ( State_d ) ; // i n t e g r a t e a G a u s s i a n Vegas_d. I n t e g r a t e ( G a u s s i a n ) ; A. Augusto Alves Jr. Hydra May 7, / 23

13 Vegas-like multidimensional numerical integration Processing a Gaussian distribution in 10 dimensions. Integral result Duration [ms] Speed-up GPU vs CPU GPU Iteration result Cumulative result Iteration System configuration: GPU model: Tesla K40c CPU: Intel R Xeon(R) CPU E GHz (one thread) GPU CPU speed-up Number of samples A. Augusto Alves Jr. Hydra May 7, /

14 Phase-Space Monte Carlo Describes the kinematics of a particle with a given four-momentum decaying to N-particle final state. No limitation on the number of particles in the final state. Support the generation of sequential decays. Generation of weighted and unweighted samples. // Masses o f t h e p a r t i c l e s hydra : : Vector4R Mother ( mother_mass, 0. 0, 0. 0, 0. 0 ) ; double Daughter_Masses [ 3 ] { daughter1_mass, daughter2_mass, daughter3_mass } ; // C r e a t e PhaseSpace o b j e c t hydra : : PhaseSpace <3> phsp ( Mother_mass, Daughter_Masses ) ; // A l l o c a t e t h e c o n t a i n e r f o r t h e e v e n t s hydra : : Events <3, d e v i c e > e v e n t s ( n d e c a y s ) ; // G e n e r a t e phsp. G e n e r a t e ( Mother, e v e n t s. b e g i n ( ), e v e n t s. end ( ) ) ; A. Augusto Alves Jr. Hydra May 7, / 23

15 Phase-Space Monte Carlo M(J/Ψπ) dalitz Entries 1e Mean x Mean y Std Dev x Std Dev y Duration [ms] Speed-up GPU vs CPU M(Kπ) System configuration: GPU model: Tesla K40c CPU: Intel R Xeon(R) CPU E GHz (one thread) GPU CPU speed-up Number of events A. Augusto Alves Jr. Hydra May 7, / 23 50

16 Interface to Minuit2 ROOT::Minuit2 is widely used in particle physics to find the minimum value of a multi-parameter function (FCN) and analyze the shape of the function around the minimum, and so to compute model s best-fit parameter values and uncertainties. Hydra implements an interface to ROOT::Minuit2 that parallelizes the FCN calculation. This dramatically accelerates the calculation over large datasets. The PDFs are normalized on-the-fly using analytical or numerical integration algorithms provided by Hydra. Data is passed using hydra::pointvector. A. Augusto Alves Jr. Hydra May 7, / 23

17 Interface to Minuit2 Model = N g Gaussian + N e Exponential G a u s s A n a l y t i c I n t e g r a l G a u s s I n t e g r a l ( min, max ) ; E x p A n a l y t i c I n t e g r a l E x p I n t e g r a l ( min, max ) ; auto Gaussian_PDF = hydra : : make_pdf ( Gaussian, G a u s s I n t e g r a l ) ; auto Exponentia_PDF = hydra : : make_pdf ( Exponentia, E x p I n t e g r a l ) ; // add t h e pds to make a e x t e n d e d pdf model s t d : : a r r a y <hydra : : Parameter, 3> y i e l d s { NGaussian, N E x p o n e n t i a l } ; auto Model = hydra : : add_pdfs ( y i e l d s, Gaussian_PDF, Exponentia_PDF ) ; model. SetExtended ( 1 ) ; // g e t t h e FCN auto Model_FCN = hydra : : make_ loglikehood_ fcn ( Model, data_d ) ; // p a s s t h e FCN to M i n u i t 2... A. Augusto Alves Jr. Hydra May 7, / 23

18 Interface to Minuit2 20 million event maximum likelihood unbinned fit. Yield data Entries 2e+07 Mean Std Dev X Timing: Fit on GPU: seconds Fit on CPU: seconds Speed-up: 62x System configuration: GPU model: Tesla K40c CPU: Intel R Xeon(R) CPU E GHz (one thread) A. Augusto Alves Jr. Hydra May 7, / 23

19 Summary Hydra s development has been supported by the National Science Foundation under the grant number PHY The project is hosted on GitHub: The package includes a suite of examples. It is being used at CERN on analyses aiming to measure the Kaon mass using large datasets. Acknowledgments To Karen Tomko and Bradley Hittle from the Ohio Supercomputer Center. To the University of Cincinnati LHCb group. Please, visit the page of the project, try it out, report bugs, make suggestions... Thanks! A. Augusto Alves Jr. Hydra May 7, / 23

20 Backup

21 Phase-Space Monte Carlo OpenMP: scalling with number of threads Duration [ms] System configuration: CPU: Intel R Xeon(R) CPU E GHz x Number of OpenMP threads A. Augusto Alves Jr. Hydra May 7, / 23

22 Phase-Space Monte Carlo CUDA OpenMP, TBB GPU vs OpenMP GPU vs TBB Duration [ms] Speed-up GPU vs CPU Duration [ms] Speed-up GPU vs CPU GPU 4 CPU speed-up Number of events 1 GPU 4 CPU speed-up Number of events A. Augusto Alves Jr. Hydra May 7, / 23

23 Vegas-like multidimensional numerical integration OpenMP: scalling with number of threads System configuration: CPU: Intel R Xeon(R) CPU E GHz x 48 Duration [ms] Number of OpenMP threads A. Augusto Alves Jr. Hydra May 7, / 23