The GAP Project: GPU for Online Processing in Low-Level Trigger

Transcription

1 Th GAP Projct: GPU for Onlin Procssing in Low-Lvl Triggr Massimiliano Fiorini! (Univrsità di Frrara and INFN Frrara)! On bhalf of th GAP Collaboration! GPU in High Enrgy Physics Pisa, Sptmbr 2014

2 GAP Projct GAP (GPU Application Projct) for ral-tim in HEP and mdical imaging is a 3 yars projct fundd by th Italian Ministry of rsarch, startd in 2013! Collaboration btwn INFN Szion di Pisa, Univrsity of Frrara and Univrsity of Roma La Sapinza! Dmonstrat th fasibility of using off-th-shlf computr commoditis to acclrat ral-tim scintific computations! Application in diffrnt filds:! q q High Enrgy Physics (low and high lvl triggrs)! Mdical Imaging (NMR, CT and PET)! 2

3 GAP Projct GAP (GPU Application Projct) for ral-tim in HEP s! n o i t a lic p p A and mdical imaging is a 3 PU G g! : t n c i g j a o Pr Im l P yars projct fundd by th a c A i G d h M T d n a r auc of rsarch, Italian: Ministry g B g. i r M T l! lk v a m T L a h b g startdfoin 2013! i n rh o C t s Fa! o c U i P n G Collaboration btwn INFN Szion gdi Pisa, Univrsity of om n i D s i u D n tio La Sapinza! c : G. dt u r r a Frrara and of Roma t s u TalkUnivrsity n q r o a c M g r CT r c nb i r t v Dmonstrat th fasibility of using off-th-shlf computr L m ra d a z i p l d l l t a arscintific ma p o commoditis to acclrat computations! t u U a P G m i o t! b l a m g o r n l i s a g a Application in diffrnt.p ard filds:! m i M w : o R t k l M g Ta N in t n t o fi i l s u q m High od Enrgy Physics diff (low and high lvl triggrs)! q Mdical Imaging (NMR, CT and PET)! 3

4 Physics cas: NA62 K+à "+## dcay (BR~ )! Hug background from kaon dcays! Vacuum Tank! SAV! Small Angl γ Vto! LAV:! Larg Angl Photon Vto! CHANTI! Targt! CEDAR! Chargd! Particl! Vto! Gigatrackr! RICH! Bam Pip! 750 MHz! Dcay Rgion: 65 m! bam rat! (~6% kaons, 75 GV/c)! Total Lngth: 270 m! CHOD! Chargd! Hodoscop! Straw! Trackr! LKr!MUV! 10 MHz rat! from dcays! 4

5 Triggr and DAQ 10 MHz RICH MUV CEDAR STRAWS LKR LAV 1 MHz 10 MHz L0 L0TP 1 MHz 100 khz GigaEth SWITCH L1/L2 PC L1/L2 PC Triggr primitivs L1/L2 PC L1/L2 PC O(KHz) CDR Data L1/L2 PC EB L0 triggr L1 triggr L1/L2 PC L1/2 L1/L2 PC L0: Hardwar synchronous lvl! q 10 MHz to 1 MHz, 1 ms max. latncy! q Primitivs (MUV, RICH, LAV, LKR)! L1: Softwar lvl! q Singl dtctor, 1 MHz to 100 khz! L2: Softwar lvl! q Complt information, 100 khz to 10 khz! 5

6 Th RICH dtctor 17 m focal lngth, ~4 m in diamtr, filld with N at 1 atm! Pion/muon sparation in th rang GV/c! Tim rsolution ~70 ps! Misidntification ! 2 spots of 1000 PMTs ach! 10 MHz vnts rat in th RICH (~20 hits/track)! q Main contribution from kaon dcays (~1 MHz from halo muons and pion dcays)! 6

7 2024 TDC channls! 4 TEL62! Th goal: GPU in L0 RICH TEL62! This is not th L0 triggr baslin vrsion for th NA62 RICH dtctor! TEL62! RO! buffr! 4 TEL62 for RICH dtctor! q 8 1Gb/s links for data r/o! q 4 1Gb/s triggr primitivs! q 4 1Gb/s GPU triggr! Evnts rat: 10 MHz! L0 triggr rat: 1 MHz! Max Latncy: 1 ms! Rducd rat! L1! L2! L0TP! 7

8 GPUs in Low Lvl Triggrs Two main issus to b solvd:! Latncy! q q Is th GPU latncy pr vnt small nough to cop with th tiny latncy of low lvl triggrs?! Is th latncy stabl nough for usag in synchronous triggr systms?! Computing powr! q Is th GPU fast nough to tak a triggr dcision at tns of MHz vnts rat?! 8

9 GPU Procssing Exampl: packt with 1404 B (fw tns of vnts in NA62 RICH application)! T=0! VRAM NIC GPU PCI xprss chip st RAM µs 0 CPU 9

10 GPU Procssing VRAM NIC GPU PCI xprss chip st 0 CPU RAM µs 10 10

11 GPU Procssing VRAM NIC GPU PCI xprss chip st 0 10 CPU RAM µs 99 11

12 GPU Procssing VRAM NIC GPU PCI xprss chip st CPU RAM µs 99 12

13 GPU Procssing VRAM NIC GPU PCI xprss chip st CPU RAM µs 13

14 GPU Procssing VRAM NIC GPU PCI xprss chip st CPU RAM µs 14

15 GPU Procssing Latncy du to data transfr from th dtctor to th systm is biggr than th latncy du to GPU computing! It scals almost linarly (apart from th ovrhads) with th data siz whil th latncy du to computing can b hiddn xploiting th hug rsourcs! Communication latncy fluctuations quit big! VRAM NIC GPU PCI xprss chip st CPU RAM µs 15

16 First solution: NANET NANET is an FPGA-basd NIC that has GPUDirct RDMA capabilitis! APEnt Rom Group! R. Ammndola t al., JINST 9 C02023, 2014! 16

17 First solution: NANET NANET is an FPGA-basd NIC that has GPUDirct RDMA capabilitis!! k r o w t N d s! a b g n i A l G b FP na A t! c. s o r t i d n r D a PU on rim L G. p h A x t i : w P E d Talk r H a n C i g c n a i f t r u t p n I om c U P G Group! m i t APEnt Rom l ra R. Ammndola t al., JINST 9 C02023, 2014! 17

18 Scond solution: PFRING PFRING DNA (Dirct NIC Accss) is a way to map NIC mmory to usrland so that thr is no additional packt copy bsids th DMA transfr don by th NIC! PFRING! 18

19 Rsults: PFRING Latncy rducd and ngligibl fluctuations! Th total latncy is givn as a function of th numbr of vnts to buffr bfor th start of GPU computation! For ral application th working point dpnds on th vnts rat and vnt dimnsion! 19

20 L0 RICH triggr algorithm Rquirmnts for an on-lin RICH rconstruction algorithm:! Tracklss! q No information from th trackr! q Difficult to mrg information from many dtctors at L0! Multi-rings! q Many-body dcay in th RICH accptanc! Fast! q Non-itrativ procdur! q Evnts rat at a lvl of ~10 MHz! Low latncy! q Onlin (synchronous) triggr! Accurat! 20

21 Almagst Nw algorithm (Almagst) basd on Ptolmy s thorm: A quadrilatr is cyclic (th vrtx li on a circl) if and only if is valid th rlation: AD*BC+AB*DC=AC*BD! Dsign a procdur for paralll implmntation! 21

22 Almagst: xampl 22

23 Almagst: xampl i) Slct a triplt (3 starting points)! A B C 23

24 Almagst: xampl i) Slct a triplt (3 starting points)! ii) Loop on th rmaining points: if th nxt point dos not satisfy th Ptolmy s condition thn rjct it B! A D C 24

25 Almagst: xampl i) Slct a triplt (3 starting points)! ii) Loop on th rmaining points: if th nxt point dos not satisfy th Ptolmy s condition thn rjct it B! iii) If th point satisfy th Ptolmy s condition thn considr it for th fit! A D C 25

26 Almagst: xampl i) Slct a triplt (3 starting points)! ii) Loop on th rmaining points: if th nxt point dos not satisfy th Ptolmy s condition thn rjct it B! iii) If th point satisfy th Ptolmy s condition thn considr it for th fit! A D C iv) again! 26

27 Almagst: xampl i) Slct a triplt (3 starting points)! ii) Loop on th rmaining points: if th nxt point dos not satisfy th Ptolmy s condition thn rjct it! iii) If th point satisfy th Ptolmy s condition thn considr it for th fit! iv) again! v) Prform a singl ring fit! 27

28 Almagst: xampl i) Slct a triplt (3 starting points)! ii) Loop on th rmaining points: if th nxt point dos not satisfy th Ptolmy s condition thn rjct it B! iii) If th point satisfy th Ptolmy s condition thn considr it for th fit! A D D D C iv) again! vi) Rpat by xcluding th alrady usd points! v) Prform a singl ring fit! 28

29 A mor complicatd xampl 29

30 Almagst on GPU Vry high paralllism! q Hug numbr of computing cors (>2000)! q Hug mmory bandwidth!... Two lvls of paralllism! q Svral triplts run in paralll! q Svral vnts at th sam tim! 30

31 Almagst: implmntation Tsts on NVIDIA Tsla K20 GPU! Total computing tim ordr a fw #s pr vnt (on singl GPU)! Good fficincy (using 8 triplts)! q Room for improvmnt! 8 triplts 4 triplts 256 vnts! Furthr tsts ongoing to study nois immunity, bias, fficincy a function of th numbr of hits, tc! 31

32 Nxt stps Rciv th TTC stram (timing and triggr) from th xprimnt! q TTC intrfac board with HSMC connctor! Intgration in th NA62 Triggr and DAQ systm! q First tst during dry run in August! q Parasitic tst during NA62 xprimntal run in Octobr! 32

33 Conclusions Th us of GPUs in HEP triggr systms could giv svral advantags, but procssing prformancs and latncis should b carfully studid! q Construction of a dmonstrator L0 procssor for th NA62 RICH is undr way! q Data transfr is th dominant contribution! Chrnkov rings pattrn rcognition within th total L0 latncy of 1 ms sms possibl! Intgration with th NA62 Triggr and DAQ systm! q q First tsts during dry run in August 2014! Parasitic data taking during NA62 xprimntal run starting Octobr 2014! 33

34 SPARES 34

35 Latncy masurmnt Evnts simulatd in TEL62! 1 Gb/s Groupd in MTP! NIC PC GPU Start signal riss with th TEL62 lpt Scop first vnt in th MTP! Start First stop: packt arrival! Stop 1 Stop 2 Buffring in th PC RAM: GMTP dpth can b changd! Scond stop: aftr xcution Dual procssor PC:! on GPU (singl ring q XEON E Ghz! rconstruction krnl)! q I350T2 Gigabit card! Th prcision of th mthod q 32 GB! has bn valuatd as bttr q GPU K20c (2496 cors) PCI than 1 µs! v2 x16! 35

36 Algorithms for singl ring domh! hough! tripl! math! 36

37 Procssing tim Using Mont Carlo data, th algorithms ar compard on Tsla C1060! For packts of >1000 vnts, th MATH algorithm procssing tim is around 50 ns pr vnt! Th prformanc on DOMH (th most rsourc-dpndnt algorithm) is compard on svral GPUs! 37

38 Singl ring algorithms Crowford mthod ( math ):! q Translat in th cntr of mass! q Last squar minimization à linar! Taubin mthod:! q Mor fficint: minimiz th bias introducd by th Kasa rlatd mthods (minimization of simpl algbraic distanc)! q Rsolution slightly bttr (on idntifid rings)! Th diffrnc of computing tim on th GPU is at th lvl of 10 ns pr vnt! 38

39 N(=hits) triplts Numbr of triplts qual to th numbr of hits.! Rlativly high fficincy.! Computing tim dpnds on numbr of rings (diffrnt numbr of GPU cors pr vnts)! Rsults on TESLA C1060 (240 cors, lss than 1 Tflops)! Room for optimization! 39

40 4 slctd triplts Only 4 triplts pr vnt ar usd: lft, right, up and down! Furthr cuts to avoid too clos hits! 40

41 4 slctd triplts Stability with small nois (studis ar ongoing)! Infficincy du to th ordr in choosing th rings.! Dpndnc on th cuts to dfin th triplts.! 41