Measuring freeze-out parameters on the Bielefeld GPU cluster

Transcription

1 Measuring freeze-out parameters on the Bielefeld GPU cluster

2 Outline Fluctuations and the QCD phase diagram Fluctuations from Lattice QCD The Bielefeld hybrid GPU cluster Freeze-out conditions from QCD BNL-Bielefeld Collaboration: A. Bazavov, H.-T. Ding, P. Hegde, O. Kaczmarek, F. Karsch, E. Laermann, S. Mukherjee, P. Petreczky, C. Schmidt, D. Smith, W. Soeldner, M. Wagner

3 Fluctuations and the QCD phase diagram different QCD phases characterized by chiral symmetry confinement aspects possible critical end-point T [GeV] 0.16 critical end-point QCD Figure from C. Schmidt quark-gluon-plasma 2nd order phase transition hadron gas divergent correlation length divergent susceptibility 0 vacuum nuclear matter neutron stars chemical potential µ B

4 Fluctuations from Lattice QCD expansion of the pressure in p 1 T 4 = X i,j,k 1 i!j!k! BQS ijk µb T i µq T j µs T k B,Q,S conserved charges (baryon number, electric charge, strangeness) generalized susceptibilities BQS ijk = 1 B /T Q /T S /T ) Z(T,µ) µ=0 related to cumulants of net charge fluctuations, e.g. VT 3 B 2 = h( N B ) 2 i = N 2 B 2N B hn B i + hn B i 2

5 Calculation of susceptibilities from Lattice QCD µ-dependence is contained in the fermion determinant Z Z = DU(det M(µ)) N f /4 exp( S g ), calculation of susceptibilities requires µ-derivatives of fermion 2 * nf ln Z (ln det det 2 = 2 + formulate all operator in terms of traces over space-time, color (and spin) evaluate using noisy estimators ensemble average large number of configurations (~3000 for each beta)

6 Noisy estimators traces required for 2 (ln det 2 =Tr =Tr. M 2 noisy estimators large number of random vectors η (~1500 / n 1 M n M M n...m 1 1 NX = n 1 M 2 N!1 k n M M n...m 1 2 k dominant operation: fermion matrix inversion (~ 99%) Tr k=1

7 Configuration generation sequential process use RHMC algorithm to evaluate the system in simulation time = P two dominant parts of the calculation (90% of the runtime) fermion force ~50% for improved actions (HISQ) fermion matrix inversion ~90% for standard action

8 staggered Fermion Matrix (Dslash) Krylov space inversion of fermion matrix dominates runtime sparse Matrix w x = D x,x 0v x 0 = 3X µ=0 n o U x,µ v x+ˆµ U x ˆµ,µ v x ˆµ memory: 8 SU(3) matrices input, 8 color vectors input, 1 color vector output 8 x ( ) + 24 bytes = 792 bytes ( 1584 for double precision) Flops: (CM = complex mult, CA = complex add) 4 x ( 2 x 3 x (3 CM + 2 CA) + 3 CA) + 3 x 3 CA = 570 flops flops / byte ratios: 0.72

9 History of QCD machines in Bielefeld special purpose APE machines optimized low-latency network fast complex a x b + c GFlops (single precision) APE100 APEmille APEnext

10 GPUs as Accelerators typical CPU data performance ~ 150 GFlops bandwidth ~ 30 GB/s power consumption ~ 125W price ~ Euro typical GPU data performance ~ 1 TFlops bandwidth ~ 200 GB/s power consumption ~ 250W price ~ 2000 Euro (Tesla 6 GB) ~ 500 Euro (GTX 3 GB)

11 GPUs are throughput processors Low Latency or High Throughput? Figure from Nvidia CPU Optimized for low-latency access to cached data sets Control logic for out-of-order and speculative execution GPU Optimized for data-parallel, throughput computation Architecture tolerant of memory latency More transistors dedicated to computation

12 The Bielefeld GPU cluster hybrid GPU / CPU cluster inauguration 2/ compute nodes in 14x19 racks 400 GPUs with 1824 GB memory 304 CPUs (1216 cores) with 7296 GB memory 7 storage nodes / 2 head nodes 1.1 million founded with federal and state government funds (~50% for GPUs)

13 Installation

14 Tesla nodes 104 Tesla Nodes (1U) Dual Intel Xeon X GB memory 2 x Nvidia Tesla M2075 (6GB - ECC) 515 Gflops double precision 1030 Gflops single precision 150 GB/s memory bandwidth

15 GTX nodes 48 GTX Nodes (4U) 7ST' Dual Intel Xeon X GB memory 4 x Nvidia GTX 580 (3GB - no ECC) 198 Gflops double precision 1581 Gflops single precision 192 GB/s memory bandwidth

16 Cooling and power consumption power consumption: ~125kW (CPU+GPU) compute nodes in 14 racks: < 10kW / rack cold aisle containment (~ 24 ) GPUs running at ~ 90 / ~ 60 (GTX/Tesla) hot aisle containment (~ 39 ) to prevent heat pollution in data center

17 Stability artificial stress tests tests based on QCD production code Tesla cards with ECC after 1/2 year (total 208 cards) 4 cards with single bit errors (corrected) 2 cards with double bit errors (program aborted) GTX cards (gamer cards) after 1/2 year (total 192 cards) 12 cards with wrong results in tests, 2 died replace faulty cards (also Tesla w/ single bit errors)

18 Bandwidth bound low flops/byte ratio GTX cards are always faster even for double precision calculations linear algebra has an even worse flop / byte ratio flops are free - but registers are limited (more later) Dslash efficiency Tesla: 0.72 flop/byte * 144 Gbytes/s = 103 Gflops (10% peak) memory bandwidth is the relevant performance indicator general issue for supercomputing Card GFlops (32 bit) GFlops (32 bit) GBytes/s Flops / byte Flops/ byte GTX Tesla M

19 optimizing memory access use coalesced memory layout: structure of arrays (SoA) instead of AoS one can reconstruct a SU(3) matrix also from 8 or 12 floats improved actions result in matrices that are no longer SU(3): must load 18 floats exploit possibility of texture access allows to get near 100% of the bandwidth ECC hurts (naive 12.5%, real world ~ %) do more work with less bytes: mixed precision inverters

20 optimizing memory access use coalesced memory layout: structure of arrays (SoA) instead of AoS one can reconstruct a SU(3) matrix also from 8 or 12 floats improved actions result in matrices that are no longer SU(3): single precision must load 18 floats 100 exploit possibility of texture access allows to get near 100% of the bandwidth 20 ECC hurts (naive 12.5%, real world ~ %) Gflop/s HISQ inverter on single GPU GTX580 (3GB) M2075 (6GB) double precision lattice size do more work with less bytes: mixed precision inverters

21 Registers... improved fermion action require sum over products of up to 7 SU(3) matrices SU(3) Matrix: 18 / 36 registers Fermi architecture: 63 registers (4 byte) / thread optimize SU(3) *= SU(3) operation for register usage spilling causes significant performance drop for bandwidth bound kernels however: spilling is often better than shared memory 48kB L1 cache precomputed products can help but must be stored somewhere

22 Impact of the cluster on our results 4 B Evolution from 2/2012 to 9/2012, Nt=8 4 Q T [MeV] T [MeV]

23 Impact of the cluster on our results 4 B Evolution from 2/2012 to 9/2012, Nt=8 4 Q T [MeV] T [MeV]

24 Status of HISQ lattice data B 2 B 4 T c =154(9) MeV N =6 8 BNL-Bielefeld preliminary B T [MeV] Q N = T c =154(9) MeV Q BNL-Bielefeld 0.25 preliminary Q T [MeV]

25 Freeze-out curve from heavy-ion collision cf. Talks by Gelis, Rischke, Klein-Bösing initial conditions T [GeV] RHIC@BNL Figure from C. Schmidt quark-gluon-plasma cf. Talk by Petreczky Lattice EoS 0.16 FAIR@GSI NICA@JINR evolution freeze-out hadron gas Ratio Hadron abundances from HRG model s NN =130 GeV freeze-out 0 vacuum nuclear matter chemical potential neutron stars µ B 10-2 Data Model 10-3 π - π + T=165.5, µ b =38 MeV - K K + p p Λ Λ Ξ Ξ Ω Ω K + π + - K π - p π - Λ π - Ξ π - Ω π - φ K - * K K - T f ( p s),µ f ( p s)

26 Freeze-out curve from heavy-ion collision cf. Talks by Gelis, Rischke, Klein-Bösing initial conditions T [GeV] RHIC@BNL Figure from C. Schmidt quark-gluon-plasma cf. Talk by Petreczky Lattice EoS 0.16 FAIR@GSI NICA@JINR evolution freeze-out hadron gas Fluctuations from Lattice QCD B 2 B 4 T c =154(9) MeV N =6 8 BNL-Bielefeld preliminary B T [MeV] Q N = T c =154(9) MeV Q BNL-Bielefeld 0.25 preliminary Q T [MeV] freeze-out 0 vacuum nuclear matter T f ( p s),µ f ( p s) chemical potential neutron stars µ B

27 Constraining isospin and strangeness Assume: homogenous model in thermal equilibrium Exploit: initial conditions strangeness neutrality: hn S i =0 isospin assymetry: hn Q i = r hn B i expand in powers of solve for µ Q,µ S µ B,µ Q,µ S µ Q (T,µ B )=q 1 (T )µ B + q 3 (T )µ 3 B µ S (T,µ B )=s 1 (T )µ B + s 3 (T )µ 3 B LO NLO µ f Q µ Q(T f,µ f B ) µ f S µ S(T f,µ f B ) two independent parameter remain: T f,µ f B

28 Constraining isospin and strangeness µ Q (T,µ B )=q 1 (T )µ B + q 3 (T )µ 3 B µ S (T,µ B )=s 1 (T )µ B + s 3 (T )µ 3 B q 1 HRG LO N =6 N =8 N =12 free s 1 HRG LO N =6 N =8 N =12 free q 3 /q 1 NLO HRG T [MeV] free s 3 /s NLO free T [MeV] LO: continuum extrapolated based on Nt=6,8,12 BNL-BI, arxiv: NLO: small cut-off dependence, continuum estimate based on Nt=6,8 HRG NLO corrections <10% for µ B /T. 1.3

29 Isospin and strangeness constrained BNL-BI, arxiv: µ Q /µ B T = 160 MeV T = 150 MeV T = 170 MeV HRG µq(t,µb)/µb bands for 3 temperatures comparison with HRG (lines) µ S /µ B T = 170 MeV T = 160 MeV T = 150 MeV HRG µ B [MeV] µs(t,µb)/µb deviations from HRG < 5-15 %

30 Pinning down the freeze-out parameters need two experimental ratios to determine (T f,µ f B ) baryon number fluctuations are not directly accessible in experiments we consider ratios of electric charge fluctuations M Q ( p s) 2 Q (p s) = hn Q Qi h( N Q ) 2 i = 1 (T,µ B) Q 2 (T,µ B) = RQ,1 12 µ B + R Q,3 12 µ3 B + = RQ 12 (T,µ B) LO linear in, fixes µ B µ f B M : mean : variance S : skewness S Q ( p s) 3 Q (p s) M Q ( p s) = ( NQ ) 3 = hn Q i Q 3 (T,µ B) Q 1 (T,µ B) = RQ, RQ,2 31 µ2 B + = RQ 31 (T,µ B) LO independent of µ B, fixes T f

31 Determination of freeze-out temperature 3.0 R Q 31 (T,µ B)=R Q, RQ,2 31 µ2 B Q R 31 small cutoff effects small NLO corrections (<10%) for μ/t < HRG µ B /T=1 µ B /T=0 N =6 N =8 S Q 3 Q /M Q T f [MeV ] 1.0 & free T [MeV] & 165 Bands: Continuum estimate

32 Determination of freeze-out chemical potential R Q 12 (T,µ B)=R Q,1 12 µ B + R Q,3 12 µ3 B Q,1 R 12 HRG LO N =6 N =8 N =12 free small cutoff effects at NLO small NLO corrections (<10%) for μ/t < Q,3 Q,1 R 12 /R12 NLO 0.04 HRG T [MeV] free Bands: LO Continuum extrapolation NLO Continuum estimate

33 Determination of freeze-out chemical potential R Q 12 (T,µ B)=R Q,1 12 µ B + R Q,3 12 µ3 B Q R µ B /T HRG Bands: LO Continuum extrapolation NLO Continuum estimate T = 160 MeV T = 170 MeV T = 150 MeV small cutoff effects at NLO small NLO corrections (<10%) for μ/t < 1.3 M Q / 2 Q µ f B /T f (for T f 160 MeV )

34 Summary Fluctuations of conserved charges with HISQ action Determination of freeze-out parameters from Lattice QCD eliminate the need of thermal model experimentally measured ratios of fluctuations can be translated to freeze-out parameters Bielefeld Hybrid-GPU Cluster 400 GPUs, combination of professional Tesla and GTX gamer cards challenges for simulating lattice QCDs