ACCURATE FLOATING-POINT SUMMATION IN CUB

Transcription

1 ACCURATE FLOATING-POINT SUMMATION IN CUB URI VERNER Summe inten

2 OUTLINE Who need accuate floating-point ummation?! Round-off eo: ouce and ecovey A new method fo accuate FP ummation on a GPU Added a a function to the open-ouce CUB libay How fat i it? Download link

3 NORMAL FLOATING-POINT SUMMATION INPUT RESULT INACCURATE RESULTS NON-DETERMINISTIC RESULTS!

4 ACCURATE FP SUMMATION INPUT EXACT SUM ACCURATE SUM AS FLOATING-POINT Ou method compute both!

5 EXISTING WORK: EXBLAS (OPENCL) By Iakymchuk, Collange, et al. Ue Kulich accumulato (vey wide fied-peciion vaiable) Ou method ue a diffeent appoach

6 WHERE IS THIS USEFUL? High Pefomance Computing application An eample coming net Co-platfom application Debugging: bit-eact eult fo floating point! if (d_eult!= d_efeence) eo( wong anwe! );

7 EXAMPLE: LATTICE QCD COMPUTATIONS QCD - Quantum Chomodynamic Decibe the tong foce that bind quak and gluon GPU acceleated QUDA libay lattice.github.io/quda Accuate ummation can potentially impove convegence and educe computation time

8 CONVERGENCE OF ITERATIVE ALGORITHM BiCGtab Algoithm: Diac equation olve convegence tagnation

9 ROUND-OFF ERROR: SOURCE AND RECOVERY

10 IEEE-754 FLOATING-POINT STANDARD 32/64 bit S EXP SIGNIFICANT DIGITS = ( 1) 2 EXP 1. D b ingle-peciion double-peciion Fomat width 32 bit 64 bit Eponent ange (8 bit) (11 bit) Significant digit 23 (+1 implicit) 52 (+1 implicit) Special cae: +/-NaN, +/-Inf, +/-0, ubnomal

11 SOURCE OF NON-REPRODUCIBILITY Non-aociative opeation Ode of opeation matte: 1,000,000 + ( ) -> 1,000,001 (1,000, ) > 1,000,000 diffeent implementation etun diffeent um value

12 SOURCE OF ACCURACY LOSS Round-off eo in compute opeation Compute um: accuate actual bigge diffeence in magnitude => moe digit lot

13 TWO-SUM ALGORITHM (KNUTH) TwoSum(a,b) Round(a+b) ound-off eo 6 FP opeation [,] = TwoSum(a,b) <- a+b z <- -b <- (b-(-z)) + (a-z)

14 FAST TWO-SUM (DEKKER) FatTwoSum(a,b) Round(a+b) ound-off eo 3 FP opeation Requie EXP(a) EXP(b) [,] = FatTwoSum(a,b) <- a+b z <- -a <- b-z

15 ERROR-FREE PARALLEL SUMMATION

16 INTEGRATION INTO CUB LIBRARY CUB: Paallel pimitive in CUDA Include paallel pimitive like Sum, Scan, Sot, etc. Pefomance tuned fo evey NVIDIA GPU achitectue Reduction Aim: ue Reduction with TwoSum() fo an eo-fee um

17 REDUCTION+TwoSum: PROBLEM #1 The output of TwoSum i two FP, intead of one! + = + = Paallel eduction TwoSum (1,2)=2Sum(1,2) (y1,y2)=2sum(1,2) 2=2+y1 (1,2)=fat(1,2) 2=2+y2 (3,3)=fat(1,2) 1. Convet (, 0.0) 2. Define (1,1) + (2,2) (3,3)

18 REDUCTION+TwoSum: PROBLEM #2 Limited accuacy E.g.: Multiple value with imila eponent can be added without oveflow

19 DIVIDE THE EXPONENT RANGE INTO BINS Numbe of EXP value: 2048 (double) add to bin #3 floo 1023 / = 3 3 (bin id)

20 HOW MANY EXPONENT VALUES PER BIN? binay digit Suppoe we add n numbe to a bin: a i = 2 e i m i, whee e l e i < e h. Ou budget i 106 digit log 2 n + e h e l a l = a h = Fo n = 2 20, e h e l = 32 diffeent eponent! e h e l

21 ALGORITHM: ERROR-FREE SUMMATION ON GPUS

22 EACH THREAD DOES THE FOLLOWING EXPONENT=11bit binid=6bit bin=5bit ead input adi-ot by binid educe-by-key binid update mem bin SHARED MEMORY

23 FINAL SUMMATION PHASE Bin Bin 0 Block 0 Block Reult SUM X (eial phae)

24 ALGORITHM SUMMARY 1. Fo each thead block: Repeat: Read input tile Radi-ot item by bin ID Compute um fo each bin with Reduce-by-key in egite in egite (+ temp buffe in haed) in egite (+ temp buffe in haed) Update bin in haed memoy in haed memoy Save bin to global memoy in global memoy 2. Mege bin with the ame bin ID 3. Nomalize bin by adding them fom low to high 4. Rounded eult i in the highet wod

25 GItem/ec PERFORMANCE (K40) Billion Item pe econd Cont Random 1 Nomal ummation i ~6 time fate 0 Numbe of item

26 DOWNLOAD AND CONTRIBUTE Get it at: Uage intuction in README.ACCUSUM It open ouce. Ue it, impove it!

27 THANK YOU

28 BACKUP