Round-off error propagation and non-determinism in parallel applications

Transcription

1 Round-off error propagation and non-determinism in parallel applications Vincent Baudoui (Argonne/Total SA) Franck Cappello (Argonne/INRIA/UIUC-NCSA) Georges Oppenheim (Paris-Sud University) Philippe Ricoux (Total SA)

2 Improve numerical simulations precision future exascale supercomputers Round-off error management at the software level Objectives: be able to validate simulation results against errors build a model for error propagation Exascale particularities: High dimension problems Different algorithms Possibly non-deterministic behavior Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

3 Outline 1 Round-off errors 2 Error propagation in LU decomposition 3 Impact of non-determinism Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

4 Outline 1 Round-off errors 2 Error propagation in LU decomposition 3 Impact of non-determinism Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

5 Round-off errors sign exponent fraction Limited number of bits to encode reals round-off errors (machine precision u for 64-bits floating point format) 0 x backward error x + x y = f (x) forward error ŷ = f (x + x) Forward error: error on the result E = ŷ y Backward error: perturbation in the data ɛ = x forward error condition number backward error Higham (2002), Accuracy and stability of numerical algorithms Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

6 Outline 1 Round-off errors 2 Error propagation in LU decomposition 3 Impact of non-determinism Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

7 LU decomposition Study round-off error propagation LU decomposition to solve linear systems in numerical simulations (e.g. wave equation in depth imaging) Decompose A R 4 4 into triangular matrices L and U A = L U a 11 a 12 a 13 a u 11 u 12 u 13 u 14 a 21 a 22 a 23 a 24 a 31 a 32 a 33 a 34 = l l 31 l u 22 u 23 u u 33 u 34 a 41 a 42 a 43 a 44 l 41 l 42 l u 44 }{{} u 11 u 12 u 13 u 14 l 21 u 22 u 23 u 24 l 31 l 32 u 33 u 34 l 41 l 42 l 43 u 44 Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

8 LU decomposition LU decomposition algorithm without pivoting (fast and simple) for i = 1..4 do for j = i..4 do for k = 1..(i 1) do t = a ik a kj a ij = a ij t for j = (i + 1)..4 do for k = 1..(i 1) do t = a jk a ki a ji = a ji t a ji = a ji /a ii Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

9 LU decomposition Instruction level graph for A R 3 3 a 11 u 11 a 12 u 12 a 13 a 21 a 22 / u 13 l 21 u 22 a 23 a 31 a 32 a 33 / / u 23 l 31 l 32 u 33 Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

10 LU decomposition Hierarchical graph i = 1 A i = 2 i = 3 L U i = 4 Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

11 Error propagation Analytical worst-case bounds Statistical analysis (CADNA, PRECISE) Error propagation through partial derivatives First order approximation of output error E y : E y i δ i y δ i with respect to instruction errors δ i : ˆx i = x i + δ i, δ i u Miller & Wrathall (1980), Software for roundoff analysis of matrix algorithms Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

12 Error propagation Partial derivatives computed using algorithmic differentiation: y δ i = (a,b) P(δ i,y) b a Instruction error δ i quantification for elementary operations +,,, / (estimation only for the division operator): s = a + b δ i = (s a) b output error E y i δ i y δ i quantified! Langlois (2001), Automatic linear correction of rounding errors Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

13 Error propagation Limitations First order approximation: exact for linear algorithms (multiplication/division by an error-free number no cross-interaction between instruction input errors) Computational cost: derivatives computation instruction error estimation Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

14 Experiments Experiments on Gaussian random matrices A N (0, I 4 ) (100,000 samples) Median output error: Last element error evolution with matrix size: n E Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

15 Experiments Semi-random matrices: A N (0, I 4 ) except for a 11 = Median output error: i = 1 Median derivatives 0.5 but: A i = 2 l 23, l 24, l 32, l 42 u 33, u 34, l 43 i = 3 u 44 i = 4 L U u 33 l 23, u 33 l 32, u 34 l 24, u 34 l 32, l 43 l 23, l 43 l 32, l 43 l 42, u 44 l 24, u 44 l Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

16 Outline 1 Round-off errors 2 Error propagation in LU decomposition 3 Impact of non-determinism Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

17 Impact of non-determinism Non-determinism in parallel applications Data may arrive in random order (delays in message passing between processes) If data is processed in order of arrival, results can be non-deterministic because of round-off errors If message order is not recorded: a failure a b f (a, b) checkpoint b f (b, a) Will we converge to the same solution in the same time? Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

18 1D stencil Example: 1D Jacobi stencil b L x 0 1 x 0 2 x 0 3 x 0 4 x 0 5 x 0 6 x 0 7 x 0 8 x 0 9 x 0 10 b R 10 elements vector X with initial state X 0 and boundary conditions b At each iteration, x i is updated as the mean of its neighbors: x n+1 i = f (xi 1 n, x i+1 n ) = x i 1 n + x i+1 n 2 Markov process: X n+1 = F(X n ) Converges if F is contracting: d(f(a), F(B)) kd(a, B), k < 1 Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

19 1D stencil Deterministic behavior initial state X 0 final state X Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

20 Non-determinism Random data order { x n+1 f (x n i = i 1, xi+1 n ) with probability p f (xi+1 n, x i 1 n ) with probability (1 p) Harmless if f is symmetrical But f can be non-symmetrical in practice because of round-off errors: { a + b (a + b)/2 when a 10 f (a, b) = 1 + a a (a + b)/3 otherwise non-deterministic results Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

21 Non-determinism Non-deterministic behavior initial state X 0 X for 10 runs Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

22 Non-determinism Converges (to a stationary distribution) if the F functions are contracting in average: E[log(k)] < 0 some functions can be non-contracting if their probability is relatively low Diaconis, P. & Freedman, D. (1999), Iterated random functions Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

23 Failure recovery If the i th cell process fails, xi n is lost and rolls back to the last checkpoint x n τ i iterations n τ to n must be recomputed Result may differ because of data order randomness system restarts from a possibly different state X n x i checkpoint failure x i recovery If X n remains in the same attraction domain as X n it will converge to the same distribution at the same speed but number of iterations may vary Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

24 Recovery Failure recovery Failure at x 500 8, checkpoint restart from X 400 X 500 before/after recovery X 1256 y goes back to the same solution after a while here Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

25 Conclusion Run massively parallel numerical simulations on future exascale supercomputers Validate simulation results against round-off errors Error propagation through derivatives Study other LU decomposition algorithms Define heuristics for graph exploration Impact of non-determinism on recovery No need to log message order under some hypothesis Study a real test-case Round-off error propagation and non-determinism Urbana-Champaign, IL, November 26, / 25

26 Round-off error propagation and non-determinism in parallel applications Thank you for your attention!