Scalable, Robust, Fault-Tolerant Parallel QR Factorization. Camille Coti

Transcription

1 Communication-avoiding Fault-tolerant Scalable, Robust, Fault-Tolerant Parallel Factorization Camille Coti LIPN, CNRS UMR 7030, SPC, Université Paris 13, France DCABES August 24th, / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

2 Roadmap Communication-avoiding Fault-tolerant 1 Failures Factorization 2 Communication-avoiding 3 Fault-tolerant 4 2 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

3 Communication-avoiding Fault-tolerant Large-scale systems are volatile Failures Factorization Life expectancy of an electronic component: the famous bathtub curve 3 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

4 Communication-avoiding Fault-tolerant Reliability of a distributed system Failures Factorization Mean Time Between Failures n 1 1 MT BF total = ( ) 1 (1) MT BF i The more components a system is made of, the more likely it is to have a failure. Mean Time Between Failures of the system (hours) i= H H H H e+06 Number of components in the system 4 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

5 Factorization Communication-avoiding Fault-tolerant Failures Factorization Matrix A: A = R is upper-triangular Q is orthogonal Unicity: Yes, if the diagonal elements of R are positive No otherwise Several algorithms: Dense linear algebra: algorithms based on unitary transformations. (Modified) Gram-Schmidt, Givens rotation, Householder projection... 5 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

6 Factorization Communication-avoiding Fault-tolerant Failures Factorization Panel-update approach Factorize a panel Update the trailing matrix Repeat recursively on the trailing matrix R ( ) A11 A 12 A = A 21 A 22 ( ) 1 R 12 = Q 1 0 A 1 22 Q

7 Factorization Communication-avoiding Fault-tolerant Failures Factorization Panel-update approach Factorize a panel Update the trailing matrix Repeat recursively on the trailing matrix R ( ) A11 A 12 A = A 21 A 22 ( ) 1 R 12 = Q 1 0 A 1 22 Q

8 Factorization Communication-avoiding Fault-tolerant Failures Factorization Panel-update approach Factorize a panel Update the trailing matrix Repeat recursively on the trailing matrix R ( ) A11 A 12 A = A 21 A 22 ( ) 1 R 12 = Q 1 0 A 1 22 Q panel trailing matrix 6 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

9 Communication-avoiding Fault-tolerant 1 Failures Factorization 2 Communication-avoiding 3 Fault-tolerant 4 7 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

10 Communication-avoiding Fault-tolerant factorization of a panel: particular shape M >> N: the matrix is tall-and-skinny Specific algorithm: TS ( A11 A 21 ) ( ) 1 = Q 1 0 P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 8 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

11 Communication-avoiding Fault-tolerant factorization of a panel: particular shape M >> N: the matrix is tall-and-skinny Specific algorithm: TS ( A11 A 21 ) ( ) 1 = Q 1 0 P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 8 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

12 Communication-avoiding Fault-tolerant factorization of a panel: particular shape M >> N: the matrix is tall-and-skinny Specific algorithm: TS ( A11 A 21 ) ( ) 1 = Q 1 0 P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 8 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

13 Communication-avoiding Fault-tolerant factorization of a panel: particular shape M >> N: the matrix is tall-and-skinny Specific algorithm: TS ( A11 A 21 ) ( ) 1 = Q 1 0 P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 8 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

14 Communication-avoiding Fault-tolerant factorization of a panel: particular shape M >> N: the matrix is tall-and-skinny Specific algorithm: TS ( A11 A 21 ) ( ) 1 = Q 1 0 P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 8 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

15 Communication-avoiding Fault-tolerant factorization of a panel: particular shape M >> N: the matrix is tall-and-skinny Specific algorithm: TS ( A11 A 21 ) ( ) 1 = Q 1 0 P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 8 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

16 Communication-avoiding Fault-tolerant After the panel factorization: Compute the compact representation of Q 1: Q 1 = I Y 1T 1Y1 T Use the Y 1 and T 1 matrices to update the trailing matrix: ( I Y1T 1Y T 1 ) ( A 12 A 22 ) = = ( A12 A 22 ( 2 A 1 22 ) ) Y 1(T T 1 (Y T 1 ( A12 A 22 ) )) R 12 : upper part of the R A 1 22 : new trailing matrix. ( ) A11 A 12 A = A 21 A 22 ( ) 1 R 12 = Q 1 0 A / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

17 Communication-avoiding Fault-tolerant Step 0 Step 1 Step 2 P0 P0 P0 P1 P1 P1 P2 P2 P2 P3 P3 P3 Cij 10 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

18 Communication-avoiding Fault-tolerant Step 0 Step 1 Step 2 P0 P0 P0 P1 P1 P1 P2 P2 P2 P3 P3 P3 Cij 10 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

19 Communication-avoiding Fault-tolerant Step 0 Step 1 Step 2 P0 P0 P0 P1 P1 P1 P2 P2 P2 P3 P3 P3 Cij 10 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

20 Communication-avoiding Fault-tolerant 1 Failures Factorization 2 Communication-avoiding 3 Fault-tolerant 4 11 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

21 Communication-avoiding Fault-tolerant Let s look at TS in details P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 12 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

22 Communication-avoiding Fault-tolerant Let s look at TS in details P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 12 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

23 Communication-avoiding Fault-tolerant Let s look at TS in details P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 12 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

24 Communication-avoiding Fault-tolerant Let s look at TS in details P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 12 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

25 Communication-avoiding Fault-tolerant Let s look at TS in details P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 12 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

26 Communication-avoiding Fault-tolerant Let s look at TS in details P0 A0 R V0 V0 V P1 A1 V1 P2 A2 V2 V2 P3 A3 V3 12 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

27 Communication-avoiding Fault-tolerant P 0 works beginning end P 2 works during the first two steps, then stops P 1 and P 3 work during the first step, then stops Let s put these lazy dudes to work! 13 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

28 Fault-tolerant TS Communication-avoiding Fault-tolerant P 0 P 1 P 2 P 3 14 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

29 Fault-tolerant TS Communication-avoiding Fault-tolerant P 0 P 1 P 2 P 3 14 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

30 Fault-tolerant TS Communication-avoiding Fault-tolerant P 0 P 1 P 2 P 3 14 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

31 Communication-avoiding Fault-tolerant Update of the trailing matrix Triggered by the tree of the panel factorization Uses the intermediate Y and T matrices Pairwise operation tree ( ) R Ĉ 0 A = Q Ĉ 1 Decompose the blocks of the trailing matrix: ( ) ( ) C C i = i Ci[: N 1] C i = C i[n :] Compute the update : ( ) ( ) C 0 C R 1 C 1 = 0 C 1 P 0 Ĉ 0 = C 0 W C 0 W P 1 T W = T T (C 0 Y T 1 C 1) Ĉ 1 = C 1 Y 1W 15 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

32 Fault-tolerant update Communication-avoiding Fault-tolerant Change the order of the operations T W = T T (C 0 Y T 1 C 1) Ĉ 0 = C 0 W P 0 C 0 C 1, Y 1 P 1 T W = T T (C 0 Y T 1 C 1) Ĉ 1 = C 1 Y 1W If a process fails Both process have the W matrix Computation recovered from the W matrix and the initial submatrix 16 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

33 Fault-tolerant update Communication-avoiding Fault-tolerant Change the order of the operations T W = T T (C 0 Y T 1 C 1) Ĉ 0 = C 0 W P 0 C 0 C 1, Y 1 P 1 T W = T T (C 0 Y T 1 C 1) Ĉ 1 = C 1 Y 1W If a process fails Both process have the W matrix Computation recovered from the W matrix and the initial submatrix 16 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

34 Communication scheme Communication-avoiding Fault-tolerant P 0 P 1 P 2 P 3 17 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

35 Soft errors Communication-avoiding Fault-tolerant Soft errors : Bit flips causing numerical errors Caused by radiations, cosmic rays, various electric disturbances... Approaches to tackle soft errors: Triple modular redundancy ECC memory Multiple executions... At several levels: hardware, system, application Idea here: Take advantage of partial redundancy to detect errors Recompute only the erroneous step 18 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

36 Soft errors Communication-avoiding Fault-tolerant exchange check check exchange check P 0 P 1 P 2 P 3 19 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

37 Soft errors Communication-avoiding Fault-tolerant exchange check check exchange check P 0 P 1 P 2 P 3 19 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

38 Soft errors Communication-avoiding Fault-tolerant exchange check check exchange check P 0 P 1 P 2 P 3 19 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

39 Soft errors Communication-avoiding Fault-tolerant exchange check check exchange check P 0 P 1 P 2 P 3 19 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

40 Soft errors Communication-avoiding Fault-tolerant exchange check check exchange check P 0 P 1 P 2 P 3 19 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

41 Communication-avoiding Fault-tolerant 1 Failures Factorization 2 Communication-avoiding 3 Fault-tolerant 4 20 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

42 Communication-avoiding Fault-tolerant Algorithm-based fault-tolerance Failure recovery is handled by the application Behavior upon failures: defined in the algorithm Goal: Minimal overhead during failure-free executions As few recomputations as possible after failures Fault-tolerant algorithms for factorization Use intrinsic redundancy or idle processes Operation reordering based on algebraic properties Small modifications in the critical path No additional computations Data exchange instead of send/recv Partial results already ready for recovery 21 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization

43 Appendix: performance Communication-avoiding Fault-tolerant Time (sec) CA FT TS FT CA TS 0 1k 4k 8k 16k 32k Matrix size 21 / 21 C. Coti Scalable, Robust, Fault-Tolerant Parallel Factorization