Avoiding Communication in Distributed-Memory Tridiagonalization

Size: px

Start display at page:

Download "Avoiding Communication in Distributed-Memory Tridiagonalization"

Michael Arnold
5 years ago
Views:

1 Avoiding Communication in Distributed-Memory Tridiagonalization SIAM CSE 15 Nicholas Knight University of California, Berkeley March 14, 2015 Joint work with: Grey Ballard (SNL) James Demmel (UCB) Laura Grigori (INRIA) Mathias Jacquelin (LBNL) H. Diep Nguyen (UCB) Edgar Solomonik (ETHZ) Nicholas Knight SIAM CSE 15 1

2 The symmetric eigenproblem Three-phase direct approach for the symmetric eigenproblem: 1 Reduce symmetric matrix to tridiagonal by orthogonal similarity 2 Solve (symmetric) tridiagonal eigenproblem 3 Apply inverse orthogonal similarity to tridiagonal eigenvectors Nicholas Knight SIAM CSE 15 2

3 The symmetric eigenproblem Three-phase direct approach for the symmetric eigenproblem: 1 Reduce symmetric matrix to tridiagonal by orthogonal similarity 2 Solve (symmetric) tridiagonal eigenproblem 3 Apply inverse orthogonal similarity to tridiagonal eigenvectors This talk: improving phase 1 (tridiagonalization) Integrate communication-efficient QR factorization (CAQR) Nicholas Knight SIAM CSE 15 2

4 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 Return (Q, R) Nicholas Knight SIAM CSE 15 3

5 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Return (Q, R) Nicholas Knight SIAM CSE 15 3

6 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 Return (Q, R) Nicholas Knight SIAM CSE 15 3

7 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 A11 (Q L, R 1 ) = QR A 21 Return (Q, R) panel factorization Nicholas Knight SIAM CSE 15 3

8 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 A11 (Q L, R 1 ) = QR A 21 R2 = Q B L T A21 A 22 Return (Q, R) trailing matrix update Nicholas Knight SIAM CSE 15 3

9 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 A11 (Q L, R 1 ) = QR A 21 R2 = Q B L T A21 A 22 (Q R, R 3 ) = QR(B) Return (Q, R) Nicholas Knight SIAM CSE 15 3

10 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 A11 (Q L, R 1 ) = QR A 21 R2 = Q B L T A21 A 22 (Q R, R 3 ) = QR(B) I 0 Q = Q L 0 Q R Return (Q, R) Nicholas Knight SIAM CSE 15 3

11 A class of algorithms for QR factorization (Q, R) = QR(A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 A11 (Q L, R 1 ) = QR A 21 R2 = Q B L T A21 A 22 (Q R, R 3 ) = QR(B) I 0 Q = Q L 0 Q R R1 R R = 2 0 R 3 Return (Q, R) Nicholas Knight SIAM CSE 15 3

12 A class of algorithms for QR factorization (Q, R) = QR b (A) A is m-by-n with m n 1 If n = 1 Compute orthogonal Q such that Q T A = R span(e 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is b-by-b 22 A11 (Q L, R 1 ) = QR b A 21 R2 = Q B L T A21 Blocking factor b: A 22 b = b(n) {1,..., n} (Q R, R 3 ) = QR b (B) b(n) > 1 b(n) > b(b(n)) I 0 Q = Q L 0 Q R R1 R R = 2 0 R 3 Return (Q, R) Nicholas Knight SIAM CSE 15 3

13 Particular choices of blocking function b b = n/2, b = n/k for k 2 b = 1 recursive right-looking (unblocked) b = max{1, n 1} { left-looking (unblocked) b 1 n > b 1 b = n b 1 1 { right-looking (1-level blocking) b = n b 1 n > b 1 n 1 n b 1 left-looking (1-level blocking) multi-level blocking... Nicholas Knight SIAM CSE 15 4

14 Householder representation for QR Use compact representation [SVL89] for orthogonal matrices arising in QR factorization (see also [SB95]). Q = H(Y ) = I YTY T Y is unit lower trapezoidal T is invertible upper triangular Q = I - Y T Y T Nicholas Knight SIAM CSE 15 5

15 Householder representation for QR Use compact representation [SVL89] for orthogonal matrices arising in QR factorization (see also [SB95]). Q = H(Y ) = I YTY T Y is unit lower trapezoidal T is invertible upper triangular T determined by Y T T + T 1 = Y T Y In practice: keep T explicit Q = I - Y T Y T T -T + T -1 = Y T Y Nicholas Knight SIAM CSE 15 5

16 Householder representation for QR Use compact representation [SVL89] for orthogonal matrices arising in QR factorization (see also [SB95]). Q = H(Y ) = I YTY T Y is unit lower trapezoidal T is invertible upper triangular T determined by Y T T + T 1 = Y T Y In practice: keep T explicit Q = I - Y T Y T T -T + T -1 = Y T Y Q, Q T can be applied efficiently Q T A = A YT T Y T A Q T A = A - Y T T Y T A well-optimized (mat. mul.) Nicholas Knight SIAM CSE 15 5

17 Householder QR factorization (Y, R) = QR(A) A is m-by-n with m n 1 If n = 1 R Compute vector Y such that H(Y ) T A = span(e 0 1 ) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 A11 (Y L, R 1 ) = QR A ( 21 ) R2 = H(Y B L ) T A21 A 22 (Y R, R ( 3 ) = QR(B) ) 0 Y = Y L Y R R1 R R = 2 0 R 3 Return (Y, R) Nicholas Knight SIAM CSE 15 6

18 Communication-efficient Householder QR (Sca)LAPACK (P)xGEQRF use 1-level blocking to improve data locality in trailing matrix update (see, e.g., [Pug92]) Nicholas Knight SIAM CSE 15 7

19 Communication-efficient Householder QR (Sca)LAPACK (P)xGEQRF use 1-level blocking to improve data locality in trailing matrix update (see, e.g., [Pug92]) But, panel factorization can still be communication bottleneck E.g., A too tall/skinny to benefit from blocking Nicholas Knight SIAM CSE 15 7

20 Communication-efficient Householder QR (Sca)LAPACK (P)xGEQRF use 1-level blocking to improve data locality in trailing matrix update (see, e.g., [Pug92]) But, panel factorization can still be communication bottleneck E.g., A too tall/skinny to benefit from blocking Tall/skinny QR (TSQR) [DGHL12] Cheaper than Householder QR when A is sufficiently tall/skinny Comm.-avoiding Householder QR (CAQR): use TSQR as base-case Nicholas Knight SIAM CSE 15 7

21 Communication-efficient Householder QR (Sca)LAPACK (P)xGEQRF use 1-level blocking to improve data locality in trailing matrix update (see, e.g., [Pug92]) But, panel factorization can still be communication bottleneck E.g., A too tall/skinny to benefit from blocking Tall/skinny QR (TSQR) [DGHL12] Cheaper than Householder QR when A is sufficiently tall/skinny Comm.-avoiding Householder QR (CAQR): use TSQR as base-case Flops Words Messages Householder QR 2mn2 2n 3 /3 n log p CAQR p 2mn 2 2n 3 /3 p 2mn+n 2 /2 p 2mn+n 2 log p p 7 2 p log 3 p Leading-order costs of QR factorization (right-looking with 1-level blocking) of m-by-n matrix distributed over p processors in 2D fashion. Nicholas Knight SIAM CSE 15 7

22 Tall/skinny QR (TSQR) Key benefits of TSQR: sequential: read A once parallel: one reduction Q represented as tree (Y i ) i : Q = i H(Y i) Nicholas Knight SIAM CSE 15 8

23 Communication-avoiding QR (CAQR) CAQR uses TSQR for panel factorization and applies Q T to the trailing matrix by tree traversal More complicated than applying I YT T Y T. Nicholas Knight SIAM CSE 15 9

24 CAQR: Householder QR factorization using TSQR ((Y i ) i, R) = QR(A) A is m-by-n with m n 1 If A sufficiently tall/skinny ((Y i ) i, R) = TSQR(A) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty ( 22 ) A11 ((Y L,i ) i, R 1 ) = QR A 21 R2 = ( B i H(Y L,i)) T A21 A 22 ((Y R,i ) i, R 3 ) = QR(B) ( ) 0 (Y i ) i = concat (Y L,i ) i, Y R,i i R1 R R = 2 0 R 3 Return ((Y i ) i, R) Nicholas Knight SIAM CSE 15 10

25 Householder reconstruction for TSQR TSQR with Householder reconstruction (TSQR-HR) [BDG + 14]: Stably convert TSQR repr. (Y i ) i to Householder repr. Y CAQR-HR: simple and well-optimized trailing matrix update Performance portability (xgemm is likely more tuned than xtprfb) Flops Words Messages Householder QR 2mn2 2n 3 /3 n log p CAQR CAQR-HR p 2mn 2 2n 3 /3 p 2mn 2 2n 3 /3 p 2mn+n 2 /2 p 2mn+n 2 log p p 7 2mn+n 2 /2 p 2 p log 3 p 6 p log 2 p Leading-order costs of QR factorization (right-looking with 1-level blocking) of m-by-n matrix distributed over p processors in 2D fashion. Nicholas Knight SIAM CSE 15 11

26 CAQR-HR: CAQR with Householder reconstruction (Y, R) = QR(A) A is m-by-n with m n 1 If A sufficiently tall/skinny (Y, R) = TSQR-HR(A) Else A11 A Partition A = 12 such that A A 21 A 11 is square, nonempty 22 A11 (Y L, R 1 ) = QR A ( 21 ) R2 = H(Y B L ) T A21 A 22 (Y R, R 3 ) = QR(B) ( ) 0 Y = Y L Y R R1 R R = 2 0 R 3 Return (Y, R) Nicholas Knight SIAM CSE 15 12

27 Householder reconstruction Key idea: Let A = QR be a (thin) QR factorization, A = QR = (I YTY1 T )R, and rearrange to obtain an LU factorization, A1 R = Y ( TY1 T R) = L U. A 2 A-R = Y T Y T 1 R = L U Nicholas Knight SIAM CSE 15 13

28 Householder reconstruction If A 1 R is nonsingular, let Y = L where A1 R (L, U) = LU A 2 A1 R = Y ( TY1 T R). A 2 A-R = Y T Y T 1 R = L U Nicholas Knight SIAM CSE 15 14

29 Householder reconstruction If A 1 SR is nonsingular for some diagonal sign matrix S, instead compute A = ˆQ ˆR = (QS)(SR), and let Ŷ = L where A1 SR (L, U) = LU A 2 A1 SR = Ŷ ( T Ŷ 1 T SR). A 2 A-SR = Y T Y T 1 S R = L U Nicholas Knight SIAM CSE 15 14

30 Householder reconstruction It is always possible ( to) pick S such that Q 1 S is nonsingular and largest elements of Q1 S on diagonal. Q 2 Q1 S (L, U) = LU Q 2 Q1 S = Ŷ ( T Ŷ 1 T S). Q 2 Nicholas Knight SIAM CSE 15 14

31 Numerical stability TSQR-HR is as stable as Householder QR (up to constants) [BDG + 14]: Use TSQR to compute (thin) QR factorization LU factorization (well conditioned, no pivoting) Nicholas Knight SIAM CSE 15 15

32 Numerical stability TSQR-HR is as stable as Householder QR (up to constants) [BDG + 14]: Use TSQR to compute (thin) QR factorization LU factorization (well conditioned, no pivoting) ρ κ(a) A QR F I Q T Q F 1e e02 2.2e e-15 1e e04 2.3e e-15 1e e06 2.4e e-15 1e e08 2.3e e-15 1e e10 2.3e e-15 1e e12 2.2e e-15 1e e14 2.7e e-14 1e e15 2.3e e-15 Errors on tall and skinny matrices (m = 1000, b = 200) Nicholas Knight SIAM CSE 15 15

33 Conclusions Use communication-efficient QR factorization for tridiagonalization [BDG + 14]: Approach based on TSQR [AHW14]: Approach based on Cholesky-QR (w/ dynamic block-size) [FNYY14]: Approach based on Cholesky-QR (w/ iterative refinement) Nicholas Knight SIAM CSE 15 16

34 Conclusions Use communication-efficient QR factorization for tridiagonalization [BDG + 14]: Approach based on TSQR [AHW14]: Approach based on Cholesky-QR (w/ dynamic block-size) [FNYY14]: Approach based on Cholesky-QR (w/ iterative refinement) Ongoing work: 3D QR factorization Use recursive approach: b = n/2 Use 3D matrix multiplication for trailing matrix update Use fewer processors on base cases (TSQR-HR) O(p 1/6 )-fold decrease in # words O(p 1/6 )-fold increase in # messages Thank you! knight@cs.berkeley.edu Nicholas Knight SIAM CSE 15 16

35 References I T. Auckenthaler, T. Huckle, and R. Wittmann. A blocked QR-decomposition for the parallel symmetric eigenvalue problem. Parallel Computing, 40(7): , G. Ballard, J. Demmel, L. Grigori, M. Jacquelin, H. D. Nguyen, and E. Solomonik. Reconstructing Householder vectors from tall-skinny QR. In Parallel and Distributed Processing Symposium, 2014 IEEE 28th International, pages IEEE, J. Demmel, L. Grigori, M. Hoemmen, and J. Langou. Communication-optimal parallel and sequential QR and LU factorizations. SIAM Journal on Scientific Computing, 34(1):A206 A239, Nicholas Knight SIAM CSE 15 17

36 References II T. Fukaya, Y. Nakatsukasa, Y. Yanagisawa, and Y. Yamamoto. CholeskyQR2: a simple and communication-avoiding algorithm for computing a tall-skinny QR factorization on a large-scale parallel system. In Proceedings of the 5th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems, pages IEEE Press, C. Puglisi. Modification of the Householder method based on the compact WY representation. SIAM Journal on Scientific and Statistical Computing, 13(3): , X. Sun and C. Bischof. A basis-kernel representation of orthogonal matrices. SIAM Journal on Matrix Analysis and Applications, 16(4): , Nicholas Knight SIAM CSE 15 18

37 References III R. Schreiber and C. Van Loan. A storage-efficient WY representation for products of Householder transformations. SIAM Journal on Scientific and Statistical Computing, 10(1):53 57, A. Tiskin. Communication-efficient parallel generic pairwise elimination. Future Generation Computer Systems, 23(2): , Nicholas Knight SIAM CSE 15 19

A communication-avoiding parallel algorithm for the symmetric eigenvalue problem

A communication-avoiding parallel algorithm for the symmetric eigenvalue problem Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign Householder Symposium XX June