An Efficient Solver for Sparse Linear Systems based on Rank-Structured Cholesky Factorization

Transcription

1 An Efficient Solver for Sparse Linear Systems based on Rank-Structured Cholesky Factorization David Bindel and Jeffrey Chadwick Department of Computer Science Cornell University 30 October 2015 (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

2 u = K \ f Great for circuit simulations, 1D or 2D finite elements, etc. Standard advice to students: Just try backslash for these problems. Standard response: What about for the 3D case? (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

3 Try PCG with a good preconditioner. Maybe start with the ones in PETSc. You ve taken Matrix Computations, right? Blah blah yadda blah... (Not an actual student) (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

4 Direct or iterative? A L CW: Gaussian elimination scales poorly. Iterate instead! Pro: Less memory, potentially better complexity Con: Less robust, potentially worse memory patterns Commercial finite element codes still use (out-of-core) Cholesky. Longer compute times, but fewer tech support hours. (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

5 Desiderata I want a code for sparse Cholesky (A = LL T ) that Handles modest problems on a desktop (or laptop?) Inside a loop, without trying my patience = Does not need gobs of memory = Makes effective use of level 3 BLAS Requires little parameter fiddling / hand-holding Works with general elliptic problems (esp. elasticity) See Sherry Li plenary (and many minisymposium talks here). (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

6 From ND to superfast ND... ND gets performance using just graph structure: 2D: O(N 3/2 ) time, O(N log N) space. 3D: O(N 2 ) time, O(N 4/3 ) space. Superfast ND reduces space/time complexity via low-rank structure. (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

7 Strategy Start with CHOLMOD (a good supernodal left-looking Cholesky) Supernodal data structures are compact Algorithm + data layout = most work in level 3 BLAS Widely used already (so re-use the API!) Incorporate compact representations for low-rank blocks Outer product for off-diagonal blocks HSS-style representations for diagonal blocks Optimize, test, swear, fix, repeat (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

8 Supernodal storage structure L D j L(C j, C j ) L D j L(C j, R j ) L O j collapsed L O j L j (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

9 Supernode factorization U D j A(C j, C j ) U O j A(R j, C j ) for each k D j do Build dense updates from L O k Scatter updates to U D j and U O j L D j cholesky(u D j ) UO j (L D j ) T L O j Initialize storage Pull Schur contributions Finish forming L D j What changes in the rank-structured Cholesky? (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

10 Off-diagonal block compression L D j V j U T j Collapsed L O j Compressed L O j L O j Collapsed off-diagonal block is a (nearly low-rank) dense matrix (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

11 Off-diagonal block compression G rand( C j, r + p) C (L O j )T G for i = 1,..., s do C (L O j )C C (L O j )T C U j = orth(c) V j = L O j U j Compress without explicit L O j : Probe (L O j )T with random G Extract orth. row basis U j L O j = V ju T j = V j = L O j U j Where do we get the estimated rank bound r? (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

12 Interaction rank Could dynamically estimate the rank of L O j. Practice: empirical rank bound α k log(k). (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

13 Optimization: Selective off-diagonal compression j 1 j 2 j 3 j 1 j 2 j 3 Compress off-diagonal blocks of sufficiently large supernodes (j 1, j 2 ). (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

14 Optimization: Interior blocks B 2 B 4 j 1 j 2 B 1 B 3 j 3 j 1 j 2 j 3 B 1 B 2 B 3 B 4 Don t store any of L O j for interior blocks (Represent as L O j = AO j (LD j ) 1 when needed) (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

15 Diagonal block compression L D j = L D j,1 0 L D j,2 L D j,3 L D j,4 0 L D j,5 0 L D j,6 L D j,7 Basic observation: off-diagonal blocks are low-rank. (H-matrix, semiseparable structure, quasiseparable structure,...) Assumes reasonable ordering of unknowns! (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

16 Diagonal block compression L D j L D j,1 0 0 Vj,2 D (UD j,2 )T L D j,3 Vj,4 D (UD j,4 )T L D j,5 0. V D j,6 (UD j,6 )T L D j,7 How do we get directly to this without forming U D j explicitly? (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

17 Forming compressed updates L D j,1 L D j,3 L D j,2 L D j,5 L D j,4 L D j,7 L D j,6 D j (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

18 Rank-structured supernode factorization Basic ingredients: Randomized algorithms form U D j Rank-structured factorization of U D j Randomized algorithm forms L O j (involves solves with LD j ) Plus various optimizations. (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

19 Example: Large deformation of an elastic block (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

20 Example: Large deformation of an elastic block Benchmark based on example from deal.ii: Nearly-incompressible hyperelastic block under compression Mixed FE formulation (pressure and dilation condensed out) Tried both p = 1 and p = 2 finite elements Two load steps, Newton on each (14-15 steps) Experimental setup: 8-core Xeon X5570 with 48 GB RAM LAPACK/BLAS from MKL 11.0 PCG + preconditioners from Trilinos (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

21 RSC vs standard preconditioners (p = 1, N = 50) Relative residual Jacobi Relative residual RSC ML ICC RSC Jacobi ICC ML Iterations 10 3 Seconds 10 3 (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

22 RSC vs standard preconditioners (p = 2, N = 35) Relative residual RSC ICC ML Jacobi Relative residual RSC ICC ML Jacobi Iterations Seconds 10 3 (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

23 Time and memory comparisons (p = 1) Solve time (s) 1, ICC ML Jacobi RSC Cholesk Memory (GB) Choles RSC n n 10 6 (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

24 Effect of in-separator ordering Relative residual Semi-sep diag relies on variable order don t want any old order! Apply recursive bisection based on spatial coords Use coordinates if known Geo. Eig. Random Else assign spectrally Iteration 10 3 (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

25 Example: Trabecular bone model ( 1M dof) Relative residual Relative residual ICC ML 10 6 RSC2RSC1 ML ICC 10 6 RSC1 RSC Iterations 10 3 Seconds 10 3 (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

26 Example: Steel flange ( 1.5M dof) Relative residual RSC1 RSC2 ML ICC Iterations 10 3 Relative residual RSC2 RSC Seconds ML ICC 10 3 (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27

27 Conclusions For more: J. Chadwick and D. Bindel. An Efficient Solver for Sparse Linear Systems Based on Rank-Structured Cholesky Factorization. (Department of Computer Science Cornell University) Rank-Structured Cholesky / 27