Multilevel low-rank approximation preconditioners Yousef Saad Department of Computer Science and Engineering University of Minnesota

Transcription

1 Multilevel low-rank approximation preconditioners Yousef Saad Department of Computer Science and Engineering University of Minnesota SIAM CSE Boston - March 1, 2013

2 First: Joint work with Ruipeng Li Work supported by NSF 2 SIAM-CSE March 1, 2013

3 Introduction Preconditioned Krylov subspace methods offer a good alternative to direct solution methods Especially for 3D problems Compromise between performance and robustness... But there are challenges: Highly indefinite systems [Helmholtz, Maxwell,...] Highly ill-conditioned systems [structures,..] Problems with extremely irregular nonzero pattern Recent: impact of new architectures [many core, GPUs] 3 SIAM-CSE March 1, 2013

4 Introduction (cont.) Main issue in using GPUs for sparse computations: Huge performance degradation due to irregular sparsity Matrices: Matrix -name N NNZ FEM/Cantilever 62,451 4,007,383 Boeing/pwtk 217,918 11,634,424 Performance of Mat-Vecs on NVIDIA Tesla C1060 Single Precision Double Precision Matrix CSR JAD DIA CSR JAD DIA FEM/Cantilever Boeing/pwtk SIAM-CSE March 1, 2013

5 Sparse Forward/Backward Sweeps Next major ingredient of precond. Krylov subs. methods ILU preconditioning operations require L/U solves: x U 1 L 1 x Sequential outer loop. for i=1:n for j=ia(i):ia(i+1) x(i) = x(i) - a(j)*x(ja(j)) end end Parallelism can be achieved with level scheduling: Group unknowns into levels Unknowns x(i) of same level can be computed simultaneously 1 nlev n 5 SIAM-CSE March 1, 2013

6 ILU: Sparse Forward/Backward Sweeps Very poor performance [relative to CPU] Matrix N CPU GPU-Lev Mflops #lev Mflops Boeing/bcsstk36 23, , FEM/Cantilever 62, , COP/CASEYK 696, COP/CASEKU 208, Prec: miserable :-) GPU Sparse Triangular Solve with Level Scheduling Very poor performance when #levs is large A few things can be done to reduce the # levels but perf. will remain poor 6 SIAM-CSE March 1, 2013

7 So... 7 SIAM-CSE March 1, 2013

8 Either GPUs must go... 8 SIAM-CSE March 1, 2013

9 or ILUs must go... 9 SIAM-CSE March 1, 2013

10 Or perhaps: Alternative preconditioners? What would be a good alternative? Wish-list: A preconditioner requiring few irregular computations Something that trades volume of computations for speed If possible something that is robust for indefinite case Good candidate: Multilevel Low-Rank (MLR) approximate inverse preconditioners 10 SIAM-CSE March 1, 2013

11 Related work: Work on HSS matrices [e.g., JIANLIN XIA, SHIVKUMAR CHAN- DRASEKARAN, MING GU, AND XIAOYE S. LI, Fast algorithms for hierarchically semiseparable matrices, Numerical Linear Algebra with Applications, 17 (2010), pp ] Work on H-matrices [Hackbusch,...] Work on balanced incomplete factorizations (R. Bru et al.) Work on sweeping preconditioners by Engquist and Ying. Work on computing the diagonal of a matrix inverse [Jok Tang and YS (2010)..] 11 SIAM-CSE March 1, 2013

12 Low-rank Multilevel Approximations Starting point: symmetric matrix derived from a 5-point discretization of a 2-D Pb on n x n y grid A = A = A 1 D 2 D 2 A 2... D D α A α D α+1 D α+1 A α D ny A ny ( ) A11 A 12 A 21 A 22 ( A11 A22) + ( A 21 A 12 ) 12 SIAM-CSE March 1, 2013

13 Corresponding splitting on FD mesh: SIAM-CSE March 1, 2013

14 A 11 R m m, A 22 R (n m) (n m) In the simplest case second matrix is: ( ) A11 A 12 = A 21 A 22 ( A11 A22) + I I Write 2nd matrix as: I I = + I + I I I I I E T = I I E E T 14 SIAM-CSE March 1, 2013

15 Above splitting can be rewritten as A = (A + EE T ) }{{} B EE T B := ( B1 B2) A = B EE T, R n n, E := ( E1 E 2 ) R n n x, Note: B 1 := A 11 + E 1 E T 1, B 2 := A 22 + E 2 E T SIAM-CSE March 1, 2013

16 Shermann-Morrison formula: {}}{ A 1 = B 1 + B 1 E( I E T B 1 E) 1 E T B 1 X A 1 B 1 + B 1 EX 1 E T B 1 X = I E T B 1 E Note: E R n n x, X R n x n x n x = number of points in separator [O(n 1/2 ) for 2-D mesh, O(n 2/3 ) for 3-D mesh] Use in a recursive framework Similar idea was used for computing the diagonal of the inverse [J. Tang YS 10] 16 SIAM-CSE March 1, 2013

17 Multilevel Low-Rank (MLR) algorithm Method: Use lowrank approx. for B 1 E B 1 E U k V T k, U k R n k, V k R n x k, Replace B 1 E by U k Vk T in X = I (ET B 1 )E: X G k = I V k U T k E, ( Rn x n x ) Leads to... Preconditioner: M 1 = B 1 + U k [V T k G 1 k V k]u T k Use recursivity 17 SIAM-CSE March 1, 2013

18 Note: From A 1 = B 1 [I + EX 1 E T B 1 ] could define: M 1 1 = B 1 [I + EG 1 k V ku T k ]. [rationale: approximation made on one side only ] It turns out M 1 and M are equal! We have: M 1 = B 1 + U k H k U T k, with H k = V T k G 1 k V k. No need to store V k : Only keep U k and H k (k k). We can show : H k = (I U T k EV k) 1... and : H k is symmetric 18 SIAM-CSE March 1, 2013

19 Recursive multilevel framework ( A i = B i + E i Ei T, B Bi1 i. Bi2) Next level, set A i1 B i1 and A i2 B i2 Repeat on A i1, A i2 Last level, factor A i (IC, ILU) Binary tree structure: SIAM-CSE March 1, 2013

20 Generalization: Domain Decomposition framework Domain partitioned into 2 domains with an edge separator Ω 2 Ω 1 Matrix can be permuted to: P AP T = ˆB 1 ˆF 1 ˆF 1 T C 1 X X T ˆF 2 T C 2 ˆB 2 ˆF 2 Interface nodes in each domain are listed last. 20 SIAM-CSE March 1, 2013

21 Each matrix ˆB i is of size n i n i (interior var.) and the matrix C i is of size m i m i (interface var.) P AP T = Let: E α = ( B1 B2) and α used for balancing 0 αi 0 X T α EE T with B i = then we have: { D1 = α 2 I D 2 = 1 α 2 X T X. ( ˆB i ˆF 1 Better results when using diagonals instead of αi ˆF T i C i + D i 21 SIAM-CSE March 1, 2013 )

22 Theory: 2-level analysis for model problem Interested in eigenvalues γ j of A 1 B 1 = B 1 EX 1 E T B 1 when A = Pure Laplacean.. They are: γ j = β j = α j = β j 1 α j, j = 1,, n x with: n y /2 k=1 n y /2 k=1 4 ( sin 2 sin 2 sin 2 n y kπ n y +1 kπ n y +1 + sin2 sin 2 n y kπ n y +1 kπ n y +1 + sin2 ) 2, jπ 2(n x +1) jπ 2(n x +1). 22 SIAM-CSE March 1, 2013

23 Decay of the γ j s when nx = ny = β j sqrt(β j ) 1 α j γ j Note β j are the singular values of B 1 E. In this particular case 3 eigenvectors will capture 92 % of the inverse whereas 5 eigenvectors will capture 97% of the inverse. 23 SIAM-CSE March 1, 2013

24 EXPERIMENTS

25 Experimental setting Hardware: Intel Xeon X5675 processor (12 MB Cache, 3.06 GHz, 6-core) C/C++; Intel Math Kernel Library (MKL,version 10.2) Stopping criteria: r i 10 8 r 0 Maximum number of iterations: SIAM-CSE March 1, 2013

26 2-D/3-D model problems (theory) u cu = ( x 2 + y 2 + c ) e xy in (0, 1) 2, + Dirichlet BC FD discret.: n x = n y = 256 Eigenvalues of B 1 i E i X 1 i i = 0, 1, 3 Rapid decay. Ei T B 1 i Lev 1 Lev 2 Lev SIAM-CSE March 1, 2013

27 3-D elliptic PDE u cu = 6 c ( x 2 + y 2 + z 2) in (0, 1) 3, + Dirichlet BC FD discret.: n x = n y = 32, nz = 64 Eigenvalues of B 1 i E i X 1 i i = 0, 1, 3 Rapid decay. Ei T B 1 i Lev 1 Lev 2 Lev SIAM-CSE March 1, 2013

28 Tests: SPD cases SPD cases, pure Laplacean (c = 0 in previous equations) MLR + PCG compared to IC + PCG 2-D problems: #lev= 5, rank= 2 3-D problems: #lev= 5, 7, 10, rank= 2 28 SIAM-CSE March 1, 2013

29 Grid N ICT-CG MLR-CG fill p-t its i-t fill p-t its i-t K K , 048K K K ,097K Set-up times for MLR preconditioners are higher Bear in mind the ultimate target architecture [SIMD...] 29 SIAM-CSE March 1, 2013

30 Symmetric indefinite cases c > 0 in u cu; i.e., shifted by si. 2D case: s = 0.01, 3D case: s = 0.05 MLR + GMRES(40) compared to ILDLT + GMRES(40) 2-D problems: #lev= 4, rank= 5, 7, 7 3-D problems: #lev= 5, rank= 5, 7, 7 ILDLT failed for most cases Difficulties in MLR: #lev cannot be large, [no convergence] inefficient factorization at the last level (memory, CPU time)

31 Grid ILDLT-GMRES MLR-GMRES fill p-t its i-t fill p-t its i-t F F F F F F F 31 SIAM-CSE March 1, 2013

32 General symmetric matrices - Test matrices MATRIX N NNZ SPD DESCRIPTION Andrews/Andrews 60, ,154 yes computer graphics pb. Williams/cant 62,451 4,007,383 yes FEM cantilever UTEP/Dubcova2 65,025 1,030,225 yes 2-D/3-D PDE pb. Rothberg/cfd1 70,656 1,825,580 yes CFD pb. Schmid/thermal1 82, ,458 yes thermal pb. Rothberg/cfd2 123,440 3,085,406 yes CFD pb. Schmid/thermal2 1,228,045 8,580,313 yes thermal pb. Cote/vibrobox 12, ,700 no vibroacoustic pb. Cunningham/qa8fk 66,127 1,660,579 no 3-D acoustics pb. Koutsovasilis/F2 71,505 5,294,285 no structural pb. 32 SIAM-CSE March 1, 2013

33 Generalization of MLR via DD DD: PartGraphRecursive from METIS balancing with diagonals higher ranks used in two problems (cant and vibrobox) Show SPD cases first then non-spd 33 SIAM-CSE March 1, 2013

34 MATRIX ICT/ILDLT MLR-CG/GMRES fill p-t its i-t k lev fill p-t its i-t Andrews cant F Dubcova cfd thermal cfd F thermal SIAM-CSE March 1, 2013

35 MATRIX ICT/ILDLT MLR-CG/GMRES fill p-t its i-t k lev fill p-t its i-t vibrobox F qa8fk F F SIAM-CSE March 1, 2013

36 Conclusion Promising approach Many more avenues to explore: Nonsymmetric case, Implementation on GPUS, Storage for 3D case SIAM-CSE March 1, 2013