Parallel Iterative Methods for Sparse Linear Systems. H. Martin Bücker Lehrstuhl für Hochleistungsrechnen

Transcription

1 Parallel Iterative Methods for Sparse Linear Systems Lehrstuhl für Hochleistungsrechnen RWTH Aachen

2 Large and Sparse

3 Small and Dense

4 Outline Problem with Direct Methods Iterative Methods Krylov Subspace Methods Selected Issues in Parallelism

5 Solution of Linear Systems A x = b A coefficient matrix size N x N regular, unique solution exists,... large sparse x, b vectors of dimension N

6 Origin of Sparse Systems Finite Element Method Finite Volume Method Finite Difference Method Further Sources as well Increase problem size N! sparsity is increased Therefore: Sparsity more and more important

7 Outline Problem with Direct Methods Iterative Methods Krylov Subspace Methods Parallelism Consider Factorization...

8 Original Ordering

9 Original Ordering

10 Reverse CMK Ordering

11 Reverse CMK Ordering

12 Column Count Ordering

13 Column Count Ordering

14 Minimum Degree Ordering

15 Minimum Degree Ordering

16 Fill-In

17 Time

18 Explicit Use of Matrix 2 Pure direct methods: Ω ( N ) storage 7 Reality check: N = 10! 600 Tera Byte Fill-in and time depend on ordering Finding ordering with minimal fill-in is hard combinatorial problem ( NP-complete ).

19 Sparse Direct Methods A. George and J.W. Liu: Computer Solution of Large Sparse Positive Definite Systems, Prentice-Hall, I.S. Duff, A.M. Erisman, and J. Reid: Direct Methods for Sparse Matrices, Clarendon Press, C.H. Bischof: Introduction to High-Performance Computing, winter 2001/02, Aachen University of Technology.

20 Outline Problem with Direct Methods Iterative Methods Krylov Subspace Methods Parallelism

21 Implicit Use of Matrix Avoid explicit use of matrix (rows & cols ops) by y A y fast matrix-vector multiplication O ( N ) for sparse matrices O ( N log N ) for dense and structured matrices

22 Classical Iterative Methods Iterative scheme with matrix splitting M x = S x + b n n-1 A = M - S -1 converges if M nonsingular and ρ (M S ) < 1. Choice of splitting! Jacobi, Gauss-Seidel,...

23 Outline Problem with Direct Methods Iterative Methods Krylov Subspace Methods Parallelism

24 Alexei Nikolaevich Krylov Maritime Engineer 300 papers and books: shipbuilding, magnetism, artillery, math, astronomy 1890: Theory of oscillating motions of the ship : Krylov subspace methods

25 Krylov Subspace Methods

26 Conjugate Gradients (CG) symmetric positive definite systems for n = 1, 2, 3, A p n-1 x = x n-1+ α n p... endfor n n-1 Optimal: x by minimizing x - x n n A Efficient: storage and work per iteration fixed

27 Generalized Minimum Residual Method (GMRES) general nonsymmetric systems for n = 1, 2, 3, A p n-1 for k = 1, n T α k n = p k v endfor... endfor

28 Let residual vector GMRES r n:= b - A x n Goal of any Krylov subspace method: r 0 n Optimal: x by minimizing b - A x n n Inefficient: storage and work per iteration! n 2

29 Classification Efficient: MatVec + O( N ) Inefficient: MatVec + O( n N ) Optimal Not Optimal

30 Classification Efficient: MatVec + O( N ) Inefficient: MatVec + O( n N ) Not Useful Optimal Not Optimal

31 Classification Efficient: MatVec + O( N ) Inefficient: MatVec + O( n N ) GMRES Optimal Not Optimal

32 Classification Efficient: MatVec + O( N ) Inefficient: MatVec + O( n N ) CG (symmetric!) Optimal Not Optimal

33 Classification Efficient: MatVec + O( N ) Inefficient: MatVec + O( n N ) Not Possible Optimal Not Optimal

34 Classification Efficient: MatVec + O( N ) Inefficient: MatVec + O( n N ) Long Recurrences Optimal Not Optimal

35 Classification Efficient: MatVec + O( N ) Inefficient: MatVec + O( n N ) Short Recurrences Optimal Not Optimal

36 Iterative Methods Y. Saad: Iterative Methods for Sparse Linear Systems, PWS Publishing, L.N. Trefethen and D. Bau, III: Numerical Linear Algebra, SIAM, H.M. Bücker: Parallel Algorithms and Software for Iterative Methods, summer 2001, Aachen University of Technology.

37 Outline Problem with Direct Methods Iterative Methods Krylov Subspace Methods Parallelism

38 Parallel Matrix-Vector Product z = A y z = a y + Σ a y i ii i k = 1 ik Distribute data and work on p processors Balancing of computational load Minimization of communication N (i,k) E k

39 Symmetric Matrix Pattern

40 Graph Representation

41 Graph Partitioning Given undirected graph G = ( V, E ), find partition of nodes V = V 1+ V Vp such that number of edges connecting nodes in different V is minimal. i Hard combinatorial problem NP-complete (for p = 2).

42 Graph Partitioning

43 Elimination of Syncs Iterative methods involve synchronization points in reduction operations such as inner products vector norms Avoid data dependencies when designing new iterative methods.

44 Convergence History r n r 0 Iteration n

45 Parallel Performance Intel Paragon, 1997 Processors

46 Parallel Performance Processors

47 Parallel Performance depends on architecture Processors

48 Summary Direct Methods! Fill-In Don t Use Classical Iterations Krylov Subspace Methods (Long vs. Short Recurrences) New Issues in Parallelism (Graph Partitioning, New Methods)

49 Graph Partitioning