Linear Systems of Equations by Gaussian Elimination

Transcription

1 Chapter 6, p. 1/32 Linear of Equations by School of Engineering Sciences Parallel Computations for Large-Scale Problems I

2 Chapter 6, p. 2/32 Outline

3 The Problem Consider the system a 1,1 x 1 + a 1,2 x a 1,N x N = b 1 a 2,1 x 1 + a 2,2 x a 2,N x N = b 2 a N,1 x 1 + a N,2 x a N,N x N = b N. In matrix notation: Ax = b The problem For given a matrix A and a given right-hand side b find the solution x! Chapter 6, p. 3/32

4 Chapter 6, p. 4/32 Idea 1 For n = 1,..., N 1: Use equation n for the elimination of x n from the equations n + 1,..., N. 2 For n = N,..., 1: Compute x n from the n-th equation. The method works for all linear systems with non-singular matrix A (possibly after some permutations of the order of the equations). Notations: 1 (Forward) elimination 2 Backsubstitution

5 Chapter 6, p. 5/32 An Example System x 1 + 2x 2 + 2x 3 = 3, 4x 1 + 4x 2 + 2x 3 = 6, 4x 1 + 6x 2 + 4x 3 = 10 Matrix notation Ax = x 1 x 2 = 3 6 = b x 3 10

6 Chapter 6, p. 6/32 Example: (0) (i) (ii) /

7 Chapter 6, p. 7/32 Example: Backsubstitution 1 x 3 = 1 x 3 = 1 2 4x 2 6x 3 = 6 4x 2 = 12 x 2 = 3 3 x 1 + 2x 2 + 2x 3 = 3 x 1 = = 1

8 Chapter 6, p. 8/32 Step (0): Step (i): Let Example: Matrix Notation M 1 = M 1 A = , M 1 b = M 2 = /2 1 Step (ii): M 2 M 1 A = 0 4 6, M 2 M 1 b =

9 Chapter 6, p. 9/32 Example: LU-decomposition Define U = M 2 M 1 A and L = M 1 1 M 1 2. Properties U is an upper triangular matrix. L is a lower triangular matrix with ones on the diagonal. It holds LU = M 1 1 M 1 2 M 2M 1A = A A = LU is the LU-decomposition of A.

10 Chapter 6, p. 10/32 LU-decomposition Theorem For every non-singular matrix A, there exists a permutation matrix P, an upper triangular matrix U, and a lower triangular matrix L with l n,n = 1, n = 1,..., N such that PA = LU. Corollary Once an LU-decomposition (without loss of generality, assume P = I ) is known, the system Ax = b can be solved in two steps: 1 Solve Ly = b for y; 2 solve Ux = y for x. This algorithm is called forward/backward subsitution.

11 Chapter 6, p. 11/32 Pivoting The algorithm presented above is numerically unstable. Pivoting is a strategy to stabilize elimination: Find an elimination order such that the sequence of pivot elements (the factors a n,n in the n-th elimination step) do not become too small. This can be achieved be a permutation of the rows (equations) and/or columns (unknowns) od A. For simplicity, we will not consider pivoting, that means P = I.

12 Chapter 6, p. 12/32 for k = 1:N-1 for m = k+1:n l(m,k) = a(m,k)/a(k,k); for n = k+1:n a(m,n) = a(m,n)-a(k,n)*l(m,k); The computational complexity is O(N 3 ). Technically, the arrays A, U, L are identified in memory. So A will be destroyed. The zeros and ones of L and U are not saved.

13 Chapter 6, p. 13/32 A More Detailed MM = NN = 1:N; for k = 1:N-1 pivot = a(k,k); MM = MM\{k}; NN = NN\{k}; % Copy the pivot row for n NN ar(n) = a(k,n); % Compute the multiplier column for m MM ac(m) = a(m,k) = a(m,k)/pivot; % for (m,n) MM NN a(m,n) = a(m,n)-ac(m)*ar(n);

14 Chapter 6, p. 14/32 Strategy A will be distributed on a P Q processor mesh using row (µ) and column (ν) data distributions ar must be known on all rows (cf. x) ac must be known on all columns (cf. y) pivot must be known on all processes

15 Chapter 6, p. 15/32 The on Processor (p, q) II =...; JJ =...; for k = 1:N-1; (pp,ii) = µ(k); (qq,jj) = ν(k); if p == pp & q == qq pivot = a(ii,jj); broadcast(pivot, all); if p == pp; II = II\{ii}; if q == qq; JJ = JJ\{jj}; if p == pp for j JJ; ar(j) = a(ii,j); broadcast(ar, (all,q)); if q == qq for i II; ac(i) = a(i,jj) = a(i,jj)/pivot; broadcast(ac, (p,all)); for (i,j) II JJ a(i,j) = a(i,j)-ac(i)*ar(j);

16 Chapter 6, p. 16/32 Performance Analysis: Memory On each processor, we have: I p J q + I p + J q + I p + J q + }{{}}{{}}{{}}{{}}{{}}{{} 1 a ac ar II JJ pivot Summing up over all processors: Conclusion: MN + 2QM + 2PN + PQ If P M and Q N, this is only slightly larger than the serial value MN. Otherwise, the number of memory locations may be as large as six times that of the serial version.

17 Chapter 6, p. 17/32 Time Complexity The time complexity is dominated by the elimination step. For each k, it holds t comp (p, q) 2 #II (p) #JJ(q) t a The values of #II and #JJ are heavily data distribution depent: For linear data distribution (or cyclic), processors with small (p, q) become idle, while for all steps but the last few it holds #II (P 1) = M P, #JJ(Q 1) = N Q So the load becomes heavily unbalanced. For the scatter data distribution, almost optimal load balance is obtained.

18 Chapter 6, p. 18/32 Notes on Pivoting In case of pivoting, a load balanced distribution is not known in advance. Assuming a random selection of pivot elements, any distribution will do. In practice, the assumption of randomness is seldom justified. Complete pivoting is often only slightly more expensive than partial pivoting!

19 The Problem Consider the system a 1,1 x 1 = b 1 a 2,1 x 1 + a 2,2 x 2 = b 2.. Examples of such systems: a N,1 x 1 + a N,2 x a N,N x N = b N Ly = b Ux = y Method of solution: forward substitution Chapter 6, p. 19/32

20 Chapter 6, p. 20/32 Mathematically, x 1 = b 1 /a 1,1 x m = (b m a m,1 x 1 a m,m 1 x m 1 )/a m,m, for m = 1:N x(m) = b(m); for n = 1:m-1 x(m) = x(m)-a(m,n)*x(n); x(m) = x(m)/a(m,m); Problem The algorithm is sequential in nature! m = 2,..., N

21 Chapter 6, p. 21/32 The Easiest Case Assume P = N and Q = 1 Data distribution: The m-th row of A as well as b m reside on processor m 1 Hence, processor m 1 will compute x m and pass it on to the subsequent processors, Definition An algorithm of this type is called a pipelined computation.

22 Chapter 6, p. 22/32 Implementation on processor m 1 x(m) = b(m); for n = 1:m-1 receive(x(n), m-2); s(x(n), m); for n = 1:m-1 x(m) = x(m)-a(m,n)*x(n); x(m) = x(m)/a(m,m); s(x(m), m); For performance reasons: Interleave communication and computation!

23 Chapter 6, p. 23/32 A Modified Version x(m) = b(m); for n = 1:m-1 receive(x(n), m-1); Is(x(n), m); % Nonblocking s! x(m) = x(m)-a(m,n)*x(n); x(m) = x(m)/a(m,m); s(x(m), m);

24 Chapter 6, p. 24/32 Using a Data Distribution µ Assume a row data distribution µ Instead of traversing the processors in their natural order, the order is given by µ: P = µ(1), µ(2),..., µ(n) m = µ 1 (p); x(m) = b(m); for n = 1:m-1 receive(x(n), µ(m-1)); Is(x(n), µ(m+1)); x(m) = x(m)-a(m,n)*x(n); x(m) = x(m)/a(m,m);

25 Chapter 6, p. 25/32 Performance Analysis The last process (P 1) must wait for x 1 to arrive x 1 is ready to s if the first processor (0) has done its work In an ideal world, further communication is completely overlaped by computation Running time (assuming linear data distribution): T P = t comp (0) + (P 1)(t startup + (N/P) t data ) + t comp (P 1) = N2 P 2 t a + (P 1)(t startup + N P t data) + N2 P (2 1 P )t a

26 Chapter 6, p. 26/32 Performance Analysis (cont) Serial time N Ts = (2(m 1) + 1)t a = N 2 t a m=1 Speedup (neglecting communication!) 1 S P P 2 1/P For large P, the efficiency is around 50%. Anyway, it scales good.

27 Chapter 6, p. 27/32 The Case P = 1 And Q = N Data distribution: processor n 1 holds the n-th column of A and all of b Starting point is the re-arranged sequential version: x = b; for n = 1:N x(n) = x(n)/a(n,n); for m = n+1:n x(m) = x(m)-a(m,n)*x(n);

28 Chapter 6, p. 28/32 The Parallel Version x = b; if n > 1 receive(x, n-2); x(n) = x(n)/a(n,n); for m = n+1:n x(m) = x(m)-a(m,n)*x(n); s(x, n); Performance This algorithm is a pure sequential algorithm! Its speed-up is only 1.

29 Chapter 6, p. 29/32 Comments It is easy to introduce a column data distribution ν In applications, the data distribution of A dictates which algorithm to use If A is distributed on a P Q processor mesh, the resulting algorithm is a combination of both ideas. Thus we may expect rather poor performance Surprisingly enough, forward/backward substitution may become slower than the LU-decomposition There is a welth of different parallelization strategies available in order to speed up the triangular solve A complete algorithm can be found in the appix

30 Chapter 6, p. 30/32 Solve Appix: Solve: General Case II = for k = 1:N (pp,ii) = µ(k); if pp == p; II = II {ii}; for k = 1:N if k > 1; (qm,jm) = ν(k-1); (qq,jj) = ν(k); if k < N; (qp,jp) = ν(k+1); (pp,ii) = µ(k); if q == qq if (k > 1) & (q ~= qp) receive(ii, (p,qp)); receive(b(ii), (p,qp)); cont...

31 Chapter 6, p. 31/32 Solve (cont) Appix: Solve: General Case if p == pp II = II\{ii}; x(jj) = b(ii)/a(ii,jj); s(x(jj), (all,q); else receive(x(jj), (pp,q); for i = II b(i) = b(i)-a(i,jj)x(jj); if (k < N) & (q ~= qm) s(ii, (p,qm)); s(b(ii), (p,qm));

32 Chapter 6, p. 32/32 Notes Appix: Solve: General Case The right-hand side vector b is assumed to be row distributed (that is, duplicated in every process column) The solution vector x is evaluated column distributed (that is, duplicated in every process row) These data distributions are a straightforward generalization of the simple approaches For every k, there is only one process column active at a time