Parallel Programming. Parallel algorithms Linear systems solvers

Transcription

1 Parallel Programming Parallel algorithms Linear systems solvers

2 Terminology System of linear equations Solve Ax = b for x Special matrices Upper triangular Lower triangular Diagonally dominant Symmetric 2010@FEUP Linear Systems Solvers 2

3 Upper Triangular @FEUP Linear Systems Solvers 3

4 Lower Triangular @FEUP Linear Systems Solvers 4

5 Diagonally Dominant @FEUP Linear Systems Solvers 5

6 Symmetric @FEUP Linear Systems Solvers 6

7 Back Substitution Used to solve upper triangular system Tx = b for x Methodology: one element of x can be immediately computed Use this value to simplify system, revealing another element that can be immediately computed Repeat 2010@FEUP Linear Systems Solvers 7

8 Back Substitution 1x 0 +1x 1 1x 2 +4x 3 = 8 2x 1 3x 2 +1x 3 = 5 2x 2 3x 3 = 0 2x 3 = @FEUP Linear Systems Solvers 8

9 Back Substitution 1x 0 +1x 1 1x 2 +4x 3 = 8 2x 1 3x 2 +1x 3 = 5 2x 2 3x 3 = 0 x 3 = 2 2x 3 = @FEUP Linear Systems Solvers 9

10 Back Substitution 1x 0 +1x 1 1x 2 = 0 2x 1 3x 2 = 3 2x 2 = 6 2x 3 = @FEUP Linear Systems Solvers 10

11 Back Substitution 1x 0 +1x 1 1x 2 = 0 2x 1 3x 2 = 3 x 2 = 3 2x 2 = 6 2x 3 = @FEUP Linear Systems Solvers 11

12 Back Substitution 1x 0 +1x 1 = 3 2x 1 = 12 2x 2 = 6 2x 3 = @FEUP Linear Systems Solvers 12

13 Back Substitution 1x 0 +1x 1 = 3 x 1 = 6 2x 1 = 12 2x 2 = 6 2x 3 = @FEUP Linear Systems Solvers 13

14 Back Substitution 1x 0 = 9 2x 1 = 12 2x 2 = 6 2x 3 = @FEUP Linear Systems Solvers 14

15 Back Substitution x 0 = 9 1x 0 = 9 2x 1 = 12 2x 2 = 6 2x 3 = @FEUP Linear Systems Solvers 15

16 Pseudocode for i n 1 down to 0 do x [ i ] b [ i ] / a [ i, i ] for j 0 to i 1 do b [ j ] b [ j ] x [ i ] a [ j, i ] endfor endfor Time complexity: (n 2 ) 2010@FEUP Linear Systems Solvers 16

17 Data Dependence Diagram We cannot execute the outer loop in parallel. We can execute the inner loop in parallel. Linear Systems Solvers 17

18 Row-oriented Algorithm Associate primitive task with each row of A and corresponding elements of x and b During iteration i, task i must compute x i and broadcast its value During iteration i, the task associated with row j computes new value of b j Agglomerate using rowwise interleaved striped decomposition 2010@FEUP Linear Systems Solvers 18

19 Interleaved Decompositions Rowwise interleaved striped decomposition Columnwise interleaved striped decomposition Process with rank r takes rows (columns) where i % p == r 2010@FEUP Linear Systems Solvers 19

20 Complexity Analysis Each process performs an average of n / (2p) iterations of loop j for each i iteration A total of n i iterations in all Computational complexity: (n 2 /p) One broadcast per i iteration Communication complexity: (n log p) 2010@FEUP Linear Systems Solvers 20

21 Gaussian Elimination Used to solve Ax = b when A is dense Reduces Ax = b to upper triangular system Tx = c Back substitution can then solve Tx = c for x 2010@FEUP Linear Systems Solvers 21

22 Gaussian Elimination 4x 0 +6x 1 +2x 2 2x 3 = 8 2x 0 +5x 2 2x 3 = 4 4x 0 3x 1 5x 2 +4x 3 = 1 8x 0 +18x 1 2x 2 +3x 3 = @FEUP Linear Systems Solvers 22

23 Gaussian Elimination 4x 0 +6x 1 +2x 2 2x 3 = 8 3x 1 +4x 2 1x 3 = 0 +3x 1 3x 2 +2x 3 = 9 +6x 1 6x 2 +7x 3 = @FEUP Linear Systems Solvers 23

24 Gaussian Elimination 4x 0 +6x 1 +2x 2 2x 3 = 8 3x 1 +4x 2 1x 3 = 0 1x 2 +1x 3 = 9 2x 2 +5x 3 = @FEUP Linear Systems Solvers 24

25 Gaussian Elimination 4x 0 +6x 1 +2x 2 2x 3 = 8 3x 1 +4x 2 1x 3 = 0 1x 2 +1x 3 = 9 3x 3 = @FEUP Linear Systems Solvers 25

26 Iteration of Gaussian Elimination i Elements that will not be changed Pivot row Elements already driven to 0 Elements that will be changed i 2010@FEUP Linear Systems Solvers 26

27 Numerical Stability Issues If pivot element close to zero, significant roundoff errors can result Gaussian elimination with partial pivoting eliminates this problem In step i we search rows i through n-1 for the row whose column i element has the largest absolute value Swap (pivot) this row with row i 2010@FEUP Linear Systems Solvers 27

28 Gauss elimination algorithm for i=0 to n-1 loc[i] = i for i=0 to n-1 magnitude = 0 for j=i to n-1 if a[ loc[ j ], i ] > magnitude magnitude = a[ loc[ j ], i ] pivot = j exchange( loc[ i ], loc[ pivot ] ) //initialize pivot array // for each row // find pivot for j=i+1 to n-1 // reduce each row below i factor = a[ loc[ j ], i ] / a[ loc[ i ], i ] for k=i+1 to n // reduce each element after i a[ loc[ j ], k ] = a[ loc[ j ], k ] - a[ loc[ i ], k ] * factor for i=n-1 downto 0 // back substitution x[ i ] = a[ loc[ i ], n] / a[ loc[ i ], i ] for j=0 to i-1 a[ loc[ j ], n ] = a[ loc[ j ], n ] x[ i ] * a[ loc[ j ], i ] 2010@FEUP Linear Systems Solvers 28

29 Row-oriented Parallel Algorithm Associate primitive task with each row of A and corresponding elements of x and b A kind of reduction needed to find the identity of the pivot row (called tournament) Tournament: want to determine identity of row with largest value, rather than the largest value itself Could be done with two all-reductions MPI provides a simpler, faster mechanism 2010@FEUP Linear Systems Solvers 29

30 MPI_MAXLOC, MPI_MINLOC MPI provides reduction operators MPI_MAXLOC, MPI_MINLOC Provide datatype representing a (value, index) pair 2010@FEUP Linear Systems Solvers 30

31 MPI (value,index) Datatypes MPI_Datatype MPI_2INT MPI_DOUBLE_INT MPI_FLOAT_INT MPI_LONG_INT MPI_LONG_DOUBLE_INT MPI_SHORT_INT Meaning Two ints A double followed by an int A float followed by an int A long followed by an int A long double followed by an int A short followed by an int 2010@FEUP Linear Systems Solvers 31

32 Example Use of MPI_MAXLOC struct { double value; int index; } local, global;... local.value = fabs(a[j][i]); local.index = j;... MPI_Allreduce (&local, &global, 1, MPI_DOUBLE_INT, MPI_MAXLOC, MPI_COMM_WORLD); 2010@FEUP Linear Systems Solvers 32

33 Second Communication per Iteration i k a[j][i] a[j][k] j a[picked][i] a[picked][k] picked 2010@FEUP Linear Systems Solvers 33

34 Communication Complexity Complexity of tournament: (log p) Complexity of broadcasting pivot row: (n log p) A total of n - 1 iterations Overall communication complexity: (n 2 log p) 2010@FEUP Linear Systems Solvers 34

35 Column-oriented Algorithm Associate a primitive task with each column of A and another primitive task for b During iteration i, task controlling column i determines pivot row and broadcasts its identity During iteration i, task controlling column i must also broadcast column i to other tasks Agglomerate tasks in an interleaved fashion to balance workloads Isoefficiency same as row-oriented algorithm 2010@FEUP Linear Systems Solvers 35

36 Comparison of Two Algorithms Both algorithms evenly divide workload Both algorithms do a broadcast each iteration Difference: identification of pivot row Row-oriented algorithm does search in parallel but requires all-reduce step Column-oriented algorithm does search sequentially but requires no communication Row-oriented superior when n relatively larger and p relatively smaller 2010@FEUP Linear Systems Solvers 36

37 Problems with these Algorithms They break parallel execution into computation and communication phases Processes not performing computations during the broadcast steps Time spent doing broadcasts is large enough to ensure poor scalability Linear Systems Solvers 37

38 Pipelined, Row-Oriented Algorithm Want to overlap communication time with computation time We could do this if we knew in advance the row used to reduce all the other rows Let s pivot columns instead of rows! In iteration i we can use row i to reduce the other rows. 2010@FEUP Linear Systems Solvers 38

39 Gauss elimination with column pivoting for i=0 to n-1 loc[i] = i for i=0 to n-1 magnitude = 0 for j=i to n-1 if a[ i, loc[ j ] ] > magnitude magnitude = a[ i, loc[ j ] ] pivot = j exchange( loc[ i ], loc[ pivot ] ) //initialize pivot array // for each row // find pivot inside the row for j=i+1 to n-1 // for each row j below i factor = a[ j, loc[ i ] ] / a[ i, loc[ i ] ] for k=i to n // reduce each column element including i a[ j, loc[ k ] ] = a[ j, loc[ k ] ] - a[ i, loc[ k ] ] * factor for i=n-1 downto 0 // back substitution x[ loc[ i ] ] = a[ i, n] / a[ i, loc[ i ] ] for j=0 to i-1 a[ j, n ] = a[ j, n ] x[ loc[ i ] ] * a[ j, loc[ i ] ] 2010@FEUP Linear Systems Solvers 39

40 Communication Pattern 0 Reducing Using Row 0 Row @FEUP Linear Systems Solvers 40

41 Communication Pattern 0 Reducing Using Row Reducing Using Row 0 Row @FEUP Linear Systems Solvers 41

42 Communication Pattern 0 Reducing Using Row Reducing Using Row 0 Row 0 2 Reducing Using Row @FEUP Linear Systems Solvers 42

43 Communication Pattern 0 Reducing Using Row 0 3 Reducing Using Row 0 1 Reducing Using Row 0 2 Reducing Using Row @FEUP Linear Systems Solvers 43

44 Communication Pattern 0 Reducing Using 3 Row 0 Reducing Using 1 Row 0 2 Reducing Using Row @FEUP Linear Systems Solvers 44

45 Communication Pattern 0 Reducing Using 3 Row 0 Reducing Using 1 Row 1 Row 1 2 Reducing Using Row @FEUP Linear Systems Solvers 45

46 Communication Pattern 0 Reducing Using 3 Row 0 Reducing Using 1 Row 1 Row 1 2 Reducing Using Row @FEUP Linear Systems Solvers 46

47 Communication Pattern 0 Row 1 Reducing Using 3 Row 1 Reducing Using 1 Row 1 2 Reducing Using Row @FEUP Linear Systems Solvers 47

48 Communication Pattern 0 Reducing Using Row 1 3 Reducing Using Row 1 1 Reducing Using Row 1 2 Reducing Using Row @FEUP Linear Systems Solvers 48

49 Analysis Total computation time: (n 3 /p) Total message transmission time: (n 2 ) When n large enough, message transmission time completely overlapped by computation time Message start-up not overlapped: (n) Parallel overhead: (np) 2010@FEUP Linear Systems Solvers 49

50 Sparse Systems Gaussian elimination not well-suited for sparse systems Coefficient matrix gradually fills with nonzero elements Result Increases storage requirements Increases total operation count Linear Systems Solvers 50

51 Example of Fill Linear Systems Solvers 51

52 Iterative Methods Iterative method: algorithm that generates a series of approximations to solution s value Require less storage than direct methods Since they avoid computations on zero elements, they can save a lot of computations 2010@FEUP Linear Systems Solvers 52

53 Linear Systems Solvers 53 Jacobi Method i j k j j i i a k i x a b x i i ) (, 1 1, i j k j j i i a k i x a b x i i ) (, 1 1, Values of elements of vector x at iteration k+1 depend upon values of vector x at iteration k Gauss-Seidel method: Use latest version available of x i

54 Jacobi Method Iterations Start: x 0 = (0,0) 4 1 x Solution: x = (2,3) 3 4 x x 3 2 x x @FEUP Linear Systems Solvers 54

55 Rate of Convergence Even when Jacobi method and Gauss- Seidel methods converge on solution, rate of convergence often too slow to make them practical We will move on to an iterative method with much faster convergence Linear Systems Solvers 55

56 Conjugate Gradient Method A is positive definite if for every nonzero vector x and its transpose x T, the product x T Ax > 0 If A is symmetric and positive definite, then the function q( x ) 2 1 x T Ax x T b c has a unique minimizer that is the solution to Ax = b Conjugate gradient is an iterative method that solves Ax = b by minimizing q(x) 2010@FEUP Linear Systems Solvers 56

57 Conjugate Gradient Convergence x 2 x Finds value of n-dimensional solution in at most n iterations x @FEUP Linear Systems Solvers 57

58 Conjugate Gradient Computations Matrix-vector multiplication Inner product (dot product) Matrix-vector multiplication has higher time complexity Must modify previously developed algorithm to account for sparse matrices Linear Systems Solvers 58

59 Algorithm b, x, d, g, t - size n vectors (linear system Ax = b) for i=0 to n-1 d[ i ] = x[ i ] = 0 g[ i ] = - b[ i ] // initializations for j=1 to n // iterations d1 = Inner_Product(g, g) g = Matrix_Vector_Product(A, x) for i=0 to n-1 g[ i ] = g[ i ] b[ i ] n1 = Inner_Product(g, g) if n1 < e break for i=0 to n-1 d[ i ] = - g[ i ] + (n1 / d1) * d[ i ] n2 = Inner_Product(d, g) t = Matrix_Vector_Product(A, d) d2 = Inner_Product(d, t) s = - n2 / d2 for i=0 to n-1 x[ i ] = x[ i ] + s * d[ i ] g k = Ax k-1 b d k = -g k + (g kt g k / g k-1t g k-1 )*d k-1 s k = - d k-1t g k / d kt Ad k x k = x k-1 + s k d k 2010@FEUP Linear Systems Solvers 59

60 Rowwise Block Striped Decomposition of a Symmetrically Banded Matrix Decomposition Matrix (a) (b) 2010@FEUP Linear Systems Solvers 60

61 Representation of Vectors Replicate vectors Need all-gather step after matrix-vector multiply Inner product has time complexity (n) Block decomposition of vectors Need all-gather step before matrix-vector multiply Inner product has time complexity (n/p + log p) 2010@FEUP Linear Systems Solvers 61

62 Summary Solving systems of linear equations Direct methods Iterative methods Parallel designs for Back substitution Gaussian elimination Conjugate gradient method Linear Systems Solvers 62