An introduction to parallel algorithms

Transcription

1 An introduction to parallel algorithms Knut Mørken Department of Informatics Centre of Mathematics for Applications University of Oslo Winter School on Parallel Computing Geilo January 20 25, /26

2 1 Introduction 2 Parallel addition 3 Computing the dot product in parallel 4 Parallel matrix multiplication 5 Solving linear systems of equations in parallel 6 Conclusion 2/26

3 Adding n numbers Suppose we have n real numbers (a i ) n i=1 their sum s. In mathematics n s = a i. i=1 and want to compute 3/26

4 Adding n numbers Suppose we have n real numbers (a i ) n i=1 their sum s. In mathematics n s = a i. i=1 and want to compute On a computer s = 0; for i = 1, 2,..., n s = s + a i ; 3/26

5 Algorithm An algorithm is a precise prescription of how to accomplish a task. 4/26

6 Algorithm An algorithm is a precise prescription of how to accomplish a task. Two important issues determine the character of an algorithm: Which operations are available to us? 4/26

7 Algorithm An algorithm is a precise prescription of how to accomplish a task. Two important issues determine the character of an algorithm: Which operations are available to us? In which order can the operations be performed? 4/26

8 Algorithm An algorithm is a precise prescription of how to accomplish a task. Two important issues determine the character of an algorithm: Which operations are available to us? In which order can the operations be performed? One at a time (sequentially). 4/26

9 Algorithm An algorithm is a precise prescription of how to accomplish a task. Two important issues determine the character of an algorithm: Which operations are available to us? In which order can the operations be performed? One at a time (sequentially). Several at once (in parallel). 4/26

10 Primitive operations We will assume that the basic operations available to us are The four arithmetic operations (with two arguments) Comparison of numbers (if-tests) Elementary mathematical functions (trigonometric, exponential, logarithms, roots,... ) 5/26

11 Primitive operations We will assume that the basic operations available to us are The four arithmetic operations (with two arguments) Comparison of numbers (if-tests) Elementary mathematical functions (trigonometric, exponential, logarithms, roots,... ) We will measure computing time (complexity) by counting the number of time steps necessary to complete all the arithmetic operations of an algorithm. 5/26

12 Order of operations In traditional (sequential) programming it is assumed that a computer can only perform one operation at a time. Adding n numbers s = 0; for i = 1, 2,..., n s = s + a i ; 6/26

13 Order of operations In traditional (sequential) programming it is assumed that a computer can only perform one operation at a time. Adding n numbers s = 0; for i = 1, 2,..., n s = s + a i ; Mathematically, however, many problems exhibit considerable freedom in the order in which operations are performed. 6/26

14 Order of operations We can add the n numbers in any order n a i = a π1 + a π2 + + a πn i=1 where (π 1,..., π n ) is any permutation of the integers 1,..., n. This can be exploited to speed up the addition if we have the possibility of performing several operations simultaneously. 7/26

15 Parallel addition Suppose that n = 10. Then we have s = a 1 + a 2 + a 3 + a 4 + a 5 + a 6 + a 7 + a 8 + a 9 + a 10, 8/26

16 Parallel addition Suppose that n = 10. Then we have s = a 1 + a }{{ 2 + a } 3 + a 4 + a }{{} 5 + a 6 + a }{{} 7 + a 8 + a }{{} 9 + a 10, }{{} a1 1 a2 1 a3 1 a4 1 a5 1 8/26

17 Parallel addition Suppose that n = 10. Then we have s = a 1 + a }{{ 2 + a } 3 + a 4 + a }{{} 5 + a 6 + a }{{} 7 + a 8 + a }{{} 9 + a 10, }{{} a1 1 a2 1 a3 1 a4 1 a5 1 = a a a a a 1 5, 8/26

18 Parallel addition Suppose that n = 10. Then we have s = a 1 + a }{{ 2 + a } 3 + a 4 + a }{{} 5 + a 6 + a }{{} 7 + a 8 + a }{{} 9 + a 10, }{{} a1 1 a2 1 a3 1 a4 1 a5 1 = a1 1 + a2 1 + a3 1 + a4 1 + a5 1, }{{}}{{}}{{} a1 2 a2 2 a3 2 8/26

19 Parallel addition Suppose that n = 10. Then we have s = a 1 + a }{{ 2 + a } 3 + a 4 + a }{{} 5 + a 6 + a }{{} 7 + a 8 + a }{{} 9 + a 10, }{{} a1 1 a2 1 a3 1 a4 1 a5 1 = a1 1 + a2 1 + a3 1 + a4 1 + a5 1, }{{}}{{}}{{} a1 2 a2 2 a3 2 = a a a 2 3, 8/26

20 Parallel addition Suppose that n = 10. Then we have s = a 1 + a }{{ 2 + a } 3 + a 4 + a }{{} 5 + a 6 + a }{{} 7 + a 8 + a }{{} 9 + a 10, }{{} a1 1 a2 1 a3 1 a4 1 a5 1 = a1 1 + a2 1 + a3 1 + a4 1 + a5 1, }{{}}{{}}{{} a1 2 a2 2 a3 2 = a1 2 + a2 2 + a3 2, }{{}}{{} a1 3 a2 3 8/26

21 Parallel addition Suppose that n = 10. Then we have s = a 1 + a }{{ 2 + a } 3 + a 4 + a }{{} 5 + a 6 + a }{{} 7 + a 8 + a }{{} 9 + a 10, }{{} a1 1 a2 1 a3 1 a4 1 a5 1 = a1 1 + a2 1 + a3 1 + a4 1 + a5 1, }{{}}{{}}{{} a1 2 a2 2 a3 2 = a1 2 + a2 2 + a3 2, }{{}}{{} a1 3 a2 3 = a a /26

22 Parallel addition This means that if we have 5 computing units at our disposal, we can add the 10 numbers in 4 time steps. In general, this technique allows us to add n numbers in log 2 n time steps if we have n/2 computing units. 9/26

23 Parallel addition This means that if we have 5 computing units at our disposal, we can add the 10 numbers in 4 time steps. In general, this technique allows us to add n numbers in log 2 n time steps if we have n/2 computing units. Adding n numbers a 0 = a; for j = 1, 2,..., log 2 n parallel: a j i/2 = aj 1 i 1 + aj 1 i, i = 2, 4, 6,..., a j 1 ; 9/26

24 Parallel addition This means that if we have 5 computing units at our disposal, we can add the 10 numbers in 4 time steps. In general, this technique allows us to add n numbers in log 2 n time steps if we have n/2 computing units. Adding n numbers a 0 = a; for j = 1, 2,..., log 2 n parallel: a j i/2 = aj 1 i 1 + aj 1 i, i = 2, 4, 6,..., a j 1 ; if a j 1 % 2 > 0 then a j 1 = aj 1 1 ; 9/26

25 Parallel addition There is a recursive version of the previous algorithm where each function call is given to a new processor. Adding n numbers (recursive version) sum(a, n) { if n > 1 then return(sum(a, 1, n//2) + sum(a, n//2 + 1, n)) else return(a 1 ); } 10/26

26 Parallel computing in practice Problem Add 1000 integers, each with 5 digits. 11/26

27 Parallel computing in practice Problem Add 1000 integers, each with 5 digits. Resources This group of people. 11/26

28 Parallel computing in practice Problem Add 1000 integers, each with 5 digits. Resources This group of people. Method 1 While more than one number: 1 I share out the numbers evenly, at least two numbers each 2 You add your numbers 3 You pass your results back to me 11/26

29 Dot product If a = (a 1, a 2,..., a n ) and b = (b 1, b 2,..., b n ) then n a b = a i b i. i=1 12/26

30 Dot product If a = (a 1, a 2,..., a n ) and b = (b 1, b 2,..., b n ) then a b = n a i b i. i=1 Provided we have n processors this can be computed in log 2 n + 1 time steps: Inner product 1 Compute the n products in parallel. 2 Compute the sum. 12/26

31 Matrix multiplication Let A R n,n and B R n,n. Then the product AB is a matrix in R n,n which is given by a 1 a AB = 2 ) (b 1 b 2 b n. a n a 1 b 1 a 1 b 2 a 1 b n a = 2 b 1 a 2 b 2 a 2 b n a n b 1 a n b 2 a n b n 13/26

32 Parallel matrix multiplication Recall that one dot product of length n can be computed in log 2 n + 1 time steps provided we have n processors at our disposal. 14/26

33 Parallel matrix multiplication Recall that one dot product of length n can be computed in log 2 n + 1 time steps provided we have n processors at our disposal. Since all the n 2 dot products in the matrix product are independent, we can also compute AB in log 2 n + 1 time steps, provided we may use n 3 processors. 14/26

34 Parallel matrix multiplication Alternative algorithm Give each of the n 2 dot products in a 1 b 1 a 1 b 2 a 1 b n a 2 b 1 a 2 b 2 a 2 b n..... a n b 1 a n b 2 a n b n to different processors. 15/26

35 Parallel matrix multiplication Alternative algorithm Give each of the n 2 dot products in a 1 b 1 a 1 b 2 a 1 b n a 2 b 1 a 2 b 2 a 2 b n a n b 1 a n b 2 a n b n to different processors. Accumulation of the dot product on one processor requires n time steps, provided we have n 2 processors. 15/26

36 Strassen s algorithm (sequential) The product of the two block matrices ( ) ( ) ( ) C 1,1 C 1,2 A1,1 A = 1,2 B1,1 B 1,2 C 2,1 C 2,2 A 2,1 A 2,2 B 2,1 B 2,2 16/26

37 Strassen s algorithm (sequential) The product of the two block matrices ( ) ( ) ( ) C 1,1 C 1,2 A1,1 A = 1,2 B1,1 B 1,2 C 2,1 C 2,2 A 2,1 A 2,2 B 2,1 B 2,2 can be computed by first calculating H 1 = A 1,1 (B 1,2 B 2,2 ), H 2 = A 2,2 (B 2,1 B 1,1 ), H 3 = (A 2,1 + A 2,2 )B 1,1, H 4 = a 2,2 (B 2,1 B 1,1 ), 16/26

38 Strassen s algorithm (sequential) The product of the two block matrices ( ) ( ) ( ) C 1,1 C 1,2 A1,1 A = 1,2 B1,1 B 1,2 C 2,1 C 2,2 A 2,1 A 2,2 B 2,1 B 2,2 can be computed by first calculating H 1 = A 1,1 (B 1,2 B 2,2 ), H 2 = A 2,2 (B 2,1 B 1,1 ), H 3 = (A 2,1 + A 2,2 )B 1,1, H 4 = a 2,2 (B 2,1 B 1,1 ), H 5 = (A 1,1 + A 2,2 )(B 1,1 + B 2,2 ), H 6 = (A 1,2 A 2,2 )(B 2,1 + B 2,2 ), H 7 = (A 1,1 A 2,1 )(B 1,1 + B 1,2 ). 16/26

39 Strassen s algorithm (sequential) The product of the two block matrices ( ) ( ) ( ) C 1,1 C 1,2 A1,1 A = 1,2 B1,1 B 1,2 C 2,1 C 2,2 A 2,1 A 2,2 B 2,1 B 2,2 can be computed by first calculating H 1 = A 1,1 (B 1,2 B 2,2 ), H 2 = A 2,2 (B 2,1 B 1,1 ), H 3 = (A 2,1 + A 2,2 )B 1,1, H 4 = a 2,2 (B 2,1 B 1,1 ), H 5 = (A 1,1 + A 2,2 )(B 1,1 + B 2,2 ), H 6 = (A 1,2 A 2,2 )(B 2,1 + B 2,2 ), H 7 = (A 1,1 A 2,1 )(B 1,1 + B 1,2 ). Then C 1,1 = H 4 + H 5 + H 6 H 2, C 1,2 = H 1,1 + H 1,2, C 2,1 = H 2,1 + H 2,2, C 2,2 = H 1 + H 1 H 3 H 7. 16/26

40 Strassen s algorithm Strassen s algorithm may be applied recursively to multiply together any two matrices. Theorem (Complexity of Strassen s algorithm) If A and B are n n matrices, then the (sequential) complexity of Strassen s algorithm is O(n 2.71 ). 17/26

41 Strassen s algorithm Strassen s algorithm may be applied recursively to multiply together any two matrices. Theorem (Complexity of Strassen s algorithm) If A and B are n n matrices, then the (sequential) complexity of Strassen s algorithm is O(n 2.71 ). More elaborate algorithms exist that have complexity O(n ) (Coppersmith-Winograd). 17/26

42 Complexity of matrix algorithms Theorem (Equivalence of matrix operations) Matrix multiplication, computation of determinants, matrix inversion and solution of a linear system of equations all have the same computational complexity (sequential and parallel). 18/26

43 Solving linear systems of equations The standard Gaussian elimination algorithm is a bit cumbersome to parallelise, we therefore consider a classical iterative algorithm instead. 19/26

44 Solving linear systems of equations Suppose we have the linear system of equations a 1,1 x 1 + a 1,2 x a 1,n x n = b 1, a 2,1 x 1 + a 2,2 x a 2,n x n = b 2,. a n,1 x 1 + a n,2 x a n,n x n = b n. 20/26

45 Solving linear systems of equations Suppose we have the linear system of equations a 1,1 x 1 + a 1,2 x a 1,n x n = b 1, a 2,1 x 1 + a 2,2 x a 2,n x n = b 2,. a n,1 x 1 + a n,2 x a n,n x n = b n. If a i,i 0 for i = 1,..., n, then we can solve equation i for x i, x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1, 20/26

46 Solving linear systems of equations Suppose we have the linear system of equations a 1,1 x 1 + a 1,2 x a 1,n x n = b 1, a 2,1 x 1 + a 2,2 x a 2,n x n = b 2,. a n,1 x 1 + a n,2 x a n,n x n = b n. If a i,i 0 for i = 1,..., n, then we can solve equation i for x i, x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1, x 2 = (b 2 a 2,1 x 1 a 2,3 x 3 a 2,n x n )/a 2,2,. 20/26

47 Solving linear systems of equations Suppose we have the linear system of equations a 1,1 x 1 + a 1,2 x a 1,n x n = b 1, a 2,1 x 1 + a 2,2 x a 2,n x n = b 2,. a n,1 x 1 + a n,2 x a n,n x n = b n. If a i,i 0 for i = 1,..., n, then we can solve equation i for x i, x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1, x 2 = (b 2 a 2,1 x 1 a 2,3 x 3 a 2,n x n )/a 2,2,. x n = (b n a n,1 x 1 a n,2 x 2 a n,n 1 x n 1 )/a n,n. 20/26

48 Solving linear systems of equations x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1 x 2 = (b 2 a 2,1 x 1 a 2,3 x 3 a 2,n x n )/a 2,2. x n = (b n a n,1 x 1 a n,2 x 2 a n,n 1 x n 1 )/a n,n If we choose an initial estimate x 0 for the solution, we can use these equations to compute a new estimate x 1, then x 2 etc. This is called Jacobi s method. 21/26

49 Solving linear systems of equations x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1 x 2 = (b 2 a 2,1 x 1 a 2,3 x 3 a 2,n x n )/a 2,2. x n = (b n a n,1 x 1 a n,2 x 2 a n,n 1 x n 1 )/a n,n If we choose an initial estimate x 0 for the solution, we can use these equations to compute a new estimate x 1, then x 2 etc. This is called Jacobi s method. Under suitable conditions on the coefficient matrix these iterations will converge to the true solution. 21/26

50 Solving linear systems of equations x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1 x 2 = (b 2 a 2,1 x 1 a 2,3 x 3 a 2,n x n )/a 2,2. x n = (b n a n,1 x 1 a n,2 x 2 a n,n 1 x n 1 )/a n,n If we choose an initial estimate x 0 for the solution, we can use these equations to compute a new estimate x 1, then x 2 etc. This is called Jacobi s method. Under suitable conditions on the coefficient matrix these iterations will converge to the true solution. Note that if the calculations are performed sequentially, we may make use of the new values of x 1, x 2,..., x i 1 when we compute x i ; this is called Gaus-Seidel iteration and converges faster than Jacobi iteration. 21/26

51 Jacobi s method in parallel Jacobi s method is perfect for parallel implementation: x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1 x 2 = (b 2 a 2,1 x 1 a 2,3 x 3 a 2,n x n )/a 2,2. x n = (b n a n,1 x 1 a n,2 x 2 a n,n 1 x n 1 )/a n,n 22/26

52 Jacobi s method in parallel Jacobi s method is perfect for parallel implementation: x 1 = (b 1 a 1,2 x 2 a 1,3 x 3 a 1,n x n )/a 1,1 x 2 = (b 2 a 2,1 x 1 a 2,3 x 3 a 2,n x n )/a 2,2. x n = (b n a n,1 x 1 a n,2 x 2 a n,n 1 x n 1 )/a n,n Each of n processors is given the task of computing one x i. Requires n time steps per iteration. 22/26

53 Hybrid Jacobi/Gaus-Seidel Suppose the system has dimension and we only have 1000 processors. Assume that an initial solution x 0 is given. 23/26

54 Hybrid Jacobi/Gaus-Seidel Suppose the system has dimension and we only have 1000 processors. Assume that an initial solution x 0 is given. First iteration: 1 For i = 1, 2,..., 100: 23/26

55 Hybrid Jacobi/Gaus-Seidel Suppose the system has dimension and we only have 1000 processors. Assume that an initial solution x 0 is given. First iteration: 1 For i = 1, 2,..., 100: 1 Set j 1 = 1000i + 1 and j 2 = 1000(i + 1) 23/26

56 Hybrid Jacobi/Gaus-Seidel Suppose the system has dimension and we only have 1000 processors. Assume that an initial solution x 0 is given. First iteration: 1 For i = 1, 2,..., 100: 1 Set j 1 = 1000i + 1 and j 2 = 1000(i + 1) 2 Give equations j 1,..., j 2 to the processors 23/26

57 Hybrid Jacobi/Gaus-Seidel Suppose the system has dimension and we only have 1000 processors. Assume that an initial solution x 0 is given. First iteration: 1 For i = 1, 2,..., 100: 1 Set j 1 = 1000i + 1 and j 2 = 1000(i + 1) 2 Give equations j 1,..., j 2 to the processors 3 Compute x 1 j 1,..., x 1 j 2 from x 1 1,..., x 1 j 1 1, x 0 j 1,..., x 0 n 23/26

58 Hybrid Jacobi/Gaus-Seidel Suppose the system has dimension and we only have 1000 processors. Assume that an initial solution x 0 is given. First iteration: 1 For i = 1, 2,..., 100: 1 Set j 1 = 1000i + 1 and j 2 = 1000(i + 1) 2 Give equations j 1,..., j 2 to the processors 3 Compute x 1 j 1,..., x 1 j 2 from x 1 1,..., x 1 j 1 1, x 0 j 1,..., x 0 n 23/26

59 Hybrid Jacobi/Gaus-Seidel Suppose the system has dimension and we only have 1000 processors. Assume that an initial solution x 0 is given. First iteration: 1 For i = 1, 2,..., 100: 1 Set j 1 = 1000i + 1 and j 2 = 1000(i + 1) 2 Give equations j 1,..., j 2 to the processors 3 Compute x 1 j 1,..., x 1 j 2 from x 1 1,..., x 1 j 1 1, x 0 j 1,..., x 0 n Repeat until convergence. 23/26

60 Conclusion The number of time steps in many (mathematical) operations can be reduced considerably if multiple operations can be performed simultaneously. 24/26

61 Conclusion The number of time steps in many (mathematical) operations can be reduced considerably if multiple operations can be performed simultaneously. Often other algorithms than the traditional sequential ones are the most efficient. 24/26

62 Conclusion The number of time steps in many (mathematical) operations can be reduced considerably if multiple operations can be performed simultaneously. Often other algorithms than the traditional sequential ones are the most efficient. The actual choice of algorithm is usually highly dependent on the type of processor and how memory is organised. 24/26

63 Parallel computing in practice Problem Add 1000 integers, each with 5 digits. 25/26

64 Parallel computing in practice Problem Add 1000 integers, each with 5 digits. Resources This group of people. 25/26

65 Parallel computing in practice Problem Add 1000 integers, each with 5 digits. Resources This group of people. Method 1 While more than one number: 1 I share out the numbers evenly, at least two numbers each 2 You add your numbers 3 You pass your results back to me 25/26

66 Parallel computing in practice Method (intelligent) Assumption: You are arranged in a strict hierarchy 1 I share out the numbers evenly 26/26

67 Parallel computing in practice Method (intelligent) Assumption: You are arranged in a strict hierarchy 1 I share out the numbers evenly 2 While more than one active computer: 26/26

68 Parallel computing in practice Method (intelligent) Assumption: You are arranged in a strict hierarchy 1 I share out the numbers evenly 2 While more than one active computer: 1 You add your numbers 26/26

69 Parallel computing in practice Method (intelligent) Assumption: You are arranged in a strict hierarchy 1 I share out the numbers evenly 2 While more than one active computer: 1 You add your numbers 2 You pass your result to someone more important than you 26/26

70 Parallel computing in practice Method (intelligent) Assumption: You are arranged in a strict hierarchy 1 I share out the numbers evenly 2 While more than one active computer: 1 You add your numbers 2 You pass your result to someone more important than you 3 I receive the result from my top assistant 26/26

71 Parallel computing in practice Method (intelligent) Assumption: You are arranged in a strict hierarchy 1 I share out the numbers evenly 2 While more than one active computer: 1 You add your numbers 2 You pass your result to someone more important than you 3 I receive the result from my top assistant 26/26

72 Parallel computing in practice Method (intelligent) Assumption: You are arranged in a strict hierarchy 1 I share out the numbers evenly 2 While more than one active computer: 1 You add your numbers 2 You pass your result to someone more important than you 3 I receive the result from my top assistant The intelligence and communication skills of our processors are important, not just their computational skills. 26/26