Algorithms PART II: Partitioning and Divide & Conquer. HPC Fall 2007 Prof. Robert van Engelen

Transcription

1 Algorithms PART II: Partitioning and Divide & Conquer HPC Fall 2007 Prof. Robert van Engelen

2 Overview Partitioning strategies Divide and conquer strategies Further reading HPC Fall

3 Partitioning Strategies Block partitioning of a 2D domain Data partitioning Perform domain decomposition to run parallel tasks on subdomains Scatter-compute-gather where local computation may require communication and scatter/gather may involve computations function f(x,y) u=g(x) v=h(y) return u+v end Thread 1 Thread 2 u=g(x) v=h(y) return u+v Task partitioning Decompose functions into independent subfunctions and execute the subfunctions in parallel HPC Fall

4 Partitioning Strategies Partitioning strategy (data partitioning): 1. Break up a given problem into P independent subproblems 2. Solve the P subproblems concurrently 3. Collect and combine the P solutions Embarrassingly parallel Is a simple form of data partitioning without initial work and no interaction between workers HPC Fall

5 Partitioning Example 1: Summation Summation of n values X = [x 1,,x n ] 1. Divide X into P equally-sized sublists X p, p = 0,,P-1 and distribute the X p sublists to the P processors 2. The processors sum the local parts s p = X p 3. Combine the local sums s = s p Algorithms: 1. Scatter list X using a scatter-tree 2. Serial summation of parts 3. Reduce local sums HPC Fall

6 Partitioning Example 1: Summation n/2 Log 2 (P) steps scatter (divide) n/8 n/4 n/4 n/8 n/8 n/8 Total amount of data transferred: n/2 log 2 (P) time Local summations: n/p steps reduce (combine) Log 2 (P) steps Total amount of data transferred: P-1 HPC Fall

7 Partitioning Example 1: Summation Communication time Scatter: t comm1 = k=1..log2(p) (t startup + 2 -k n t data ) = log 2 (P)t start + n(p-1)/p t data Reduce: t comm2 = log 2 (P) (t start + t data ) Total: t comm = 2 log 2 (P)t start + ( n(p-1)/p + log 2 (P) ) t data Computation time Local sum: Global sum: Total: t comp1 = n/p t comp2 = log 2 (P) t comp = n/p + log 2 (P) Speedup, assuming t startup = 0 Sequential time: t s = n-1 Parallel time: t P = ( n(p-1)/p + log 2 (P) ) t data + n/p + log 2 (P) Speedup: S P = t s /t P = O(n / (n + log(p))) Best speedup w/o communication: S P = O(P/log(P)) HPC Fall

8 General M-Ary Partitioning divide Example: partitioning an image, e.g. to compute histogram in parallel Second division First division time compute Third division combine 3-level 4-ary partitioning for 4 3 = 64 processors HPC Fall

9 Partitioning Example 2: Parallel Bucket Sort Bucket sort of a list of values bounded within a range [lo hi] 1. Partition n values in n/p segments 2a. Sort each segment into P small buckets (local computation) 2b. Send content of small buckets to P large buckets 3. Sort P large buckets and merge lists Unsorted values P processors Small buckets Empty small buckets into large buckets Sort content of buckets and merge lists Sorted values HPC Fall

10 Partitioning Example 2: Parallel Bucket Sort Input: list X of length n with minimum value L and maximum U Output: sorted list X def function bucket(x) = P*(x-L)/(U-L); scatter list X to local X p lists each of size n/p forall processors p = 0,,P-1 for i = 0,,n/P-1 x = X p [i] put x into small bucket b p [bucket(x)] all-to-all of small buckets b p into large buckets B p sort values in B p [0,,P-1] using a sequential sort algorithm gather X from B p into a merged sorted list HPC Fall

11 Partitioning Example 2: Parallel Bucket Sort Communication time (assuming uniform distribution in X) Scatter: t comm1 = log 2 (P)t startup + n(p-1)/p t data All-to-all: t comm2 = (P-1)(t startup + n/p 2 t data ) Gather: t comm3 = log 2 (P)t startup + n(p-1)/p t data Computation time (assuming uniform distribution in X) Small bucket sort: Large bucket sort: Speedup t comp1 = n/p Sequential time: t s = n log 2 (n/p) Parallel time: Speedup w/o communication: t comp2 = n/p log 2 (n/p) (with P buckets) t P = 2 log 2 (P)t startup + 2 n(p-1)/p t data + (P-1)(t startup + n/p 2 t data ) + n/p (1 + log 2 (n/p)) S P = O(P) HPC Fall

12 Partitioning Example 3: Barnes Hut Algorithm Direction of the force between two bodies at points p and q HPC Fall

13 .... Partitioning Example 3: Barnes Hut Algorithm Quadtree Parent computes M and C Particle at (x,y) and mass m. A square w/o particle is deleted Particles in 2D space Mass of parent is sum of masses of children Center of mass HPC Fall

14 Partitioning Example 3: Barnes Hut Algorithm for (t = 0; t < tmax; t++) { Build_tree(); Compute_Total_Mass_Center(); Compute_Force(); Update_Positions(); } Sequential time is O(n log n) Assuming P = n then t P = O(log P) (*) (**) Compute_Force() { for (i = 0; i < n; i++) Compute_Tree_Force(i,root) } Compute_Tree_Force(i,node) { if (box at node contains one particle) F = force using eq (**) else { r = distance from i to C (*) of box D = size of box if (D/r < theta) F = force using eq (**) with total M else for (all children c of box) F = F + Compute_Tree_Force(i,c); } return F; } HPC Fall

15 Divide and Conquer Divide and conquer strategy (definition by JáJá 1992) 1. Break up a given problem into independent subproblems 2. Solve the subproblems recursively and concurrently 3. Collect and combine the solutions into the overall solution In contrast to the partitioning strategy, divide and conquer uses recursive partitioning with concurrent execution to divide the problem down into independent subproblems In deeper levels of recursion the number of active processors may increase or decrease HPC Fall

16 Divide & Conquer Example 1: Parallel Recursive Matmul Block matrix multiplication in recursion by decomposing matrix in 2 2 submatrices and computing the submatrices recursively Mat matmul(mat A, Mat B, int s) { if (s == 1) C = A * B; else { s = s/2; P0 = matmul(a p,p, B p,p, s); P1 = matmul(a p,q, B q,p, s); P2 = matmul(a p,p, B p,q, s); P3 = matmul(a p,q, B q,q, s); P4 = matmul(a q,p, B p,p, s); P5 = matmul(a q,q, B q,p, s); P6 = matmul(a q,p, B p,q, s); P7 = matmul(a q,q, B q,q, s); C p,p = P0 + P1; C p,q = P2 + P3; C q,p = P4 + P5; C q,q = P6 + P7; } return C; } P0 P7 computed in parallel Level of parallelism increases with deepening recursion Suitable for shared memory systems HPC Fall

17 Divide and Conquer Example 2: Parallel Convex Hull Algorithm The planar convex hull of a set of points S={p 1,p 2,,p n } of p i =(x,y) coordinates is the smallest convex polygon that encompasses all points S on the x-y plane y x HPC Fall

18 Divide and Conquer Example 2: Parallel Convex Hull Algorithm The upper convex hull spans points {q 1,,q s } S from point q 1 with minimum x to q s with maximum x The convex hull = upper convex hull + lower convex hull Problem: Given points S = {p 1,,p n } such that x(p 1 ) < x(p 2 ) < < x(p n ), construct the upper convex hull in parallel Upper convex hull y q 1 q s x HPC Fall

19 Divide and Conquer Example 2: Parallel Convex Hull Algorithm Parallel convex hull: 1. Divide the x-sorted points S into sets S 1 and S 2 of equal size 2. Compute upper convex hull recursively on S 1 and S 2 3. Combine UCH(S 1 ) and UCH(S 2 ) by computing the upper common tangent to form UCH(S) Upper common tangent S 2 S 1 HPC Fall

20 Divide and Conquer Example 2: Parallel Convex Hull Algorithm Base case of recursion: two points, which are returned as UCH(S) The line segment (a,b) can be computed sequentially in O(log n) time with n = UCH(S 1 ) + UCH(S 2 ) using a binary search method Line segments can be implemented as linked list of points, thus UCH(S 1 ) and UCH(S 2 ) can be connected using one pointer change of a to point to b in O(1) time Parallel convex hull time complexity recurrence relation: T(n) < T(n/2) + a log n with solution: T(n) = O(log 2 n) Parallel convex hull operations recurrence relation: W(n) < 2W(n/2) + b n with solution: W(n) = O(n log n) which is cost optimal, since sequential algorithm is O(n log n) HPC Fall

21 Divide and Conquer Example 3: First-Order Linear Recurrences First-order linear recurrence y 1 = b 1 y i = a i y i-1 + b i Example applications: 2 < i < n Prefix sum y i = j=1..i b j is a special case of a first-order linear recurrence with a i = 1 (the multiplicative unit element) n-th order polynomial evaluation using Horner s rule p(x) = (((b 1 x + b 2 ) x + b 3 ) x + + b n-1 ) x + b n is a special case of a first-order linear recurrence with a i = x Solving a bi-diagonal system By = c, let a i = l i /d i b i = c i /d i then solve linear recurrence to obtain solution y d 1 l 2 d 2 l 3 d 3 l n d n y 1 y 2 y 3 y n = c 1 c 2 c 3 c n HPC Fall

22 Divide and Conquer Example 3: First-Order Linear Recurrences Rewrite y i = a i y i-1 + b i into y i = a i (a i-1 y i-2 + b i-1 ) + b i This equation defines a linear recurrence of size n/2 for even index i z 1 = b 1 z i = a i z i-1 + b i 2 < i < n/2 1. Let a i = a 2i a 2i-1 b i = a 2i b 2i-1 + b 2i 2. Solve z i recursively 3. For 1 < i < n set y i = z i/2 if i is even y i = a i z (i-1)/2 +b i if i is odd > 1 y i = b 1 if i = 1 HPC Fall

23 Recursion level Divide and Conquer Example 3: First-Order Linear Recurrences log 2 n recursive steps Parallel algorithm: linrecsolve(a[], b[], y[], n) { if (n==1) { y[1] = b[1]; return; } forall (i = 1 to n/2) { a new [i] = a[2*i]*a[2*i-1]; b new [i] = a[2*i]*b[2*i-1]+b[2*i]; } linrecsolve(a new, b new, z, n/2); forall (i = 1 to n) { if (i == 1) y[1] = b[1]; else if (even(i)) y[i] = z[i/2]; else y[i] = a[i]*z[(i-1)/2]+b[i]; } } b 1 b 1 = a 2 b 1 + b 2 b 1 = a 2 b 1 + b 2 = ((a 2 b 1 + b 2 ) a 3 + b 3 ) a 4 + b 4 b 1 = a 2 b 1 + b 2 = ((a 2 b 1 + b 2 ) a 3 + b 3 ) a 4 + b 4 = ((((a 2 b 1 + b 2 ) a 3 + b 3 ) a 4 + b 4 ) a 5 + b 5 ) a 6 + b 6 ) a 7 + b 7 ) a 8 + b 8 HPC Fall

24 Divide and Conquer Example 4: Triangular Matrix Inversion Consider Ax = b with triangular matrix A a 11 a 21 a 22 a 31 a 32 a 33 a n1 a n2 a nn Partition A into (n/2) (n/2) blocks A 1 A 2 A 3 Then A -1 is given by A A 3-1 A 2 A 1-1 A 3-1 HPC Fall

25 Divide and Conquer Example 4: Triangular Matrix Inversion Parallel algorithm: 1. Divide A into A 1, A 2, A 3 2. Recursively compute inverses of A 1 and A 3 in parallel 3. Multiply -A 3-1 A 2 A 1-1 and combine with A 1-1 and A 3-1 to get A -1 Time complexity is given by the recurrence relation T(n) = T(n/2) + c n with P=n 2 processors to compute -A 3-1 A 2 A 1-1 in O(n) operations in parallel, thus T(n) = O(n) time HPC Fall

26 Divide and Conquer Example 5: Banded Triangular Systems Consider Ax = b with banded matrix A with m=3 a 11 a 21 a 22 a 31 a 32 a 33 a 42 a 43 a 44 a 53 a 54 a 55 a 64 a 65 a 66 a 75 a 76 a 77 a 86 a 87 a 88 a 97 a 98 a 99 a 11 a 21 a 22 a 31 a 32 a 33 a 42 a 43 a 44 a 53 a 54 a 55 a 64 a 65 a 66 a 75 a 76 a 77 a 86 a 87 a 88 a 97 a 98 a 99 Define block diagonal D and inverse D -1 A 11 A A22 A D = D -1 = An/m,n/m An/m,n/m -1 HPC Fall

27 Divide and Conquer Example 5: Banded Triangular Systems Compute d = D -1 b and B = D -1 A where B i,i-1 = A ii -1 A i,i-1 d = D -1 b = d 1 d 2 d n/m B = D -1 A = I m B 21 I m B 32 I m B n/m,n/m-1 I m Solve first-order linear recurrence on m m matrices B i,i-1 x 1 = d 1 x i = -B i,i-1 x i-1 + d 1 2 < i < n/m Parallel time O(m + m log (n/m)) with P=nm processors Compute all A ii -1 (each requiring O(m) operations) in parallel with parallel matrix inversion algorithm Compute all B i,i-1 = A ii -1 A i,i-1 in O(m) operations in parallel Recurrence depth is log 2 (n/m), each step has O(m) operations HPC Fall

28 Divide and Conquer Example 6: LU of Tridiagonal Matrix Consider tridiagonal matrix LU decomposition a 1 c 1 b 2 a 2 c 2 b 3 a 3 c 3 b n a n = 1 l 2 1 l 3 1 l n 1 d 1 u 1 d 2 u 2 d 3 u 3 d n The LU decomposition A = L U satisfies a 1 = d 1 c i = u i a i = d i + l i u i-1 b i = l i d i-1 thus d 1 = a 1 d i = a i - l i u i-1 = a i - u i-1 b i /d i-1 = [ a i d i-1 - b i c i-1 ] / d i-1 HPC Fall

29 Divide and Conquer Example 6: Let LU of Tridiagonal Matrix R 1 = a a i -b i c i-1 R i = 1 0 T i = R i R i-1 R 1 From the Möbius transformation we have d i = Algorithm: Set up matrices R T T T i T i Solve first-order linear recurrence (prefix sum) of T Compute d i From the solution of d i compute l i = b i /d i-1 HPC Fall

30 Further Reading [PP2] pages [PSC] pages HPC Fall