Parallel Prefix Algorithms 1. A Secret to turning serial into parallel

Transcription

1 Parallel Prefix Algorithms. A Secret to turning serial into parallel 2. Suppose you bump into a parallel algorithm that surprises you there is no way to parallelize this algorithm you say 3. Probably a variation on parallel prefix!

2 Sum Prefix Input x = (x, x 2,..., x n ) Output y = (y, y 2,..., y n ) Example y i = Σ j= x j x = (, 2, 3, 4, 5, 6, 7, 8 ) y = (, 3, 6, 0, 5, 2, 28, 36) i Prefix Functions-- outputs depend upon an initial string

3 Clearly only Serial Right?

4 Prefix Functions -- outputs depend upon an initial string Suffix Functions -- outputs depend upon a final string Other Notations +\ plus scan APL MPI_scan y = x y() = x() for i = 2 : n y(i) = x(i) + y(i-) end MATLAB: y=cumsum(x)

5 2log 2 n Parallel Steps on n Processors Algorithm:. Pairwise sum 2. Recursively Prefix 3. Pairwise Sum (Recursively Prefix)

6 Parallel Prefix prefix( [ ])=[ ] Any associative operator Pairwise sums Recursive prefix Update odds AKA: +\ (APL), cumsum(matlab), MPI_SCAN, 0 0 0

7 Notice # adds = 2n # required = n Parallelism at the cost of more work!

8 Any Associative Operation on any inputs works + Associative: (a + b) + c = a + (b + c) Sum (+) Product (*) Max Min Input: Reals All (=and) Any (= or) Input: Bits (Boolean) MatMul Inputs: Matrices

9 Fibonacci via Matrix Multiply Prefix F n+ = F n + F n- = + n- n n n F F 0 F F Can compute all F n by matmul_prefix on [,,,,,,,, ] then select the upper left entry

10 Arithemetic Modulo 2 (binary arithmetic) 0+0=0 0*0=0 0+= 0*=0 +0= *0=0 +=0 *= Add = exclusive or Mult = and

11 Carry-Look Ahead Addition (Babbage 800 s) Example Goal: Add Two n-bit Integers 0 Carry 0 First Int 0 0 Second Int Sum

12 Carry-Look Ahead Addition (Babbage 800 s) Goal: Add Two n-bit Integers Example Notation 0 Carry c 2 c c 0 0 First Int a 3 a 2 a a Second Int a 3 b 2 b b Sum s 3 s 2 s s 0

13 s n =c n- Carry-Look Ahead Addition (Babbage 800 s) Goal: Add Two n-bit Integers Example Notation 0 Carry c 2 c c 0 0 First Int a 3 a 2 a a Second Int a 3 b 2 b b Sum s 3 s 2 s s 0 c - = 0 (addition mod 2) for i = 0 : n- s i =a i + b i +c i- c i =a i b i +c i- (a i + b i ) end

14 s n =c n- Carry-Look Ahead Addition (Babbage 800 s) Goal: Add Two n-bit Integers Example Notation 0 Carry c 2 c c 0 0 First Int a 3 a 2 a a Second Int a 3 b 2 b b Sum s 3 s 2 s s 0 c - = 0 (addition mod 2) for i = 0 : n- s i =a i + b i +c i- c i =a i b i +c i- (a i + b i ) end c i a i + b i a i b i c i- = 0

15 Carry-Look Ahead Addition (Babbage 800 s) Goal: Add Two n-bit Integers Example Notation 0 Carry c 2 c c 0 0 First Int a 3 a 2 a a Second Int a 3 b 2 b b Sum s 3 s 2 s s 0 c - = 0 (addition mod 2) for i = 0 : n- s i =a i + b i +c i- c i =a i b i +c i- (a i + b i ) end s n =c n- c i a i + b i a i b i c i- = 0 Matmul prefix with binary arithmetic is equivalent to carry-look ahead! Compute c i by prefix, then s i = a i + b i +c i- in parallel

16 a b c a 2 b 2 Tridiagonal Factor Determinants (D 0 =, D =a ) (D k is the det of the kxk upper left): T = c 2 a 3 b 3 c 3 a 4 b 4 D n- D n c 4 a 5 D n = a n D n- -b n- c n- D n-2 Compute D n by matmul_prefix D n a = n -b n- c n- D n- D n- 0 D n-2 T = 3 embarrassing Parallels + prefix d b l d 2 b 2 l 2 d 3 d n =D n /D n- l n =c n /d n

17 The Myth of log n The log 2 n parallel steps is not the main reason for the usefulness of parallel prefix. Say n = 000p (000 summands per processor) Time = (2000 adds) + (log 2 P message passings) fast & embarrassingly parallel (2000 local adds are serial for each processor of course)

18 0, 000 adds + 3 communication hops total speed is as if there is no communication Myth of log n Example 80, , , 000 0, log 2 n = number of steps to add n numbers (NO!!)

19 Any Prefix Operation May Be Segmented!

20 Segmented Operations Inputs = Ordered Pairs (operand, boolean) e.g. (x, T) or (x, F) Change of segment indicated by switching T/F + 2 (y, T) (y, F) (x, T) (x + y, T) (y, F) (x, F) (y, T) (x y, F) e. g Result T T F F F T F T

21 Copy Prefix: x + y = x (is associative) Segmented T T F F F T F T

22 High Performance Fortran SUM_PREFIX ( ARRAY, DIM, MASK, SEG, EXC) T T T T T A = M = F F T T T T F T F F SUM_PREFIX(A) = SUM_SUFFIX(A) SUM_PREFIX(A, DIM = 2) = SUM_PREFIX(A, MASK = M) =

23 More HPF Segmented A = S = T T F F F F T T F F T T T T T T T F T T F F Sum_Prefix (A, SEGMENTS = S)

24 Example of Exclusive A = Sum_Prefix(A) Sum_Prefix(A, EXCLUSIVE = TRUE) (Exclusive: Don t count myself)

25 Parallel Prefix prefix( ( [ ])=[ ] Any associative operator Pairwise sums Recursive prefix Update evens AKA: +\ (APL), cumsum(matlab), MPI_SCAN, 0 0 0

26 Variations on Prefix exclusive( [ ])=[ ] )Pairwise Sums 2)Recursive Prefix 3)Update odds

27 Variations on Prefix exclusive( [ ])=[ ] Directions Left Inclusive Exc=0 Prefix The Family... Exclusive Exc= Exc Prefix )Pairwise Sums 2)Recursive Prefix 3)Update odds

28 Variations on Prefix exclusive( [ ])=[ ] Directions Left Right Inclusive Exc=0 Prefix Suffix The Family... Exclusive Exc= Exc Prefix Exc Suffix )Pairwise Sums 2)Recursive Prefix 3)Update evens

29 Variations on Prefix reduce( [ ])=[ ] Directions Left Right Left/Right Inclusive Exc=0 Prefix Suffix Reduce The Family... )Pairwise Sums 2)Recursive Reduce 3)Update odds Exclusive Exc= Exc Prefix Exc Suffix Exc Reduce

30 Variations on Prefix exclusive( [ ])=[ ] Directions Left Right Left/Right Inclusive Exc=0 Prefix Suffix Reduce The Family... )Pairwise Sums 2)Recursive Prefix 3)Update evens Exclusive Neighbor Exc Exc= Exc=2 Exc Prefix Left Multipole Exc Suffix Right " " " Exc ReduceMultipole

31 Multipole in 2d or 3d etc Notice that left/right generalizes more readily to higher dimensions Ask yourself what Exc=2 looks like in 3d Directions Left Right Left/Right Inclusive Exc=0 Prefix Suffix Reduce The Family... Exclusive Neighbor Exc Exc= Exc=2 Exc Prefix Left Multipole Exc Suffix Right " " " Exc ReduceMultipole

32 Not Parallel Prefix but PRAM Only concerned with minimizing parallel time (not communication) Arbitrary number of processors

33 Csanky s (977) Matrix Inversion Lemma : ( - ) in O(log 2 n) (triangular matrix inv) Proof Idea: A 0 - A - 0 Lemma 2: Cayley - Hamilton = C B -B - CA - B - p(x) = det (xi - A) = x n + c x n c n (c n =deta) 0 = p(a) = A n + c A n c n I A - = (A n- + c A n c n- )(-/c n ) Powers of A via Parallel Prefix ±

34 Lemma 3: Leverier s Lemma c s s 2 c 2 s 2 s 2 s. c 3 = - s 3 s k =tr(a k ) : :.. : : s n-.. s n c n s n Csanky ) Parallel Prefix powers of A 2) s k by directly adding diagonals 3) c i from lemas and 3 4) A - obtained from lemma 2 Horrible for A=3I and n>50!!

35 Matrix multiply can be done in log n steps on n 3 processors with the pram model Can be useful to think this way, but must also remember how real machines are built!