EE/CSCI 451: Parallel and Distributed Computation

Transcription

1 EE/CSCI 451: Parallel and Distributed Computation Lecture #19 3/28/2017 Xuehai Qian University of Southern California 1

2 From last class PRAM Shared memory access Example PRAM algorithms BSP model Today Example PRAM Algorithms Prefix sum List ranking Finding maximum Sorting Outline 2

3 See for example References COMP 633: Parallel Computing PRAM Algorithms Siddhartha Chatterjee, Jan Prins, Fall 2009 Introduction to Parallel Computing, (Second Edition) Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar Explict Mutli-Threading (XMT): A PRAM-On-Chip Vision Uzi Vishkin, University of Maryland 3

4 Prefix sum problem (also called parallel prefix or scan) Input: 1. an input sequence x = x #,, x & of elements for an set T 2. a binary associative operator : T T T. Output: 2 sequence s = s #,, s &, for 1 k n, s 2 = 34# Serial solution Analysis: Example 2: Prefix Sum s 5 = 0 (zero element) s 37# = s 3 x 37#, 0 i n T 9 n = O(n) W n = O(n) a b c = (a b) c x 3. 4

5 Application of Prefix Sum Carry Lookahead adder + x &A# y &A# x 5 y 5 binary numbers z & z &A# z 5 x 3 y 3 FA Sum z 3 c 3 Carry input c 3 = f(x 3A#, y 3A#,, x 5, y 5, c 5 ) Prefix computation 5

6 Prefix Sum (PRAM Solution) (1) Algorithm 3 SCAN(T) if n = 1 then s # x # return s endif forall i 1: n/2 pardo y 3 x K3A# x K3 enddo < z #,, z & K > SCAN(< y #,, y & K >, ) forall i 1: n do if even(i) then s 3 z 3 K elsif i = 1 then s # x # else endif enddo return s 6 s 3 z (3A#) K x 3 Recursive call n/2 values Combine

7 Prefix Sum (PRAM Solution) (2) x # x K x P x Q x R x S x T x U y # y K y P y Q z # z K z P z Q Recursive call using y Result of the call s # s K s P s Q s R s S s T s U 7

8 Prefix Sum (PRAM Solution) (3) Example x # x K x P x Q x R x S x T x U Input y i z i s i

9 Prefix Sum (PRAM Solution) (4) Theorem: Algorithm 3 correctly computes the prefix sum of sequence x with time complexity O log n and work complexity O n, n = 2 k. Proof (correctness): 1) Base case: k = 0, which is correct by line 2. 2) Assume the correctness for inputs of size 2 2, and consider an input of size 2 27#. z 3 = y # y 3 = x # x K3, and If i = 2j (line 10), then s 3 = z i = x # x Ki = x # x 3 ; If i = 2 (line 11), then s 3 = x # ; If i = 2j + 1 (line 12), then s 3 = z i x 3 = x # x 3A# x 3 ; 9

10 Prefix Sum (PRAM Solution) (5) Theorem: Algorithm 3 correctly computes the prefix sum of sequence x with time complexity O log n and work complexity O n, n = 2 k. Proof (resource bounds): T n = T & K + O(1) W n = W & K + O(n) T n = O log n and W n = O n 10

11 PRAM Processor index Each Processor knows its index For example: a variable " myindex " This can be used for accessing various locations in memory For example: Processor(i) accesses i no row of an array Or Processor (i, j) access (i, j) no entry of a two dimensional array 11

12 Example 3: List Ranking List ranking problem Input: a list of pointers x 0, x 1,, x n 1, x 0 =NULL (end of the list) Note: initially, ith processor does not know it is the ith from the end of the list Output: rank of the element denoted count i, 0 i n 1 (number of elements in the list from that element till the end of the list, including that element) Null i End of the list Output: count(i)

13 List Ranking PRAM Solution (1) Algorithm PJ Given x 0, x 1,, x(n 1) [ Pointers] n processors, the i no processor is responsible for the i no pointer x(i) 1. for i = 0 to n 1 do in parallel 2. count i 1 3. while there exists x i Null 4. for i = 0 to n 1 pardo 5. if x i Null then 6. count i count i + count x i 7. x i x x i 8. end if 9. end for All PEs execute synchronously Will repeat log K n times Pointer Jumping 13

14 List Ranking PRAM Solution (2) Example for n = 8 Count: Null Step 0 Count: Step 1 Count: Step 2 Count: Step 3 (log K 8 = 3) 14

15 List Ranking PRAM Solution (3) CREW PRAM Theorem: Algorithm PJ computes the rank of each element in O(log n) time. T & n = O(log n) This technique is also called Pointer Jumping 15

16 List Ranking PRAM Solution (4) Analysis (with p = n) Total parallel time: T ~ n = O log n Total work done: W n = O n log n All n PEs active over log n cycles 16

17 Example 4: Finding Maximum (1) Simple EREW algorithm Input: (Max1) a set of distinct numbers x 0, x 1,, x n 1 Output: the maximum of these numbers Use a binary tree 17

18 Example 4: Finding Maximum (2) (Max1) Given n numbers, x 0, x 1,, x(n 1) p = n processors 1. for i = 0 to log n 1 do 2. for j = 0 to n 1 do in parallel 3. if j%2 37# = 0 4. x j = max (x j, x j ) 5. endif 6. endfor 7. endfor x 0 x 1 x 2 x 3 Analysis For p = n, T n = O log n W n = O n log n Note: We do not need all n processors in each level (can make the algorithm work optimal) 18

19 1. Assign log n numbers to each processor 2. for i = 0 to p 1 do in parallel 3. p(i) = max of assigned numbers 4. endfor 5. for i = 0 to log p 1 do 6. for j = 0 to p 1 do in parallel 7. if j%2 37# = 0 //mod operation // 8. p j = max (p j, p j ) 9. endif 10. endfor 11. endfor Example 4: Finding Maximum (3) Work Optimal Max 1 Given n numbers, x 0, x 1,, x(n 1) p = n/ log n processors 19 1 n/ log n blocks log n Note: At the beginning of Line 5, there are p numbers left.

20 Example 4: Finding Maximum (4) Work Optimal Max 1 1 n/ log n blocks log n p = n/ log n T n = O(log n) W n = O(p log n) = O(n) Work optimal 20

21 i A Faster Algorithm (Max2) (1) An algorithm on the COMMON CRCW model Common CRCW: 1 processors can write into the same memory location in a step All values written into a location in a cycle should be the same n 2 processors O(1) time solution j 1, if x i x(j) Bigger i, j = Š 0, otherwise Note: If x i is the maximum element, then row(i) is all 1 s This is the only row with all 1 s. 21

22 A Faster Algorithm (Max2) (2) Max2 algorithm j 0 x(i) is not Max 1. for i = 0 to n 1 pardo 2. A i = 1 i 3. Each processor P(i, j) pardo 4. compute Bigger(i, j) n 2 ops in parallel 5. if Bigger i, j = 0 then 6. A i = 0 Concurrent write possible 7. end 8. Each processor P(i) pardo 9. if A i = 1, then output x(i) Exactly one output 10.end 22

23 A Faster Algorithm (Max2) (3) Example: x 0 = 2, x 1 = 1, x 2 = 4, x 3 = 5 A P(0,0) 1. for i = 0 to n 1 pardo 2. A i = 1 3. Each processor P(i, j) pardo 4. compute Bigger(i, j) i j 5. if Bigger i, j = 0 then 6. A i = 0 7. end 8. Each processor P(i) pardo 9. if A i = 1, then output x i 10.end

24 A Faster Algorithm (Max2) (4) Analysis T n = O 1 W n = O n K p = n K Common CRCW model 24

25 An Improved Algorithm (Max2 ) Max2 algorithm (less total work) n x 0, x 1, n blocks Size: n 1. Apply Max2 in parallel to get n output values 2. Apply Max2 for those n values O 1 time p = n K = n O 1 time p = n n K = n n In total, T n = O 1 W n = O n n 25

26 Input: a sequence of distinct numbers X = [x 0, x 1,, x n 1 ] Output: rearranged sequence in ascending order A solution: merge sort Serial Merge Sort (X) 1. if X == 1, return X 2. else Example 5: Sorting 3. {X # = Serial Merge Sort(x 0, x 1,, x & K 1 ) 4. X K = Serial Merge Sort(x & K, x & K 5. Merge(X #, X K ) } 6. end if + 1,, x n 1 ) 26

27 Serial Merge Sort Merge Analysis 0 ( & K 1) n 2 T n # n = 2 T # 2 T # 1 = 1 + O n n 1 T # n = O n log n 27

28 Parallel Merge Sort (X) 1. if X == 1, return X 2. else pardo 3. {X # = Parallel Merge Sort(x 0, x 1,, x & 1 ) K 4. X K = Parallel Merge Sort(x &, x & + 1,, x n 1 ) } K K 5. end pardo 6. Merge(X #, X K ) 7. end if Parallel Merge Sort Parallel? p = n? (lowest level of the tree) EREW model? 28

29 Parallel merge sort with serial merge T ~ n = T ~ T 1 = 1 n 2 + O(n) Analysis T ~ n = O n, p = n W n = 2 W n 2 + O(n) W 1 = 1 W n = O(n log n) Note: we can do parallel merge, i.e., we can merge two & elements sorted K sequence into a sorted sequence faster by using more processors. (Later bitonicmerge network) 29

30 PRAM algorithm: T ~ (n) Network Model (a realistic model) p processors PRAM to Real Machines Communication primitives Random access read Random access write Time in network model to perform the same algorithm = O(T ~ ž (n) T ~ (n))? T ~ ž (n) Loss in performance 30

31 PRAM Conclusion Synchronous execution model Simple model Analysis Thinking parallel Time # of Processors Total Work PRAM Pthreads? Shared variables? Asynchronous execution 31

32 Backup Slides 32

33 Another Algorithm (Max3) (1) Max3 (x) n = x If n = 1 Return x 1. for i = 0 to n 1 pardo 2. {m i = Max3 X i n,, i n + n 1 } 3. end for 4. Maximum = Max2(m) Max2 using n values O 1 time, W T n = T n + O 1 = O(loglog n) W n = nw n + O n = O(n loglog n) Note: Total number of processors used p = O(n) Recursive call n = O(n) 33

34 Another Algorithm (Max3) (2) Analysis Assume n = 2 K Tree with variable # of children at each level Root has 2 K = n children Each node at the i-th level has 2 K children for 0 i k 1 Each node at level k has two children 34

35 Another Algorithm (Max3) (3) Ex. n = 2 K = 16, k = 2 Level 0, # of children = 2 K = 4 Level 1, # of children = 2 K =2 Level 2, # of children = k=2 n = 16 35

36 Another Algorithm (Max3) (4) Summary T n = O(log log n) W n = O n log log n p = n Not Work Optimal 36

37 Work Optimal Algorithm (Max4) (1) n log log n x 0, x 1, Size: log log n Each block has loglog n elements, & ««& blocks Max4 algorithm 1. Use 1 processor per block to find max of each block. & ««& elements lefttotal # of processors = n/ log log n 2. Run Max3 on the resulting array of size & ««& 37

38 Work Optimal Algorithm (Max4) (2) Analysis 1. Use 1 processor per block to find the maximum of each block p = n/ log log n Time Work O(log log n) O(n) 2. Run Max3 on the resulting array of size Problem size is & ««& & ««&, so we have p = n/ log log n Time O log log & ««& = O(log log n) Work O & log log( & ) = O(n) ««& ««& 38

39 Work Optimal Algorithm (Max4) (3) To conclude: Summary T n = O(loglog n) W n = O(n) p = & ««& Note: For Max3 W n = O(n loglog n) 39

40 Accelerated Cascading (1) Fast, work optimal Faster, not work optimal Combine Fast, work optimal Max4 idea is called Accelerated Cascading Start with (work) optimal algorithm initially to reduce problem size Then use faster (work) suboptimal algorithm on smaller problem n input Fast Smaller problem Faster Solution 40

41 Accelerated Cascading (2) In Max4 Algorithm: Stage 1: (Fast, work optimal) Start with binary tree algorithm with input size n Each level up the tree, reduce input size by factor of 2 In log log log n levels, the input size reduced to & K ± = & ²³ ²³ & W n = O n Stage 2: (Faster, suboptimal) and T n = O(logloglogn) Then switch to faster algorithm with n µ = W n = O n loglogn = O n T n = O loglogn µ 41 = O(loglogn) & ²³ ²³ & processors

42 EREW model Max1 p = & «& Common CRCW model Max Finding Summary (1) Input: n distinct numbers T n = O log n 42 W n = O n p T n W n Max2 n K O 1 O n K Max2 n n O 1 O n n Max3 n O log log n O n log log n Max4 n loglog n O log log n O n

43 Assume n = 2 K Max3 Details (1) Tree with variable # of children at each level Root has 2 K = n children Each node at the i-th level has 2 K children for 0 i k 1 Each node at level k has two children 43

44 Ex. n = 2 K = 16, k = 2 Max3 Details (2) Root has 2 K = 2 K = 4 children Each node at the i-th level has 2 K children for 0 i k 1 Each node at level 2 has two children Level 0, # of children = 4 Level 1, # of children = 2 Level 2, # of children = 2 n = 16 44

45 Max3 Details (3) Some Properties The depth of the tree is k. Since n = 2 K, k = O(log log n) The number of nodes at the i-th level is 2 K AK, for 0 i < k (Can prove this by induction) 45

46 Bottom up, level by level Max3 Details (4) The Algorithm At every level, compute maximum of all the children of an internal node O(1) time algorithm (Max2) Time complexity = O(log log n) [the depth of the tree] 46

47 Max3 Details (5) Total Work Max2 (O(1) time algorithm) needs O(m K ) processors for m elements Node at i-th level: 2 K children Total work for each node at the i-th level: O(2 K ) K 2 K AK nodes at the i-th level Total work for the i-th level is: O(2 K ) K 2 K AK = O 2 K = O(n) For O(log log n) levels, the total work W n = O(n log log n). (suboptimal) Total # of processor p = n (amount of work at i-th level) 47