CS 770G - Parallel Algorithms in Scientific Computing

Transcription

1 References CS 770G - Parallel Algorthms n Scentfc Computng Parallel Sortng Introducton to Parallel Computng Kumar, Grama, Gupta, Karyps, Benjamn Cummngs. A porton of the notes comes from Prof. J. Demmel s CS267 course at UC Berkeley. July 18, 2001 Lecture 14 2 Issues n Sortng on Parallel Computers Compare-Exchange Where the nput and output sequences are stored. Input: unsorted sequence dstrbuted unformly among processors. Output: sorted sequence across processors. Global orderng - proc enumeraton. Compare-exchange or compare -splt on nonlocal elements. 3 a a j a, a j a, a j mn{a, a j } max{a, a j } P P j P P j P P j One element per processor. P & P j compare ther elements a & a j. Send ther element to each other. P keeps mn(a,a j ). P j keeps max(a,a j ). Communcaton tme: T comm = t s + t w. 4 1

2 1, 3 2, 4 P P j 2, 4 2, 4 1, 3 1, 3 P Compare-Splt More than one element per processor. P & P j compare ther blocks A & A j. P j 1, 2, 3, 4 Sort ther block locally. Send ther block to each other. Each proc merges the two sorted blocks, and retans the approprate half. Communcaton tme: T comm = t s + t w (n/p). P 1, 2, 3, 4 P j 1, 2 3, 4 P 5 P j Algorthm: Bubble Sort BUBBLE_SOR T (n) begn for = n -1 to 1 for j = 1 to compare - exchange(a end j - loop end - loop end O(n) per teraton + n teratons complexty = O(n 2 ). Inherently sequental -- compare adjacent pars n order.,a j j+ 1 ) 6 Odd-Even Transposton Odd-Even Transposton (cont.) Bubble sort varant. Sort n elements n n phases. Each phase requres n/2 compare-exchange operatons. Alternate between 2 phases -- odd & even. Let {a 1, a 2,, a n } be the sequence to be sorted. Odd phase: compare-exchange the pars (a 1,a 2 ), (a 3, a 4 ),, (a n-1,a n ). Even phase: compare-exchange the pars (a 2,a 3 ), (a 4, a 5 ),, (a n-2,a n-1 ). n comparsons per phase + n phases complexty = O(n 2 ). 7 Sequental algorthm: ODD_EVEN(n) begn for = 1 to n f s odd then for j = 0 to n/2-1 compare-exchange(a 2j+ 1,a 2j+ 2) end j -loop f s even then for j = 1 to n/2-1 compare-exchange(a 2j,a 2j+ 1) end j -loop end -loop end 8 2

3 Example Example (cont.) Phase 1 (odd) Phase 5 (odd) Phase 2 (even) Phase 6 (even) Phase 3 (odd) Phase 7 (odd) Phase 4 (even) Phase 8 (even) 9 10 Parallel Implementaton One element per processor. Compare-exchange operatons on pars of elements are done smultaneously. Odd phase: proc 2-1 compare-exchanges ts element wth proc 2. Even phase: proc 2 compare-exchanges ts element wth proc 2+1. Parallel Complexty In each phase, the complexty of compare-exchange = O(1). A total of n phases complexty = O(n). Sequental complexty of the best sortng algorthm = O(n log n). Hence, odd-even transposton sort s not costoptmal because processor-tme product = O(n 2 )

4 Parallel Implementaton (cont.) Parallel Performance More than one element per processor, p < n. Complexty of local sort = O(n/p log(n/p) ). P phases: (p/2 odd & p/2 even) Odd phase: proc 2-1 compare-splts ts element wth proc 2. Even phase: proc 2 compare-splts ts element wth proc Parallel run-tme: n n T p = O( log ) p p + O( n) + O( n) Speedup: S Effcency: E Ts O( nlogn) = = T n n p O( log ) + O( n) p p Cost optmal p=o(log n). p p local sort comparsons communcatons S p 1 = = p log p p 1 O( ) + O( ) logn logn 14 Dvde-and-conquer. (Average) complexty = O(n log n). Let the sequence be A[1..n]. Two steps: Dvde: gven A[q..r], dvde nto 2 subarrays A[q..s] & A[s+1..r] such that each element of A[q..s] each element of A[s+1..r]. Conquer: apply Qucksort to the subsequences. Parttonng Select a pvot x. Qucksort A subsequence contan elements x; another subsequnce contans elements > x. 15 Qucksort Algorthm QUICKSORT (A,q, r) begn f q < r then x = A[q]; s = q; for = q + 1 to r f A[] x then s = s + 1; swap(a[s], A[]); endf end -loop swap(a[q], A[s]); QUICKSORT(A,q, s); QUICKSORT(A,s + 1,r); endf end 16 4

5 Parallel Qucksort Parallel Complexty Perform qucksort on the subsequences n parallel. Start wth a sngle processor. Assgn one of the subproblems to another processor. Each of these processors sort ts array by usng qucksort and assgns one of ts subproblems to other processors. Algorthm termnates when the arrays cannot be further parttoned. Problem: partton s done by a sngle processor. In the begnnng, the complexty of partton = O(n). Hence the lower bound = O(n). Processor-tme product = O(n 2 ) not cost-optmal. Needs parallel parttonng PRAM model Parallel CRCW PRAM Model Parallel PRAM Algorthm Concurrent-read, concurrent-wrte parallel randomaccess machne. Wrte conflcts are resolved arbtrarly. Qucksort can be nterpreted as constructng a bnary tree. Pvot s the root. Elements pvot go to the left subtree. Elements > pvot go to the rght subtree. Sorted sequence obtaned by norder trasversal. 19 Select a pvot. Partton nto 2 parts. Subsequent pvot elements, one for each new subtree, are then selected n parallel. In each teraton, a level of the tree s constructed n O(1) tme. Thus, the averge complexty = depth of tree = O(log n). The sorted squence s obtaned by norder trasversal n O(1) tme. Thus, t s cost-optmal. 20 5

6 BUILD_TREE(A[1..n]) begn for BuldTree Algorthm each proc do root = ; parent = root; leftchld[] = rghtchld[] = n+ 1; end for repeat for each proc root f (A[] < A[parent ]) or (A[] = A[parent leftchld[parent ] = ; f = leftchld[parent ] then ext elseparent = leftchld[parent ]; else rghtchld[parent ] = ; f = rghtchld[parent ] then ext elseparent = rghtchld[parent ]; endf end repeat end ]and < parent ) then 21 Btonc Sort A btonc sortng network sorts n elements n O(log 2 n). The key operaton s rearrange a btonc sequence nto a sorted sequence. Btonc sequence: {a 0, a 1,, a n-1 } wth the property that ether: (1) there exsts such that {a 0,, a } s monotoncally ncreasng and {a +1,, a n-1 } s monotoncally decreasng, or (2) there exsts a cyclc shft of ndces so that (1) holds. 22 PRAM Sorts Sortng Networks Sortng on Dfferent Networks MEM p p p M P LogPSorts M P network M P Sortng on Machne X Sortng on Network Y Btonc Seq. to Increasng Seq. Let s 1 = {a 0, a 1,, a n-1 } be a btonc sequence such that a 0 a 1 a n/2-1 and a n/2 a n/2+1 a n-1. Defne 2 subsequences: s = (mn{ a, a },mn{ a, a },..., mn{ a, a }) 1 0 n/ 2 1 n/ 2+ 1 n/ 2 1 n 1 s = (max{ a, a }, max{ a, a },...,max{ a, a }) 2 0 n/ 2 1 n/ 2+ 1 n/ 2 1 n 1 In sequence s 1, there s an element b =mn{a,a n/2+ } such that all the elements before b are from the ncreasng part of the orgnal sequence, and all those after are from the decreasng part. 24 6

7 Btonc Seq. to Increasng Seq. (cont.) Btonc Seq. to Increasng Seq. (cont.) In sequence s 2, the element b =max{a,a n/2+ } s such that all the elements before b are from the decreasng part of the orgnal sequence, and all those after are from the ncreasng part. Every element of the frst sequence every elment of the second sequence. Both s 1 and s 2 are btonc sequences. Thus, the ntal problem of rearrangng a btonc sequence of sze n s reduced to that of rearrangng 2 smaller btonc sequences of sze n/2 and concatenatng the results. 25 Repeat the process recursvely untl we obtan subsequences of sze 1. At that pont, the output s sorted n monotoncally ncreasng order. Snce after each btonc splt, the sze of the problem s halved, the number of splts = log n. Sortng a btonc sequence usng btonc splts s called btonc merge. Can be mplemented easly on a network of comparators. 26 Sortng Unordered Elements Btonc Sort Algorthm A sequence of 2 elements forms a btonc sequence. Hence, any unsorted sequence s a concatenaton of btonc sequence of sze 2. Merge adjacent btonc sequences n ncreasng and decreasng order. By defnton, the sequence obtaned by concatenatng the ncreasng and decreasng sequences s btonc. By mergng larger and larger btonc sequences, we eventually obtan a btonc sequence of sze n

8 Parallel Btonc Sort Parallel Btonc Sort (cont.) Parallel Performance Parallel Sortng Comparson