Plan. Analysis of Multithreaded Algorithms. Plan. Matrix multiplication. University of Western Ontario, London, Ontario (Canada) Marc Moreno Maza

Transcription

1 Pla Aalysis of Multithreaded Algorithms Marc Moreo Maza Uiversity of Wester Otario, Lodo, Otario (Caada) CS Pla (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Matrix multiplicatio c 11 c 12 c 1 c 21 c 22 c 2 a 11 a 12 a 1 a 21 a 22 a 2 = b 11 b 12 b 1 b 21 b 22 b 2 c 1 c 2 c a 1 a 2 a b 1 b 2 b C A B We will study three approaches: a aive ad iterative oe a divide-ad-coquer oe a divide-ad-coquer oe with memory maagemet cosideratio (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27

2 Naive iterative matrix multiplicatio Naive iterative matrix multiplicatio cilk_for (it i=1; i<; ++i) { cilk_for (it j=0; j<; ++j) { for (it k=0; k<; ++k { C[i][j] += A[i][k] * B[k][j]; Work:? Spa:? Parallelism:? cilk_for (it i=1; i<; ++i) { cilk_for (it j=0; j<; ++j) { for (it k=0; k<; ++k { C[i][j] += A[i][k] * B[k][j]; Work: Θ( 3 ) Spa: Θ() Parallelism: Θ( 2 ) (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Matrix multiplicatio based o block decompositio (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Divide-ad-coquer matrix multiplicatio C 11 C 12 A 11 A 12 B 11 B 12 = C 21 C 22 A 21 A 22 B 21 B 22 A 11 B 11 A 11 B 12 A 12 B 21 A 12 B 22 = + A 21 B 11 A 21 B 12 A 22 B 21 A 22 B The divide-ad-coquer approach is simply the oe based o blockig, preseted i the first lecture. (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 // C <- C + A * B void MMult(T *C, T *A, T *B, it, it size) { T *D = ew T[*]; //base case & partitio matrices cilk_spaw MMult(C11, A11, B11, /2, size); cilk_spaw MMult(C12, A11, B12, /2, size); cilk_spaw MMult(C22, A21, B12, /2, size); cilk_spaw MMult(C21, A21, B11, /2, size); cilk_spaw MMult(D11, A12, B21, /2, size); cilk_spaw MMult(D12, A12, B22, /2, size); cilk_spaw MMult(D22, A22, B22, /2, size); MMult(D21, A22, B21, /2, size); MAdd(C, D,, size); // C += D; delete[] D; Work? Spa? Parallelism? (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27

3 Divide-ad-coquer matrix multiplicatio Divide-ad-coquer matrix multiplicatio: No temporaries! void MMult(T *C, T *A, T *B, it, it size) { T *D = ew T[*]; //base case & partitio matrices cilk_spaw MMult(C11, A11, B11, /2, size); cilk_spaw MMult(C12, A11, B12, /2, size); cilk_spaw MMult(C22, A21, B12, /2, size); cilk_spaw MMult(C21, A21, B11, /2, size); cilk_spaw MMult(D11, A12, B21, /2, size); cilk_spaw MMult(D12, A12, B22, /2, size); cilk_spaw MMult(D22, A22, B22, /2, size); MMult(D21, A22, B21, /2, size); MAdd(C, D,, size); // C += D; delete[] D; A p () ad M p (): times o p proc. for Add ad Mult. A 1 () = 4A 1 (/2) + Θ(1) = Θ( 2 ) A () = A (/2) + Θ(1) = Θ(lg ) M 1 () = 8M 1 (/2) + A 1 () = 8M 1 (/2) + Θ( 2 ) = Θ( 3 ) M () = M (/2) + Θ(lg ) = Θ(lg 2 ) M 1 ()/M () = Θ( 3 / lg 2 ) (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Divide-ad-coquer matrix multiplicatio: No temporaries! void MMult2(T *C, T *A, T *B, it, it size) { //base case & partitio matrices cilk_spaw MMult2(C11, A11, B11, /2, size); cilk_spaw MMult2(C12, A11, B12, /2, size); cilk_spaw MMult2(C22, A21, B12, /2, size); MMult2(C21, A21, B11, /2, size); cilk_spaw MMult2(C11, A12, B21, /2, size); cilk_spaw MMult2(C12, A12, B22, /2, size); cilk_spaw MMult2(C22, A22, B22, /2, size); MMult2(C21, A22, B21, /2, size); Work? Spa? Parallelism? Pla (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 void MMult2(T *C, T *A, T *B, it, it size) { //base case & partitio matrices cilk_spaw MMult2(C11, A11, B11, /2, size); cilk_spaw MMult2(C12, A11, B12, /2, size); cilk_spaw MMult2(C22, A21, B12, /2, size); MMult2(C21, A21, B11, /2, size); cilk_spaw MMult2(C11, A12, B21, /2, size); cilk_spaw MMult2(C12, A12, B22, /2, size); cilk_spaw MMult2(C22, A22, B22, /2, size); MMult2(C21, A22, B21, /2, size); MA p (): time o p proc. for Mult-Add. MA 1 () = Θ( 3 ) MA () = 2MA (/2) + Θ(1) = Θ() MA 1 ()/MA () = Θ( 2 ) Besides, savig space ofte saves time due to hierarchical memory. (Moreo Maza) Aalysis of Multithreaded Algorithms CS / (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27

4 Mergig two sorted arrays Merge sort void Merge(T *C, T *A, T *B, it a, it b) { while (a>0 && b>0) { if (*A <= *B) { *C++ = *A++; a--; else { *C++ = *B++; b--; while (a>0) { *C++ = *A++; a--; while (b>0) { *C++ = *B++; b--; Time for mergig elemets is Θ(). merge merge merge (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Parallel merge sort with serial merge (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Parallel merge sort with serial merge void MergeSort(T *B, T *A, it ) { if (==1) { else { T* C[]; cilk_spaw MergeSort(C, A, /2); MergeSort(C+/2, A+/2, -/2); Merge(B, C, C+/2, /2, -/2); Work? Spa? void MergeSort(T *B, T *A, it ) { if (==1) { else { T* C[]; cilk_spaw MergeSort(C, A, /2); MergeSort(C+/2, A+/2, -/2); Merge(B, C, C+/2, /2, -/2); T 1 () = 2T 1 (/2) + Θ() thus T 1 () == Θ( lg ). T () = T (/2) + Θ() thus T () = Θ(). T 1 ()/T () = Θ(lg ). Puy parallelism! We eed to parallelize the merge! (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27

5 Parallel merge Parallel merge A Recursive P_Merge B 0 ma = a/2 a A[ma] A[ma] Biary Search A[ma] A[ma] Recursive P_Merge 0 mb-1 mb b a b Idea: if the total umber of elemets to be sorted i = a + b the the maximum umber of elemets i ay of the two merges is at most 3/4. (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Parallel merge void P_Merge(T *C, T *A, T *B, it a, it b) { if (a < b) { P_Merge(C, B, A, b, a); else if (a==0) { retur; else { it ma = a/2; it mb = BiarySearch(A[ma], B, b); C[ma+mb] = A[ma]; cilk_spaw P_Merge(C, A, B, ma, mb); P_Merge(C+ma+mb+1, A+ma+1, B+mb, a-ma-1, b-mb); Oe should coarse the base case for efficiecy. Work? Spa? (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Aalyzig parallel merge void P_Merge(T *C, T *A, T *B, it a, it b) { if (a < b) { P_Merge(C, B, A, b, a); else if (a==0) { retur; else { it ma = a/2; it mb = BiarySearch(A[ma], B, b); C[ma+mb] = A[ma]; cilk_spaw P_Merge(C, A, B, ma, mb); P_Merge(C+ma+mb+1, A+ma+1, B+mb, a-ma-1, b-mb); Let PM p () be the p-processor ruig time of P-Merge. I the worst case, the spa of P-Merge is PM () PM (3/4) + Θ(lg ) = Θ(lg 2 ) The worst-case work of P-Merge satisfies the recurrece PM 1 () PM 1 (α) + PM 1 ((1 α)) + Θ(lg ) (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Recall PM 1 () PM 1 (α) + PM 1 ((1 α)) + Θ(lg ) for some 1/4 α 3/4. To solve this hairy equatio we use the substitutio method. We assume there exist some costats a, b > 0 such that PM 1 () a b lg holds for all 1/4 α 3/4. After substitutio, this hypothesis implies: PM 1 () a b lg b lg + Θ(lg ). We ca pick b large eough such that we have PM 1 () a b lg for all 1/4 α 3/4 ad all > 1/ The pick a large eough to satisfy the base coditios. Fially we have PM 1 () = Θ(). (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27

6 Parallel merge sort with parallel merge Parallel merge sort with parallel merge void P_MergeSort(T *B, T *A, it ) { if (==1) { else { T C[]; cilk_spaw P_MergeSort(C, A, /2); P_MergeSort(C+/2, A+/2, -/2); P_Merge(B, C, C+/2, /2, -/2); Pla Work? Spa? (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 void P_MergeSort(T *B, T *A, it ) { if (==1) { else { T C[]; cilk_spaw P_MergeSort(C, A, /2); P_MergeSort(C+/2, A+/2, -/2); P_Merge(B, C, C+/2, /2, -/2); The work satisfies T 1 () = 2T 1 (/2) + Θ() (as usual) ad we have T 1 () = Θ(log()). The worst case critical-path legth of the Merge-Sort ow satisfies T () = T (/2) + Θ(lg 2 ) = Θ(lg 3 ). The parallelism is ow Θ( lg )/Θ(lg 3 ) = Θ(/ lg 2 ). (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Tableau costructio Costructig a tableau A satisfyig a relatio of the form: The work is Θ( 2 ). A[i, j] = R(A[i 1, j], A[i 1, j 1], A[i, j 1]). (1) (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27

7 Recursive costructio A more parallel costructio I III II IV Parallel code I; cilk_spaw II; III; IV; I II IV III V VII VI VIII IX I; cilk_spaw II; III; cilk_spaw IV; cilk_spaw V; VI; cilk_spaw VII; VIII; IX; T 1 () = 4T 1 (/2) + Θ(1), thus T 1 () = Θ( 2 ). T () = 3T (/2) + Θ(1), thus T () = Θ( log 2 3 ). Parallelism: Θ( 2 log 2 3 ) = Ω( 0.41 ). (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Ackowledgemets T 1 () = 9T 1 (/3) + Θ(1), thus T 1 () = Θ( 2 ). T () = 5T (/3) + Θ(1), thus T () = Θ( log 3 5 ). Parallelism: Θ( 2 log 3 5 ) = Ω( 0.53 ). This ie-way d--c has more parallelism tha the four way but exhibits more cache complexity (more o this later). (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27 Charles E. Leiserso (MIT) for providig me with the sources of its lecture otes. Matteo Frigo (Itel) for supportig the work of my team with Cilk++ ad offerig us the ext lecture. Yuzhe Xie (UWO) for helpig me with the images used i these slides. Liyu Li (UWO) for geeratig the experimetal data. (Moreo Maza) Aalysis of Multithreaded Algorithms CS / 27