Overall Problem. Parallel Program Design Patterns and Strategies. Contents. 1. Patterns for Functional Decomposition

Transcription

1 Parallel Program Desgn Patterns and Strateges 1 ontents 1. Patterns for Functonal Decomposton. Patterns for Dstrbutng Tasks to Processors Mke Baley mjb@cs.oregonstate.edu 3. Patterns for Data Decomposton The goal of ths secton s to look at some of the common desgn and programmng patterns one encounters n parallel programmng and to understand some of the nuances one encounters. parallel.desgn.pptx mjb Aprl 17, 017 mjb Aprl 17, 017 The Functonal Decomposton Desgn Pattern 3 The Functonal (or Task) Decomposton Desgn Pattern 4 lmate Anmals Plants Money Overall Problem Thread 0 Thread 1 Thread Thread 3 mjb Aprl 17, 017 redt: Maxs (Sm Park) mjb Aprl 17, Thread-to-Thread Decentralzed (Peer) Broadcast Reducton Scatter Gather mjb Aprl 17, 017 Peer-threads mjb Aprl 17, 017 1

2 7 8 Manager / workers Map-Reduce Manager-thread Map-thread Accumulate-thread Worker-threads Worker-threads mjb Aprl 17, 017 mjb Aprl 17, Multcore Block Data Decomposton: 1D Heat Transfer Example 10 Ppelne Requres some sort of queue between the stages You have a steel bar. Each secton of the bar starts out at a dfferent temperature. There are no ncomng heat sources or outgong heat snks (.e., gnore boundary condtons). Ready, go! How do the temperatures change over tme? The fundamental dfferental equaton here s: T T k( ) t x where: ρ s the densty n kg/m 3 s the specfc heat capacty measured n Joules / (kg K) k s the coeffcent of thermal conductvty measured n Watts / (meter K) (These unts work because a Watt s a Joule/second.) mjb Aprl 17, 017 In plan words, ths all means that temperatures, left to themselves, try to even out. The greater the temperature dfferental, the faster the evenng-out process goes. mjb Aprl 17, 017 Numercal Methods: hangng a Dervatve nto Dscrete Arthmetc 11 Multcore Block Data Decomposton: 1D Heat Transfer Example 1 Δ T T k( ) t x k T T T t T k T 1 1 ( ) T t x x T 1 T T 1 T 1 T T 1 mjb Aprl 17, 017 As a sde note: the quantty k/(ρ) has the unlkely unts of m /sec! mjb Aprl 17, 017

3 1D Data Decomposton: Parttonng Strateges 13 Allocate as One Large ontnuous Global or Malloc ed Array 14 T 1 T T 1 T 1 T T 1? T 1 T T 1? ore #0 ore #1 ore # ore #3 Should you allocate the data as one large global-memory block (.e., shared)? Or, should you allocate t as separate arrays, each dedcated to ts own core? Does t matter?. mjb Aprl 17, 017 ore #0 ore #1 ore # ore #3 float Temps[ARRAYSIZE]; float *Temps = (float *)malloc( ARRAYSIZE*szeof(float) ); float *Temps = new float[ ARRAYSIZE ]; << allocate a new[ ] array the same way >> Pck one way of allocatng global or heap data omp_set_num_threads( 4 ); for( nt t = 0; t < NUM_TIME_STEPS; t++ ) { #pragma omp parallel for default(none), shared(temperatures) for( nt = 1; < ARRAYSIZE-1; ++ ) { What happens when computng at the boundares? << compute T usng T -1, T, and T +1 >> Two cores are accessng the new[ ] = Temps[ ] + T ; same cache lne. } False Sharng! Oregon State << Unversty copy the new[ ] array to the Temps[ ] array >> } mjb Aprl 17, 017 Allocate as Separate Sub-arrays 15 1D ompute-to-ommuncate Rato 16 T 1 T T 1? T 1 T T 1? ore #0 ore #1 ore # ore #3 We could make each sub-array as a thread-local (.e., prvate) varable. Ths would put each sub-array on each thread s ndvdual stack. But, let s not do that just n case these arrays mght be large enough to overflow the stack. Although, f we dd, t wouldn t change ths story. Be sure to start each sub-array on ts own cache lne boundary. (See cache notes.) But, now when we << compute T usng T -1, T, and T +1 >> at the boundares, T -1 or T +1 mght be n another sub-array. So, we need some logc to reach nto the other sub-array to get the adjacent temperature. It s no longer as easy as sayng Temps[-1] or Temps[+1].. mjb Aprl 17, 017 Intracore computng Intercore communcaton ompute : ommuncate rato = N : where N s the number of compute cells per core In the above drawng, ompute : ommuncate s 4 : mjb Aprl 17, 017 How do more ores Interact wth the ompute-to-ommuncate Rato? 17 Performance as a Functon of Number of Nodes 18 In ths case, wth 4 cores, ompute : ommuncate = 4 : In ths case, wth 8 cores, ompute : ommuncate = : Thnk f t as a Goldlocks and the Three Bears sort of thng. :-) Too lttle ompute : ommuncate and you are spendng all your tme sharng data values across threads and dong too lttle computng MegaNodes omputed Per Second Too much ompute : ommuncate and you are not spreadng out your problem among enough threads to get good parallelsm. It s dffcult to fnd the sweet spot wthout runnng experments mjb Aprl 17, 017 # of Nodes to ompute # of Threads mjb Aprl 17, 017 3

4 Performance as a Functon of Number of Threads 19 D Heat Transfer Equaton 0 MegaNodes omputed Per Second T k( T T ) t x y T k T T ( ) t x y k T1, jt, jt1, j T, j1 T, jt, j1 T, j t x y j # of Threads # of Nodes mjb Aprl 17, 017 mjb Aprl 17, 017 D Doman (Data) Decomposton In addton to the ssues of sze of the compute block, you also have ssues of drecton. D *,Block D Block,* D Block, Block 1 mjb Aprl 17, 017 Drecton Issue: Decomposton Order Matters (thnk cache) float Array[A][B]; B B 1 A In D problems, ths s often (but not 1 B 1 always) thought of as: float Array[NY][NX];... A 1 0 A 1 1 A 1 A 1 3 A 1... A1 B1 mjb Aprl 17, 017 D ompute-to-ommuncate Rato 3 3D Heat Transfer Equaton 4 k( T T T ) t x y z k T1, j, kt, j, kt1, j, k T, j1, kt, j, kt, j1, k T, j, k1 T, j, kt, j, k1 T, j, k t x y z T k T T T ( ) t x y z k Intracore computng Intercore communcaton ompute : ommuncate rato = N : 4N = N : 4 where N s the dmenson of compute nodes per core j The D ompute : ommuncate rato s sometmes referred to as Area-to-Permeter mjb Aprl 17, 017 mjb Aprl 17, 017 4

5 Y 3D Doman (Data) Decomposton 5 Drecton Issue: Decomposton Order Matters (thnk cache) 6 Z X float Array[A][B][]; B A 3D Block, *, * 3D *,Block, * 3D *,*,Block In 3D problems, ths s often (but not always) thought of as: float Array[NZ][NY][NX]; 3D Block, Block, Block mjb Aprl 17, 017 mjb Aprl 17, 017 3D ompute-to-ommuncate Rato 7 ompute : ommuncate rato = N 3 :6N = N : 6 where N s the dmenson of compute nodes per core In 3D the ompute : ommuncate rato s sometmes referred to as Volume-to-Surface mjb Aprl 17, 017 5