A New Approach to Increase Parallelism for Dependent Loops

Size: px

Start display at page:

Download "A New Approach to Increase Parallelism for Dependent Loops"

Dorthy Gilmore
6 years ago
Views:

1 A New Apprach t Increase Parallelism fr Dependent Lps Yeng-Sheng Chen, Department f Electrical Engineering, Natinal Taiwan University, Taipei 06, Taiwan, Tsang-Ming Jiang, Arctic Regin Supercmputing Center, University f Alaska, Fairbanks, AK 99775, and Sheng-De Wangj, Department f Electrical Engineering, Natinal Taiwan University, Taipei 06, Taiwan ABSTRACT: Lps are the primary surce f parallelism in parallel prcessing. Tw iteratins in a lp are flw dependent if the results cmputed at ne iteratin are used by the ther. Otherwise, they are independent. Independent iteratins in lps can be scheduled in any rders r partitined t any prcessrs withut explicit synchrnizatins. Fr dependent iteratins, they are partitined int sets (ne r many) such that iteratins in different sets are independent (e.g., the minimum distance methd). These sets can be executed in parallel withut explicit synchrnizatins. The degree f parallelism is the number f sets. This paper presents a new apprach which can significantly increase the parallelism by adding apprpriate synchrnizatins. The implementatin feasibility and perfrmance benefits f this apprach are demnstrated n the CRAY MPP which has very fast synchrnizatin mechanisms. Intrductin Lps prvide the mst fruitful surce f parallelism fr parallel prcessing systems. S, it is well recgnized that ne f the mst crucial prblems fr parallel executin f prgrams n multiprcessr systems is t partitin lps int independent grups s as t explit parallelism within lps [2-6], [8-4]. Since synchrnizatin verhead amng prcessrs will deterirate the perfrmance, many researches are fcused n the develpment f techniques fr partitining nested lps s that when the lps are executed n synchrnizatin amng prcessrs is incurred [3], [8], [2]. Minimum distance methd is a typical example. It partitins lps int ttally independent sets f cmputatins. Each set f cmputatins is then assigned t a prcessr. All the sets can be executed in parallel withut explicit synchrnizatin. Hwever, in practical implementatin n distributed memry multiprcessr systems, n synchrnizatin amng prcessrs des nt imply n data cmmunicatin amng prcessrs. If the cmputatins that can be executed in parallel are nt aggregated in a regular regin f fixed size and shape, then it is very difficult t partitin the data s that there is n data cmmunicatin amng prcessrs. Mstly, data distributin can be dne nly in blck r cyclic manner [7]. S, fr an arbitrary independent set f cmputatins that are executed in a particular prcessr, it is very difficult t distribute all the crrespnding data t the same prcessr s that there is n need t access data lcated in ther prcessrs. In ther wrds, althugh all the cmputatins are partitined int independent sets, in practice, there will exist data cmmunicatins amng prcessrs. Observing the abve facts, in this paper, we pay ur attentins t anther imprtant factr which can enhance perfrmance expliting mre parallelism. We add barrier synchrnizatins int lps t increase the degree f parallelism. Barrier synchrnizatins are very efficient in the distributed memry multiprcessr systems that have hardware supprt fr them. Fr example, Cray T3D MPP has such a mechanism []. We prpse an methd t aggregate as many independent cmputatins as pssible int a blck fr maximizing parallelism. At first, all the cmputatins that are independent when the lps just begin their executin are identified. They are called initially independent cmputatins. We shw that all the initially independent cmputatins can be aggregated int rectangular blcks. S, based n them, the riginal lps can be partitined int rectangular blcks t maximize parallelism. The crrespnding cmputatins f a blck is assigned t different prcessrs fr parallel executin. A barrier CUG 995 Fall Prceedings 33

2 synchrnizatin is added in the end f each blck, and all blcks are executed sequentially. In the prpsed methd, since each partitined regin is blck-shaped, the parallel cde generatin is straightfrward and easy. And, it is very likely that when the lps are partitined int blcks, there is better chance t distribute data int prcessrs s that data cmmunicatin amng different prcessrs is minimized. (Mstly, data distributin can be dne nly in blck r cyclic manner.) Sme experiments are cnducted n CRAY T3D MPP. As will be discussed in Sectin Fur, it demnstrates the implementatin feasibility and perfrmance benefits f ur prpsed methd. 2 Partitining Dependent Lps 2. Basic terminlgy The prgram mdel cnsidered in this paper is the lps with unifrm inter-iteratin dependences [2-6, 8-4]. That is, the dependence pattern is the same fr each iteratin f lps. Such lps are called unifrm dependence lps. Like mst ther related wrk, a lp iteratin is cnsidered as the basic scheduling unit. A lp nest with n levels frms an n-dimensinal iteratin space, which is a subspace f Z n bunded by the lp bunds. This space is cmpsed f pints each f which represents an iteratin f the nested lps. Each pint in the iteratin space is identified by the index vectr (p, p 2,, p n ) T if its crrespnding lp iteratin is f lp index (p, p 2,, p n ). Definitin. Unifrm dependence lp nests: A unifrm dependence lp nest L n (V, D) is perfectly nested lps f depth n where. V is the set f indexed pints each f which represents ne lp iteratin and is called a cmputatin; 2. D is the dependence matrix in which each clumn is a dependence vectr [32] that describes the inter-iteratin dependence f the lps; 3. all the cmpnents f the dependence vectrs are cnstants; 4. v V, v takes ne unit f time t execute; and 5. v V, v depends n {w w V, v-w is a dependence vectr in D} Withut lss f generality, we assume that the lp nest has been nrmalized. That is, fr each lp level i, i n, the lwer bund f lp index is 0 and the lp index increment is. 2.2 Minimum Distance Methd The minimum distance methd [3, 8] partitins the iteratin space f a lp nest L int independent sets f cmputatins. Let these sets are P i, i k, k N. The basic idea behind this methd is that p P i, p must be a linear cmbinatin f the dependence vectrs f L frm an initial rigin-pint p 0 P i. That is, with matrix ntatin, it can be btained as P i =P i,0 +A D, where A is an integer matrix and D is the dependence matrix. T generate the parallel cdes that crrespnd t the partitined lps, the dependence matrix D is transfrmed int an upper triangular matrix D s s that the shifting distance in each dimensin can be easily derived. Hwever, the transfrmatin frm D t D s is frmulated as a NP prblem. D Hllander [3] imprves it by using a direct and inexpensive partitining algrithm based n unimdular transfrmatins and verrides the drawbacks f it when the dependence vectrs are nt linearly independent. Althugh it is shwn that the minimum-distance methd can generate the maximal number f independent sets in the unbunded iteratin space, the independent cmputatins are nt aggregated int regular regins f fixed size and shape. S, the transfrmed parallel cdes cannt be efficiently executed due t data distributin and prcessr lad balance prblems. As discussed in the previus sectin, fr an arbitrary set f cmputatins, it is very difficult t partitin the crrespnding data int the same prcessr. Thus, althugh, theretically, minimum distance methd can partitin lps int independent sets, in practice, data cmmunicatins amng prcessrs are necessary. Many researches have shwn that the abve prblems can be relieved by partitining lps int blcks (tiles) [2, 6, 0,, 4]. Tiles becme natural units fr scheduling parallel tasks. The cmpiler can easily generate parallel cdes fr tiles s that prcessrs are lad balanced when executing the cdes. Mrever, mst the existing parallelizing cmpilers supprt blck r cyclic (blck size is ne) data distributin [7]. S, in additin t parallelism explitatin, partitining lps int blcks is als a very imprtant ptimizatin issue fr efficiently executing prgrams in distributed memry parallel systems. In this paper, we will partitin lps int rectangular blcks while maximizing parallelism. 3 The Prpsed methd 3. Sme definitins Intuitively, t explit the ptimal (maximal) parallelism within a lp nest is t identify all the cmputatins that can be executed in the first time unit, and then all thse can be executed in the secnd time unit,..., and s n. S, identifying all the independent cmputatins at the first time unit prvides a gd start pint fr expliting maximal parallelism. Definitin 2. Initially independent cmputatins, initially independent cmputatin sets: A cmputatin is initially independent iff all its real dependences are satisfied when the lps just begin their executin. An initially independent cmputatin set (IICS) is an independent cmputatin set in which every cmputatin is initially independent. An initially independent cmputatin set is maximal iff it aggregates all the initially independent cmputatins. Definitin 3. Bundary blck cmputatin sets: A bundary blck cmputatin set R=[r, r 2,, r n ] f a lp nest L n is a set f cmputatins where ( i n) 34 CUG 995 Fall Prceedings

3 (a) r i Z { }; ( dentes the unknwn lp upper bund.) (b) a cmputatin (p, p 2,, p n ) T is in R iff it is initially independent and meets the fllwing cnditins: () if r i <0 then u i +r i + p i u i, (u i dentes the upper bund f the ith lp) (2) if r i >0 then 0 p i r i -, (3) if r i = then 0 p i u i ; and (c) if r i =0, then R is an empty set; Nte that a bundary blck cmputatin set gathers initially independent cmputatins arund the bundaries f the iteratin space and is a blck-shaped IICS. Example 3.. The iteratin space f a lp nest L 2 is shwn in Fig.. Assume that R and R 2 are bundary blck cmputatin sets. They are dented as [, 3] and [2, -3], respectively. And, the cmputatin sets crrespnding t the regins S and S 2 must nt be bundary blck cmputatin sets. u i I R S 2 S Figure : Bundary blck cmputatin sets. Definitin 4. Maximal bundary blck cmputatin sets: Let expand(r i,)=r i + if r i >0, and expand(r i,)=r i - if r i <0. A bundary blck cmputatin set R=[r, r 2,, r n ] is maximal iff i, i n, [r,, expand(r i, ),, r n ], which is a prper superset f R, is nt an IICS (i.e., it will include a cmputatin that is nt initially independent). (Nte that i, i n, r i 0, since R is empty if r i =0.) In ther wrds, Definitin 4 says that R is the blck-shaped IICS that has been expanded as much as pssible. 3.2 Frmulatin f the basic steps Lemma 3.. Fr the lp nest L n (V, D) which has a dependence vectr d i =(d i, d 2i,, d ni ) T, R di =[d i,,, ] [, d 2i,, ] [,,, d ni ] is an initially independent cmputatin set with respect t d i. ([d i,,, ],, [,,, d ni ] are bundary blck cmputatin sets defined abve.) Lemma 3.2. A cmputatin v f a lp nest L n is an initially independent cmputatin with respect t the dependence vectr d i iff v is in R di. R 2 0 J 0 u j Frm Lemmas 3. and 3.2, it is clear that R di is the maximal initially independent cmputatin set f L n with respect t the dependence vectr d i. Therem 3.. Fr a lp nest L n with m dependence vectrs d i, i m, m R 0 = R di i= is the maximal initially independent cmputatin set f L n. The fllwing algrithm derives the maximal initially independent cmputatin set fr a given lp nest. Prcedure IICS Input: L n with dependence matrix D=[d, d 2,, d m ] (d i =(d i, d 2i,, d ni ) T, i m) Output: R 0 Begin R d = [d,,, ] [, d 2,, ] [,,, d n ]; R d2 = [d 2,,, ] [, d 22,, ] [,,, d n2 ];... R dm = [d m,,, ] [, d 2m,, ] [,,, d nm ]; R 0 = R d R R d2 d m ; Return(R 0 ); End. Therem 3.2. The utput f Prcedure IICS, R 0, is the unin f all the maximal bundary blck cmputatin sets. Nw, we shuld shw that R 0 always cntains a maximal bundary blck cmputatin set with size greater than ne if a prper wavefrnt transfrmatin is applied t the riginal lps. Lemma 3.3. Any unifrm dependence lp nest can be transfrmed (by skewing) int a cannical frm fully permutable lp nest. (A fully permutable lp nest is a lp nest whse dependence vectrs have n negative elements.) Therem 3.3. Cnsider a fully permutable lp nest L n with m dependence vectrs d i =(d i, d 2i,, d ni ) T, i m. If a prper wavefrnt transfrmatin is applied t the riginal lp nest L n, then R t =[ m (d i +d 2i + +d ni ),,, ] R 0, where R 0 is the max i= maximal IICS derived frm Prcedure IICS. 3.3 An illustrative example We want t explit parallelism within lps by partitining them int blcks s that all the blcks are identical by translatin and each f them aggregates as many independent cmputatins as pssible. T facilitate the presentatin, we define the fllwing term. Definitin 5. Valid independent cmputatin sets: Fr a lp nest L, an independent cmputatin set (ICS) R is valid iff L can be partitined int blcks s that each f them is disjunctive, atmic and identical by translatin t R [4]. We call this situatin tiling L with R. Being disjunctive, each cmputatin is executed exactly nly nce; and, being atmic, n tw blcks are CUG 995 Fall Prceedings 35

4 cyclically dependent n each ther and therefre can be scheduled withut vilating dependences [8]. (R is said t be invalid if it is nt valid.) Fr a lp nest, if it can be tiled with an ICS R, clearly, R must be a subset f the maximal initially independent cmputatin set R 0. Frm Therem 3.2, we see that R 0 is the unin f all the maximal bundary blck cmputatin sets. S, amng them, we can chse the maximal bundary blck cmputatin set that is valid and has the largest size t tile the riginal lps fr maximizing parallelism. We describe hw t tile the lps with a valid ICS by ging thrugh the fllwing example. Example 3.2. Cnsider the nested lps L 3 (V, D) with V={(i, j, k) T 0 i, j, k } and D = 2 0. With Prcedure IICS, R =([,, ] [, 2, ] [,, 4]) ([2,, ] [, 4, ] [,, ]) ([, 4, ] [,, 2]) = [, 4, ] [, 4, 4] [2,, 2]. We intend t tile the lps with a valid independent cmputatin set R t. Since [, 4, ], [, 4, 4] and [2,, 2] are all the maximal bundary blck cmputatin sets, R t can be [, 4, ], [, 4, 4] r [2,, 2]. Each case is discussed as fllws:. R t = [, 4, ]: Let the lp nest be tiled with [, 4, ]. As shwn in Fig. 2(a), this is t strip-mine dimensins and 2 with strip lengths and 4, respectively. (Dimensin 3 is nt strip-mined.) After tiling the lps, since all blcks are identical by translatin, each f them can be represented by an arbitrary pint within it (the s-called tile rigin) []. Thus, the tile rigins define a lattice (see Fig. 2(b)). It is easy t see that, in this lattice, the new dependence vectrs are e =(, 0, 0) T 2, e =(, -, 0) T, e 2 =(2,, 0) T, and e 3 =(0,, 0) T. In the sense that tiling is a transfrmatin f the riginal lps, we cnsider e 2,, e 2, and e 3 the transfrmed dependence vectrs. e Since all the transfrmed dependence vectrs are lexicgraphically psitive [3], such a tiling is valid. We shuld explain the ntatin e that we are using: When the riginal lps are tiled with R t =[r, r 2,, r n ], e i =(e i, e 2i,, e ni ) T represents the dependence vectr in the lattice f tile rigins that crrespnds t the riginal dependence vectr d i =(d i, d 2i,, d ni ) T ( i m, m is the number f dependence vectrs). Fr cnvenience, we may call e i the crrespnding blck dependence vectr f d i. 2. (2) R t = [, 4, 4]: Tiling the lps with [, 4, 4] is strip-mining dimensins 2 and 3 bth with strip length 4. (Dimensin is nt strip-mined.) After tiling, the blck dependence vectrs are =(0, -, ) T, 2=(0, 0, ) T, e =(0,, e e 2 -) T, 2=(0,, 0) T and e 3 =(0,, ) T. Since e = -, tw 2 e e 2 blcks are cyclically dependent n each ther. This is the s-called dead-lcked cnditin [3]. S, the lps cannt be tiled with [, 4, 4]. In fact, such a tiling is invalid because nt all e s are lexicgraphically psitive. (Lexicgraphically psitive dependence vectrs will never cause dead-lck.) 3. (3) R t = [2,, 2]: As the same reasning in (), the lp nest can be tiled with [2,, 2]. I I 0 (a) Figure 2: Illustratin f Examples 3.2 S, we have R t = [, 4, ] r R t = [2,, 2]. (They have the same number f cmputatins, 4. dentes the unknwn lp upper bund). With R t =[, 4, ], the riginal lps can be partitined int the fllwing parallel frm. (On the ther hand, ne may let R t =[2,, 2]. Since it has the same degree f parallelism, we mit the discussin f it.) D 200 SI=0, U i - D 200 SJ=0, U j /4 - DAll 00 I= SI,SI DAll 00 J= SJ 4, min(sj 4+3,U j ) DAll 00 K=, U k Lp Bdy; 00 Cntinue Barrier(); 200 Cntinue In cntrast, with the minimum distance methd, the shifting distance in each dimensin is, 4, and 3, respectively [3], [8]. Hence, the degree f parallelism is limited t 52. The riginal lps are partitined int the fllwing parallel frm. (Nte that t cnfrm t the riginal papers, in the fllwing lps, the lwer bund f every lp index is.) DAll 00 I0=, DAll 00 J0=,4 DAll 00 K0=,3 D 00 I=I0, U i, Y=(I-I0)/ Jmin=+(J0--2*Y) md 4 D 00 J= Jmin, U j, 4 J 0 (: tile rigin) J (b) 36 CUG 995 Fall Prceedings

5 Y2=(J-J0+2*Y)/4 Kmin=+(K0-+4*Y+2*Y2) md 3 D 00 K=Kmin,U k,3 Lp Bdy; 00 Cntinue Obviusly, in the abve cdes, there are index calculatin verheads. And, each independent set is nt in a regin f regular shape and size. S, in practical implementatin, it is very difficult t ptimize the data distributin amng prcessrs. 4 Experimental Results We cnduct sme experiments n CRAY T3D. The test lps are mdified frm the example given in [8] whse dependence matrix is the same as that f Example 3.2. As discussed in Example 3.2, the lps are partitined int blcks whse size is 4 ( dentes the lp upper bund). That is, at mst, the lps can be executed in parallel n 4 prcessrs. Hwever, with minimum distance methd, the shifting distances [8] in the first, the secnd and the third dimensins are, 4, and 3, respectively. That is, there are at mst 52 prcessrs that can execute in parallel. S, the methd prpsed in this paper has much mre parallelism, but it requires a glbal synchrnizatin in each blck f lps. Althugh the minimum distance methd prduces cmplete independent sets f cmputatins (n synchrnizatin is required), its parallelism is limited. Table shws the time results f sme experiments n the CRAY T3D. All time are shwn in clcks, which is abut 6.7 nsec. In Table, Private means that data is nt shared amng prcessrs, and (:, :blck(4), :blck()) stands fr the data distributin pattern: the data array is nt divided in the first dimensin, is divided with blck size 4 in the secnd dimensin, and is cyclically divided in the third dimensin. Frm the results, we see that the prpsed methd can run faster than minimum distance methd when the prcessr number is small (, 2, 4, 8 and 6). This is due t the cde simplicity and lw barrier verhead. As the number f prcessrs increases, the barrier verhead increases, and s the prpsed methd lses its advantage. Hwever, with minimum distance methd, the degree f parallelism is limited (52 in this example). That is t say, it cannt take advantage f massively parallel prcessrs, which have hundreds r thusands f prcessrs. In cntrast, with the prpsed methd, the degree f parallelism is 4 N (N is the lp bund), which is usually very large. S, its executin time shuld be bunded by the size f the machine as lng as the barrier verhead is lw and the prblem size is large enugh. As shwn in Table, the prpsed methd runs faster again at 28 prcessrs due the degree f parallelism. 5 Cnclusins Fr efficiently executing lps n distributed memry multiprcessr system, sme existing methds partitin lps int ttally independent sets f cmputatins [3, 8, 2]. Hwever, fr an arbitrary independent set f cmputatins that are executed in Table. Sme experimental results n T3D N. private (:, :blck(4), :blc f PEs MD I I MD I I (II: the prpsed methd, MD: minimum distance methd) a particular prcessr, it is very difficult t distribute all the crrespnding data t the same prcessr s that there is n need t access data lcated in ther prcessrs. S, althugh all the cmputatins are partitined int independent sets, in practice, there will exist data cmmunicatins amng prcessrs. In this paper, we fcus t expliting mre parallelism, which is als an imprtant factr fr enhancing perfrmance. The prpsed methd can increase parallelism within lps by adding apprpriate barrier synchrnizatins. The riginal lps are partitined int rectangular blcks. Each blck aggregates as many independent cmputatins as pssible fr maximizing parallelism. Since each partitined regin is blck-shaped, the parallel cde generatin is straightfrward and easy. With the prpsed methd, the degree f parallelism usually is a functin f lp bunds, and therefre can be very large. Thus, it can take advantage f massively parallel prcessr systems, which have hundreds r thusands f prcessrs. And, its executin time shuld be nly bunded by the size f the machine as lng as the barrier verhead is lw and the prblem size is large enugh. The experiments cnducted n CRAY T3D MPP, which has very fast barrier synchrnizatin, shws the implementatin feasibility and perfrmance benefits f ur prpsed apprach. 6 Acknwledgments The authrs wuld like t thank the Arctic Regin Supercmputing Center fr access its CRAY T3D. The prject was funded in part by the University f Alaska and the Natinal Science Cuncil in Taiwan. 7 References [] Cray T3D System Architecture Overview Manual, Cray Research, Inc., HR-04033, 993 [2] Y. S. Chen, S. D. Wang, and C. M. Wang, Cmpiler Techniques fr Maximizing Fine-grain and Carse-grain Parallelism in Lps with Unifrm Dependences, in Prc. 8th ACM Cnf. Supercmputing, Jul. 994, pp [3] E. H. D'Hllander, Partitining and labeling f lps by unimdular transfrmatins, IEEE Trans. Parallel Distributed Syst. Vl. 3, N. 4, pp , July 992. [4] F. Irigin and R. Trilet, Supernde partitining, in Prc. 5th annual ACM SIGACT-SIGPLAN Sympsium n Principles Prgramming Languages, Jan. 988, pp CUG 995 Fall Prceedings 37

6 [5] C. King and L. M. Ni, Gruping in nested lps fr parallel executin n multicmputers, in Prc. Int. Cnf. Parallel Prcessing, Vl. II, 989, pp [6] A. Niclau, Lp quantizatin: A generalized lp unwinding technique, J. Parallel Distributed Cmput., Vl. 5, N. 0, pp , 988. [7] D. M. Pase, T. MacDnald, A. Meltzer, MPP Frtran Prgramming Mdel, Cray Research, Inc., SN-253, Feb [8] J.-K. Peir and R. Cytrn, Minimum Distance: A methd fr partitining recurrences fr multiprcessrs, IEEE Trans. Cmput., Vl. 38, N. 8, pp , Aug [9] C. D. Plychrnpuls, Cmpiler ptimizatins fr enhancing parallelism and their impact n architecture design, IEEE Trans. Cmput. Vl. 27, N. 8, pp , Aug [0]J. Ramanujam and P. Sadayappan, Tiling multidimensin-al iteratin space fr multicmputers, J. Parallel Distributed Cmput., Vl. 6, N. 2, pp , Oct []R. Schreiber and J. Dngarra, Autmatic blcking f nested lps, Technical Reprt CS-90-08, Univ. f Tennessee, Knxville, TN, 990. [2]W. Shang and J. A. B. Frtes, Independent partitining f algrithms with unifrm dependencies, IEEE Trans. Cmput., Vl. 4, N. 2, pp , Feb [3]M. E. Wlf and M. S. Lam, A lp transfrmatin thery and an algrithm t maximize parallelism, IEEE Trans. Parallel Distributed Syst., Vl. 2, N. 4, pp , Oct. 99. [4]M. Wlfe, Mre iteratin space tiling, in Prc. Supercmputing '89, Nv CUG 995 Fall Prceedings

CS 477/677 Analysis of Algorithms Fall 2007 Dr. George Bebis Course Project Due Date: 11/29/2007

CS 477/677 Analysis of Algorithms Fall 2007 Dr. George Bebis Course Project Due Date: 11/29/2007 CS 477/677 Analysis f Algrithms Fall 2007 Dr. Gerge Bebis Curse Prject Due Date: 11/29/2007 Part1: Cmparisn f Srting Algrithms (70% f the prject grade) The bjective f the first part f the assignment is