A Parallel Two Level Hybrid Method for Diagonal Dominant Tridiagonal Systems

Transcription

1 Paralll wo Lvl Hybrid Mthod for Diagonal Dominant ridiagonal Systms Xian-H Sun and Wu Zhang Dpartmnt of Computr Scinc Illinois Institut of chnology Chicago, IL 6066 bstract nw mthod, namly th Paralll wo-lvl Hybrid (PH) mthod, is dvlopd to solv tridiagonal systms on paralll computrs. PH is dsignd basd on Paralll Diagonal Dominant (PDD) algorithm. Li PDD, PH is highly scalabl. It provids accurat solutions whn PDD may not b applicabl and maintains a nar PDD prformanc whn th undrlying machin nsmbl siz is larg. By controlling its two-lvl partition, PH can dlivr optimal prformanc for diffrnt machin nsmbl and problm sizs. hortical analyss and numrical xprimnts indicat that PH is significantly bttr than xisting mthods for many scintific and nginring applications.. Introduction Solving tridiagonal systms is on of th y issus of numrical simulations in many scintific and nginring problms. Paralll tridiagonal algorithms hav bn studid xtnsivly and rmain an activ rsarch ara. h algorithms of Lawri and Samh [5] and Wang [8] ar calld dividd-and-conqur mthods, which partition th original problm into sub-problms. h sub-problms ar thn solvd in paralll, and th final solution is obtaind by combining th solutions of th sub-problms. Latr, Sun, Zhang and Ni [6,7] proposd thr paralll algorithms for solving tridiagonal systms. ll thr algorithms ar dividd-and-conqur mthods and ar basd on Shrman-Morrison matrix modification formula []. wo of thm, th Paralll Parition LU (PP) algorithm and th Paralll Partition Hybrid algorithms, ar fast and abl to incorporat limitd pivoting. h third algorithm, th Paralll Diagonal Dominant (PDD) algorithm, is dsignd for diagonal dominant systms. PDD algorithm is th most fficint among all thr algorithms. Compard with othr tridiagonal solvrs, which usually rquir O(log p) communications, PDD algorithm has only two communications indpndnt of th numbr of procssors, and has a balancd worload on procssors. Most paralll tridiagonal solvrs trad computation with paralllism. For solving multipl tridiagonal systms or systms with multipl right-hand-sids, piplining can b introducd with th bst squntial algorithm. h piplining approach is computationally fficint but has a communication cost of O(p). It is a good choic whn p, th numbr of procssors, is small. Whn p is larg, its prformanc drops dramatically du to communication costs and piplining dlay. novl Paralll wo-lvl Hybrid (PH) mthod for diagonal dominant tridiagonal systms is proposd in this study basd on PDD algorithm. With a two-lvl partition, PH has two lvls of paralllism. h first lvl (outr lvl) is basd on PDD. h scond lvl (innr lvl) can choos diffrnt paralll tridiagonal solvrs basd on th undrlying application. For instanc, PP is a good candidat for singl systms and piplining is a natural choic for solving multipl systms. PH algorithm ovrcoms th shortcoming of PDD and th piplind mthod and mas us of th mrits of both. It is highly fficint and mor applicabl than PDD. PH provids th bst prformanc for many applications. 2. xisting Paralll ridiagonal Solvrs tridiagonal systm is a linar systm of quations Whr x x = d () and d ar n-dimnsional vctors, and = [ ai, bi, ci ], is a tridiagonal matrix with dimnsion b > a i + c i, for i n. is calld diagonal dominant if i 0 < n. ssum that, x and d hav ral cofficints, to solv () fficintly on paralll computrs, w partition into two parts, th main part and th rsidu. = + (2) Whr is a bloc diagonal matrix with diagonal submatrics i ( i = 0,, Λ, p ). W assum that /02 $ I

2 n = p m. hus, i ( i = 0,, Λ, p ) ar m m tridiagonal matrics. Lt i b a column vctor with its ith, 0 i < n-, lmnt bing on and all th othr ntris ar zro. W hav = V (3) whr V [ a, c, a2 2,..., c( ) ( ) ] = m m m m m m p m p m and = [ m, m,..., ( p ) m, ( p ) m ] ar n 2( p ) matrics. hus, w hav + V = (4) Basd on th matrix modification formula originally dfind by Shrman and Morrison [] for ran-on changs. quation () can b solvd by x d V ( I + V ) = (5) Not that I is an idntity matrix. Z = ( I + V ) is a pntadiagonal matrix of ordr 2(p-). W introduc a prmutation matrix P such that Pz = z, z, z, z, Λ, z, z ), (6) ( p 3 2( p 2) From th proprty that x d P = VP( P + P d, quation (5) bcoms = (6) Z VP) h intrmdiat matrix, = ( + ), is a 2( p ) 2( p ) tridiagonal systm, which lads to a rducd computation cost. h solving squnc of (6) is, P d VP x = d (7) Y = VP (8) h = x (9) Z = P + Y (0) Zy = h () x = Yy (2) x = x x (3) In (7) and (8), x and Y ar solvd by th LU dcomposition mthod. By th structur of and V, ths ar quivalnt to solving i [ x, v, w ] = [ d, a 0, c im ( i+ ) m m ], Hr (i) x and i = 0,, Λ, p (4) (i) d ar th ith bloc of x and d, (i) rspctivly, and v (i) and w ar possibl no-zro column vctors of th ith row bloc of Y. quation (4) implis that w only nd to solv thr linar systms of ordr m with th sam LU dcomposition for ach i( i = 0,, Λ, p ). 2.. PP: h Paralll Partition LU lgorithm Basing on th matrix partitioning tchniqu abov, using p procssors to solvr (), PP consists of th following stps, Stp. llocat i, d and lmnts aim, c( i+ ) m to th ith nod, whr 0 i p. Stp 2. Us th LU dcomposition mthod to solvr (4). ll computations can b xcutd in paralll and indpndntly on p procssors. Stp 3. Snd x, x, v 0 0, v, w0, w (0 i p m m m ) to all othr nods from th ith nod to form matrix Z and vctor h (9-0) on ach nod. Hr ar throughout th subindx indicats th componnt of th vctor. Stp 4. Us th LU dcomposition mthod to solvr () on all nods simultanously. Not that Z is a 2(p- ) dimnsional tridiagonal matrix. Stp 5. Comput (2) and (3) in paralll on p procssors. W hav x x = [ v = x y, w ] y x 2i 2i 2.2 PDD: Paralll Diagonal Dominant lgorithm Whn is diagonal dominant, th most intrsting mathmatical proprtis is that th off diagonal cofficints of th matrix V hav an xponntially dcay to 0. hrfor, th cofficints of vm, w0,(0 < i < p ) can b droppd within machin accuracy whn p << n. s shown in [6,7], for most diagonal dominant systms, whn th subsystm siz is gratr than 64, th rducd Z with th dropping is quivalnt to Z within machin accuracy for numrical computing. PDD uss th dropping for th solution and nds only two nighboring communications. h optimal and simpl communication /02 $ I

3 proprty mas PDD algorithm an idal algorithm for massivly paralll computing. h rsulting PDD algorithm is similar with PP xcpt that Stp 3 and Stp 4 ar modifid as givn blow. i ( ) Stp 3. Snd x 0, v0 from th ith nod to th (i-)th nod for i p. Stp 4. Solv ( i wm ) v ( i+ ) 0 y y2 2i i+ xm = ( i+ ) x0 0 p in paralll on all ith componnts for i. hn snd y 2 i+ from th ith nod to (i+)th nod for 0 i p h Piplind Mthod for Multipl Systms PDD can b applid to ithr singl tridiagonal systms or multipl systms whr multipl indpndnt systms or a systm with multipl right-hand-sids hav to b solvd. Li most paralll tridiagonal solvrs, PDD has a non-optimal computation count. For solving multipl systms, howvr, optimal squntial algorithm can b usd to achiv paralll procssing via piplining [2]. Piplining wors by passing th intrmdiat rsults from solving a subst of th systm onto th nxt procssor bfor continuing. Lt K b th numbr of systms to b solvd. W partition th K systms into m sts. ach st has L systms. h piplining procdur is givn blow. In th first pass, procssor 0 solvs th first part of th first L systms whras procssors 2 to p- idl. In th scond pass, procssor wors on th scond part of th first L systm (using th rsults of th first pass) whil procssor 0 wors on th first part of th scond L systms; procssors 2 to p- idl. In th ith pass, procssor i- solvs th ith part of th first L systms, procssor i-2 solvs th (i-)th part of th scond L systms,, procssor 0 solvs th first part of th ith L systms. h subsqunt passs continu until vntually running out of wor, and procssors on by on (starting with procssor 0) go idl. hr ar thr rounds of computations for solving a tridiagonal systm (or systms) via th convntionally usd tridiagonal solvr, th homas algorithm [6]. On round is for LU dcomposition, on is for forward substitution, and anothr on is for bacward substitution. ach round of computation rquirs a p- communication. h communication cost is high. It incrass linarly with th numbr of procssors. In addition, thr is a piplin dlay of (p-). h trad-off is that th bst squntial mthod can b usd. h computation is optimal. Whn th nsmbl siz is small, th piplind mthod is a good candidat for paralll procssing. 3. PH: h Paralll wo-lvl Hybrid Mthod h Paralll wo-lvl Hybrid (PH) mthod is proposd in this study to combin th mrits of both PDD and Piplind mthods. h basic ida of PH is to mbd an innr tridiagonal solvr into PDD to form a two-lvl hirarchical paralllism. h bas algorithm is PDD. h tridiagonal systm is first partitiond basd on PDD. Howvr, th subsystms may b too small for th accuracy concrn if w us PDD dirctly with th onprocssor on-subsystm approach. o ovrcom th limitation of PDD, w group ach procssors togthr to solv a supr-subsystm (s Figur ). ach suprsubsystm is an indpndnt tridiagonal systm and can b solvd by any dirct paralll tridiagonal solvr that dos not introduc approximation rror. For singl tridiagonal systms, a good choic for th innr tridiagonal solvr would b PP algorithm [7]. PP introducs a good paralllism and has log(p) communication cost. For multipl tridiagonal systms, th piplind mthod would b a natural candidat for th innr solvr. Both of PP and th piplind mthod rquir global communications and othrwis fficint. Whn thy ar mbddd into PDD, thy ar usd on a small numbr of procssors for solving th suprsubsystms, and th communication cost is small. h two-lvl hybrid mthod tas th advantag of PDD and innr solvrs. PDD tas car of th scalability issu and can b scald fficintly on massivly paralll machins. h innr solvrs conduct fficint computation at th local lvl and provid an adquat solution for accuracy concrn. In addition, by adjusting th siz of th suprsubsystm, PH mthod can b scal-up and scal-down on diffrnt paralll machins basd on th numbr of procssors availabl. Whn th numbr of suprsubsystm quals to on, PH is th innr tridiagonal solvr. Whn th numbr of supr-subsystm quals to p, th numbr of procssors, PH bcoms PDD. h optimal numbr of th supr-subsystm is a function of machin paramtrs and rror tolranc. Whn th piplind algorithm is chosn as th innr solvr, w call th rsulting algorithm Partition Piplind diagonal Dominant (PPD) algorithm /02 $ I

4 lgorithm Computation Communication Piplining PDD PPD 8n 7 ( n + p ) p ( n + p) ( 3α + 2β ) (7 n p ( 2α + 2n ) n β 4 ) 3 n 4n [( + ) + n ( p p ( n + )( 3α + 2 β ) n + 4)] ( 2 + log( ))( α + 2 n + β ) abl. Formulas of computation and communication Figur. PH: a wo-lvl hirarchical algorithm h rsulting PH mthod can b dscribd in th following two stps. Stp. Us an accurat paralll tridiagonal solvr to solv th m supr-subsystms concurrntly, ach with procssors, whr p = m. Stp 2. Modify th solutions of Stp with Stps 3-5 of PDD algorithm and considr communications only btwn th m supr-subsystms. If th accurat paralll tridiagonal solvr of Stp is th piplind mthod, th rsulting PH algorithm is th PPD algorithm. W us PPD for our prformanc analysis. abl givs th computation and communication counts of th piplind mthod, PDD and PPD algorithm, rspctivly. α is th communication startup tim. β is th data transmission tim pr byt, normalizd to th n, n and in abl computing tim. h paramtrs stand for th numbr of tridiagonal systms, th ordr of th systms and th numbr of subsystms (or th numbr of procssors) for ach supr-subsystm, rspctivly. h computation and communication costs for solving tridiagonal systms incras with th paramtr n for all algorithms. Compard with th piplind algorithm, PPD algorithm rducs th communication cost significantly whn p is big du to th fact that << p. In gnral, in PPD w choos th smallst that maintains th accuracy. From abl, w can s that whn p is small, th piplind mthod is th bst among th thr. Whn p is big, PDD provids th bst prformanc. PDD, howvr, may los accuracy and, thrfor, bcom inapplicabl whn p is big. Whn PDD is inapplicabl, PPD is th lading algorithm. h rang of p for th prformanc ran chang is machin and application dpndnt. It can b dtrmind whn th application and th undrlying hardwar ar givn. 4. Numrical xprimnts ridiagonal solvrs hav many applications. Hr w prsnt th xprimntal tsting of on application, solving Poisson quations. Hocny s fast Poisson solvr, th Fourir nalysis and Cyclic Rduction (FCR) algorithm [3,4], is a most accptd dirct solvr. W us FCR as an application of our tridiagonal solvrs. 2-D Poisson quation, writtn in Cartsian coordinats, is 2 2 ϕ ϕ + = f ( x, in Ω, 2 2 x y whr Ω is a rctangl rgion and ϕ ( f ( f ( x, is a givn function. Lt and b th Fourir cofficints of th th ϕ( x, wav numbr of and f ( x, Fourir xpansion in x dirction, rspctivly. ϕ ( In th cas that th function has prscribd valus at th boundaris in y dirction, w can solv th problm with th following stps [4], which is calld th fast Poisson solvr or FCR /02 $ I

5 Stp. Conduct Fast Fourir transform on th givn function f ( x, f ( Stp 2. Solv n indpndnt tridiagonal systms ach f ( y ϕ y with ordr of n, ) ( ). Stp 3. Conduct Fast Fourir transform on th function ϕ ( ϕ( x,. h bst way to solv FCR on paralll machins is to solv FF squntially on ach procssor and solv th tridiagonal systms in paralll. o solv th FFs in ach machin locally, howvr, mas th rsulting tridiagonal systms having a vry distinguishd data distribution: ach procssor has a submatrix (in th ordr of n/p) from ach of th n tridiagonal systms, so that ach procssor solvrs n subsystms squntially. vn p is clos to n, ach procssor still has nough computing to do. Whn p is clos to n, howvr, PDD bcoms inapplicabl. application, howvr, whn p is gratr than 96, th subsystm siz is lss than 48. PDD dos not provid accurat rsults whn p rachs 96. It is not applicabl for FCR whn p is larg. W now us PPD for FCR tridiagonal systms and choos, th numbr of subsystms in a supr-subsystm, to b 6. W ta th sam st of numbrs of procssors as abov for PPD tsting, that is, numbrs of procssors, 2, 24, 48, 96, 92, 384 and 52 ar considrd, corrsponding to th subsystm sizs of 4608, 384, 92, 96, 48, 24, 2 and 9. Figur 3 shows th runtims of thr tridiagonal solvrs for th FCR tridiagonal systms. From p = to p =48, PPD chooss to ta on supr-subsystm. hat mans PPD is th piplind mthod for p < 96. Starting at 4. xprimntal sting xprimntal tsting prformd on th NPCI IBM Blu-Horizon at th San Digo Suprcomputing Cntr (SDSC). h Blu Horizon is a traflop-scal Powr3 basd clustr. h machin contains,52 procssors and 576 GByts of main mmory, arrangd as 44 Symmtric Multiprocssing (SMP) comput nods. ach nod is quippd with 4 GByts of mmory shard among its MHz Powr3 procssors. ach nod also has svral GByts of local dis spac. It is wll suitd to run straight MPI applications. W us FCR to tst PH. Sinc th tridiagonal systms in FCR ar multipl systms, PPD is th chosn PH. For th xprimnts, w choos th wav numbr n = 52 in x-dirction and msh points n = 4608 in y- dirction. hrfor, w nd to solv 52 tridiagonal systms, which ach has an ordr of h siz of th subsystms is a function of th numbr of procssors usd. h prformanc of, 2, 24, 48, 96, 92, 384 and 52 procssor implmntations ar masurd, which has th subsystm sizs of 4608, 384, 92, 96, 48, 24, 2, and 9, rspctivly. h xprimntal rsults for solving th FCR tridiagonal systms with PDD and th Piplind mthod ar givn in Figur 2. s shown in Figur 4, th piplind mthod prforms bttr for small nsmbl siz. t p = 96, PDD starts to ovr prform of th piplind mthod. h prformanc gap btwn PDD and th piplind mthod bcoms largr and largr, as th numbr of procssors usd bcoms biggr and biggr. t p =52 PDD is mor than tn tims fastr than th piplind mthod. It is a vry imprssiv rsult. h xprimntal rsults confirm PDD is highly scalabl. For th givn Figur 2. ridiagonal solvr runtim: Piplining (squar) and PDD (dlta) Figur 3. ridiagonal Solvr im: Piplining (squar), PDD (dlta), PPD (circl) p = 96, PPD groups ach 6 subsystms to form a suprsubsystm and us 6 procssors to solv ach suprsubsystm. In this way, PPD rachs th optimal prformanc. W can s PPD achivs a nar PDD /02 $ I

6 prformanc whn p is larg and th sam prformanc as th piplind mthod whn p is small. Figur 4 shows th masurd accuracy of PDD, PPD and th piplind mthod compard with th squntial algorithm with th L norm. h accuracy of PPD coincids with that of th piplind mthod in L norm. xprimntal rsults confirm that PPD is scalabl and applicabl. It is a good algorithm for FCR. h masurd runtims of FCR for solving th Poisson quation with thr diffrnt tridiagonal solvrs ar givn in Figur 5. Plas notic that th FCR-PDD implmntation dos not provid an accurat solution. It only can b usd as a pr-conditionr for furthr computing. computing. h xponntial dcay rat of th dropping lmnts has bn mathmatically provn. Howvr, it is also nown that PDD is inapplicabl whn th siz of th partitiond subsystms is small. In this study, a nw mthod, th Paralll wo-lvl Hybrid (PH) mthod, is proposd to ovrcom th shortcoming of PDD. PH consists of two paralll tridiagonal solvrs: th outr solvr and th innr solvr. h outr solvr is PDD. h innr solvr is opn and can b application spcific. Whn th innr solvr is th piplind mthod, th rsulting PH is calld th Partition Piplind diagonal Dominant (PPD) algorithm. PPD maintains PDD's scalability and is fasibl whil PDD is not. PPD has bn xamind closly and xprimntally tstd for th wll-nown fast Poisson solvr originally proposd by Hocny [3,4]. xprimntal analyss show that PPD is fundamntally mor appropriat for th fast Poisson solvr than xisting tridiagonal algorithms. PPD nds to b furthr studid to improv its prformanc in solving Poisson quations and othr applications. PPD is on of th many possibl algorithms that can b gnratd from PH mthod. h potntial of PH also should b furthr invstigatd. cnowldgmnts Figur 4. ccuracy: Piplining (squar), PDD (dlta), PPD (circl) his rsarch was supportd in part by ONR undr P/Logicon and by NSF undr NSF grant CCR ll numrical xprimnts wr prformd on th IBM Blu Horizon opratd by th San Digo Suprcomputing Cntr (SDSC). h authors ar gratful to NPCI and SDSC for providing th accss to this facility. Rfrncs Figur 5. otal runtim: Piplining (squar), PDD (dlta), PPD (circl) 5. Conclusions PDD algorithm is an fficint algorithm for diagonal dominant tridiagonal systms. Partitiond via th Shrman and Morrison formula [], PDD can drop communications without losing accuracy for numrical [] I. Duff,. risman and J. Rid, Dirct Mthods for Spars Matrics (Clarndon Prss, Oxford, 986) [2].M. dison and G. rlbachr, Implmntation of a fullybalancd priodic tridiagonal solvr on a paralll distributrd mmory architctur, Concurrncy: Practics and xprinc, 995 [3] R.W. Hocny, fast dirct solution of Poisson s quation using Fourir analysis, J. ssoc. Comput. Mach., 2 (965), 95-3 [4] R.W. Hocny, Paralll computrs 2, rchitctur, Programming and algorithms, dam Hilgr, 988 [5] D. Lawri and. Samh, h computation and communication complxity of a paralll bandd systm solvr, CM rans. Math. Soft. 0 (2), Jun 984, [6] X.-H. Sun, pplication and accuracy of th paralll diagonal dominant algorithm, Paralll Computing, 8(995), [7] X.-H. Sun, H. Zhang and L. Ni, fficint tridiagonal solvrs on multicomputrs, I rans. Comput., 4(3) (992), [8] H. Wang, paralll mthod for tridiagonal quations, CM rans. Math Softwar, 7(98), /02 $ I