This is a repository copy of Parallel-in-Space-and-Time Simulation of the Three-Dimensional, Unsteady Navier-Stokes Equations for Incompressible Flow.

Transcription

1 This is a reository coy of Parallel-in-Sace-and-Time Simlation of the Three-Dimensional, Unsteady Naier-Stokes Eqations for Incomressible Flow. White Rose Research Online URL for this aer: htt://erints.whiterose.ac.k/90737/ Version: Acceted Version Proceedings Paer: Croce, R, Rrecht, D orcid.org/ and Krase, R (2014) Parallel-in-Sace-and-Time Simlation of the Three-Dimensional, Unsteady Naier-Stokes Eqations for Incomressible Flow. In: Bock, HG, Hoang, XP, Rannacher, R and Schlöder, JP, (eds.) Proceedings of the Fifth International Conference on High Performance Scientific Comting, March 5-9, 2012, Hanoi, Vietnam, Modeling, Simlation and Otimization of Comlex Processes - HPSC Fifth International Conference on High Performance Scientific Comting, Mar 2012, Hanoi, Vietnam. Modeling, Simlation and Otimization of Comlex Processes. Sringer International Pblishing, ISBN htts://doi.org/ / _2 Rese Unless indicated otherwise, flltext items are rotected by coyright with all rights resered. The coyright excetion in section 29 of the Coyright, Designs and Patents Act 1988 allows the making of a single coy solely for the rose of non-commercial research or riate stdy within the limits of fair dealing. The blisher or other rights-holder may allow frther rerodction and re-se of this ersion - refer to the White Rose Research Online record for this item. Where records identify the blisher as the coyright holder, sers can erify any secific terms of se on the blisher s website. Takedown If yo consider content in White Rose Research Online to be in breach of UK law, lease notify s by ing erints@whiterose.ac.k inclding the URL of the record and the reason for the withdrawal reqest. erints@whiterose.ac.k htts://erints.whiterose.ac.k/

2 Parallel-in-Sace-and-Time Simlation of the Three-Dimensional, Unsteady Naier-Stokes Eqations for Incomressible Flow Roberto Croce and Daniel Rrecht and Rolf Krase Abstract The Parareal arallel-in-time method is combined with satial arallelization by domain decomosition into a sace-time arallel scheme for the three-dimensional incomressible Naier-Stokes eqations. Parallelization of timesteing roides a new direction of arallelization and allows to emloy additional cores to frther seed simlations after satial arallelization has satrated. We reort on nmerical exeriments erformed on a Cray XE6, simlating a drien caity flow with and withot obstacles. Distribted memory arallelization is sed in both sace and time, featring to 2048 cores in total. It is confirmed that the sace-time-arallel method can roide seed beyond the satration of the satial domain decomosition. 1 Introdction Simlating three-dimensional flows by nmerically soling the time-deendent Naier-Stokes eqations leads to hge comtational costs. In order to obtain a reasonable time-to-soltion, massiely arallel comter systems hae to be tilized. This reqires sfficient arallelism to be identifiable in the emloyed soltion algorithms. Decomosition of the satial comtational domain is by now a standard techniqe and has roen to be extremely owerfl. Neertheless, for a fixed Roberto Croce Institte of Comtational Science, Via Gisee Bffi 13, CH-6906 Lgano, Switzerland, roberto.croce@si.ch Daniel Rrecht Institte of Comtational Science, Via Gisee Bffi 13, CH-6906 Lgano, Switzerland, daniel.rrecht@si.ch Rolf Krase Institte of Comtational Science, Via Gisee Bffi 13, CH-6906 Lgano, Switzerland, rolf.krase@si.ch 1

3 2 R. Croce, D. Rrecht, R. Krase roblem size, this aroach can only sh the time-to-soltion down to some fixed threshold, below which the comtation time for each sbdomain becomes comarable to the commnication time. While re satial arallelization can roide satisfactory rntime redction, time-critical alications may reqire larger seed and hence need additional directions of arallelism in the sed nmerical schemes. One aroach that has receied increasing attention oer recent years is arallelizing the time-steing rocedre tyically sed to sole time-deendent roblems. A olar algorithm for this is Parareal, introdced in [10] and comrehensiely analyzed in [6]. Its erformance has been inestigated for a wide range of roblems, see for examle the references in [11, 12]. A first alication to the 2D-Naier-Stokes eqations, focssing on stability and accracy withot reorting rntimes, can be fond in [5]. Some exeriments with a combined Parareal/domaindecomosition method for the two-dimensional Naier-Stokes eqations hae been condcted on to 24 rocessors in [14, 15]. While they sccessflly established the general alicability of sch a sace-time arallel aroach for the Naier-Stokes eqations, the obtained seeds were ambigos: Best seeds were achieed either with a re time-arallel or a re sace-arallel aroach, deending on the roblem size. In the resent aer, the combined Parareal/domain-decomosition sace-time arallel aroach is sed to sole a qasi-2d and a flly 3D drien caity flow roblem on a state-of-the-art HPC distribted memory architectre, sing to 2,048 cores. The caability of the aroach to redce time-to-soltion below the satration oint of a re satial arallelization is demonstrated. Also it is shown that the addition of obstacles into the comtational domain, leading to more trblent flow, leads to slower conergence of Parareal. This is likely de to the reorted stability isses for hyerbolic and conection-dominated roblems, see [4, 12]. 2 Physical Model and its discretization and arallelization The behaior of three-dimensional, incomressible Newtonian flids is described by the incomressible Naier-Stokes eqations. In dimensionless form the according momentm- and continm eqation read t + = 1 Re =0 (1) with = (,,w) being the elocity field consisting of the Cartesian elocitycomonents, being the ressre and Re the dimensionless Reynolds nmber. The Naier-Stokes soler is based on the software-ackage NaSt3DGP [2, 8] and we frther extended it by an MPI-based imlementation of Parareal [10]. In NaSt3DGP, the nsteady 3D-Naier-Stokes eqations are discretized ia standard finite olme/finite differences sing the Chorin-Temam [1, 13] rojection method

4 Sace-Time-Parallel Naier-Stokes 3 on a niform Cartesian staggered mesh for robst ressre and elocity coling. A first order forward Eler scheme is sed for time discretization and as a bilding block for Parareal, see the descrition in Second order central differences are sed for the ressre gradient and diffsion. The conectie terms are discretized with a second order TVD SMART [7] wind scheme, which is basically a bonded QUICK [9] scheme. Frthermore, comlex geometries are aroximated sing a first order cell decomosition/enmeration techniqe, on which we can imose sli as well as no-sli bondary conditions. Finally, the Poisson eqation for the ressre arising in the rojection ste is soled sing a BiCGStab [16] iteratie method. 2.1 Parallelization Both the satial as well as the temoral arallelization are imlemented for distribted-memory machines sing the MPI-library. The nderlying algorithms are described in the following Parallelization in sace ia domain decomosition We niformly decomose the discrete comtational domain Ω h into P sbdomains by first comting all factorizations of P into three comonents, i.e. P=P x P y P z, with P x,p y,p z N. Then we se or re-comted factorizations of P as argments for the following cost fnction C with resect to commnication C(P x,p y,p z )= I P x J P y + J P y K P z + I P x K P z (2) with I,J and K as the total nmber of grid-cells in x-, y- and z-direction. Finally, we aly that factorization for the domain decomosition for which C is minimal, i.e. the sace decomosition is generated in iew of the oerall srface area minimization of neighboring sbdomains. Here, P is always identical to the nmber of rocessors N sace, so that each rocessor handles one sbdomain. Since the stencil is fie grid-oints large for the conectie terms and three grid-oints for the Poisson eqation, each sbdomain needs two ghost-cell rows for the elocities and one ghost-cell row for the ressre Poisson eqation. Ths or domain decomosition method needs to commnicate the elocities once at each time-ste and the ressre once at each ressre Poisson iteration Parallelization in time with Parareal For a gien time interal[0,t], we introdce a coarse temoral mesh 0= t 0 < t 1 <...< t Nc = T (3)

5 4 R. Croce, D. Rrecht, R. Krase with a niform time-ste size t = t i+1 t i. Frther, we introdce a fine time-ste δt < t and denote by N f the total nmber of fine stes and by N c the total nmber of coarse stes, that is N c t = N f δt = T. (4) Also assme that the coarse time-ste is a mltile of the fine, so that t δt =: N r N. (5) Parareal relies on the iteratie se of two integration schemes: a fine roagator F δt that is comtationally exensie, and a coarse roagator G t that is comtationally chea. We sketch the algorithm only ery briefly here, for a more detailed descrition see for examle [10]. Denote by F(y,t n+1,t n ), G(y,t n+1,t n ) the reslt of integrating from an initial ale y at time t n to a time t n+1, sing the fine or coarse scheme, resectiely. Then, the basic iteration of Parareal reads y k+1 n+1 = G t(y k+1 n,t n+1,t n )+F δt (y k n,t n+1,t n ) G t (y k n,t n+1,t n ) (6) with ser-scrits referring to the iteration index and y n corresonding to the aroximation of the soltion at time t n. (6) conerges to a soltion y n+1 = F δt (y n,t n+1,t n ), (7) that is a soltion with the accracy of the fine soler. Here, we always erform some rescribed nmber of iterations N it. We se a forward Eler scheme for both F δt and G t and simly se a larger time-ste for the coarse roagator. Exerimenting with the combination of schemes of different and/or higher order is left for ftre work. Once the ales y k n in (6) from the reios iteration are known, the comtationally exensie calclations of the ales F δt (y k n,t n+1,t n ) can be erformed in arallel for mltile coarse interals [t n,t n+1 ]. In the re time-arallel case, the time-slices are distribted to N time cores assigned for the time-arallelization. Note that in the sace-time arallel case, the time-slices are not handled by single cores bt by mltile cores, each handling one sbdomain at the secific time, see Section The theoretically obtainable seed with Parareal is bonded by s(n ) N time N it (8) with N time denoting the nmber of rocessors in the temoral arallelization and N it the nmber of iterations, see for examle [11]. From (8) it follows that the maximm achieable arallel efficiency of the time arallelization is bonded by 1/N it. Parareal is hence considered as an additional direction of arallelization to be sed when the satial arallelization is satrated bt a frther redction of time-

6 Sace-Time-Parallel Naier-Stokes 5 sace t 0 t t t t 1 N c 2 N c 1 Nc time Fig. 1 Decomosition of the time interal [0,T] into N c time-slices. The satial mesh at each oint t i is again decomosed into P sbdomains, assigned to N sace cores. Becase the satial arallelization does not need to commnicate across time-slices, the cores from eery satial mesh are ooled into one MPI commnicator. Also, in the time arallelization, only cores handling the same sbdomain at different times hae to commnicate. Note that for readability the sketched satial mesh is 2D, althogh the simlations se a flly 3D mesh. to-soltion is reqired or desirable. Some rogress has recently been made deriing time-arallel schemes with less strict efficiency bonds [3, 11] Combined arallelization in sace and time In the combined sace-time arallel aroach as sketched in Figre 1, each coarse time interal in (6) is assigned not to a single rocessor, bt to one MPI commnicator containing N sace cores, each handling one sbdomain of the corresonding time-slice. The total nmber of cores is hence N total = N time N sace. (9) Note that the commnication in time in (6) is local in the sense that each rocessor has only to commnicate with the cores handling the same sbdomain in adjacent time-slices. Also, the satial arallelization is not commnicating across time-slices, so that for the ealation of F or G in (6), no commnication between rocessors assigned to different oints in time is reqired. We ths organize all aailable cores into two tyes of MPI commnicators: (i) Satial commnicators collect all cores belonging to the soltion at one fixed time-slice, bt handling different sbdomains. They corresond to the distribted reresentation of the soltion at one fixed time-slice. There are N time satial commnicators and each contains N sace cores. (ii) Time commnicators collect all cores dealing with the same satial sbdomain, bt at different time-slices. They are sed to erform the iteratie date in (6) of the local soltion on a satial sbdomain. There are N sace time commnicators, each ooling N time cores. No secial attention was aid to how different MPI tasks are assigned to cores. Becase of the ery different commnication attern of the sace- and time-arallelization, this can resmably hae a significant effect on the oerall erformance. More detailed inestigation of the otimal lacement of tasks is lanned for ftre stdies with the here resented code.

7 6 R. Croce, D. Rrecht, R. Krase Table 1 Simlation arameters for the qasi-2d drien caity flow (Simlation 1) and the flly 3D drien caity flow with obstacles (Simlation 2). Sim. 1 : Ω h = [0,1] [0,1] [0,0.1] Sim. 2 : Ω h = [0,1] [0,1] [0,1] Sim. 1 : N x N y N z = Sim. 2 : N x N y N z = Sim. 1 : T = 80 Sim. 2 : T = 24 Both : t = 0.01 Both : δt = Both : Re = 1000 Both : bondary = 1 Sim. 1 : N sace = 1,2,4,8 Sim. 2 : N sace = 1,...,128 Sim. 1 : N time = 4,8,16 Sim. 2 : N time = 8,16,32 Fig. 2 Arrows and color lot of the Eclidean norm of the qasi two-dimensional drien caity flow field along the center lane at three oints in time t = 0.8 and t = 8.0 and t = Nmerical Examles In the following, we inestigate the erformance of the sace-time arallel aroach for two nmerical examles. The first is the classical drien-caity roblem in a qasi-2d set. Figre 2 shows the flow in a xy-lane with z = 0.05 at times t = 0.8,8.0 and The second examle is an extension where seeral obstacles are inserted into the domain, leading to flly 3D flow. Figre 3 sketches the obstacles and the flow at T = 24. Both roblems are osed on a 3D-domain with eriodic bondary conditions in z-direction (note that x and z are the horizontal coordinates, while y is the ertical coordinate). Initially, elocity and ressre are set to zero. At the er bondary, a tangential elocity bondary = 1 is rescribed, which generates a flow inside the domain as time rogresses. No-sli bondary conditions are sed at the bottom and in the two yz-bondary lanes located at x = 0 and x = 1 as well as on the obstacles. The arameters for the two simlations are smmarized in Table 1. To assess the temoral discretization error of F δt, the soltion is comared to a reference soltion comted with F δt/10, giing a maximm error of for the fll 3D flow with obstacles. That means that once the iteration of Parareal has to redced the maximm defect between the serial and arallel soltion below this threshold, the time-arallel and time-serial soltion are of comarable accracy. We se this threshold also for the qasi-2d examle, bearing in mind that the simler strctre of the flow in this case most likely renders the estimate too conseratie. The code is rn on a Cray XE6 at the Swiss National Sercomting

8 Sace-Time-Parallel Naier-Stokes 7 Fig. 3 Arrows and color lot of the Eclidean norm of the flly three-dimensional drien caity flow field with obstacles along the center lane at time t = Max. Defect to Serial Soltion Max. Defect to Serial Soltion Max. Defect to Serial Soltion Fig. 4 Maximm difference to time-serial soltion erss nmber of Parareal iterations for the qasi 2D drien caity roblem at time t = 80.0 for N time = 4 (left), N time = 8 (middle) and N time = 16 (right). Centre, featring a total of 1496 nodes, each with two 16-core 2.1GHz AMD Interlagos CPUs and 32GB memory er node. Nodes are connected by a Gemini 3D tors interconnect and the theoretical eak erformance is 402 TFlos. 3.1 Qasi-2D Drien Caity Flow Figre 4 shows the maximm difference between the time-arallel and the timeserial soltion at the end of the simlation erss the nmber of iterations of Parareal. In all three cases, the error decreases exonentially with N it. The threshold of is reached after a single iteration, indicating that the erformance of Parareal cold robably be otimized by sing a larger t. Figre 5 shows the total seed roided by the time-serial scheme rnning F δt with only sacearallelism (black dashed line) as well as by the sace-time arallel method for different ales of N time (grey lines). All seeds are measred against the rntime of the time-serial soltion rn on a single core. The re satial arallelization reaches a maximm seed of a little oer 6 sing 8 cores. For N it = 1, the sace-time ar-

9 8 R. Croce, D. Rrecht, R. Krase Total seed Time serial =4 =8 = Total nmber of cores Total seed Time serial =4 =8 = Total nmber of cores Fig. 5 Total seed of the combined sace-time arallelization for qasi 2D drien-caity flow with N it = 1 (left) and N it = 2 (right) iterations. allel scheme reaches a seed of 14 sing 64 cores. This amonts to a seed of roghly 14/ from Parareal alone. For N it = 2 the seed is down to 8, bt still noticeably larger than the satration oint of the re sace-arallel method. Note that becase of the limited efficiency of the time arallelization, the sloes of the sace-time arallel scheme are lower for larger ales of N time. 3.2 Fll 3D Drien Caity Flow with Obstacles Figre 6 shows the maximm difference between the time-arallel and the timeserial soltion deending on the nmber of Parareal iterations for for different ales of N time. In general, as in the qasi-2d case, the error decays exonentially with the nmber of iterations, bt now, articlarly rononced for N time = 32, a small nmber of iterations has to be erformed withot large effect before the error starts to decrease. This is likely de to the increased trblence cased by the obstacles, as it is known that Parareal exhibits instabilities for adection dominated roblems or hyerbolic roblems [6, 12]. A more detailed analysis of the erformance of Parareal for trblent flow and larger Reynolds nmbers is left for ftre work. Figre 7 shows the total seed measred against the rntime of the soltion rnning F δt serially with N sace = 1. The black line shows the seed for a re satial arallelization, which scales to N sace = 16 cores and then satrates at a seed of abot 18. Adding time-arallelism can significantly increase the total seed, to abot 20 for N time = 4, abot 27 for N time = 8 and to almost 40 for N time = 16 for a fixed nmber of N it = 3 iterations (left figre). Howeer, as can be seen from Figre 6, the soltion with N = 32 is significantly less accrate. The right figre shows the total seed for a nmber of iterations adjsted so that the defect of Parareal in all cases is below 10 5 in all soltion comonents (cf. Figre 6). This illstrates that there is a sweet-sot in the nmber of concrrently treated time-slices: At some oint the otential increase in seed is offset by the additional iterations reqired.

10 Sace-Time-Parallel Naier-Stokes 9 Max. defect to time serial soltion w Max. defect to time serial soltion w Max. defect to time serial soltion w Fig. 6 Maximm difference to time-serial soltion at end of simlation erss nmber of Parareal iterations for the 3D drien caity flow with obstacles for N time = 8 (left), N time = 16 (middle), N time = 32 (right). The horizontal line indicates an error leel of Total seed Time serial =8 with 3 It. =16 with 3 It. =32 with 3 It Total nmber of cores Total seed Time serial =8 with 3 It. =16 with 3 It. =32 with 6 It Total nmber of cores Fig. 7 Total seed of the combined sace-time arallelization for 3D caity flow with obstacles for a fixed nmber of N it = 3 iterations (left) and a nmber of iterations chosen to achiee a defect below 10 5 in all soltion comonents (right). Note that the soltions in the left figre are not comarable in accracy. In the resented examle, the soltion with N = 16 is clearly more efficient than the one with N = Conclsions A sace-time arallel method, coling Parareal with satial domain decomosition, is resented and sed to sole the three-dimensional, time-deendent, incomressible Naier-Stokes eqations. Two sets are analyzed: A qasi-2d drien caity examle and an extended set, where obstacles inside the domain lead to flly 3D flow. The conergence of Parareal is inestigated and seeds of the sace-time arallel aroach are comared to seeds from a re sace-arallel scheme. It is fond that Parareal conerges ery raidly for the qasi-2d case. It also conerges in the 3D case, althogh for larger nmbers of Parareal time-slices, conergence starts to stagnate for the first few iterations, likely becase of the known stability isses of Parareal for adection dominated flows. Reslts are reorted from rns on to 128 nodes with a total of 2,048 cores on a Cray XE6, illstrating the feasibility of

11 10 R. Croce, D. Rrecht, R. Krase the aroach for state-of-the-art HPC systems. The reslts clearly demonstrate the otential of time-arallelism as an additional direction of arallelization to roide additional seed after a re satial arallelization reaches satration. While the limited arallel efficiency of Parareal in its crrent form is a drawback, we exect the scalability roerties of Parareal to direct ftre research towards modified schemes with relaxed efficiency bonds. Acknowledgements This research is fnded by the Swiss High Performance and High Prodctiity Comting initiatie HP2C. Comtational resorces were roided by the Swiss National Sercomting Centre CSCS. References 1. Chorin, A.J.: Nmerical soltion of the NaierStokes eqations. Math. Comt. 22(104), (1968) 2. Croce, R., Engel, M., Griebel, M., Klitz, M.: NaSt3DGP - a Parallel 3D Flow Soler. URL htt://wissrech.ins.ni-bonn.de/research/rojects/nast3dgp/index.htm 3. Emmett, M., Minion, M.L.: Toward an efficient arallel in time method for artial differential eqations. Comm. A. Math. and Com. Sci. 7, (2012) 4. Farhat, C., Chandesris, M.: Time-decomosed arallel time-integrators: Theory and feasibility stdies for flid, strctre, and flid-strctre alications. Int. J. Nmer. Methods Engrg. 58, (2005) 5. Fischer, P.F., Hecht, F., Maday, Y.: A arareal in time semi-imlicit aroximation of the Naier-Stokes eqations. In: R. Kornhber, et al. (eds.) Domain Decomosition Methods in Science and Engineering, Lectre Notes in Comtational Science and Engineering, ol. 40, Sringer, Berlin (2005) 6. Gander, M.J., Vandewalle, S.: Analysis of the arareal time-arallel time-integration method. SIAM J. Sci. Com. 29(2), (2007) 7. Gaskell, P., La, A.: Cratre-comensated conectie transort: SMART a new bondedness-resering transort algorithm. Int. J. Nmer. Methods Fids 8, (1988) 8. Griebel, M., Dornseifer, T., Nenhoeffer, T.: Nmerical Simlation in Flid Dynamics, a Practical Introdction. SIAM, Philadelhia (1998) 9. Leonard, B.: A stable and accrate conectie modelling rocedre based on qadratic stream interolation. Comt. Methods Al. Mech. Eng. 19, (1979) 10. Lions, J.L., Maday, Y., Trinici, G.: A arareal in time discretization of PDE s. C. R. Acad. Sci. Ser. I Math. 332, (2001) 11. Minion, M.L.: A hybrid arareal sectral deferred corrections method. Comm. A. Math. and Com. Sci. 5(2), (2010) 12. Rrecht, D., Krase, R.: Exlicit arallel-in-time integration of a linear acostic-adection system. Comters & Flids 59, (2012) 13. Temam, R.: Sr l aroximation de la soltion des eqations de Naier-Stokes ar la methode des as fractionnaires II. Arch. Rational Mech. Anal. 33, (1969) 14. Trindade, J.M.F., Pereira, J.C.F.: Parallel-in-time simlation of the nsteady Naier-Stokes eqations for incomressible flow. Int J. Nmer. Meth. Flids 45, (2004) 15. Trindade, J.M.F., Pereira, J.C.F.: Parallel-in-time simlation of two-dimensional, nsteady, incomressible laminar flows. Nm. Heat Trans., Part B 50, (2006) 16. an der Vorst, H.: BiCGStab: A fast and smoothly conerging ariant of BiCG for the soltion of nonsymmetric linear systems. SIAM J. Sci. Stat. Comt. 13, 631 (1992)