A method to reduce load imbalances in simulations of phase change processes with FS3D Johannes Müller Philipp Offenhäuser Martin Reitzle Workshop on Sustained Simulation Performance HLRS, Stuttgart, Germany October 9th and 10th, 2018
Introduction Institute of Aerospace Thermodynamics Droplet Dynamics Group Simulations of Multiphase flows with Free Surface 3D (FS3D) Phase changes Dynamic of Free Surface Droplet Splashing Droplet Dynamics Freezing Evaporation Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 2/18
Motivation Phase change process: supercooled water to ice Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 3/18
Motivation Numerical simulation of supercooled droplet Prozess 1 Prozess 2 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 4/18
Motivation Numerical simulation of supercooled droplet Prozess 1 Prozess 2 Load Imbalance: Prozess 1 Prozess 2 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 4/18
Outline Motivation Solidification simulations with Free Surface 3D Load-Balancing method Results Conclusions Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 5/18
Solidification simulation Volume of Fluid method (VOF) liquid 0 0 0 0 0 0 0.9 0.8 0.1 0 1 1 1 1 0.8 0.2 0 0 1 0.9 0.4 0 0 0 Fractional volume: f( x, t) = V 0 in the liquid phase solid = 0 < f < 1 in the boundary cells V cell 1 in the solid phase 1 1 1 1 0.8 0 1 1 1 1 0.9 0 solid Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 6/18
Solidification simulation Solidification model Calculation of normal vectors and curvature Evaluate surface temperatur Calculate temperature gradients at interface Solve heat conduction equation Advance interface Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 7/18
Solidification simulation Solidification model Calculation of normal vectors and curvature Evaluate surface temperatur Calculate temperature gradients at interface Solve heat conduction equation Advance interface Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 7/18
Solidification simulation Parallelization Discretized computational domain: Indexing of control volumina y n j (1, 4) (1, 3) (1, 2) (2, 4) (2, 3) (2, 2) (3, 4) (3, 3) (3, 2) (4, 4) (4, 3) (4, 2) Structured, equidistant grid Data structure is based on 3-dimensional arrays Decomposition in contiguous subarrays (1, 1) (2, 1) n i (3, 1) (4, 1) x Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 8/18
Solidification simulation Parallelization Decomposition of computational domain in subdomains cj max cj max cj min cj min ci min ci max ci min ci max Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 8/18
Solidification simulation Parallelization Evaluation of a stencil cj max cj max cj min cj min ci min ci max ci min ci max Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 8/18
Solidification simulation Parallelization Introduction of ghost cells cj max cj max cj min cj min ci min ci max ci min ci max Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 8/18
Solidification simulation Parallelization Data exchange between domains cj max cj max cj min cj min ci min ci max ci min ci max Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 8/18
Solidification simulation Load Imbalance Symmetrical domain decomposition #2 #1 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 9/18
Solidification simulation Load Imbalance Additional operations in domains with surface cells Symmetrical domain decomposition 1: 2: 3: 4: 5: 6: 7: 8: #2 9: 10: #1 11: 12: 13: 14: SUBROUTINE energy jump solid Communication tr_start = MPI_Wtime() loop k = 1,NK loop j = 1,NJ loop i = 1,NI if 0 < f (i, j, k) < 1 then Calculation end if end loop end loop end loop tr_end = MPI_Wtime() Communication Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 9/18
Solidification simulation Load Imbalance Additional operations in domains with surface cells Symmetrical domain decomposition 1: 2: 3: 4: 5: 6: 7: 8: #2 9: 10: #1 11: 12: 13: 14: SUBROUTINE energy jump solid Communication tr_start = MPI_Wtime() loop k = 1,NK loop j = 1,NJ loop i = 1,NI if 0 < f (i, j, k) < 1 then Calculation end if end loop end loop end loop tr_end = MPI_Wtime() Communication Work Load Imbalance: t2 > t1 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes (1) 9/18
Load-Balancing method Optimal Work Load Compute Blade of Cray XC40: 4 compute nodes with 2 processors each containing 12 cores. Optimal workload per core Pp 1 Lopt = Lmean = n=0 LT ptotal (2) Workload per Domain LT = XGN w (3) 1 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 10/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Decomposition by Bisection 1: assign cells with computational weight w e(i, j, k) << w o(i, j, k) n j n i 2: while D temp < D total do 3: Find Domain with highest Work Load 4: Copy Bisection Coordinates 5: Calculate expansion of domain 6: Bisection of Domain 7: old subdomain c max halved 8: new subdomain c min corrected 9: end while Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 11/18
Load-Balancing method Load Balanced Domain Decomposition Equalized Work Load Implementation of process communication: Data exchange to at least 26 neighbors (27-point stencil) Construction of neighborhood (source and destination) Construction of coordinates for send and receive arrays Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 12/18
Results Testcases Name Total number of cells Initial diameter % Surface cells Test Case 1 128 3 d = d 0 0.09 Test Case 2 128 3 d = 2d 0 0.37 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 13/18
Results Testcases Name Total number of cells Initial diameter % Surface cells Test Case 1 128 3 d = d 0 0.09 Test Case 2 128 3 d = 2d 0 0.37 Indicator for load imbalance: % Surface cells(pmax) % Surface cells(pmean) Imbalance = % Surface cells(p mean) (4) Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 13/18
Results Domain decompositions for Test Case 1 Load Balancing LB 1 LB 2 LB 3 LB 4 Weight per cell w e = w o w e < w o w e << w o w e << w o Number of processes with surface cells 2 4 8 Imbalance 7 3 1 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 14/18
Results Domain decompositions for Test Case 1 Load Balancing LB 1 LB 2 LB 3 LB 4 Weight per cell w e = w o w e < w o w e << w o w e << w o Number of processes with surface cells 2 4 8 Imbalance 7 3 1 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 14/18
Results Domain decompositions for Test Case 1 Load Balancing LB 1 LB 2 LB 3 LB 4 Weight per cell w e = w o w e < w o w e << w o w e << w o Number of processes with surface cells 2 4 8 Imbalance 7 3 1 LB 1 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 14/18
Results Domain decompositions for Test Case 1 Load Balancing LB 1 LB 2 LB 3 LB 4 Weight per cell w e = w o w e < w o w e << w o w e << w o Number of processes with surface cells 2 4 8 Imbalance 7 3 1 LB 1 LB 2 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 14/18
Results Domain decompositions for Test Case 1 Load Balancing LB 1 LB 2 LB 3 LB 4 Weight per cell w e = w o w e < w o w e << w o w e << w o Number of processes with surface cells 2 4 8 Imbalance 7 3 1 LB 1 LB 2 LB 3 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 14/18
Results Domain decompositions for Test Case 1 Load Balancing LB 1 LB 2 LB 3 LB 4 Weight per cell w e = w o w e < w o w e << w o w e << w o Number of processes with surface cells 2 4 8 Imbalance 7 3 1 increases LB 1 LB 2 LB 3 Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 14/18
Results Evaluation of domain decompositions t SY M : Time per timestep with symmetrical domain decomposition t LB : Time per timestep with load balanced domain decomposition Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 15/18
Results Evaluation of domain decompositions t SY M : Time per timestep with symmetrical domain decomposition t LB : Time per timestep with load balanced domain decomposition constant number of cores Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 15/18
Results Strong Scaling Test Case 2 Speedup = t SYM t P (5) Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 16/18
Results Strong Scaling Test Case 2 Speedup = t SYM t P (5) Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 16/18
Conclusions Load-Balanced Domain Decomposition Nearest neighbor process communication for solidification simulation Optimized performance if percentage of boundary cells is high Imbalance only in specific Subroutines Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 17/18
Conclusions Load-Balanced Domain Decomposition Nearest neighbor process communication for solidification simulation Optimized performance if percentage of boundary cells is high Imbalance only in specific Subroutines Further work: Optimization of domain decomposition Investigate optimal cell weight w opt = f(percentage of boundary cells) Optimization of the complex communication pattern Implementation for other phase change processes Johannes Müller, Institute of Aerospace Thermodynamics (ITLR), University of Stuttgart: A method to reduce load imbalances in simulations of phase change processes 17/18
Thank you! Johannes Müller email johannes.mueller@itlr.uni-stuttgart.de University of Stuttgart Institute of Aerospace Thermodynamics (ITLR) Pfaffenwaldring 31 70569 Stuttgart