Available online at ScienceDirect. Procedia Engineering 61 (2013 ) 94 99

Transcription

1 Available online at ScienceDirect Procedia Engineering 6 (203 ) Parallel Computational Fluid Dynamics Conference (ParCFD203) Simulations of three-dimensional cavity flows with multi relaxation time lattice Boltzmann method and graphic processing units Hung-Wen Chang a, Pei-Yao Hong a, Li-Song Lin a, Chao-An Lin a, a Department of Power Mechanical Engineering, National Tsing Hua University, Hsinchu 3003, TAIWAN Abstract Three-dimensional square cavity flows are simulated using multi-relaxation lattice Boltzmann method and graphic processing units. It was found that transition takes place between 700 < Re < Parallel computations are conducted on a single node multi graphic processing unit (GPU) system, consisting of three nvidia M2070 or GTX560 devices using OpenMP. Results show that the speedup performance is strongly dependent on problem size and the precision adopted. For single precision computation with grid density, about 59 times speedup can be obtained, while for double precision computation, this is slightly degraded to 2. Both are achieved with three Tesla M2070. c 203 The Authors. Published by Elsevier Ltd. Selection Open access and/or under peer-review CC BY-NC-ND under license. responsibility of the Hunan University and Selection National Supercomputing and peer-review Center under inresponsibility Changsha (NSCC). of the Hunan University and National Supercomputing Center in Changsha (NSCC) Keywords: multi relaxation time (MRT); lattice Boltzmann model; graphic processing unit (GPU); three dimensional lid-driven cavity flow; high Reynolds number flows.. Introduction Three-dimensional cavity flows were frequently investigated both by experiments and numerical simulations. For example, Iwatsu et al.[], Guj and Stella[2] and Mei et al.[3] adopted different numerical schemes to simulate cubic cavity flows, where steady solution was shown to exist at Reynolds number being Re = Using Chebyshevcollocation technique, Albensoeder and Kuhlmann[4] presented solutions for cavity flows at various aspect ratio at Reynolds number up to 000. Feldman et al.[5] also investigated numerically the critical Reynolds number for cubic cavity and showed that the oscillatory instability occurs at Re 94. This result was later on supported by a PIV measurement[6], which concluded that the critical Reynolds number locates in region 700 < Re < 970. On the other hand, lattice Boltzmann method (LBM)[7, 8, 9, 0] has been successfully applied to various hydrodynamic problems and the major advantage of the LBM is explicit formulation. Thus, the present study aims to examine the range of critical Reynolds number on a square cavity using multi-relaxation lattice Boltzmann model due to its enhanced stability at high Reynolds number flows. Also, as an explicit numerical scheme with intensive local computation, the LBM algorithm is very suitable for parallelization. This can be achieved using the Graphical Processing Unit (GPU) through the Compute Unified Device Architecture (CUDA). The computation platform is a single node multi-gpu system consisting of three nvidia M2070 devices with OpenMP framework and its performance relative to CPU will also be addressed. Corresponding author. Tel.: ; fax: address: calin@pme.nthu.edu.tw The Authors. Published by Elsevier Ltd. Open access under CC BY-NC-ND license. Selection and peer-review under responsibility of the Hunan University and National Supercomputing Center in Changsha (NSCC) doi:0.06/j.proeng

2 Hung-Wen Chang et al. / Procedia Engineering 6 ( 203 ) Multi Relaxation Time Lattice Boltzmann Model and Boundary conditions The multi relaxation time (MRT) lattice Boltzmann method [] can be expressed as collision and streaming steps, respectively: fi = f i( x, t) Mil S lj [m j ( x, t) m eq j ( x, t)], () f i ( x + e i t, t + t) = fi ( x, t) (2) where M is a matrix that transforms the distribution function f to the velocity moment, m=mf, and S is the relaxation matrix. These will be defined later. Based on the particle distribution functions, the macroscopic density and velocity can be obtained as: f i = ρ, f i e i = ρ u (3) i i For the present 3D applications, the D3Q9 multi relaxation time model is adopted. The transform matrix M of this model is given as[2], M = and the velocity moments are correspondingly defined as, m = (ρ, e,ε, j x, q x, j y, q y, j z, q z, 3p xx, 3π xx, p ww,π ww, p xy, p yz, p zx, m x, m y, m z ) T (5) As suggested in[2], the adopted equilibrium moments are, e eq = ρ + 9 ρ ( j2 x + j 2 y + j 2 z ), ε eq = ρ ( j2 x + j 2 y + j 2 z ) q eq x = 2 3 j x, q eq y = 2 3 j y, q eq z = 2 3 j z, (4) 3 xx = ρ [2 j2 x ( j 2 y + j 2 z )], ww = ρ ( j2 y j 2 z ), xy = ρ j x j y, 3π eq xx = π eq ww = 0, yz = ρ j y j z, m eq x = m eq y xz = ρ j x j z = m eq z = 0 (6)

3 96 Hung-Wen Chang et al. / Procedia Engineering 6 ( 203 ) CPU Thread 0 The plane of symmetry Lid velocity U_T = 0. hostmem = NX*NY*NZ*sizeof() malloc(f_h), malloc(u_h),..., malloc(rho_h) init(f_h, u_h,..., rho_h) h st Primary vortex CPU Thread 0 CPU Thread cudasetdevice(0) devicemem = NX*NY*(NZ/N)*sizeof() cudasetdevice() devicemem = NX*NY*(NZ/N)*sizeof() CPU Thread N- cudasetdevice(n-) devicemem = NX*NY*(NZ/N)*sizeof() K 2nd Primary vortex Collision<<<>>>() Streaming<<<>>>() Collision<<<>>>() P2P Memcpy P2P Memcpy Streaming<<<>>>() Collision<<<>>>() P2P Memcpy Streaming<<<>>>() Boundary<<<>>>() MacroCompute<<<>>>() Boundary<<<>>>() MacroCompute<<<>>>() Boundary<<<>>>() MacroCompute<<<>>>() Y X Z CPU Thread 0 Write data Post process free(f_h), free(u_h),..., free(rho_h) Fig.. Geometric setup of 3D cavity flow Fig. 2. The simulation procedure for multi-gpu implementation Here, the relaxation matrix S is a diagonal matrix, i.e., S = diag[0, s, s 2, 0, s 4, 0, s 4, 0, s 4, s 9, s 0, s 9, s 0, s 3, s 3, s 3, s 6, s 6, s 6 ] (7) where the kinematic viscosity is given by ν = 3 ( s 9 2 ) = 3 ( s 3 2 ) (8) The relaxation parameters for density and momentum are set equal to zero in order to conserve mass and momentum, and the rest of the relaxation parameters are chosen to be 0.7 to enhance stability at high Reynolds number. For the present lid driven cavity shown in Fig., two types of boundary conditions are adopted. The first one is used for the top lid boundary which moves at a constant velocity, while the second one is applied on the stationary boundary along the remaining five walls. Boundary conditions proposed in [3, 4, 5] are employed to determine the unknown particle density distribution functions along the boundary, which are expressed as a combination of the local known value and a corrector, f i ( x, t) = f ( x, e i, t) + ω i C e i Q (9) where Q is the force like corrector to enforce the required momentum. This resembles the modification of momentum due to the presence of a body force, though this only applies to the unknown particle density distribution functions along the boundary. On the other hand, the equilibrium distribution functions with a constant density are imposed along the top moving lid as suggested by Hou et al.[6]. 3. GPU Implementation The focus of the present study is the first step towards the multi graphic processing unit(gpu) computations. There are several possibilities to enable the multi-gpu computational capability, such as Message Passing Interface (MPI) or POSIX threads programming. Here, the Open Multiprocessing (OpenMP) is adopted for single node multi-gpu implementation consisting of three GPUS. There are two types of GPU adopted, i.e. nvidia Tesla M2070 and Geforce GTX560 Ti. Each Tela M2070 has 6G GDDR5 memory with 448 CUDA cores and Geforce GTX560 Ti has G GDDR5 memory with 384 CUDA cores. Fig. 2 shows the simulation procedure for current LBM computations. First, variables including microscopic distribution functions f and macroscopic quantities, such as density and velocities, are declared and initialized in the host end. A set of variables which will be used for computation on GPU are then allocated in the global memory by each device. Because of the presence of multiple GPUs, each device only needs to compute a portion of the original

4 Hung-Wen Chang et al. / Procedia Engineering 6 ( 203 ) U 28 3 V 28 3 U 60 3 V 60 3 Albensoeder and Kuhlmann (2005) Re=000 Re=700 Re= U / U 0 U X Fig. 3. Simulation comparison with Albensoeder et al.[4] at Re = 000, cubic cavity Time Step Fig. 4. Time evolutions of horizontal velocity at the monitor point, K= fluid domain. For example, when simulating a fluid problem with NX*NY*NZ problem size on N graphic cards, each device is responsible for only NX*NY*(NZ/N) grids. Therefore, required memory space for each device is reduced and larger fluid problems can be simulated. Under the OpenMP structure, each device is controlled by a corresponding thread when performing the kernel executions. For convenience, in current GPU implementations, each thread is responsible for the computational tasks on the single grid. Tasks for each kernel are described as follows: Collision step: In this step, processors will execute collision operator which is the right hand side of Eq. (). To perform the collision step, only local computations will take place and no interaction between grid appears. Distribution functions in the post collision state, which can be denoted as f, are stored in a temporary array for the usage of streaming step. Streaming step: In the streaming function, each post distribution function f will move to the neighbor grid by following its lattice velocity direction, i.e. in Eq. (2). Due to the requirement of information in the neighbor gird, data transfers between different device will take place. The memory copies between devices are performed by using the cudamemcpypeerasync() command. After the data transfers between graphic cards, the local propagation is then performed in separate device and communications between neighboring grid will show up. In each device, for grids belong to different thread blocks, it is unavoidable to access data through global memory, and the parallel performance will then be affected owing to the large latency of global memory. Boundary step: Boundary condition scheme is imposed in this function. For the five stationary walls, boundary condition proposed by Ho et al.[3] is adopted. While for the moving lid wall, the equilibrium boundary condition is used[6]. Macro-computing step: Density and velocities in each computational grid are calculated in this step. Those updated values will then be used for the next iteration. When maximum time step is reached, these macroscopic quantities will be transferred back to host memory and the whole simulation loop is closed. In this study, the number of threads in a block and the number of blocks in each device are chosen by following the principle that one thread is responsible for all computation executions in one grid, which makes the GPU implementation much straightforward and convenient. Additionally, single and double precision computations are adopted, where the latter is required for deep cavity flow[7].

5 98 Hung-Wen Chang et al. / Procedia Engineering 6 ( 203 ) Numerical results The geometric setup of three-dimensional cavity is shown in Fig.. Here, W is the width in the x direction, L is the length in the z direction and H is the depth in the y direction. The ratios between W and L and W and H are fixed to one. The cavity Reynolds number is defined as Re = U lid W/ν. Firstly, grid sensitivity test is conducted to ensure the simulation is grid independent. Cubic cavity flow at Reynolds number being 000 is simulated and the results are compared with the solutions presented by Albensoeder et al.[4]. As shown in Fig. 3, LBM solutions become grid independent with grid density 28 3 and 60 3, and the results are in good agreement with the benchmark solutions. Therefore, in the following simulations, grid density is adopted. One of the issue addressed in the present study is to explore the region of the critical Reynolds number where the first Hopf bifurcation takes place. This is achieved by monitoring the time history of the horizontal velocity components at the location (x,y,z)=(0.2,0.2,0.5). Fig. 4 shows the time evolutions of the horizontal velocity inside the cubic cavity. At Reynolds number 000 and 700, the velocity profiles remains unchanged as time proceeds, indicating that steady solutions have reached. When the Reynolds number increases to 2000, the time evolution pattern becomes periodic and the flow field is unsteady. 5. Parallel performance The performance of current GPU implementation is addressed here. Cubic cavity flow at Reynolds number being 00 is used as the test problem. Each examined case will run 5000 iterations and the required computational times are documented and used to investigate the performance. The speedup performance as well as the scalability of multi- GPU implementation are studied. Three different sets of grid density are selected, i.e. 96 3, 92 3, and The multi-gpu platform is made up of three M2070 or three GTX560 Ti graphics processing units. The compared CPU counterpart is the Intel Core-i7 990 processor. The CUDA program is compiled using nvcc 4.2 version, while the CPU program is compiled with the gcc and icc. Figs. 5 and 6 show the acceleration of computational time with different problem sizes with single and double precision, respectively. For single precision computation with 96 3 grid density, about 38 times speedups can be obtained using three GTX560 GPUS, and 35 times speedups are obtained with three M2070 devices. By increasing the problem sizes, the Tesla M2070 outperforms GTX560 due the larger on card memory size. For problem size equals to 240 3, this value is enhanced to 59 usimg three M2070 devices. On the other hand, double precision computation results show a slightly degraded performance compared with the single precision counterpart, and 2 times acceleration is achieved under the biggest examined problem size using M2070. It should be noted that more GTX560 devices are required to accommodate the memory requirements as the problem size increases. 6. Conclusion The multi-relaxation lattice Boltzmann model is adopted to simulate three-dimensional cubic cavity flows. It was found that transition takes place between 700 < Re < 2000, which is consistent with the results published by Liberzon et al.[6]. Also, the computations of the present study are conducted on a single node multi-gpu system, where three nvidia M2070 or GTX560 devices execute simultaneously using OpenMP. Results show that the speedup performance is strongly dependent on problem size and the precision adopted. For single precision computation with grid density, about 59 times speedup can be obtained, while for double precision computation, this is slightly degraded to 2. Both are achieved with three Tesla M2070. GTX560 s performance is compatible with Tesla M2070. It should be noted that more GTX560 devices are required to accommodate the memory requirements as the problem size increases. Acknowledgments The authors gratefully acknowledge the supports by the Taiwan National Science Council and Low carbon energy research center of National Tsing Hua University and the computational facilities provided by the Taiwan National Center for High-Performance Computing.

6 Hung-Wen Chang et al. / Procedia Engineering 6 ( 203 ) CPU(GCC) M2070 x 560Ti x CPU(ICC) M2070 x2 560Ti x2 5.2x M2070 x3 560Ti x CPU(GCC) M2070 x 560Ti x CPU(ICC) M2070 x2 M2070 x3 560Ti x2 560Ti x3 3.33x 3.8x Log 0 (Time) x 49.62x 49.92x 96.62x 02.53x 35.07x 38.04x 52.23x 50.97x 09.07x 0.86x 58.62x 54.74x 52.94x 08.04x 58.76x x Log 0 (Time) x 35.3x 39.67x 70.4x 77.77x 95.99x 99.47x 36.33x x 78.05x 07.4x 38.39x 77.23x.74x 00.57x Problem Size Problem Size Fig. 5. Parallel performance under different problem sizes: single precision Fig. 6. Parallel performance under different problem sizes: double precision References [] Iwatsu R., Ishii K., Kawamura T., Kuwahara K., Hyun J. M., Numerical simulation of three-dimensional flow structure in a driven cavity, Fluid Dyn. Res., 5, 73, (989). [2] Guj G., Stella F., A vorticity-velocity method for the numerical solution of 3D incompressible flows, J. Comput. Phys., 06, 286, (993). [3] Mei R., Shyy W., Yu D., Luo L. S., Lattice Boltzmann method for 3-D flows with curved boundary, J. Comput. Phys., 6, 680, (2000). [4] Albensoeder S., Kuhlmann H. C., Accurate three-dimensional lid-driven cavity flow, J. Comput. Phys., 206, 536, (2005). [5] Feldman Y., Gelfgat A. Yu., Oscillatory instability of a three-dimensional lid-driven flow in a cube, Phys. Fluids, 22, , (200). [6] Liberzon A., Feldman Yu., Gelfgat A. Yu., Experimental observation of the steady-oscillatory transition in a cubic lid-driven cavity, Phys. Fluids, 23, 08406, (20). [7] Yu D. Z., Mei R. W., Luo L. S., Shyy W., Viscous flow computations with the method of lattice Boltzmann equation, Prog Aeosp Sci., 39, 329 (2003). [8] Lin L. S., Chen Y. C., Lin C. A., Multi relaxation time lattice Boltzmann simulations of deep lid driven cavity flows at different aspect ratios, Computers & Fluids, 45, 233, (20). [9] Shih C. H., Wu C. L., Chang L. C., Lin C. A., Lattice Boltzmann simulations of incompressible liquid-gas systems on partial wetting surfaces. Philos, Trans R Soc A-Math Phys Eng Sci., 369, 250 (20). [0] Lin K. H., Liao, C. C., Lien, S. Y., Lin, C. A., Thermal lattice Boltzmann simulations of natural convection with complex geometry, Computers & Fluids, 69, 35 (202). [] Lallemand P., Luo L. S., Theory of the lattice Boltzmann method: dispersion, dissipation, isotropy, Galilean invariance, and stability, Physical Review E, 6, 6546, (2000). [2] d Humières D., Ginzburg I., Krafczyk M., Lallemand P., Luo L. S., Multiple-relaxation-time lattice Boltzmann models in three dimensions, Philosophical Transaction of the Royal Society of London Series A-Mathematical Physical and Engineering Sciences, 360, 437, (2002). [3] Ho C. F., Chang C., Lin K. H., Lin C. A., Consistent boundary conditions for 2D and 3D laminar Lattice Boltzmann Simulations, CMES- Computer Modeling in Engineering and Sciences, 44, 37, (2009). [4] Chang C., Liu C. H., Lin C. A., Boundary conditions for lattice Boltzmann simulations with complex geometry flows, Computers and Mathematics with applications, 58, 940, (2009). [5] Liu C. H., Lin K. H., Mai H. C., Lin C. A., 200, Thermal boundary conditions for thermal Lattice Boltzmann simulations, Computers and Mathematics with applications, 59, 278, (200). [6] Hou S., Zou Q., Chen S., Doolen G., Cogley A. C., Simulation of cavity flow by the lattice Boltzmann method, J. Comput. Phys., 8, 329, (995). [7] Lin L. S., Chang H. W., Lin C. A., Multi relaxation time lattice Boltzmann simulations of transition in deep 2D lid driven cavity using GPU, Computers & Fluids (202), doi:0.06/j.compfluid