Applications of Lattice Boltzmann Methods

Size: px

Start display at page:

Download "Applications of Lattice Boltzmann Methods"

Chad Simpson
6 years ago
Views:

1 Applications of Lattice Boltzmann Methods Dominik Bartuschat, Martin Bauer, Simon Bogner, Christian Godenschwager, Florian Schornbaum, Ulrich Rüde Erlangen, Germany March 1, 2016 NUMET 2016

2 D.Bartuschat, M. Bauer, S. Bogner, C. Godenschwager, F. Schornbaum, U.Rüde Chair for System Simulation, FAU Erlangen-Nürnberg

3 Outline The walberla Simulation Framework The Lattice Boltzmann Method and Complex Flows Fluid-Particle Interaction Charged Particles in Fluid Flow Free Surface Flow 3

4 The walberla Simulation Framework

5 walberla Widely applicable Lattice Boltzmann framework. Suited for various flow applications. Large-scale, MPI-based parallelization. Dynamic application switches for heterogeneous architectures and optimization. 5

walberla Concepts Block concept: Domain partitioned into cartesian grid of blocks. Blocks can be assigned to different processes. Blocks contain: cell data, e.g. fluid density, electric potential.

6 walberla Concepts Block concept: Domain partitioned into cartesian grid of blocks. Blocks can be assigned to different processes. Blocks contain: cell data, e.g. fluid density, electric potential. global information e.g. MPI rank, location. Communication concept: Simple communication mechanism on uniform grids, utilizing MPI. Ghost layers to exchange cell data with neighboring blocks. Sweep concept: Sweeps are work steps of a time-loop, performed on block-parallel level. Example: MG sweep, contains sub-sweeps (restriction, prolongation, smoothing). 6

7 The Lattice Boltzmann Method and flow in complex geometries

8 Lattice Boltzmann Method f q (x i + c q dt, t n + dt) f q (x i, t n )=dt C q (f q (x i, t n )) dt C q Discrete lattice Boltzmann equation with collision operator Note: two relaxation time (TRT) model used for most simulations. Domain discretized in cubes (cells). Discrete velocities cq and associated distribution functions fq per cell. D3Q19 model 8

9 Stream-Collide The equation is solved in two steps: Stream step: f q (x i + e q, t n + dt) = f q (x i, t n ) Collide step (SRT): f q (x i, t n )=f q (x i, t n ) 1 τ fq (x i, t n ) fq eq (x i, t n ) 9

10 Flow in Complex Geometries Flow through Coronary Arteries Sparse, but coherent geometry Volume fraction 0.3% Multiple blocks per process Load balancing required 10

11 Complex Geometry Initialization 11

Domain Partitioning Domain partitioning of coronary tree dataset Partitioning for aim of one block per process JUQUEEN nodeboard 512 processes, 485 blocks JUQUEEN, full machine 458 752 processes, 458

12 Domain Partitioning Domain partitioning of coronary tree dataset Partitioning for aim of one block per process JUQUEEN nodeboard 512 processes, 485 blocks JUQUEEN, full machine processes, blocks Excellent scaling on JUQUEEN with TRT on up to 1 trillion lattice cells* * C. Godenschwager et al. A Framework for Hybrid Parallel Flow Simulations with a Trillion Cells in Complex Geometries (2013), doi: /

Blood Flow and Perfusion Flow simulation in arteries + myocardium as porous medium modeled by means of LBM forcing term (SRT+Guo forcing*) Pressure boundary conditions at aorta and

13 Blood Flow and Perfusion Flow simulation in arteries + myocardium as porous medium modeled by means of LBM forcing term (SRT+Guo forcing*) Pressure boundary conditions at aorta and endocardium Blood flow visualization by particle tracing * Z. Guo, T. Zhao. Lattice Boltzmann model for incompressible flows through porous media (2002), doi: /physreve

14 Perfusion Results 14

15 Fluid-Particle Interaction with LBM and tumbling spherocylinders in Stokes flow

16 Fluid-Particle Interaction 4-Way Coupling Particles mapped onto lattice Boltzmann grid Each lattice node with cell center inside object is treated as moving boundary Hydrodynamic forces of fluid on particle computed by momentum exchange method* * N. Nguyen, A. Ladd. Lubrication corrections for lattice-boltzmann simulations of particle suspensions (2002), doi: /physreve

17 Moving Obstacle Treatment Influence of particle on fluid Particles act on fluid by moving wall boundary condition: f q (x f, t n + dt) = f q (x f, t n ) 2 ω q c 2 s ρ 0 c q u s Reconstruction of PDFs (f eq ) for new fluid cells dependent on uw:,- indicates opposing direction Influence of fluid on particle Hydrodynamic force on all particle surface cells s: F h = s q D s 2 f q (x f, t n ) 2 ω q c 2 s ρ 0 c q u s c q dx 3 dt cq: discrete lattice velocity ωq: lattice weight (s. LBM model) ρ0: fluid reference density Ds: set of directions from which cell s is accessed from fluid cells 17

Tumbling Spherocylinders in Stokes Flow Tumbling motion of elongated particles in Stokes flow Four spherocylinders in periodic domain, aspect ratio 1/ε = length radius = 12 LBM simulations with TRT

18 Tumbling Spherocylinders in Stokes Flow Tumbling motion of elongated particles in Stokes flow Four spherocylinders in periodic domain, aspect ratio 1/ε = length radius = 12 LBM simulations with TRT operator and comparison against slender body formulation (examining influence of inertia, wall effects, and particle shape)* * D. Bartuschat et al. Two Computational Models for Simulating the Tumbling Motion of Elongated Particles in Fluids (2016), doi: /j.compfluid

Two Tumbling Spherocylinders Motion dependent on aspect ratio 1/ε 250 10 6 ε =1/10 ε =1/12 ε =1/14 Flow Field around two spherocylinders,1/ε =12 200 x [m] 150 100 50 200 10 6 ux [m/s] uz [m/s] 100 0

19 Two Tumbling Spherocylinders Motion dependent on aspect ratio 1/ε ε =1/10 ε =1/12 ε =1/14 Flow Field around two spherocylinders,1/ε = x [m] ux [m/s] uz [m/s] t [s] Convergence to preferred distance xmax due to inertia Max. velocity uz for horizontal particle orientation, min. uz for vertical orientation Domain size: [576 dx] 3 Fluid: Water (T=20 C) Time steps: dx=4.98µm, dt= s, τ =6 Particle density: 1492 kg/m 3, radius = 4dx Runtime on LiMa: 16h, 768 cores 19

20 Charged Particles in Fluid Flow for particle-laden electrokinetic flows

Motivation Simulating separation (or agglomeration) of charged particles in micro-fluid flow, influenced by external electric fields Medical

21 Motivation Simulating separation (or agglomeration) of charged particles in micro-fluid flow, influenced by external electric fields Medical applications: Optimization of Lab-on-a-Chip systems: Separation of different cells Trapping cells and viruses Deposition of charged aerosol particles in respiratory tract (e.g. drug delivery) Industrial applications (agglomeration): Filtering particulates from exhaust gases Charged particle deposition in cooling systems of fuel cells Kang and Li Electrokinetic motion of particles and cells in microchannels Microfluidics and Nanofluidics 21

22 Multi-Physics Simulation electrostatic force Rigid body dynamics charge density Electro (quasi) statics object movement hydrodynamic force ion convection force on ions Fluid dynamics 22

Poisson Equation and Force on Particles Electric potential described by Poisson equation, with particle s charge density on RHS: Φ(x) = ρ particles (x) r 0 Discretized by finite volumes

23 Poisson Equation and Force on Particles Electric potential described by Poisson equation, with particle s charge density on RHS: Φ(x) = ρ particles (x) r 0 Discretized by finite volumes Solved with cell-centered multigrid solver implemented in walberla Subsampling for computing overlap degree to set RHS accordingly Electrostatic force on particle: F q = q particle Φ(x) 23

24 6-Way Coupling for Charged Particles charge distribution velocity BCs Finite volumes MG treat BCs V-cycle iter at. object motion hydrodynam. force LBM treat BCs stream-collide step electrostat. force Newtonian mechanics collision response object distance correction force Lubrication correction D. Bartuschat, U. Rüde. Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows (2015), doi: /j.jocs

25 Charged Particles Algorithm foreach time step, do // solve Poisson equation with particle charge density set RHS of Poisson equation while residual too high do perform multigrid v-cycle to solve Poisson equation // solve lattice Boltzmann equation considering particle velocities begin perform stream step compute macroscopic variables perform collide step end // couple potential solver and LBM with pe begin apply hydrodynamic force to particles apply electrostatic force to particles pe moves particles depending on forces end 25

26 Charged Particles Multigrid Solver

27 Multigrid Iterative method for efficient solution of sparse linear systems Based on Smoothing principle: High-frequency error elimination by iterative solvers (e.g. GS) Coarse grid principle: Restriction to coarser grid transforms low-frequency error components to relative higher-frequency ones Smoothing on coarse grids. Prolongation of obtained correction terms to finer grid Applied recursively, V(νpre, νpost)-cycle 27

28 Cell-Centered Multigrid - Implementation All operations implemented as compact stencil operations Design goals: Efficient and robust black-box solver Handling complex boundary conditions on coarse levels Naturally extensible to jumping coefficients Method of choice: Galerkin coarsening (FV) Stencils stored for each unknown On finest level: quasi-constant stencils P1 P3 P2 P4 Averaging restriction, constant prolongation Preserves D3Q7 stencil on coarse grids Convergence rate deteriorates Workaround for Poisson problem: Overrelaxing prolongation* * Mohr, Wienands Cell-centered Multigrid Revisited, Comput. Vis. Sci. (2004) 28

29 Charged Particles Validation

30 Validation of Electric Potential Φ/V Analytical solution for homogeneously charged particle: if r R 10 3 Φ(r) = 1 4πε e q e r 1 4πε e q e 2R 3 r R 2 Analytical solution Numerical solution Sphere surface if r <R Particle in domain MG: 5 V(2,2)-cycles Dirichlet BCs: exact solution Relative error: in order of x L Radius: 60 µm Charge: 8000 e Subsampling: factor 3 30

31 Validation of Electric Potential Determination of residual threshold: Error Residual L2 norm Iteration Error hardly reduced after residual norm smaller than 10-9 Residual threshold for simulations: V(2,2) cycles Conv. rate

32 Charged Particles Results

33 Charged Particles in Fluid Flow 33

Charged Particles in Fluid Flow Computed on 144 cores (12 nodes) of RRZE - LiMa 132 600 time steps TRT LBM with optimal Poiseuille parameter 64 3 unknowns per core 6 MG levels Channel: 2.

34 Charged Particles in Fluid Flow Computed on 144 cores (12 nodes) of RRZE - LiMa time steps TRT LBM with optimal Poiseuille parameter 64 3 unknowns per core 6 MG levels Channel: 2.56 x 5.76 x 2.56 mm Dx=10µm, Dt= s, τ =1.7 Particle radius: 80µm Particle charge: ±10 5 e Inflow:1mm/s, Outflow:0Pa Other walls: No-slip BCs Potential: ±51.2V Else: homogen. Neumann BCs 34

Scaling Setup and Environment Weak scaling: Costant size per core: 128 3 cells 9.

35 Scaling Setup and Environment Weak scaling: Costant size per core: cells 9.4% moving obstacle cells Size doubled (y-dimension) MG (Residal L2-Norm ): V(3,3) with 7 levels 10 to 45 CG coarse-grid iterations Convergence rate: x4x2 cores per node Executed on LRZ s SuperMUC: 9216 compute nodes (thin islands), each: 2 Xeon "Sandy Bridge-EP" GHz, 32 GB DDR3 RAM Infiniband interconnect Currently ranked #20 in TOP500 35

36 Weak Scaling for 240 Time Steps 400 Total runtimes/s Number of nodes Overall parallel nodes: 83% LBM Map Lubr HydrF pe MG SetRHS PtCm ElectF cores 7.1M particles D. Bartuschat, U. Rüde. Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows (2015), doi: /j.jocs

37 Mega fluid lattice site updates per sec. Weak Scaling for 240 Time Steps 10 3 MFLUPS (LBM) LBM Perform. MG Perform Number of nodes Parallel nodes: LBM 91 % MG - 1 V(3,3) 64 % MG performance restricted by coarsest-grid solving MLUPS (MG) Mega lattice site updates per sec cores 7.1M particles D. Bartuschat, U. Rüde. Parallel Multiphysics Simulations of Charged Particles in Microfluidic Flows (2015), doi: /j.jocs

38 Free Surface Flow and effects of surface tension

Free Surface Extension Volume of Fluid Approach* Only liquid phase has to be simulated** Cells are classified as either solid, liquid, gas or interface LBM only performed in fluid cells Suitable for

39 Free Surface Extension Volume of Fluid Approach* Only liquid phase has to be simulated** Cells are classified as either solid, liquid, gas or interface LBM only performed in fluid cells Suitable for phases with high density differences *C. Hirt, B. Nichols. Volume of Fluid (VOF) Method for the Dynamics of Free Boundaries (1981), doi: / (81) **C. Körner et al. Lattice Boltzmann Model for Free Surface Flow for Modeling Foaming (2005), doi: /s

Free Surface Extension Geometry Reconstruction: calculation of interface normals compute normals at triple points compute local curvature to account for surface tension Surface

40 Free Surface Extension Geometry Reconstruction: calculation of interface normals compute normals at triple points compute local curvature to account for surface tension Surface Dynamics: non-free-surface boundary treatment free surface boundary treatment LBM streaming step ( advection ) fill level updates ( mass advection ) LBM collision step conversion of cell types 40

Drop on Inclined Plane kin. viscosity: surf. tension: drop diameter: dx: τ: dt: 7.9 10-6 m 2 /s 4.82 10-2 N/m 5 mm 100 cells 0.526 2.

41 Drop on Inclined Plane kin. viscosity: surf. tension: drop diameter: dx: τ: dt: m 2 /s N/m 5 mm 100 cells s SRT with Guo forcing term* * Z. Guo et al. Discrete lattice effects on the forcing term in the lattice Boltzmann method (2002), doi: /PhysRevE

42 Domain Setup Restriction to L-shaped domain No inclined geometry, use inclined gravity instead: gravity Full free surface calculations are costly: find interface cells reconstruct interface normals and curvature mass advection cell conversions 42

43 Domain Setup Full free surface calculations are costly Only fraction of complete domain covered with fluid/interface Fluid covered region is moving Dynamic load balancing required: 43

$Domain Setup Full free surface calculations are costly Only fraction of complete domain covered with fluid/interface Fluid covered region is$

44 Domain Setup Full free surface calculations are costly Only fraction of complete domain covered with fluid/interface Fluid covered region is moving Dynamic load balancing required Simulation on 200 cores (on Emmy, RRZE) Drop resolution of 100 cells Runtime between 1 and 2 hours 44

45 Boundary Setup No-slip boundary: solid wall, enforces zero velocity at boundary Pressure boundary: pressure Dirichlet (set to capillary pressure) 45

46 Varying Inclination Angle Higher inclination causes drop to move further before being absorped 46

47 Varying Contact Angle Higher contact angle causes drop to move further before being absorped 47

48 Free Surface Flow with Particles

49 Floating Objects S. Bogner, U. Rüde. Simulation of floating bodies with lattice Boltzmann (2012), doi: /j.camwa

50 Rising Bubble S. Bogner, U. Rüde. Simulation of floating bodies with lattice Boltzmann (2012), doi: /j.camwa

51 Thank you for your attention!

A Framework for Hybrid Parallel Flow Simulations with a Trillion Cells in Complex Geometries

A Framework for Hybrid Parallel Flow Simulations with a Trillion Cells in Complex Geometries SC13, November 21 st 2013 Christian Godenschwager, Florian Schornbaum, Martin Bauer, Harald Köstler, Ulrich