Coupled Lattice Boltzmann and Molecular Dynamics simulations on massively parallel computers

Transcription

1 Coupled Lattice Boltzmann and Molecular Dynamics simulations on massively parallel computers Jens Harting 1,2, Stefan Frijters 1, Florian Janoschek 1,2, and Florian Günther 1 1 Department of Applied Physics, Eindhoven University of Technology, Den Dolech 2, 5600MB Eindhoven, The Netherlands {f.a.r.janoschek, s.c.j.frijters, f.s.guenther}@tue.nl 2 Institute for Computational Physics, University of Stuttgart, Pfaffenwaldring 27, Stuttgart, Germany jens@icp.uni-stuttgart.de Complex colloidal fluids play an important role in many branches of industry. Examples include particle-stabilized emulsions in cosmetics, food and oil industries. Their understanding requires a study which resolves their microscopic structure, while still attaining sufficiently large length scales. The lattice Boltzmann method, owing to its high degree of locality, allows for such studies, when parallelized on modern supercomputers. However, exploiting these possibilities on hundreds of thousands of cores is a non-trivial task. We report on our experiences when employing large fractions of the IBM Blue Gene/P system at the Jülich Supercomputing Centre for our simulations and summarize recent results on particle-stabilized emulsions. 1 Introduction Stabilizing emulsions by employing colloidal particles is a very attractive tool in the food, cosmetics and medical industries. The underlying microscopic processes of this stabilization can be explained by assuming an oil-water mixture. Without any additives, these liquids will phase separate, but the mixture can be stabilized by adding small particles. If these particles diffuse to the interface they will adsorb there, reducing interfacial free energy and stabilizing the mixture. The particles in these mixtures block Ostwald ripening, which is one of the main processes leading to drop coarsening in emulsions. Thus, blocking this process allows for long-term stabilization of these emulsions. The stabilized drops can be used to produce new materials with complex hierarchical structure. Interestingly, many of the properties of such systems can not be explained with theories derived for surfactantstabilized systems. This is due to the larger size of the particles compared to the surfactant molecules, and their lack of amphiphilic properties. New theoretical models have been developed (and have been underlined experimentally), incorporating specific features of particle-stabilized systems that have no direct analogue in surfactant systems. Quantitatively, however, the description of these systems still leaves to be desired. A promising method to understand the dynamic properties of particle-stabilized multiphase flows can be found in computer simulations. However, a suitable simulation algorithm must be able to deal not only with simple fluid dynamics, but it is also required to simulate several fluid species and particle-particle and particle-fluid interactions. Some recent approaches trying to solve these problems utilize the lattice Boltzmann (LB) method for the description of the solvents 1. The LB method can be seen as an alternative to conventional Navier-Stokes solvers and is well-established in the literature. A number of multiphase and multicomponent models that are comparably straightforward to implement exist, making it attractive 1

2 for our current application. In addition, boundary conditions have been developed to simulate suspended finite-size particles in flow. Microscopic studies such as these allow for a direct link to macroscopic experimental data and will lead to a sustantial improvement of our understanding of particle-laden multiphase flows. These simulations remain computationally very challenging because typical particles need to be at least 10 LB length units in diameter and typical droplet diameters should be larger by an order of magnitude. To reduce finite-size effects and to acquire a statistically relevant number of droplets, the side length of the volume of interest easily reaches LB length units touching the limit of what is currently possible on high-end supercomputers. Another constraint is given by the very small time step in the simulations. The hydrodynamic interactions between the particles need to be resolved on the same scale as the particle motion, pushing our time resolution down to the nanosecond scale. To still be able to reach steady states, typical simulation runs require millions of LB time steps. This means that 0.5 to 1 rack months on the Blue Gene/P system JUGENE in Jülich might be required for a single simulation. 2 Simulation method and implementation We use the lattice Boltzmann (LB) method as the basis for our simulations 1. In our simulations, we effect the fluid-fluid interactions through the Shan-Chen multicomponent model 2, 3, while we use a variant of Ladd s method to couple the particles to our fluid 4, 5. Particle-particle interactions and particle dynamics are modeled by a molecular dynamics algorithm. The code has recently been expanded to allow for spherical particles as well as ellipsoidal ones. For a more detailed description of our simulation methods the reader is referred to 6, 7. The development of our simulation code LB3D started already in 1999 as a parallel LB solver. As the scientific focus at that time was on the behaviour of complex fluid mixtures under shear, the ability to model up to three fluid species, one of them with amphiphilic properties, using Shan and Chen s aforementioned approach was one the first features implemented. Already in 2004, LB3D was awarded a gold star rating by the Edinburgh Parallel Computing Centre for scaling almost linearly to 1024 processors and it also won several further prizes as part of the TeraGyroid project 8. Further applications of the code encompass flow in porous media 9 and fluid interactions with rough and hydrophobic surfaces In the meantime, the application was ported to most supercomputing platforms available. A parallel molecular dynamics (MD) code was integrated into LB3D in After considerable changes and extensions both parts of the code now use the same 3-dimensional spatial decomposition scheme, the Message Passing Interface (MPI) for communication, and modern Fortran 95 language elements. The simultaneous availability of Lagrangian particles in the same code as the Eulerian LB fluid opened up new applications for LB3D, such as tracking the velocity field in micromixers using massless tracer particles 14, a coarse-grained model for red blood cells 15, or colloidal particles with variable wettability as described in this report 6. Due to the strong locality of the LB equation and because the relevant interactions in colloidal systems are short-range, an efficient parallelization of such systems can be expected. In the past, LB3D indeed showed very good scalability. On a machine such as JUGENE, the IBM Blue Gene/P system at Jülich Supercomputing Centre (JSC), with cores, however, it is crucial to minimize communication costs and serial parts of a code even further in order to maintain high efficiency when running on large portions of the machine. Initially, LB3D showed only low effi- 2

3 speedup normalized to 1024 cores before after ideal number of cores [1024] speedup normalized to 1024 cores before after ideal number of cores [1024] (a) (b) Figure 1. Strong scaling of LB3D on the Blue Gene/P before and after our optimizations. (a) relates to a system with only one fluid component so the effect of matching or mismatching topologies of network and domain decomposition can be examined better. (b) refers to a system with two fluid species and suspended particles as they are of interest in this paper. The absolute execution times for small core counts did not change significantly (taken from Ref. 7 ). ciency there in strong scaling beyond cores. At the Jülich Blue Gene/P Extreme Scaling Workshop 2011 we could relate this to a mismatch of the network topology of the domain decomposition in the code and the network actually employed for point-to-point communication. The Blue Gene/P provides direct links only between direct neighbors in a three-dimensional torus, so a mismatch can cause severe performance losses. We investigated this issue for a system of lattice sites carrying only one fluid species and no particles. Limiting ourselves to this simpler system compared to the systems of interest later on has the benefit that the reduced ratio of computation to communication costs makes possible bottlenecks more visible. We find that if we allow MPI_Cart_create() to reorder process ranks and manually choose a domain decomposition based on the known hardware topology, efficiency can be brought close to ideal for this system. A comparison of the speedup before and after this optimization is shown in Fig. 1(a). Systems containing colloids and two fluid species were known to slowly degrade in parallel efficiency when the number of cores was increased. This is demonstrated in Fig. 1(b) for a benchmark system of lattice nodes and uniformly distributed particles with a radius of 5 lattice units (resulting in a volume concentration of about 20%). Since the degradation was not visible for a pure LB system (Fig. 1(a)), it initially was attributed to load imbalances and communication overhead related to the suspended particles. During the Scaling Workshop we identified a non-parallelized loop over all particles in one of the subroutines implementing the coupling of the colloidal particles and the two fluids as the actual reason. Due to the low computational cost per iteration compared to the overall coupling costs for colloids and fluids, at smaller numbers of particles or CPU cores this part of the code was not recognized as a possible bottleneck. A complete parallelization of the respective parts of the code produced a nearly ideal speedup up to cores also for this system. Both 3

4 speedup curves are depicted in Fig. 1(b). Actual production runs require checkpointing and the output of physical observables to disk, which is not accounted for in the benchmarks above. Pickering systems require output at most once per 100 to 1000 LB time steps, so the performance of the output routines is of inferior importance. Using parallel HDF5 output allows us to store fluid density fields of 4.6GB size for a system of lattice sites and particles within on average 29s when using the whole system ( cores). This corresponds to the time required to simulate 100 LB steps, so even at maximum core count possible and maximum I/O frequency not more than 50% of the time is spent on storing output. Particle configurations are typically smaller by two orders of magnitude than a density field. Therefore, this information is still written serially by the root process. Since strongly asymmetric point-to-point communication patterns are likely to produce buffer overflows and fail at high core counts, collective MPI operations are applied for data accumulation. However, the intuitive choice of MPI_Gatherv() is not optimal since on Blue Gene/P, the specially optimized implementation of MPI_Allgatherv() proved to be significantly faster at the cost of requiring one receive buffer per task. For MPI_Allgatherv() we measure an absolute time required to write one complete particle configuration of approximately 10s and a speedup of 77 on cores compared to MPI_Gatherv(). For even larger systems, however, MPI_Allgatherv() imposes a lower limit regarding the maximum number of particles, as receive buffers able to store the whole particle configuration need to be allocated four times per node (once per core) instead of only once on the node running the root process. Figure 2. Droplets subjected to shear but stabilized by particles deform and might even break for high shear rates. 3 Simulation results We apply our method to study the deformation of a liquid droplet under steady shear. Here, the simulation setup is a typical Couette flow where the upper and lower boundary 4

5 are moved by Lees-Edwards boundary conditions. A spherical droplet with radius R is initially placed at the centre of the system, which can be characterized by its capillary number Ca ηγ R σ. The surface tension σ can be related to the Shan-Chen parameter gcc0, η is the dynamic viscosity and γ is the shear rate. The deformation of the droplet is measured by fitting an ellipse to a two-dimensional projection of the droplet, where L and B are L B the length and breadth of the ellipse, respectively: D L+B. As shown by Taylor and reconfirmed in our simulations, for two fluids of equal density and equal viscosity one 16, 17. While there have been numerous analytical studies, obtains D = Ca for D 1 experiments and simulations on the deformation of simple liquid droplets within the last century, the case of particle covered droplets has not been covered in such detail and it is still not fully clear how the stability and deformability changes when particles have adsorbed to the liquid-liquid interface. In our simulations we can study these effects in detail by varying the droplet size, the number of particles, the particle size, their contact angle, as well as the shear rate and interfacial tension. Fig. 2 shows the deformation of a particle-covered droplet for various moderate capillary numbers. As can be observed from the plot, the deformation shows an almost parabolic behaviour and a non-uniform particle distribution can be observed at the interface (see inset figures). For even larger shear rates, the droplets break up. The capillary numbers have been computed based on the radius of the large original droplet and if one would rescale it with the radius of the small droplets occurring after breakup, the curves would be shifted towards smaller Ca. Figure 3. Snapshots of a 3D Bijel (a) and a Pickering emulsion (b): particles (green) self-assemble at the interface between two immiscible fluids (shaded in red and blue) and stabilize a bicontinuous interface (a) or droplets (b). We studied in detail how particles travel towards the fluid-fluid interface and get jammed there generating fluid-bicontinuous gels (so-called Bijels 18 ) as shown in Fig. 3a. Further, it was demonstrated that by modifying the lyophobic/lyophilic properties of the solved particles, the particles travel towards the fluid-fluid interface and accumulate at the surface of a droplet thus forming a Pickering emulsion (see Fig. 3b). Large scale parameter studies as well as investigations of the time dependent domain or droplet growth in Bijels and Pickering emulsions were performed. All simulations start from a random mixture of fluids and particles, modelling a temperature quench. At the beginning of the simulation, 5

6 (a) (b) Figure 4. Two snapshots of emulsions stabilized by particles are shown for an aspect ratio m = 2, a volume concentration of C 0.2 and a contact angle θ = 90. (a) Pickering emulsion for a fluid radio of 5 : 2. (b) Bijel for a fluid radio of 1 : 1 7. small scale droplets nucleate due to ballistic motion of particles and fluids. After the nucleation regime, the droplets grow due to spinodal decomposition. At some point the phase separation comes to a halt and droplets can only continue to grow due to coalescence (Ostwald ripening), which is limited due to the particles that were captured at the fluid-fluid interface. Ostwald ripening is one of the main processes leading to drop coarsening in emulsions and foams; hence stopping it allows long-term stabilization. Time-dependent measurements like these are very hard or even impossible to perform experimentally, but the processes involved determine the final properties of the emulsion. Therefore, we expect to be able to contribute to a better understanding of how the final product depends on fluid composition, fluid properties, particle concentration, particle size, or the contact angle the fluids form with the particle surface. We have studied in detail the phase behaviour of our system and determined a phase diagram demonstrating for which values of the particle concentration, fluid decomposition or contact angle either a Bijel or a Pickering emulsion is formed 6. It is our current main focus for these phases to understand the rheological properties of Bijels and Pickering emulsions in detail. For this, further large-scale sheared simulations will be performed in order to explore the complex non-newtonian properties of these systems. Collecting the data from a number of such large-scale runs will allow us to quantitatively compare the size distribution of the stabilized droplets as well as the rheological properties of such systems with experimental data. We also studied emulsions stabilized by non-spherical, anisotropic colloidal particles: ellipsoids with an aspect ratio of m = 2. Also for these ellipsoids, we observe two different phases: the Pickering emulsion (see Fig. 4(a)) and the Bijel (see Fig. 4(b)). Fig. 5 shows the transition from a Bijel to a Pickering emulsion for these ellipsoids with an aspect ratio of m = 2 and a particle volume concentration of C 0.2. The two control parameters used for the study of the phase transition are the fluid ratio and the contact angle θ. The squares denote the configurations which lead to a Bijel whereas the circles denote a resulting Pickering emulsion. If the amount of the two fluids present in the simulation is equal or not too 6

7 5:2 Fluid ratio 9:5 4:3 1: Contact angle θ[ ] Figure 5. Phase diagram demonstrating the transition from a Bijel to a Pickering emulsion. The contact angle θ and the density ratio between the two fluids are varied. The squares denote the configurations which lead to a Bijel whereas the circles denote a Pickering emulsion 7. different (e.g. a ratio of 4 : 3) we find a Bijel for all considered contact angles. However, if the fluid ratio is increased we find a Pickering emulsion. For intermediate fluid ratios the obtained phase depends on the chosen contact angle. For example for a ratio of 9 : 5 we get a Bijel for a contact angle of 90 and a Pickering emulsion for all higher values of θ. 4 Conclusion and outlook Suspensions of colloids in multiphase flows can exhibit many surprising effects, which are of potential interest to both fundamental science and industrial applications. Our investigation of the effect of colloids on the deformability of droplets has shown that when the fluid-fluid interface is almost saturated with colloids, the deformability of a droplet at a constant capillary number increases dramatically, and the breakup behaviour of those droplets at higher capillary number is also changed quantitatively. The results obtained for the particle-covered droplets will be compared to surfactant-covered droplets in similar systems, making a quantitative link between the two. On slightly larger scale, we investigated the behaviour of the particle-stabilized Pickering emulsions and Bijels, and found that the arrival at either phase after a temperature quench depends on many properties of the system, such as fluid ratio, and size and contact angle of the colloids. Our simulation code has been extended to include ellipsoidal colloids, which will introduce another parameter that can influence the formation of a particular phase. In the future, Janus particles will also be considered, having different wetting properties on different parts of the surface of the particle. Acknowledgments Besides support from JSC and IBM experts present at the Blue Gene/P Extreme Scaling workshop 2011 we acknowledge computing resources that were granted via GSC grants hss08 and hss10 and by PRACE (project pra036). 7

8 References 1. S. Succi, The Lattice Boltzmann Equation for Fluid Dynamics and Beyond, Numerical Mathematics and Scientific Computation. Oxford University Press, Oxford, X. Shan and H. Chen, Lattice Boltzmann model for simulating flows with multiple phases and components, Phys. Rev. E, 47, no. 3, , Mar H. Chen, B. M. Boghosian, P. V. Coveney, and M. Nekovee, A ternary lattice Boltzmann model for amphiphilic fluids, Proc. R. Soc. Lond. A, 456, 2043, C.K. Aidun, Y. Lu, and E.-J. Ding, Direct analysis of particulate suspensions with inertia using the discrete Boltzmann equation, J. Fluid Mech., 373, 287, A.J.C. Ladd and R. Verberg, Lattice-Boltzmann simulations of particle-fluid suspensions, J. Stat. Phys., 104, no. 5/6, , F. Jansen and J. Harting, From bijels to Pickering emulsions: A lattice Boltzmann study, Phys. Rev. E, 83, no. 4, , Apr F. Günther, F. Janoschek, S. Frijters, and J. Harting, Complex colloidal fluids: enabling a lattice Boltzmann solver for massively parallel simulations on the Blue Gene/P, submitted for publication, J. Chin, P. V. Coveney, and J. Harting, The TeraGyroid project collaborative steering and visualization in an HPC grid for modelling complex fluids, in: Proceedings of the UK e-science All Hands Meeting, A. Narváez, T. Zauner, F. Raischel, R. Hilfer, and J. Harting, Quantitative analysis of numerical estimates for the permeability of porous media from lattice-boltzmann simulations, J. Stat. Mech: Theor. Exp., 2010, P211026, J. Harting, C. Kunert, and H.J. Herrmann, Lattice Boltzmann simulations of apparent slip in hydrophobic microchannels, Europhys. Lett., 75, , C. Kunert and J. Harting, Roughness induced apparent boundary slip in microchannel flows, Phys. Rev. Lett., 99, , J. Hyväluoma and J. Harting, Slip flow over structured surfaces with entrapped microbubbles, Phys. Rev. Lett., 100, , C. Kunert, J. Harting, and O.I. Vinogradova, Random-roughness hydrodynamic boundary conditions, Phys. Rev. Lett., 105, , A. Sarkar, A. Narváez, and J. Harting, Quantification of the degree of mixing in chaotic micromixers using finite time lyapunov exponents, submitted for publication; arxiv: , F. Janoschek, F. Toschi, and J. Harting, Simplified particulate model for coarsegrained hemodynamics simulations, Phys. Rev. E, 82, , G.I. Taylor, The Viscosity of a Fluid Containing Small Drops of Another Fluid, Proc. R. Soc. Lond. A, 138, no. 834, 41 48, G.I. Taylor, The Formation of Emulsions in Definable Fields of Flow, Proc. R. Soc. Lond. A, 146, no. 858, , K. Stratford, R. Adhikari, I. Pagonabarraga, J.-C. Desplat, and M.E. Cates, Colloidal Jamming at Interfaces: A Route to Fluid-Bicontinuous Gels, Science, 309, no. 4, ,