Dipartimento di Scienze Matematiche

Transcription

1 Exploiting parallel computing in Discrete Fracture Network simulations: an inherently parallel optimization approach Stefano Berrone Team: Matìas Benedetto, Andrea Borio, Claudio Canuto, Corrado Fidelibus, Sandra Pieraccini, Stefano Scialò, Marco Tamburini, Fabio Vicini Dipartimento di Scienze Matematiche Workshop - 12 marzo 2014 Stefano Berrone (DISMA) Parallel Computing and DFN simulations - 12 marzo / 28

2 Outline 1 Introduction to DFNs 2 Scalability Parallel environment Scalability results 3 Uncertainty Quantification in Discrete Fracture Network Models Stochastic fracture transmissivity Stochastic geometry Stefano Berrone (DISMA) Parallel Computing and DFN simulations HPC@POLITO - 12 marzo / 28

3 Discrete Fracture Networks (DFN) DFN models can be used in simulating underground flows in fractured media. Possible applications: simulation of underground displacement of pollutant, water in aquifers, super-critical carbon dioxide... Challenges: Main features of the DFN models addressed herein: 3D network of intersecting fractures Fractures represented as planar polygons Quantity of interest is the hydraulic head evaluated with Darcy law Flux balance and hydraulic head continuity imposed across trace intersections Very large domain: high computational cost and memory requirements Complex domain: difficulties in good quality mesh generation Stefano Berrone (DISMA) Parallel Computing and DFN simulations - 12 marzo / 28

4 The problem of mesh generation Typical approach: Discretization with finite elements on totally conforming meshes geometry modifications to conform fracture intersections to the mesh Resolution of a large linear system Mortar methods to allow partial non-conformity of the mesh at fracture intersections Stefano Berrone (DISMA) Parallel Computing and DFN simulations HPC@POLITO - 12 marzo / 28

5 [B., Pieraccini, Scialò (SISC2013a-b, JCP2014)] Our approach: reformulate the problem as a PDE constrained optimization problem Independent mesh on each fracture Coupling given by the minimization of a cost functional Independent resolution of small linear systems on the fractures Stefano Berrone (DISMA) Parallel Computing and DFN simulations HPC@POLITO - 12 marzo / 28

6 Problem formulation: a coupled system of PDEs Local problems Find i I h i = h 0 i + R h id, with h 0 i V i := H 1 D,0(F i) such that: ( Ki h 0 i, v ) = (q i, v) + u i, v Si U S i,u S i + hin, v ΓiN H 1 2 (Γ in ),H 1 2 (Γ in ) (Ki R h id, v), v V i where q i L 2 (F i) is a source term and F i = Γ in Γ id with Γ in Γ id = and Γ id. Coupling (matching) conditions h i Sm h j S m = 0, for i, j I Sm, m M, u Sm i + u Sm j = 0, for i, j I Sm, m M, Stefano Berrone (DISMA) Parallel Computing and DFN simulations HPC@POLITO - 12 marzo / 28

7 PDE constrained optimization approach [BPS ] Let u be the tuple of all u i, i I, and define J : U R as J(u) = ) ( h i(u i) S h j(u j) S 2H S + us i + u S j 2U S S S Remark u being the tuple of all control variables u i S = {set of all traces} U S = H 1 2 (S), H S = H 1 2 (S) = U S the discrete problem is written as a constrained minimization problem a gradient-like method is applied U S = L 2 (S), H S = L 2 (S) = U S The core of the solution process is the (repeated) solution of local and independent problems on each fracture. This makes the approach nearly inherently parallel. Stefano Berrone (DISMA) Parallel Computing and DFN simulations HPC@POLITO - 12 marzo / 28

8 The space discretization: XFEM Extended Finite Elements have been used in [BPS ] to catch the nonsmooth behavior of the solution across the traces, without losing accuracy and still preserving a full independence of meshing process on the fractures. Proper shape functions are added, which mimic the nonsmooth behavior of the solution. Interface S Enriched DoF Reproducing el. Blending el Stefano Berrone (DISMA) Parallel Computing and DFN simulations HPC@POLITO - 12 marzo / 28

9 The space discretization: VEM In [BBPS 2014] flexibility of VEM has been used in order to catch the behavior of the solution across the traces, allowing for a partially conforming mesh, but still maintaining an independent meshing process on each fracture. Stefano Berrone (DISMA) Parallel Computing and DFN simulations HPC@POLITO - 12 marzo / 28

10 Part I - Scalability Stefano Berrone (DISMA) Part I - Scalability HPC@POLITO - 12 marzo / 28

11 Parallel Approach [BSTV] Serial algorithm 1 Build the triangulation on the fractures 2 Evaluate discrete operators on the fractures 3 Solve with a conjugate gradient (CG) iterative method 4 Post-processing - graphical representation Parallel implementation with MPI (Octave+mpi/openmpi ext package) 1 Partitioning the DFN 2 Organize communications 3 Preliminary results Stefano Berrone (DISMA) Part I - Scalability HPC@POLITO - 12 marzo / 28

12 Parallel implementation - MPI 1 Partitioning the DFN Define N p the number of processes available The DFN is seen as a graph in which fractures represents the nodes and traces the edges balanced graph partitioning: split the DFN into subsets of fractures P k, k = 1,..., N p 1 in such a way that the workload of the N p 1 processes is balanced and the number edge cut (i.e. communications) is minimized Fracture Trace Figure: Balanced Graph partitioning Stefano Berrone (DISMA) Part I - Scalability HPC@POLITO - 12 marzo / 28

13 Parallel implementation - MPI 2 Organize communications Hierarchical (modular) Master/Slave structure Figure: The Master/Slave architecture Stefano Berrone (DISMA) Part I - Scalability HPC@POLITO - 12 marzo / 28

14 Scalability results 1 3 Preliminary results Scalability tests performed on a machine with 2 6-core processors (12 physical - 24 virtual cores), before the implementation on the cluster Figure: Time reduction factor vs number of Slave processes - CG algorithm Figure: Processes independence test Stefano Berrone (DISMA) Part I - Scalability HPC@POLITO - 12 marzo / 28

15 Scalability results 2 Figure: Time reduction factor vs number of Slave processes - full algorithm Observations the full algorithm scales worse than the CG resolution phase more inherently parallel, but higher memory occupation conflicts in accessing memory have a more relevant effect Future developments Non-blocking communication routines Implementation in C language Implementation on the cluster Stefano Berrone (DISMA) Part I - Scalability HPC@POLITO - 12 marzo / 28

16 Part II - Uncertainty Quantification Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

17 Motivation DFN simulations are largely interesting in those situations in which the discrete nature of the fractures strongly impacts on the directionality of the flow. DFN are usually applied to simulate the underground displacement of pollutant, water or super-critical carbon dioxide. The simulations mainly aim at estimating the flux entity, the resulting directionality of the flux, and characteristic time. In this context we assume as characteristic quantity the flux that is affecting a fixed boundary of the DFN (fractures F 2 and F 3). A possible scenario is as follows: Assume information is available about the probability distribution of certain fracture features, such as their density, orientation, size, aspect ratio, aperture (these data may affect transmissivity). Assume a borehole is pumping some fluid underground (e.g., carbon dioxide) We are interested in evaluating the probability that the flux of carbon dioxide reaches a certain region of the underground basin, where a large outcropping fault can be a carrier for a dangerous leakage. Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

18 Toy networks for our numerical experiments [BCPS 2014] Figure: An example of toy network. Left: 3D view of the network (traces in red); right: projection on x 1 -x 2 plane Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

19 General setting We consider several toy networks with: 1 An horizontal fracture F 1 2 Two vertical orthogonal fractures F 2, F 3 (with reference to previous 2D figure: F 2 on the right of F 1, F 3 above F 1) 3 A set of additional fractures connecting the network, orthogonal to F 1, and with arbitrary orientation. Boundary conditions: 1 The east edge of F 1 acts as a source (Neumann boundary conditions set to 10) 2 On south edge of F 2 and west edge of F 3 (constant non-homogeneous) Dirichlet b.c. are set 3 All other edges are assumed to be insulated: homogeneous Neumann conditions. We consider the problem of measuring the overall flux entering fractures F 2 and F 3 through their traces, respectively. Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

20 Strategy for UQ Monte Carlo methods Stochastic Collocation Approach (Smolyak-type sparse grids): moments (mean values, variance) are computed with properly chosen weighted quadrature formulas, the weight functions corresponding to the underlying probability law. In both cases, each outcome concurring to the computation of the moments, corresponds to a simulation on the whole network. Two different frameworks: Deterministic geometry, stochastic trasmissivity Deterministic trasmissivity, stochastic geometry Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

21 We consider #I = 7 fractures. We set K 1 = K 2 = K 3 = 10 L, K i = 10 L min+(l max L min )Y i, Y i U(0, 1) for i = 4,..., N, with L min = 4, L max = 0, L = 2. Figure: DFN configuration Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

22 Figure: Flux entering F 2 (left) and F 3 (right) versus one selected stochastic variable y i, the one associated with the fracture with smallest distance from the interesection F 2 F 3 (all others set to 0.5) Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

23 Stochastic geometry We consider a geometry in which the orientation of the fractures is non-deterministic. K is fixed for all fractures. Fracture F i, for i 4, forms an angle α i with the x 1 axis which is α i = ᾱ i + α i(2y 1), y U(0, 1). Figure: Non-deterministic configuration. Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

24 Figure: Flux entering F 2 (left) and F 3 (right) versus the stochastic variable y Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

25 Another test geometry Figure: Non-deterministic configuration: increasing some fractures length. Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

26 Figure: Flux entering F 2 (left) and F 3 (right) versus the stochastic variable y Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

27 Figure: Errors on mean value (left) and variance (right) versus number of grid points. Stochastic collocation (solid lines) vs Monte Carlo (dotted lines) Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28

28 Stochastic collocation on sparse grids Number of outcomes to be considered for different numbers d of stochastic parameters Level d = 1 d = 2 d = 3 d = 4 d = Stefano Berrone (DISMA) Part II - Uncertainty Quantification HPC@POLITO - 12 marzo / 28