Hierarchical Parallel Solution of Stochastic Systems

Transcription

1 Hierarchical Parallel Solution of Stochastic Systems Second M.I.T. Conference on Computational Fluid and Solid Mechanics Contents: Simple Model of Stochastic Flow Stochastic Galerkin Scheme Resulting Equations Iterative solvers Parallel Solution Andreas Keese, Hermann G. Matthies Institute of Scientific Computing, Technical University Braunschweig

2 Model problem Geometry.5.5 Aquifier System D Model Simple Model: «κ (x) κ (x) κ (x) κ (x) «u(x) = f(x) + b.c., x R R d u hydraulic head, κ hydraulic conductivity, f sinks and sources Questions, eg: Outflow over border R, s(u) = R R n (κ(x)u(x)) dx

3 3 Stochastic Model Uncertain knowledge of System parameters e.g. κ = κ (x, ω) «κ (x, ω) κ (x, ω) κ (x, ω) random field ω Ω = probability space of all possible system s realisations. Need < κ κ(x, ω). Assume κ(x, ω) = φ(x, γ(x, ω)) of Gaussian field γ with known nd order statistics. E.g. marginal lornormal distribution if k(x, ω) = a(x) + exp(γ(x, ω))

4 4 Samples of field κ(x)

5 5 Stochastic PDE Inserting stochastic parameters into PDE gives stochastic PDE: (κ(x, ω) u(x, ω)) = f(x, ω), n (κ(x, ω)u(x, ω)) = f N(x, ω) u(x, ω) = f D (x, ω) x R x Γ N R x Γ D R Functional of interest, e.g. s(u) = Z R n (κ(x, ω)u(x, ω)) dx Compute: statistics of the solution, e.g.: E (s(u)), var(s(u)), P {s(u) s }

6 6 Example of solution Geometry.5 flow = flow out Sources 5 Dirichlet b.c. 5.5 Mean of solution 5 Realization of κ Variance of solution Realization of solution

7 7 Karhunen Loève Expansion KL mode KL mode Solving eigenvalue problem Z cov κ (x, y)g i (x) dx = λ i g j (y) R yields spectrum {λ i } and K-L eigenfunctions g i (x) representation κ(x, ω) = E (κ(x)) + X p λi g i (x)ξ i (ω) =: i= X p λi g i (x)ξ i (ω) i= with centered uncorrelated unit-variance random variables ξ i. Truncation after r largest eigenvalues Optimal expansion

8 8 Discretisation of Probability Space Represent {ξ i } in (uncorrelated ) independent Gaussian random variables ω = (ω,..., ω m ) κ(x, ω) = κ(x, ω,..., ω m ) Approximation of SPDE (κ(x, ω) u(x, ω)) = f(x, ω) in R (Ω (m), B, Γ m ). 6 4 Realisation m = 5 Ω (m) = R m with Gaussian measure Γ m (dω) = (π) m/ exp( ω /)dω. Approximation in d + m dimensional space 6 4 Realisation m =

9 9 Spatial Discretisation FEM-ansatz in spatial dimension, N(x) = (N (x),..., N n (x)) u n (x, ω) = nx N i (x)u i (ω) = N(x)u(ω) i= and Galerkin-Conditions in space K(ω)u(ω) = f(ω). Semi-discretisation, linear equations with stochastic coefficients.

10 Stochastic Discretisation Recipe: ansatz and projection in stochastic dimensions u(ω) = X β H β (ω)u (β) =: H(ω)u, H(ω) = (...H β (ω)...), u = (...u (β)...) T. H β : Hermite-polynomials (Wiener chaos, orthonormal, E (H β H γ ) = δ βγ ). Goal: Compute coefficients u (β). By stochastic Galerkin-method, γ : E`(K(ω)H(ω)u f(ω))h γ (ω) =, #dof space #dof stoch linear equations.. Non-intrusively/uncoupled, direct projection on orthonormal ansatz: β : u (β) = E (u(ω)h β (ω)), many problems of size #dof space.

11 Stochastic Galerkin Method X Z β R N(x) E (κ(x, ω)h β (ω)h γ (ω)) N(x) t dx u (β) = E (f(ω)h γ (ω)) {z } =:f (γ) To compute residual, need to evaluate expectation, e.g. by high-dimensional integration (e.g. Monte Carlo, Smolyak quadrature) here: by expanding κ in Wiener chaos and analytic integration Rewrite Karhunen-Loève expansion: κ(x, ω) = = rx p λi ξ i (ω)g i (x) i= rx X i= α p λi ξ (α) i H α (ω)g i (x)

12 Resulting equations After inserting expansion of κ: X X X α β i Z ξ (α) i E (H α H β H γ ) {z } =: (α) β,γ N(x) p λ i g i (x) N(x) t dx {z } K i u (β) = f (γ) K i stiffness matrix of FEM discretisation for material parameter g i (x). Use deterministic FEM program in black-box-fashion. For this, we require:. Ability to set material-parameters. Function returning residuum and Jacobian K i 3. Function solving a realisation, i.e. u(ω) = K (ω)f(ω)

13 3 Tensor product structure The equations have tensor product structure K u = X i X α ξ (α) i (α) K i u = f X X α i ξ (α) i (α) β,γ K i (α) β,γ K i. β,γ K i C B β,γ K i (α) (α) u (β ). u (β N ) C A = f (γ ). f (γ N ) C A. #dof space #dof stoch linear equations. Exploit parallelism in the multiplication: parallel operator sum, distribute block vector Block matrix stored efficiently in tensor form.

14 4 Block Sparsity structure Block sparsity structure for increasing number of H α

15 5 Properties of equations K u = X i X α ξ (α) i (α) K i u = f Each K i symmetric. Block matrix K symmetric. SPDE is positive definite. But expansion in polynomials Definiteness depends on chosen stochastic fields and on stochastic expansion. To solve: Use K only as multiplication. Never construct block matrix explicitly. Krylov subspace methods (CG or MINRES) with preconditioning

16 6 Block diagonal preconditioners Let K = K = stiffness matrix for mean κ(x). Use deterministic solver as preconditioner: P = A = I K... K Good preconditioner if variance of κ not too large. Well suited for parallelisation.

17 7 Parallelisation of Matrix-Vector Product (K u) (γ) = rx i n βx β nαx α ξ (α) i (α) β,γ K i u β K i = deterministic solver. Deterministic solver may be parallel program. Parallelise Operator sum in i Run different realisations of deterministic solver in parallel. Distribute u and f Parallelise sum in β

18 8 Examples Assume: enough processors to run 4 instances of deterministic solver in parallel..) One K i per processor-group, block-vector distributed pg K u () u (3) f () f (3) pg K u (4) u (6) f (4) f (6) pg K u (7) u (9) f (7) f (9) pg 3 K 3 u () u () f () f () Parallel overhead: complete cyclic shift of u for each configuration of u: complete cyclic shift of f. Good memory usage. Much parallel overhead. Does not require to switch material parameters in running solver.

19 9 Examples.) Each K i held once, block-vector replicated 4 times. Parallel overhead: parallel sum over RHS pg K u () u () f () f () pg K u () u () f () f () pg 3 K u () u () f () f () pg 4 K 3 u () u () f () f () Poor memory usage, little parallel overhead. Does not require to switch material parameters in running solver.

20 Examples 3a.) K i replicated, block-vector distributed pg K K u () u (3) f () f (3) pg K K 3 u (4) u (6) f (4) f (6) pg 3 K K u (7) u (9) f (7) f (9) pg 4 K K 3 u () u () f () f () Parallel overhead: Cyclic shift of u, Shifts of f inside each matrix group 3b.) Matrix replication necessary if more processor groups than K i : pg K u () u (3) f () f (3) pg K u (4) u (6) f (4) f (6) pg 3 K u (7) u (9) f (7) f (9) pg 4 K u () u () f () f ()

21 More Examples 3c.) K i are replicated on all processors groups, block-vector is not replicated Parallel overhead: Cyclic shift of u, pg K K 3 u () u (3) f () f (3) pg K K 3 u (4) u (6) f (4) f (6) pg 3 K K 3 u (7) u (9) f (7) f (9) pg 4 K K 3 u () u () f () f () Allows large ansatz-spaces with not too much parallel communication. Requires to switch material parameters in running solvers.

22 Parallelisation (K u) (γ) = X i X β X α ξ (α) i (α) β,γ K i u β Assume that n spatial solvers run in parallel. Block-vectors u and f distributed May be replicated to reduce communication. The matrices K i are distributed over the spatial solvers. May be replicated some times to reduce parallel communication. allow n > number of K i. n groups of (parallel) deterministic solvers each holding a subset of matrices K i and a subset of u and f.

23 3 Parallelisation of iterative solver We use portable communication based on MPI (Message Passing Interface) Solver: Conjugate Gradients. Parallelised by parallel matrix-vector multiplication. Preconditioner: deterministic solver, one instance on each processor group Memory requirements on each processor group: Hold local K i (i.e. hold local material properties) Hold preconditioner K (i.e. hold mean material property) Hold local part of u and f

24 4 Speedup Measurements Done on Cray T3E Constant problem size, N space = 75, N stoch = 54, (9 KL-terms, order 3 polynomial chaos), total equations, 55, 5. operators K i and mean. Distributed operator, distributed block-vector, (most efficient in terms of used memory): Efficiency Relative time used Number of processors 5 5 Number of processors

25 5 Speedup Measurements Replicated operator, distributed block-vector: Efficiency Number of processors Distributed block-vector, operators replicated Efficiency Number of processors

26 6 Conclusions Linear Stochastic PDEs were solved by a stochastic Galerkin scheme. Deterministic solver is used in black-box fashion. A large linear problem results which may be stored and solved efficiently. Iterative solvers have been presented. Parallelisation is exploited on different levels hierarchical parallelisation Shows good speedup if operator or vectors are replicated. Solver written in C ++ using portable communication (MPI). Massively parallel solver allows to tackle large problems. Thank you for your attention. a.keese@tu-bs.de