A simple FEM solver and its data parallelism

Transcription

1 A simple FEM solver and its data parallelism Gundolf Haase Institute for Mathematics and Scientific Computing University of Graz, Austria Chile, Jan. 2015

2 Partial differential equation

3 Considered Problem Classes Find u such that Lu(x) = f (x) x Ω lu(x) = g(x) variational formulation x Ω Find u V : a(u, v) = F, v v V FEM, FDM FVM, FIT Solve K h u h = f h u h R N h (linear) 2 nd order problem. Poisson equation (temperature) Lamé equation (deformation) Maxwell s equations (magnetic field) Matrix K h is sparse, positive definite (symmetric, large dimension) non-linear and time-dependent problems.

4 Second order PDE Find u X := C 2 (Ω) C 1 (Ω Γ 2 Γ 3) C(Ω Γ 1) such that the partial differential equation m x i i,j=1 ( a ij (x) u ) + x j m i=1 a i (x) u x i + a(x)u(x) = f (x) (1) holds for all x Ω and that the Boundary Conditions (BC) u(x) = g 1(x), x Γ 1 (Dirichlet (1 st -kind) BC), u := m a N ij (x) u(x) x j n i (x) = g 2(x), x Γ 2 i,j=1 (Neumann (2 nd -kind) BC), u + α(x)u(x) = g3(x), x N Γ3 (Robin (3rd -kind) BC). are satisfied. with u(x) as classical continuous solution of the PDE.

5 Variational formulation Choose the space of test functions V 0 = {v V = H 1 (Ω) : v = 0 on Γ 1}, where V = H 1 (Ω) is the basic space Find u V g such that a(u, v) = F, v v V 0, where ( m ) u v m u a(u, v) := a ij + a i v + auv dx + x j x i x i Ω i,j=1 i=1 Γ 3 F, v := fv dx + g 2v ds + g 3v ds, Ω Γ 2 Γ 3 V g := {v V = H 1 (Ω) : v = g 1 on Γ 1}, V 0 := {v V : v = 0 on Γ 1}. αuv ds, (2) with u(x) as weak continuous solution of the PDE.

6 Finite Elements Continuous solution u(x) discrete solution u h from the finite dimensional space } V h = span {ϕ (i) : i ω h = v h = v (i) ϕ (i) = span Φ V (3) i ω h spanned by the (linear independent) basis functions Φ = [ϕ (i) : i ω h ] = [ϕ 1,..., ϕ Nh ] with ω h as indices of basis functions. 1D linear basis functions with finite support on the neighboring elements are presented in the following picture: ϕ (1) ϕ (2) ϕ (3) 0 Basis functions 1

7 Our example: Laplace equation Find u such that u(x) = f (x) x Ω = [0, 1] 2 u(x) = 0 x Ω variational formulation Find u V : a(u, v) := T v(x) u(x)dx Ω F, v := f (x)v(x)dx Ω FEM, FDM FVM, FIT Solve K h u h = f h u h R N h with K ij := T ϕ j (x) ϕ i (x)dx = Ω T ϕ j (x) ϕ i (x)dx τ e τ e supp ϕ i supp ϕ j

8 How to solve Laplace equation? 1. Generate a finite element mesh. 2. Determine matrix pattern (sparse matrix!) and allocate storage. 3. Calculate Matrix K h and r.h.s. f h for each element. τ e T ϕ j (x) ϕ i (x)dx 4. Accumulate the element entries. τ e supp ϕ i supp ϕ j 5. Solve the system of equations K h u h = f h.

9 Discretizing the domain [xl, xr] [yb, yt] nx=ny=4 intervals Ω trangular elements linear shape functions GetMesh(nx, ny, xl, xr, yb, yt, nnode, xc, nelem, ia); OUTPUT: nnode : number of nodes xc[2*nnode] : node coordinates nelem : number of finite elements ia[3*nelem] : element connectivity (3 node numbers per element)

10 Storing the sparse matrix CRS: compressed row storage The matrix K n m = can be stored using just two integer vectors and one real/double vector. Values : sk = Column index : ik = Starting index of row : id = Dimensions for n rows and nnz non-zero elements in matrix: sk[nn], ik[nn], id[n+1] Note that (in C/C++) id[n] = nnz. also: Compressed Column Storage (CCS), Compressed Diagonal Storage (CDS), Jagged Diagonal Storage (JDS), ELLPACK,...

11 Matrix generation in code Determine matrix pattern and allocate memory for CRS Get Matrix Pattern(nelem, 3, ia, nnz, id, ik, sk); nnz : number of non-zereo elements in matrix id[nnode+1], ik[nnz] allocated and initialized sk[nnz] allocated Calculate Matrix entries and accumulate them GetMatrix (nelem, 3, ia, nnode, xc, nnz, id, ik, sk, f); sk[nnz] matrix values initialized f[nnode] r.h.s. initialized Apply Dirichlet boundary conditions ApplyDirichletBC(nx, ny, neigh, u, id, ik, sk, f); sk[nnz] matrix values adapted to B.C. f[nnode] r.h.s. adapted to B.C. nx, ny represent the geometry a input neigh represents neighboring domains in parallel context

12 Solve the system of equations via Jacobi iteration We solve Ku = f by the Jacobi iteration (ω = 1) u k+1 := u k+1 + ωd 1 ( f K u k) JacobiSolve(nnode, id, ik, sk, f, u ); until the relative error in the KD 1 K-norm is smaller than ε = D := diag(k) u := 0 r := f K u 0 w := D 1 r σ := σ 0 := (w, r) k := 0 while σ > ε 2 σ 0 do k := k + 1 u k := u k 1 + ω w // vector arithmetics r := f K u k // sparse matrix-times-vector + vector arithmetics w := D 1 r // vector arithmetics σ := (w, r) // inner product end

13 Data Parallelism for distributed memory

14 Decomposing the mesh The f.e. mesh is partitioned into P non-overlapping subdomains. (METIS,PARMETIS; SCOTCH, PT-SCOTCH) Unique mapping of an element to exacly one subdomain. Decompose linear system K ij = τ h τ h ϕ i ϕ j into two subsystems K 0 and K 1: 1. Non-overlapping decomposition of finite elements. 2. Overlapping nodes on boundary between subdomains.

15 Decomposition of matrix I Local system K ij s = τ h Ω s assembled locally: τ h ϕ i ϕ j Distribute geometry Compute local stiffness matrix Assemble local distributed equation system.

16 Decomposition of matrix II

17 Data representations accumulated distributed [j] w = 3 [i,i] M = 5 [j] [j] r s = 1 r q = [i,i] [i,i] K K q = 2 s = 3 u s = A su K s = A ska T s K ij = τ h ϕ i ϕ j τ h r = K = K ij s = τ h Ω s A T s r s s=1 A T s K sa s s=1 τ h ϕ i ϕ j

18 Parallel Linear Algebra Global-to-local map 1 A i =... 1 Scalar product w, r = w T r = w T A T i r i = i=1 Matrix-vector product (A i w) T r i = i=1 w i, r i i=1 Jacobi iteration f := A T i f i = A T i K i u i = A T i K i A i u = K u i=1 i=1 i=1 u := u + ωd 1 A T k (f k K k u k ) k=1

19 Parallel Linear Algebra no communication global communication next neighbor comm. v K s r f + α v w u + α s r R 1 w w, r = w s, r s s=1 r s A s A T k r k k=1 K s A s( A T k K k A k )A T s k=1 R = diag{r ii } N i=1 = P A s A T s s=1 s=1 and R 1 I = P A si sa T s (partition of unity)

20 Our example: Domain Decomposition Ω 3 Ω 4 Ω 1 Ω 2 Figure : Non-overlapping elements.

21 Parallel matrix generation Each process s posesses the elements of Ω s. GetMesh(nx, ny, xl, xr, yb, yt, nnode, xc, nelem, ia); with individual xl, xr, yb, yt in our example The local (distributed) matrix K ij s := τ h Ω s τ h ϕ i ϕ j is calculated by using directly the sequential routines Get Matrix Pattern(nelem, 3, ia, nnz, id, ik, sk); GetMatrix (nelem, 3, ia, nnode, xc, nnz, id, ik, sk, f); ApplyDirichletBC(nx, ny, neigh, u, id, ik, sk, f);

22 Parallel Jacobi iteration for decomposed domain We solve Ku = f by the Jacobi iteration (ω = 1) u k+1 := u k+1 + ωd 1 ( f K u k) on P processes with distributed data. JacobiSolve(nnode, id, ik, sk, f, u ); D := P A T s diag(ks)as s=1 u := 0 r := f K u 0 w := D 1 A T s r s // next neighbor comm. s=1 σ := σ 0 := (w, r) // parallel reduction k := 0 while σ > ε 2 σ 0 do k := k + 1 u k := u k 1 + ω w // no comm. r := f K u k // no comm. end // next neighbor comm. of a vector w := D 1 A T s r s // next neighbor comm. s=1 σ := (w, r) // parallel reduction