GRUPO DE GEOFÍSICA MATEMÁTICA Y COMPUTACIONAL MEMORIA Nº 6

Transcription

1 GRUPO DE GEOFÍSICA MATEMÁTICA Y COMPUTACIONAL MEMORIA Nº 6 MÉXICO 2014 MEMORIAS GGMC INSTITUTO DE GEOFÍSICA UNAM

2 MIEMBROS DEL GGMC Dr. Ismael Herrera Revilla Dr. Luis Miguel de la Cruz Salas Dr. Guillermo Hernández García Dra. Graciela Herrera Zamarrón Dr. Norberto Vera Guzmán Dr. Álvaro Alberto Aldama Rodríguez Depto. Recursos Naturales Instituto de Geofísica UNAM Publicado y distribuido por: Grupo de Geofísica Matemática y Computacional GGMC Apartado Postal: CP Unidad de Apoyo Editorial eliedit@geofisica.unam.mx ii

3 Four General Purposes Massively Parallel DDM Algorithms By Ismael Herrera and Alberto Rosas-Medina del 2012 GRUPO DE GEOFÍSICA MATEMÁTICA Y COMPUTACIONAL GGMC iii

4 CONTENTS Abstract 1. INTRODUCTION 1 2. OVERVIEW OF THE DVS FRAMEWORK 4 3. THE ORIGINAL PROBLEM AND THE ORIGINAL NODES 6 4. DERIVED-NODES 8 5. THE DERIVED VECTOR-SPACE (DVS) THE GENERAL PROBLEM WITH CONSTRAINTS THE DVS-ALGORITHMS WITH CONSTRAINTS NUMERICAL PROCEDURES COMMENTS ON THE DVS NUMERICAL PROCEDURES APPLICATION OF S APPLICATION OF S APPLICATION OF a NUMERICAL RESULTS CONCLUSIONS 24 REFERENCES 25 iv

5 GGMC v

6 Four General Purposes Massively Parallel DDM Algorithms By Ismael Herrera and Alberto Rosas-Medina Instituto de Geofísica Universidad Nacional Autónoma de México (UNAM) Apdo. Postal , México, D.F Abstract Parallel computers available nowadays require massively parallel software. DDMs are the approach to this problem; they seek algorithms capable of constructing the global solution by solving local problems, in each partition subdomain, exclusively. At present, however, competitive algorithms need to incorporate constraints, such as continuity on primalnodes. This poses an additional challenge, which has been difficult to overcome. This paper is devoted to discuss four general-purposes preconditioned algorithms with constraints in which such a goal is achieved. Each one of them can be applied to symmetric, nonsymmetric and indefinite problems, and here it is shown that they have update numerical efficiency and the global solution is obtained by solving local problems, exclusively. Furthermore, the outstanding uniformity of the formulas from which they derive yields clear advantages for code development; for example, the construction of very robust codes. The new algorithms have been derived in the realm of the DVS-framework, recently introduced. Two of them are the DVS versions of BDDC and FETI-DP, respectively, which in this manner have been extended to non-symmetric and indefinite matrices. The originality of the other two is more complete, since we have not identified in the literature algorithms with clear similarities to them. Keywords: Massively-parallel algorithms; parallel-computers; non-overlapping DDM; DDM with constraints; BDDC; FETI-DP Equation Section INTRODUCTION Mathematical models of many systems of interest, including very important continuous systems of Engineering and Science, lead to a great variety of partial differential equations whose solution methods are based on the computational processing of large-scale algebraic systems. Furthermore, the incredible expansion experienced by the existing computational hardware and software has made amenable to effective treatment problems of an ever increasing diversity and complexity, posed by engineering and scientific applications. Parallel computing is outstanding among the new computational tools, especially at present when further increases in hardware speed apparently have reached

7 Ismael Herrera and Alberto Rosas Medina insurmountable barriers. Therefore, the emergence of parallel computing during the last twenty or thirty years, prompted on the part of the computational-modeling community a continued and systematic effort with the purpose of harnessing it for the endeavor of solving partial differential equations [1]. Very early after such an effort began, it was recognized that domain decomposition methods (DDM) were the most effective technique for applying parallel computing to the solution of partial differential equations, since such an approach drastically simplifies the coordination of the many processors that carry out the different tasks and also reduces very much the requirements of information-transmission between them. At the beginning much attention was given to overlapping domain decomposition methods, but soon it was realized that iterative substructuring methods in nonoverlapping partitions, known as non-overlapping DDM, were more effective [2]. Hence, domain decomposition methods (DDM) have been developed mainly as a tool for applying parallel computers to the solution of partial differential equations and to this end non-overlapping DDM formulations split the domain of definition of the partial differential equation at hand, or system of such equations, into many subdomains with the intention of developing procedures that permit solving the global problem by exclusively solving problems separately defined in each one of the partition subdomains, frequently called: local problems. The basic idea is to assign to each subdomain a different processor capable of solving the corresponding local problem. However, at present it is recognized that competitive algorithms need to incorporate constraints, such as continuity on primal-nodes [2-6]. This poses an additional challenge for developing 100% parallelizable algorithms, which has been difficult to overcome; indeed, up to now the most effective DDM-algorithms do not fully achieve the above-stated goal [7] of solving the global problem by exclusively solving local problems. The present paper is devoted to discussing four general non-overlapping DDMalgorithms recently introduced and summarized in [8], which are referred to as preconditioned DVS-algorithms with constraints. They are applicable to a wide 2

8 Four General Purposes Massively Parallel DDM Algorithms. range of matrices that stem from the discretization of partial differential equations; such matrices may be symmetric-definite, indefinite or non-symmetric. In the present paper we show that each one of these algorithms enjoys the following properties: I. Possesses update numerical efficiency. Here, by update numerical efficiency it is meant that their efficiency is at least as good that of the best domain decompositions formulations existing today. Comparisons of numerical performances were made with BDDC and FETI-DP [2-6]; and II. The global solution is obtained by exclusively solving local problems. The algorithms here discussed are formulated in the realm of the derived-vector space (DVS) framework, which was recently introduced by I. Herrera and coworkers [8-13]. The DVS, in the sense here used, was explained in [8]; the reader is referred to [8-10] for the basic developments of this approach and to [11-13] for background material on which the DVS-framework was built. The developments to be presented are based on the extensive work on DDM done during the last twenty years or so by many authors (see, for example [1,2]), albeit what is special of our approach is the use of the DVS-framework, in which our developments are set. In this respect, we point out that two algorithms of the set of preconditioned DVSalgorithms to be discussed are the DVS-versions of two well-known, very efficient non-overlapping DDM approaches: the balancing domain decomposition with constraints (BDDC) and the dual-primal finite-element tearing and interconnecting (FETI-DP) methods, respectively [14]. The balancing domain decomposition method was originally introduced by Mandel [15,16] and more recently modified by Dohrmann who introduced constraints in its formulation [3,4], while the original finite-element tearing and interconnecting method was introduced by Farhat [17,18] and later modified by the introduction of a dual-primal approach [19-21]. It should be mentioned that the BDDC and FETI-DP methods were originally developed for 3 Memorias GMMC

9 Ismael Herrera and Alberto Rosas Medina definite (positive definite) symmetric problems, although certain number of applications to non-symmetric and indefinite matrices have been made (see, for example, [6, 22-25]). On the other hand, the algorithms derived from the DVSversions of BDDC and FETI-DP are applicable to non-symmetric matrices independently of the particular problem considered. As for the other two preconditioned DVS-algorithms, their novelty is more complete, since they do not correspond to any DDM previously reported in the literature [8]. Equation Section OVERVIEW OF THE DVS FRAMEWORK The derived vector space framework (DVS-framework) starts with the system of linear equations that is obtained after the partial differential equation, or system of such equations, has been discretized; this system of discrete equations is referred to as the original problem. Independently of the discretization method used, it is assumed that a set of nodes and a domain-partition have been introduced and that both the nodes and the partition-subdomains have been numbered. Generally, some of these original-nodes belong to more than one partition-subdomain (Fig.1). For the formulation of non-overlapping domain decomposition methods it would be better if each node would belong to one and only one partition-subdomain, and a new set of nodes the derived-nodes- that enjoy this property is introduced. This new set of nodes is obtained by dividing each original-node in as many pieces as required to obtain a set with the desired property (Fig.2). Then, each derived-node is identified by means of an ordered-pair of numbers: the label of the original-node, it comes from, followed by the partition-subdomain label, it belongs to. A derivedvector is simply defined to be any real-valued function 1 defined in the whole set of derived-nodes; the set of all derived-vectors constitutes a linear space: the derived-vector space (DVS). The Euclidean inner-product, for each pair of derived-vectors, is defined to be the product of their components summed over all the derived-nodes. The DVS constitutes a finite-dimensional Hilbert-space, with 1 For the treatment of systems of equations, vector-valued functions are considered, instead 4

10 Four General Purposes Massively Parallel DDM Algorithms. Ω Ω Γ Ω3 Ω4 Figure 1. The original nodes 25 Ω 1 ( 1,1 ) ( 2,1 ) ( 3,1 ) ( 3, 2 ) ( 4, 2) ( 6,1) Γ ( 7,1) ( 11,1 ) ( 12,1) ( 8,1) ( 13,1 ) ( 13, 2) ( 5, 2) ( 8,2 ) ( 9,2 ) ( 10,2) ( 14,2 ) ( 15,2) Ω 2 ( 11,3 ) ( 12,3 ) ( 13,3 ) ( 13, 4) ( 14,4 ) ( 15,4) ( 16,3 ) ( 17,3 ) ( 18,3 ) ( 18,4 ) ( 19,4 ) ( 20,4) ( 21, 3 ) ( 22, 3 ) ( 23, 3 ) ( 23, 4 ) ( 24, 4 ) ( 25, 4) Ω3 Ω4 Figure 2. The derived nodes respect to the Euclidean inner-product. A new problem, which is equivalent to the original problem, is defined in the derived-vector space. Of course, the matrix of 5 Memorias GMMC

11 Ismael Herrera and Alberto Rosas Medina this new problem is different to the original-matrix, which is only defined in the original-vector space, and the theory supplies a formula for deriving it [8]. From there on, in the DVS framework, all the work is done in the derived-vector space and one never goes back to the original vector-space. In a systematic manner, this framework led to the construction of the following preconditioned DVS-algorithms: DVS-primal formulation #1, DVS-dual formulation #1, DVS-primal formulation #2 and DVS-dual formulation #2. However, the DVS-primal formulation #1 and DVSdual formulation #1 were later identified as DVS-versions of BDDC and FETI-DP, respectively. With this adjustment in nomenclature, in [8] they were summarized as it is indicated next. Thereby, it should be mentioned that standard versions of BDDC and FETI-DP were originally formulated for definite-symmetric matrices (positive-definite or negative-definite) [3,4,14-20]; thus, their extension to nonsymmetric and indefinite matrices is one of the achievements of the DVSframework. The Four General Preconditioned Algorithms with Constraints that will be studied in this paper are: a).- The DVS-version of BDDC; b).-the DVS-version of FETI-DP; c).- DVS-primal formulation #2; d).- DVS-dual formulation #2. All these algorithms are preconditioned and are formulated in vector-spaces subjected to constraints; so, they are preconditioned and constrained algorithms as their titles indicate. Equation Section THE ORIGINAL PROBLEM AND THE ORIGINAL NODES It will be assumed that the system of linear algebraic equations: Au = f (3.1) is the result that has been obtained after the (linear) partial differential equation, or system of such equations has been discretized. Eq.(3.1) will be referred to as the original problem and, as said before, it will be the starting point of our discussions. 6

12 Four General Purposes Massively Parallel DDM Algorithms. The DVS-framework is an axiomatic approach; so, its developments are based on a set of assumptions fulfilled by the linear system of Eq.(3.1). In this system of equations the original matrix A, and the original vectors u and f are: A A, u u, f f with i, j = n ( ij ) ( j ) ( j ) 1,..., (3.2) Here, n is the number of original nodes occurring in the original discretization of the problem. Let Ω be the domain of definition of the continuous problem, before discretization. Then, it is assumed that the family { Ω,..., 1 Ω E} of sub-domains of Ω constitutes a non-overlapping domain decomposition of Ω ; i.e., the relations are satisfied. Here, that the two sets of numbers: α β α α = 1 E Ω Ω = and Ω Ω (3.3) Ω α stands for the closure of { 1,..., n } and ˆ { 1,..., E} Ω α. Furthermore, it is assumed Ν ˆ Ε (3.4) label the original nodes and the partition-subdomains, respectively. The notation Ν ˆ α, α = 1,..., E, will be used for the subset of original-nodes that pertain to Ω α. Original nodes will be classified into internal and interface-nodes : a node is internal if it belongs to only one partition-subdomain closure and it is an interfacenode, when it belongs to more than one. Furthermore, for the development of domain decomposition methods with restrictions it will be necessary to choose a subset of interface-nodes: the set of primal nodes. We define: Νˆ ˆ Ι Ν as the set of internal-nodes; Νˆ ˆ Γ Ν as the set of interface-nodes; Νˆ ˆ π Ν as the set of primal-nodes 2 ; and 2 In order to mimic standard notations, we should have used Π instead of the low-case π. However, the modified definitions here given yield some convenient algebraic properties. 7 Memorias GMMC

13 Ismael Herrera and Alberto Rosas Medina Νˆ Νˆ Νˆ Ν ˆ as the set of dual-nodes. Γ π Γ For the time being, the set Ν ˆ π Ν ˆ may be any, depending on the specific Γ problem treated 3. Each one of the following two families of node-subsets is disjoint: { Νˆ, ˆ Ι ΝΓ} and { ˆ ˆ ˆ Ι, π, } Ν Ν Ν. Furthermore, these node subsets fulfill the relations: Ν=Ν ˆ ˆ Ν ˆ =Νˆ Νˆ Νˆ and Ν ˆ =Νˆ Ν ˆ (3.5) Ι Γ Ι π Γ π Using the notations thus far introduced, the following assumption is adopted: AXIOM 1.- Let the indices i ˆ α Ν and j ˆ β Ν be internal original-nodes, then: A = 0, whenever α β (3.6) ij The real-valued functions defined in ˆ { 1,..., n} Ν= constitute a linear vector-space that will be denoted by W and referred to as the original vector-space. When u W, we write ui u( i) for the value of u at any original-node i Ν. ˆ As said before, we use the notation u ( u,..., 1 u n ) for the original-vectors. Thereby, we observe the duality of interpretation: original-vectors can also be thought as real- Ν ˆ 1,...,n. valued functions defined in the finite-set { } Equation Section DERIVED-NODES As said before, when a non-overlapping partition is introduced some of the nodes used in the discretization belong to more than one partition-subdomain. To overcome this inconvenient feature in the DVS-framework, besides the originalnodes, another set of nodes is introduced, called the derived nodes. The general developments are better understood, through a simple example that we explain first. 3 Extensive research has been carried out to establish criteria for the choice of Ν ˆ π, which will not be discussed here. 8

14 Four General Purposes Massively Parallel DDM Algorithms. Consider the set of twenty five nodes of a non-overlapping domain decomposition, which consists of four subdomains, as shown in Fig.1. Thus, we have a set of nodes and a set of subdomains, which are numbered using of the Ε ˆ 1,2,3,4, respectively. Then, the sets of nodes index-sets: ˆ { 1,...,25} Ν and { } corresponding to such a non-overlapping domain decomposition is actually overlapping, since the four subsets Νˆ Νˆ { } { } { } { } 1, 2,3, 6, 7,8,11,12,13 Νˆ 3, 4,5,8,9,10,13,14,15 11,12,13,16,17,18, 21, 22, 23 Νˆ 13,14,15,18,19, 20, 23, 24, 25 (4.1) are not disjoint (see, Fig.1). Indeed, for example: ˆ 1 ˆ 2 Ν Ν = 3,8,13 (4.2) { } In order to obtain a truly non-overlapping decomposition, we replace the set of original nodes by another set: the set of derived nodes ; a derived node is defined to be a pair of numbers: ( p, α ), where p corresponds a node that belongs to Ω. In symbols: a derived node is a pair of numbers (, ) α p α such that p Ν ˆ. α We denote by Χ the set of derived nodes; we observe that the total number of derived-nodes is 36 while that of original-nodes is 25. Then, we define α Χ as the set of derived-nodes that can be written as ( p, α ), where α is kept fixed. Taking α successively as 1, 2, 3 and 4, we obtain the family of four subsets, {,,, } Χ Χ Χ Χ, which is a truly disjoint decomposition of Χ, in the sense that (see, Fig.2): 4 α α β Χ= Χ and Χ Χ =, when α β (4.3) α = 1 Of course, the cardinality (i.e., the number of members) of each one of these subsets is 36 4 equal to 9. The above discussion had the sole purpose of motivating the more general and formal developments that follow. So, now we go back to the general case introduced in Section 3, in which the sets of node-indexes and subdomain-indexes 9 Memorias GMMC

15 Ismael Herrera and Alberto Rosas Medina are Ν= ˆ { 1,..., n} and ˆ { 1,..., E} Ε=, respectively, and define a derived-node to be any pair of numbers, ( p, α ), such that p ˆ α Ν. Then, the total set of derived-nodes, is given by: When ( p, α ) Χ, (, ) {( ) ˆ ˆ α p, α α & p } Χ Ε Ν (4.4) p α is said to be a descendant of the full set of descendants of p is given by { } ( p) ( p, α) ( p, α) The multiplicity, m( p ), of any original-node of Ζ ( p). ˆ p Ν. Given any ˆ p Ν, Ζ Χ (4.5) p Nˆ is defined to be the cardinality In order to avoid too many repetitions in the developments that follow, where we deal extensively with derived-nodes, the notation ( p, α ) is reserved for pairs such that ( p, α ) Χ; i.e., such that With each α = 1,..., E The family of subsets { 1 E,..., } set of derived-nodes, in the sense that: p ˆ α Ν. Some subsets of Χ are defined next: {(, α) ˆ Ι } (, α) p p and { p p ˆ Γ} {( p, ) p ˆ } {(, ) ˆ π and p p } Ι Ν Γ Ν (4.6) π α Ν α Ν (4.7), we associate a unique local subset of derived-nodes : {( p, α )} α Χ (4.8) Χ Χ, is a truly disjoint decomposition of Χ, the whole E α α β Χ= Χ and Χ Χ =, when α β (4.9) α = 1 Equation Section THE DERIVED VECTOR-SPACE (DVS) Firstly, we recall from Section 3 the definition of the vector space W. Then, for each α = 1,..., E, we define the vector-subspace α W W, which is constituted by 10

16 Four General Purposes Massively Parallel DDM Algorithms. the vectors that have the property that, for each i ˆ α Ν, its i - component vanishes. With this notation, the product-space W, is defined by E α 1 E W W = W... W (5.1) α = 1 As explained in Section 2, the original problem of Eq.(2.4) is a problem formulated in the original vector-space W and in the developments that follow we transform this problem into one that is formulated in the product-space W, which is a space of discontinuous functions. By a derived-vector we mean a real-valued function 4 defined in the set of derivednodes, Χ. The set of derived-vectors constitute a linear space, which will be referred to as the derived-vector space. Corresponding to each local subset of derived-nodes, α Χ, there is a local subspace of derived-vectors, W α W, which is defined by the condition that vectors of W α vanish at every derived-node that does not belong to α u W W α Χ. A formal manner of stating this definition is, if and only if, ( ) u p, β = 0 whenever β α An important difference between the subspaces W α and W α should be observed: α W α W, while W W. In particular, 1 E W W... W = (5.2) W W. In words: the space W is the direct sum of the family of subspaces { 1 E,..., } Using Eq.(5.2), it is straightforward to establish a bijection (actually, an isomorphism) between the derived-vector space and the product-space. E W α. α = 1 For every pair of vectors, u W and w W, the Euclidean inner product is defined to be 4 For the treatment of systems of equations, such as those of linear elasticity, such functions are vector-valued. 11 Memorias GMMC

17 Ismael Herrera and Alberto Rosas Medina uw = u( p, α) w( p, α) (5.3) ( p, α ) Χ In applications of the theory to systems of equations, when u( p, α ) itself is a vector, Eq.(5.3) is replaced by uw = u( p, α) w( p, α) (5.4) ( p, α ) Χ Here, u w means the inner product of the vectors involved. An important property is that the derived-vector space, W, constitutes a finite dimensional Hilbert-space with respect to the Euclidean inner product. We observe that this definition of the Euclidean inner product is independent of the original matrix A, which can be nonsymmetric or indefinite. The natural injection, R: W W, of W into W, is defined by the condition that, for every u W, one has Ru p, α = u p, p, α Χ (5.5) ( )( ) ( ) ( ) A derived-vector, u, is said to be continuous when u( p, α ) is independent of α, whenever ( p, α ) Χ. The subset of W, of continuous vectors, constitutes a linear subspace that is denoted by W 12. It can be seen that R: W W defines a bijection of W in W 12 ; thus we write R : W W for the inverse of R, when restricted to 12 W 12. The subspace W 11 W is defined to be the orthogonal complement of W 12, with respect to the Euclidean inner-product. In this manner the space W is decomposed into two orthogonal complementary subspaces: W11 W12 W. They fulfill W and W = W11 W12 (5.6) 12

18 Four General Purposes Massively Parallel DDM Algorithms. The derived-vectors that belong to W 11 vectors. Two matrices a: W W and j: W W will be said to be zero-average W are introduced; they are the projections operators, with respect to the Euclidean inner-product, on W 12 and W 11, respectively. The first one will be referred to as the average operator and the second one will be the jump operator, respectively. We observe that in view of Eq.(5.6), every derived-vector, u W, can be written in unique manner as the sum of a zero-average vector plus a continuous vector (we could say: a zero-jump vector); indeed: u ju W u = u11 + u 12 with u 12 au W 12 (5.7) The vectors ju and au are said to be the jump and the average of u, respectively. The following linear subspaces of W will be used in the sequel and are defined by imposing some restrictions. The subspace W r W is the restricted subspace. Vectors of W I vanish at every derived-node that is not an internal node; W Γ vanish at every derived-node that is not an interface node; W π vanish at every derived-node that is not a primal node; and W vanish at every derived-node that is not a dual node. Furthermore, it assumed that W W + aw + W ; r W Π W Ι + aw π. Ι r r π Here, r a stands for the projection operator on W r. Thus far, only dual-primal restrictions have been implemented. In that case, r W W + aw + W ; Ι π 13 Memorias GMMC

19 Ismael Herrera and Alberto Rosas Medina W Π W Ι + aw π. We observe that each one of the following families of subspaces are linearly independent: And also that { W, W }, { W, W, W }, { W, W } Ι Γ Ι π Π W = W + W = W + W + W and W = W + W (5.8) Ι Γ Ι π r Π The above definition of W r is appropriate when considering dual-primal formulations; other kinds of restrictions require changing the term aw π by r aw π, where r a is a projection on the restricted subspace. Equation Section THE GENERAL PROBLEM WITH CONSTRAINTS The following result is similar to results shown in [9,10]. A proof, as well as the definition of the matrix AW : r Wr here used, is given in Appendix A of [8]. A vector u W is solution of the original problem, if and only if, u' = Ru W fulfills the equalities: The vector ( ) 12 and f W. aau ' = f and ju ' = 0 (6.1) f Rfˆ W W r, will be written as f f + f, with f W Π We remark that this problem is formulated in W : the derived-vector space. In what follows, the matrix A, when restricted to AW W r (i.e., : r r Π Π W), is assumed to be invertible. In many cases this can be granted when a sufficiently large number of primal-nodes, adequately located, are taken. Let u' Wr be solution of it, then u' W W necessarily, since ju ' = 0, and one can apply the inverse of the 12 natural injection to obtain uˆ = R u' (6.2) 14

20 Four General Purposes Massively Parallel DDM Algorithms. Thereby, we observe that the mapping 1 R is only defined when u ' is a continuous vector. Due to this fact, Eq.(6.2) is not adequate for numerical applications, in which u ' is only evaluated in an approximate manner and, therefore, usually will not be continuous; i.e., Eq.(6.2) cannot be applied to such a u '. For its numerical application, it is better to replace Eq. (6.2) by u = R au' (6.3) This replacement is very convenient, since au ' is the continuous vector closest to u ' (here, closest is with respect to the Euclidean distance). Finally, we notice that in all the algorithms to be presented all the operations are carried out in the derived-vectors space, since the problem of Eq.(6.1) is formulated in W ; in particular, we will never return to the original vector space, W, except at the end when we apply Eq.(6.3). Equation Section THE DVS-ALGORITHMS WITH CONSTRAINTS The matrix A of Eq.(6.1), can be written as (see [8,9]): A A ΠΠ Π A = A A Π Using this notation, we define the Schur-complement matrix with constraints, by Π ΠΠ Π (7.1) S A A A A (7.2) Furthermore, in what follows: f Rf A A Rf ( ) ( ) Π ΠΠ Π (7.3) 7.1- THE DVS VERSION OF THE BDDC ALGORITHM The DVS-BDDC algorithm is [8]: Find u Once u W such that as asu = as f and ju = 0 (7.4) W has been obtained, u W solution of Eq.(3.1) is given by 15 Memorias GMMC

21 Ismael Herrera and Alberto Rosas Medina {( ) ( ) } ~1 ~1 u= R a Ι A A u + A Rf ΠΠ Π ΠΠ Π (7.5) 7.2- THE DVS VERSION OF FETI-DP ALGORITHM The DVS-FETI-DP algorithm is [ ]: Given f aw, find λ W such that js js λ = js js f and aλ = 0 (7.6) Once λ jw has been obtained, u aw is given by: u = as f λ (7.7) 1 ( ) Then, Eq.(7.5) can be applied to obtain u W solution of Eq. (3.1) THE DVS PRIMAL-ALGORITHM #2 This algorithm consists in searching for a function v W, which fulfills Once v S jsw has been obtained, then S js jv = S js js f and asv = 0 (7.8) ( ) u = a S f v (7.9) and Eq.(7.5) can be applied to obtain u W solution of Eq. (3.1) THE DVS DUAL-ALGORITHM #2 This algorithm consists in searching for a function µ W, which fulfills Once SaS aµ = SaS as js f and js µ = 0 (7.10) µ SaS W has been obtained, u aw is given by: u = as f + µ (7.11) and Eq.(7.5) can be applied to obtain u W solution of Eq. (3.1). Equation Section NUMERICAL PROCEDURES 1 ( ) 16

22 Four General Purposes Massively Parallel DDM Algorithms. The numerical experiments were carried out with each one of the four preconditioned DVS-algorithms with constraints enumerated in Section 7; they are: and as asu = as f and ju = 0; DVS - BDDC (8.1) js js λ = js js f and a λ = 0; DVS - FETI - DP (8.2) S js jv = S js js f and as v = 0; DVS - PRIMAL - 2 (8.3) SaS aµ = SaS as js f and js µ = 0; DVS - DUAL - 2 (8.4) 8.1- COMMENTS ON THE DVS NUMERICAL PROCEDURES The outstanding uniformity of the formulas given in Eqs.(8.1) to (8.4) yields clear advantages for code development, especially when such codes are built using object-oriented programming techniques. Such advantages include: I. The construction of very robust codes. This is an advantage of the DVSalgorithms, which stems from the fact the definitions of such algorithms exclusively depend on the discretized system of equations (which will be referred to as the original problem) that is obtained by discretization of the partial differential equations considered, but that is otherwise independent of the problem that motivated it. In this manner, for example, essentially the same code was applied to treat 2-D and 3-D problems; indeed, only the part defining the geometry had to be changed, and that was a very small part of it; II. The codes may use of different local solvers, which can be direct or iterative solvers; III. Minimal modifications are required for transforming sequential codes into parallel ones; and IV. Such formulas also permit to develop codes in which the global-problemsolution is obtained by exclusively solving local problems. 17 Memorias GMMC

23 Ismael Herrera and Alberto Rosas Medina This last property must be highlighted, because it makes the DVS-algorithms very suitable as a tool to be used in the construction of massively-parallelized software, which is needed for efficiently programming the most powerful parallel computers available at present. Thus, procedures for constructing codes possessing Property IV are outlined and analyzed next. All the DVS-algorithms of Eqs.(8.1) to (8.4) are iterative and can be implemented with recourse to Conjugate Gradient Method (CGM), when the matrix is definite and symmetric, or some other iterative procedure such as GMRES, when that is not the case. At each iteration step, one has to compute the action on a derivedvector of one of the following matrices: as as, 1 js js, S js j or SaS a, depending on the DVS-algorithm that is applied. Such matrices in turn are different permutations of the matrices S, 1 S, a and j. Thus, to implement any of the preconditioned DVS-algorithms, one only needs to separately develop codes capable of computing the action of one of the matrices S, 1 S, a or j on an arbitrary vector of W, the derived-vector-space. Therefore, next we separately explain how to compute the application of each one of the matrices S and 1 S. As for a and j, as it will be seen, their application requires exchange of information between derived-nodes that are descendants of the same original-node.and that is a very simple operation for which such exchange of information is minimal APPLICATION OF S From Eq.(7.2), we recall the definition of the matrix S : S A A A A (8.5) Π ΠΠ Π 18

24 Four General Purposes Massively Parallel DDM Algorithms. In order to evaluate the action of S on any derived-vector, we need to successively evaluate the action of the following matrices A Π, ~1 A ΠΠ, A Π and A. Nothing special is required except for ~1 A ΠΠ, which is explained next. We have A ΠΠ t t r A A A A a ΙΙ Ιπ ΙΙ Ιπ = r t r t r A A a A a A a πι ππ πι ππ (8.6) Let w W, be an arbitrary derived-vector, and write Then, W ( π ) v is characterized by π σ ~1 ( ) π v A w (8.7) ΠΠ A v = w A A w, subjected to j π v = 0 (8.8) ππ ΠΠ π πι ΙΙ and can obtained iteratively. Here, σ ππ ( A ~1 ) { A A A A } ΠΠ ππ πι ΙΙ Ιπ and, with a π as the projection-matrix into Wr ( π ), Ι (8.9) j π π π Ι a (8.10) We observe that the computation in parallel of the action of because 1 A ΙΙ is straightforward Once ( π ) π W r A ΙΙ E α = 1 ( A α ) 1 = (8.11) v has been obtained, to derive v Ι one can apply: This completes the evaluation of S APPLICATION OF S -1 ΙΙ 1 ( Ι ) vι = A w A v π (8.12) ΙΙ Ιπ 19 Memorias GMMC

25 Ismael Herrera and Alberto Rosas Medina We define Σ Ι (8.13) A property that is relevant for the following discussion is: Therefore, the matrix Then, S: W 1 A A W fulfills For any w W, let us write Then, v π fulfills Wr ( ) W( ) can be written as: ( A ) ( A ) ΠΠ ( ) ( ) Σ = Σ (8.14) ( A ) ( A ) ( ) ( ) = = Π ΣΣ Σπ 1 1 A A A A Π πσ ππ S ( A ) 1 1 (8.15) = (8.16) v A w (8.17) ( ) ( t ) 1 r A = w π A A w Σ, subjected to j = π 0 v v (8.18) σ ππ π πσ ΣΣ Here, j r Ι a r, where the matrix r a is the projection operator on W r, while σ Furthermore, we observe that ππ t ( ) ( ) 1 A A A A A (8.19) ππ πσ ΣΣ Σπ E ( A t ) ( A α ) ΣΣ 1 = (8.20) Eq.(8.18) is solved iteratively. Once v π has been obtained, we apply: Σ α = 1 ΣΣ t ( A ) ( wσ A π ) v = v (8.21) ΣΣ Σπ This procedure permits obtaining ( A ) w. We observe that A w in general; however, what we need only is ( A 1 ) w ( A w ) = (8.22) 20

26 Four General Purposes Massively Parallel DDM Algorithms. The vector A w can be obtained by the general procedure presented above. Thus, take w w W W and Therefore, v A w (8.23) ( ) π ( ) 1 t t t r v + v = v = A A v = A A a v (8.24) Ι Σ ΣΣ Σπ ΣΣ Σπ Using Eq.(8.20), these operations can be fully parallelized. However, the detailed discussion of such procedures will be presented separately [ ]. π 8.4- APPLICATION OF a AND j. We use the notation a = ( a( i, α )( j, β ) ) (8.25) Then [10]: 1 a = δij, α Ζ( i ) and β Ζ ( j α β ) (8.26) ( i, )( j, ) mi ( ) While Therefore, j =Ι a (8.27) jw = w aw, for every w W (8.28) As for the right hand-sides of Eqs.(7.4), (7.6), (7.8) and (7.10), all they can be obtained by successively applying to f some of the operators that have already been discussed. Recalling Eq.(7.3), we have f Rf A A Rf ( ) ( ) Π ΠΠ Π (8.29) The computation of Rf does present any difficulty and the evaluation of the actions of 1 A ΠΠ and A Π were already analyzed. 21 Memorias GMMC

27 Ismael Herrera and Alberto Rosas Medina Equation Section 9 9- NUMERICAL RESULTS All the partial differential equations treated had the form: 2 a u + b u + cu = f ( x) x Ω u= gx ( ) x Ω d Ω= i= 1 ( α, β ) i i (9.1) where ac>, 0 are constants, while b= ( b1,..., bdim) is a constant vector and dim 2,3 =. The family of subdomains { } Ω,..., 1 Ω E is assumed to be a partition of the set Ω { 1,..., d} of original nodes (this count does not include the nodes that lie on the external boundary). In the applications we present, n is equal to the number of degrees of freedom (dof) of the original problem; we use linear functions and only one of them is associated with each original node. As for the original problems treated, they have the standard form of Eq.(3.1): Au = f (9.2) They were obtained by discretization of three different differential equations, in two and three dimensions, of the above boundary value problem with a = 1. Choosing b = 0 symmetric matrices were obtained, which correspond to the choices c = 1 and to c = 0, respectively; in this latter case, the differential operator treated is the Laplacian. The choices b = ( 1,1) and ( 1,1,1) b =, with c = 0, yield non-symmetric matrices. As for the right-hand side, the following choices were made: For the Laplacian operator Other differential operators ( π ) sin( π ) ( π ) ( π ) ( π ) f sin n x n y, in 2D f sin n x sin n y sin n z, in 3 D 2 2 ( ){ } ( ) ( ) ( ) f exp xy 1 x y, in 2D f exp xyz { yz 2 + xyz + x y + z }, in 3D (9.3) (9.4) 22

28 Four General Purposes Massively Parallel DDM Algorithms. Discretization was accomplished using central finite differences, but streamline artificial viscosity was incorporated in the treatment when b 0. The DQGMRES algorithm [33] was implemented for the iterative solution of the non-symmetric problems. The restrictions used were continuity on the primal nodes. The numerical results that were obtained are reported in six tables that follow. Tables 1 to 3 refer to 2D problems while Tables 4 to 6 to 3D problems. TABLE 1 Symmetric 2-D Eps=1 e-6 Subdomains Dof Primal DVS-BDDC Primal #2 DVS-FETI-DP Dual #2 2x2 X 2x x4 X 4x x6 X 6x x8 X 8x x10 X 10x x12 X 12x x14 X 14x x16 X 16x x18 X 18x x20 X 20x x22 X 22x x24 X 24x x26 X 26x x28 X 30X x30 X 30x TABLE 2 Non-Symmetric 2-D Eps=1 e-6 Subdomains Dof Primal DVS-BDDC Primal #2 DVS-FETI-DP Dual #2 2x2 X 2x x4 X 4x x6 X 6x x8 X 8x x10 X 10x x12 X 12x x14 X 14x x16 X 16x x18 X 18x x20 X 20x x22 X 22x x24 X 24x x26 X 26x x28 X 30X x30 X 30x Memorias GMMC

29 Ismael Herrera and Alberto Rosas Medina TABLE 3 Laplacian 2-D Eps=1 e-6 Subdomains Dof Primal DVS-BDDC Primal #2 DVS-FETI-DP Dual #2 2x2 X 2x x4 X 4x x6 X 6x x8 X 8x x10 X 10x x12 X 12x x14 X 14x x16 X 16x x18 X 18x x20 X 20x x22 X 22x x24 X 24x x26 X 26x x28 X 30X x30 X 30x TABLE 4 Symmetric 3-D Eps 1 e-6 Number of Iterations Partition Subdomains Dof Primals DVS-BDDC Primal #2 DVS-FETI-DP Dual #2 (2x2x2) X (2x2x2) (3x3x3) X (3x3x3) (4x4x4) X (4x4x4) (5x5x5) X (5x5x5) (6x6x6) X (6x6x6) (7x7x7) X (7x7x7) (8x8x8) X (8x8x8) (9x9x9) X (9x9x9) (10x10x10) X (10x10x10) TABLE 5 Non-Symmetric 3-D Eps 1 e-6 Number of Iterations Partition Subdomains Dof Primals DVS-BDDC Primal #2 DVS-FETI-DP Dual #2 (2x2x2) X (2x2x2) (3x3x3) X (3x3x3) (4x4x4) X (4x4x4) (5x5x5) X (5x5x5) (6x6x6) X (6x6x6) (7x7x7) X (7x7x7) (8x8x8) X (8x8x8) (9x9x9) X (9x9x9) (10x10x10) X (10x10x10) TABLE 6 Laplacian 3-D Eps 1 e-6 Number of Iterations Partition Subdomains Dof Primals DVS-BDDC Primal #2 DVS-FETI-DP Dual #2 (2x2x2) X (2x2x2) (3x3x3) X (3x3x3) (4x4x4) X (4x4x4)

30 Four General Purposes Massively Parallel DDM Algorithms. (5x5x5) X (5x5x5) (6x6x6) X (6x6x6) (7x7x7) X (7x7x7) (8x8x8) X (8x8x8) (9x9x9) X (9x9x9) (10x10x10) X (10x10x10) Equation Section CONCLUSIONS In this article, numerical tests were carried out for each one of the preconditioned DVS-algorithms with constraints [8]: a).- The DVS-version of BDDC, Eq.(8.1); b).-the DVS-version of FETI-DP, Eq.(8.2); c).- DVS-primal formulation #2, Eq.(8.3); d).- DVS-dual formulation #2, Eq.(8.4). Each one of these algorithms is applicable to symmetric-definite, indefinite and non-symmetric problems [8]. In the sequence of numerical experiments that was carried out all they have exhibited equivalent efficiencies. Furthermore, in two previous paper [9,10] the DVS-versions of BDDC and FETI-DP, were compared with standard versions BDDC and FETI-DP and it was shown that they are, at least, as efficient as them. Thus, the results of the present paper indicate that each one of the four preconditioned DVS-algorithms here discussed has efficiency comparable with the best non-overlapping DDM available at present. Thus, another important conclusion of this paper is that the four preconditioned DVS-algorithms with constraints studied in this paper numerically have update efficiency. This fact, together with their excellent parallelization properties indicate that they are very suitable as a tool to be used in the construction of massively-parallelized software that is needed for efficiently programming the most powerful parallel computers available at present. REFERENCES 25 Memorias GMMC

31 Ismael Herrera and Alberto Rosas Medina 1. DDM Organization, Proceedings of 20 International Conferences on Domain Decomposition Methods Toselli A. and O. Widlund, Domain decomposition methods- Algorithms and Theory, Springer Series in Computational Mathematics, Springer-Verlag, Berlin, 2005, 450p. 3. Dohrmann C.R., A preconditioner for substructuring based on constrained energy minimization. SIAM J. Sci. Comput. 25(1): , Mandel J. and C. R. Dohrmann, Convergence of a balancing domain decomposition by constraints and energy minimization, Numer. Linear Algebra Appl., 10(7): , Mandel J., Dohrmann C.R. and Tezaur R., An algebraic theory for primal and dual substructuring methods by constraints, Appl. Numer. Math., 54: , Da Conceição, D. T. Jr., Balancing domain decomposition preconditioners for nonsymmetric problems, Instituto Nacional de Matemática Pura e Aplicada, Agencia Nacional do Petróleo PRH-32, Rio de Janeiro, May. 9, Widlund O., Personal communication, Herrera, I., Carrillo-Ledesma A. & Rosas-Medina Alberto A Brief Overview of Non-overlapping Domain Decomposition Methods, Geofisica Internacional, Vol. 50(4), pp , Herrera, I. & Yates R. A. The Multipliers-Free Dual Primal Domain Decomposition Methods for Nonsymmetric Matrices NUMER. METH. PART D. E. 2009, DOI /Num , (Published on line April 28, 2010). 10. Herrera, I. & Yates R. A. The Multipliers-free Domain Decomposition Methods NUMER. METH. PART D. E. 26: July 2010, DOI /num (Published on line Jan 28, 2009) 11. Herrera I. and R. Yates Unified Multipliers-Free Theory of Dual-Primal Domain Decomposition Methods. NUMER. METH. PART D. E. Eq. 25: , May 2009, (Published on line May 13, 08) DOI /num Herrera, I. New Formulation of Iterative Substructuring Methods without Lagrange Multipliers: Neumann-Neumann and FETI, NUMER METH PART D E 24(3) pp , 2008 (Published on line Sep 17, 2007) DOI NO Herrera, I. Theory of Differential Equations in Discontinuous Piecewise-Defined- Functions, NUMER METH PART D E, 23(3), pp , 2007 DOI NO Brenner S. and Sung L. BDDC and FETI-DP without matrices or vectors Comput. Methods Appl. Mech. Engrg. 196(8): Mandel J. Balancing domain decomposition, Commun. Numer. Methods Engrg. 1(1993) Mandel J. and Brezina M. Balancing domain decomposition for problems with large jumps in coefficients. Math. Comput. 65, pp , Farhat Ch., and Roux F. A method of finite element tearing and interconnecting and its parallel solution algorithm. Internat. J. Numer. Methods Engrg. 32: , Mandel J. and Tezaur R. Convergence of a substructuring method with Lagrange multipliers. Numer. Math 73(4): ,

32 Four General Purposes Massively Parallel DDM Algorithms. 19. Farhat C., Lessoinne M. LeTallec P., Pierson K. and Rixen D. FETI-DP a dualprimal unified FETI method, Part I: A faster alternative to the two-level FETI method. Int. J. Numer. Methods Engrg. 50, pp , Farhat C., Lessoinne M. and Pierson K. A scalable dual-primal domain decomposition method, Numer. Linear Algebra Appl. 7, pp , Mandel J. and Tezaur R., On the convergence of a dual-primal substructuring method, SIAM J. Sci. Comput., 25, pp , Cai, X-C. & Widlund, O.B., Domain Decomposition Algorithms for Indefinite Elliptic Problems, SIAM J. Sci. Stat. Comput. 1992, Vol. 13 pp Farhat C., and Li J. An iterative domain decomposition method for the solution of a class of indefinite problems in computational structural dynamics. ELSEVIER Science Direct Applied Numerical Math. 54 pp Li J. and Tu X. Convergence analysis of a Balancing Domain Decomposition method for solving a class of indefinite linear systems. Numer. Linear Algebra Appl. 2009; 16: Tu X. and Li J., A Balancing Domain Decomposition method by constraints for advection-diffusion problems Memorias GMMC