IAMG Proceedings. The projective method approach in the electromagnetic integral equations solver. presenting author

Transcription

1 The projective method approach in the electromagnetic integral equations solver M. KRUGLYAKOV 1 AND L. BLOSHANSKAYA 2 1 Lomonsov Moscow State University, Russia m.kruglyakov@gmail.com 2 SUNY New Paltz, USA presenting author Abstract We present a new parallel (MPI+OpenMP) solver for the 3D volumetric integral equations (IE) of electrodynamics. The solver requires only half of an amount of memory compared with the other IE solvers and has high degree of parallelization. The computational experiments, including the ones performed for the high conductivity contrast problems, show solver s efficiency. The developed software is distributed under the GPLv2 license. 1 Introduction We developed a new iterative numerical solver for the 3d integral equations (IE) of electrodynamics. While the other solvers are based on the collocation method we utilize the projective method for IE solving. Instead of obtaining the general matrices as in the collocation method, we compute the symmetric and the antisymmetric matrices. This allows us to store only the upper-triangular parts of these matrices. Hence, compared to the collocation method for the same number of cells, we reduce the RAM usage almost by half. Another feature of the proposed solver is a high degree of parallelism. The computational experiments performed by Bluegene/P and Lomonosov supercomputers from Moscow State University (MSU) show that it makes the best usage of nodes for calculation at a single frequency and a single source. The maximum number of the used nodes is 2max(N x,n y ), where N x and N y are the numbers of model cells in X and Y direction respectively. This paper is organized as follows. In Section 2 we briefly describe the contracting IE (CIE) approach and the construction of the system of linear equations approximating the CIE. In Section 3 we describe the way to reduce the memory requirments by half compared to the analogus IE solvers and our approach to matrix coefficients computing. We also demonstrate the experimental results on the parallelizm efficiency. In Section 4 we provide the example of computations based on the high (more than ) conductivity contrast COMMEMI3d3 model. 2 Projection method for CIE 2.1 Contracting integral equation Let σ(m) describe the 3D conductivity distribution in space. We further assume that the electromagnteic (EM) field is induced by the external electric currents J ext. Moreover, assume that the EM fields are time dependent as e iωt, where ω is angular frequency, i = 1. Then the electrical field E and the magnetic field H give the solution of the system of Maxwell s equations curlh = σe+j ext curle = iωµ 0 H (1) E(r),H(r) 0 as r σ = σ +iωεµ 0 ISBN (DVD) 673

2 The projective method approach in the electromagnetic integral equations... σ b (z) σ a (M) T Z Figure 1: Typical model Here ε is dielectric permeability and µ 0 is magnetic permeability. For simplicity we assume that iωεµ 0 1 and that the conductivity σ is a real value function. Let T R 3 be some bounded domain and σ(m) = σ b (z) for M T and σ(m) = σ a (M) as shown in the Figure 1. In this case E(M), H(M) are expressed in terms of the integrals E(M) = E N (M)+ Ĝ E (M,M 0 ) σ (M 0 )E(M 0 )dt M0 H(M) = H N (M)+ σ (M 0 ) = σ a (M 0 ) σ b (M 0 ) T T Ĝ H (M,M 0 ) σ (M 0 )E(M 0 )dt M0 Here ĜE, ĜH are electrical and magnetic Green s tensors respectively, Pankratov et al. (1995), Dmitriev et al. (2002). The terms E N, H N are called normal electric and magnetic fields correspondingly and form the solution of the system curlh N = σ b (z)e N +J ext curle N = iωµ 0 H N (3) E N (r),h N (r) 0 as r (2) We define the operator G m E as G m E V = σ b Ĝ E [2 σ b V]+V, (4) whereĝe isanintegraloperatorfromthefirstequationin(2). TheoperatorG m E isacontracting operator in L 2 [T], see Pankratov et al. (1995). Using (2) and (4) we obtain the so-called contracting integral equation (CIE) for E: where I is the identity operator. 674 ( I G m E b a ) Ẽ = σ b E N, Ẽ = ae, a = σ a +σ b 2 σ b b = σ a σ b 2 σ b, (5)

3 2.2 Projective method approach for CIE solving Suppose the domain T is divided in nonoverlapping cells T = T k, k = 1...N and σ b (M) = σb k, σ a (M) = σa k for M T k,k = 1...N. For each cell T k we define the function W k (M) as 1,M T k W k (M) = V k, V k = dt M, k = 1...N (6) 0,M T k T k Let W N bealinearspanofthevectorfunctionsw k, W k = (W kx,w ky,w kz ),k x,k y,k z = 1...N and P N be a projection operator from L 2 [T] to W N : [ F L 2 [T] P N [F] ] N F γ = γ (M)dT M α γ k W k, α γ k = T k, (7) V k where γ = x,y,z. Note that P N = 1. Applying P N to (5) we obtain the following operator equation in W N : k=1 W P N G m b E a W = W0 W 0 = P N (8) σ b E N Since b a < 1, Gm E is a contracting operator and P N = 1, it can be easily shown that (8) has a unique solution W in W N, see Singer (2008), Kruglyakov (2011). Moreover, W approximates E with the first order of d in L 2 [T], where d = max d k, d k is a diameter of the k=1...n cell T k, Kruglyakov (2011). Using the definition of W N we can express the components of W = (W x,w y,w z ) as W γ = N U γ k W k, γ = x,y,z (9) k=1 Using (7),(8),(9) and taking into account that σ a,σ b are piecewise functions, we obtain the following system of linear equations for the coefficients U n = (U x n,u y n,u z n), n = 1...N: where U n + N ˆγ k Kk n U k = U 0 n, (10) k=1 K n k = Î + 2 σ V b kσn ˆB b n k n B n k = Ĝ E (M,M 0 )dt M dt M0 T k T n ˆγ k = σk a σb k σa k +σb k σ U 0 n n = b V n T n E N (M)dT M Note that K k n, B k n, Î, ˆγk are 3 3 matrices, Î is an identity matrix, ˆγk is a diagonal matrix. It is clear that the system (10) has a unique solution. The main goal of this paper is to present a new effective high parallel solver for this system. ISBN (DVD) 675 (11)

4 The projective method approach in the electromagnetic integral equations... 3 Implementation issues and features 3.1 Memory requirements The main challenge of using the integral equation approach for numerical solving of 3D EM problems is the necessity to solve the system of linear equations with dense matrices (10). Moreover, the challenge is not only to solve such systems, but to store their large matrices in the RAM. The standard approach, see Avdeev et al. (1997), is to use the following property of ĜE: Ĝ E (M,M 0 ) = ĜE(x x 0,y y 0,z,z 0 ) (12) To implement the method we now consider T R 3 to be a rectangular domain. As before T is divided in N = N x N y N z cells T k, k = 1,...,N, where N x,n y,n z is the number of cells in XYZ directions respectively. Suppose in addition that each T k has the same size h x h y in XY plane. Hence, B n k = Ĝ E (M,M 0 ) = B ( ) n k Ix n Ix,I k y n Iy,I k z,i n z k, (13) T n T k where Ix n,k = 1...N x, Iy n,k = 1...N y, Iz n,k = 1...N z, n,k = 1...N. Therefore B n k is a block Toeplitzmatrixinducedbytheblockvector(C y (N y 1),Cy (N y 2),...Cy N y 2,Cy N y 1 ). Eachblock C y i, i = (N y 1)...N y 1 is also a block Toeplitz matrix and is induced by the block vector (D (N i x 1),Di (N x 2),...Di N x 2,Di N ). The x 1 Di j is a 3 3 block matrix with the structure Q xx Q xy Q xz Dj i = Q(i,j) = Q yx Q yy Q yz (14) Q zx Q zy Q zz Here Q αβ are the matrices of the order N z, α,β = x,y,z, i = (N y 1)...N y 1, j = (N x 1)...N x 1. Let A be the matrix corresponding to the system of linear equations (10). We then can express A as A = S +R 1 BR 2, (15) } where S, R 1, R 2 are the diagonal matrices; B = { Bk n is the block Toeplitz matrix described above. In view of (15) it follows that we need only 36 N x N y Nz 2 16+O(N x N y N z ) bytes to store matrix A in double precision. Using the equivalence G xy = G yx this requirement can be reduced to 32 N x N y Nz 2 16+O(N x N y N z ) bytes as in Avdeev et al. (1997). Now we show how to reduce this memory requirement by half. Our result is based on the following Lemma 1. Lemma 1. If ĜE(M,M 0 ) is an electrical Green s tensor of any layered media, then in Cartesian coordinates it possesses the following symmetric and antisymmetric properties G E zx(x x 0,y y 0,z,z 0 ) = G E xz(x x 0,y y 0,z 0,z) G E zy(x x 0,y y 0,z,z 0 ) = G E yz(x x 0,y y 0,z 0,z) G E xx(x x 0,y y 0,z,z 0 ) = G E xx(x x 0,y y 0,z 0,z) G E yy(x x 0,y y 0,z,z 0 ) = G E yy(x x 0,y y 0,z 0,z) G E zz(x x 0,y y 0,z,z 0 ) = G E zz(x x 0,y y 0,z 0,z) G E xy(x x 0,y y 0,z,z 0 ) = G E yx(x x 0,y y 0,z,z 0 ) G E xy(x x 0,y y 0,z,z 0 ) = G E xy(x x 0,y y 0,z 0,z) (16) 676

5 This lemma is a trivial corollary from Lorentz reciprocity, see Ward and Hohmann (1988) and (12). Using (13) and (16) we obtain the following properties of the blocks of matrix Q: Q zx = Q T xz,q zy = Q T yz, Q xx = Q T xx,q yy = Q T yy,q zz = Q T zz, Q xy = Q T xy = Q yx = Q T yx, (17) where T indicates a matrix transpose. Therefore, we need to store only Q xz,q yz and upper diagonal parts of Q xx,q xy,q yy,q zz. Hence, we need 8 N x N y N z (2N z +1) 16 bytes to store B k n which is only one half of the memory requirements to Avdeev et al. (1997). 3.2 The computation of B k n The next computational challenge of projective approach is an evaluation of the coefficients B k n, n,k = 1...N, i.e. the double volumetric integrals of the ĜE in the RHS of (13). The components of ĜE are the improper integrals containing the Bessel functions. The integration in vertical direction is performed analytically using the fundamental function of layered media approach from Dmitriev et al. (2002). The main problem, however, is the integration over the horizontal domains. In this case we need to compute with the appropriate accuracy the fifth-order integrals over the fast-oscillating functions. To compute these integrals we change the order of integration and make the appropriate substitution. This allows us to convert the fifth-order integral to the convolution with the special kernel. Then we compute the spectrum of this kernel and build the quadrature formula based on Shannon s interpolation. It is important to emphasize that both the nodes and the weights in the obtained formula significantly depend on the integration domains. At the same time their computational cost is independent of the integration domains. Moreover, the integration over different horizontal domains is completely data independent, which we use in our parallel algorithm. 3.3 Parallelization The most essential part of any iterative method for solving a system of linear equations is the matrix-vector multiplication. Since matrix B is a block Toeplitz matrix, we can use the 2D Fast Fourier Transform (FFT) to speed up this operation, see Avdeev et al. (1997). Therefore, instead of matrices Q(i,j) we need to store their 2D discrete Fourier transformations Q(i,j). It is obvious that storing Q(i,j) requires the same amount of memory as storing Q(i,j) itself. The algorithm for the multiplitcation of block Toeplitz matrix B on some vector V W N can be described as follows: 1. Compute 3N z forward 2D FFT of vector V; 2. Compute 4N x N y algebraic matrix-vector multiplication to obtain vector Ṽ; 3. Compute 3N z backward 2D FFT of vector Ṽ. In the second step of the algorithm above all 4N x N y multiplications are data independent. This allows us to use the following scheme of distributed data storage and parallel algorithm of IE solver. Assume, for the sake of simplicity, that we have 2N y nodes. The distributed storage of matrix is organized as follows. The block-vector Q(k,j), j = 0...2N x 1 is stored at kth node, k = 1...2N y (see Table 1). ISBN (DVD) 677

6 The projective method approach in the electromagnetic integral equations... Node 1 Node 2... Node 2N y Q(0,0) Q(1,0)... Q(2Ny 1,0) Q(0,1) Q(1,1)... Q(2Ny 1,1).... Q(0,2N x 1) Q(1,2Nx 1)... Q(2Ny 1,2N x 1) Table 1: Matrix storage organization Using this storage organization, we developed the solver with the following features of parallelization: 1. The computation of the coefficients of matricies B, S, R 1, R 2 stored at different nodes is completely data independent. It is performed simultaneously for all these matrices. 2. The computation of Fourier transform of B is done by distributed code from well-known FFTW3 library; 3. The interative method is implemented using the distributed FGMRES implementation by Fraysse et al. (2003); 4. The distributed matrix-vector calculation is processed as described above using FFTW3 library for distributed FFT and OpenBlas for algebraic matix-vector calculations; 5. The hybrid MPI+OpenMP scheme is used at all stages described above. The computational experiments performed at Bluegene/P and HPC Lomonosov from MSU showed good increment in speed depending on the number of processes for different models (see Figure 2). The matrix calculation time includes time of FFT calculation of B. One can see that the speed increment is close to a linear Matrix calculation 10 3 Equation solving Time, s 10 2 Time, s Number of processes Number of processes Bluegene/P Bluegene/P Lomonosov N x =1024 N x =128 N x =128 N Linear speed up N y =1024 N y =1024 y =128 N z =10 N z =50 N z =50 Figure 2: Efficiency of parallelization 678

7 4 High conductivity contrast modeling The accurate modeling of the EM field in a high conductivity contrast media is one of the most complex problems of EM modeling. We used the COMMEMI3D3 model (see Figure 3) as one of the test models for our solver. This model consists of 7 blocks placed in layered media. Their resistivity is stated at the Figure 3 legend. The normal vertical section of the model consists of three layers with the resistivity of 10 3, 10 4 and 10 Ω m and with thickness of the first and the second layers of 1 and 6.5 km respectively. The maximum conductivity contrast in this model is more than D view 6 Top view 5 Z,km Y,km 300 Ω m Ω m X,km 100 Ω m Ω m Y,km X,km Ω m 0.1 Ω m 0.3 Ω m Profile Figure 3: Model COMMEMI3D3 The results of magnetotellurics (MT) modeling at profile y = 3.8 km for 1Hz are presented in Figures 4, 5. The solid lines are the results of modeling with different cell sizes by our IE solver. The green crosses are the result of modeling by the high-order finite elements solver from Grayver and Kolev (in review). One can see that for the large cells (black curves) the correspondence between the above methods is bad. However, decreasing the cell sizes results in a good fit between the methods. To decrease the cell sizes means to increase the number of cells. Thus the best fit requires more than cells and more than 2TB of RAM to store the matrix. This emphasizes the importance of good parallelization and decreasing of memory requirements for IE solvers ρ xy 10 3 ρ yx m m 50m 50m 50m 25m 25m 25m 6.25m 25m 12.5m 25m 6.25m 12.5m High-order FE Figure 4: Apparent resistivity ρ xy, ρ yx at profile y = 3830 for 1 Hz depends on cell size ISBN (DVD) 679

8 The projective method approach in the electromagnetic integral equations Φ xy 96 Φ yx m m 50m 50m 50m 25m 25m 25m 6.25m 25m 12.5m 25m 6.25m 12.5m High-order FE Figure 5: Impedance tensor phases Φ xy, Φ yx at profile y = 3830 for 1 Hz depends on cell size 5 Conclusion The presented solver for 3D integral equations of electrodynamics requires only half of an amount of RAM compared with the other existing 3D IE solvers. The high parallelization degree allows us to efficiently use the modern HPC systems and to perform computations for large scale models (up to 10 8 cells) with a complex conductivity distribution. The good fit with the high-order finite elements method demonstrates the high accuracy of our solver, including the solving of the high conductivity contrast problems. The developed solver named Gnu Integral Equation Modeling in ElectroMagnetic Geophysics (GIEM2G) is implemented as hybrid MPI+OpenMP software on Fortran language. The GIEM2G is an open source software distributed under the GPLv2 license and it is available at 6 Acknowledgements This work has been supported by the Russian Foundation for Basic Research under grant no Authors acknowledge the team of HPC CMC Lomonosov MSU for the access to Bluegene/P HPC and the team of the Lomonosov MSU Research Computing Center for the access to HPC Lomonosov Sadovnichy et al. (2013). References Avdeev, D. B., A. V. Kuvshinov, O. V. Pankratov, and G. A. Newman(1997). High-performance three-dimensional electromagnetic modelling using modified neumann series. wide-band numerical solution and examples. J. Geomagn. Geoelectr. 49, Dmitriev, V., A. Silkin, and R. Farzan (2002). Tensor green function for the system of maxwellś equations in a layered medium. Computational Mathematics and Modeling 13(2), Fraysse, V., L. Giraud, S. Gratton, and J. Langou (2003). A set of GMRES routines for real and complex arithmetics on high performance computers. 680

9 Grayver, A. and T. Kolev (in review). Large-scale 3d geo-electromagnetic modeling using parallel adaptive high-order finite element method. Geophysics. Kruglyakov, M. (2011). Modified integral current methods in electrodynamics of nonhomogeneous media. Computational Mathematics and Modeling 22(3), Pankratov, O., D. Avdeyev, and A. Kuvshinov (1995). Electromagnetic-field scattering in a heterogeneous Earth: A solution to the forward problem. Physics of the Solid Earth 31(3), Sadovnichy, V., A. Tikhonravov, V. Voevodin, and V. Opanasenko (2013). Lomonosov : Supercomputing at Moscow State University. In Contemporary High Performance Computing: From Petascale toward Exascale, Chapman& Hall/CRC Computational Science, pp Boca Raton, United States. Singer, B. (2008). Electromagnetic integral equation approach based on contraction operator and solution optimization in Krylov subspace. Geophys. J. Int. 175, Ward, S. H. and G. W. Hohmann (1988). 4. Electromagnetic Theory for Geophysical Applications, Chapter 4, pp ISBN (DVD) 681