FEMAG: A High Performance Parallel Finite Element Toolbox for Electromagnetic Computations

Transcription

1 Internationa Journa of Energy and Power Engineering 2016; 5(1-1): Pubished onine November 28, 2015 ( doi: /j.ijepe.s ISSN: X (Print); ISSN: X (Onine) FEMAG: A High Performance Parae Finite Eement Toobox for Eectromagnetic Computations Tao Cui 1, Xue Jiang 2, Weiying Zheng 1, * 1 Nationa Center for Mathematics and Interdiscipinary Sciences, State ey Laboratory of Scientific and Engineering Computing, Institute of Computationa Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China 2 Department of Mathematics, Beijing University of Posts and Teecommunications, Beijing, China Emai address: tcui@sec.cc.ac.cn (Tao Cui), jxue@sec.cc.ac.cn (Xue Jiang), zwy@sec.cc.ac.cn (Weiying Zheng) To cite this artice: Tao Cui, Xue Jiang, Weiying Zheng. FEMAG: A High Performance Parae Finite Eement Toobox for Eectromagnetic Computations. Internationa Journa of Energy and Power Engineering. Specia Issue: Numerica Anaysis, Materia Modeing and Vaidation for Magnetic Losses in Eectromagnetic Devices. Vo. 5, No. 1-1, 2016, pp doi: /j.ijepe.s Abstract: This paper presents a parae finite eement toobox for computing arge eectromagnetic devices on unstructured tetrahedra meshes, FEMAG Fem for EectroMagnetics on Adaptive Grids. The finite eement toobox deas with unstructured tetrahedra meshes and can sove eectromagnetic eddy current probems in both frequency domain and time domain. It adopts high-order edge eement methods and refines the mesh adaptivey based on reiabe and efficient finite eement a posteriori error estimates. We demonstrate the competitive performance of FEMAG by extensive numerica experiments, incuding TEAM (Testing Eectromagnetic Anaysis Methods) Probem 21 and the simuation for a singe-phase power transformer. eywords: FEMAG, Eddy Current Probem, Adaptive Finite Eement Method, Parae Computation, Large Eectromagnetic Device 1. Introduction Large devices in eectric engineering usuay have very compicated structures and are made of anisotropic and noninear materias. A arge power transformer, for exampe, consists of nonmagnetic pates, magnetic oi tank, grain-oriented stee aminations, compex exciting cois and so on. It is extremey difficut to simuate the whoe structure of a arge power transformer. Commercia software is usuay not competent for such a task because of memory imitation and ow scaabiity for arge computers. Moreover, the inefficiency of agebraic system sover aso imits their appications to arge-scae computing. The purpose of this paper is to propose a parae finite eement toobox, FEMAG (Fem for EectroMagnetics on Adaptive Grids), and to demonstrate the competitive performance of our finite eement agorithms and the FEMAG. We propose to study the foowing eddy current probem: (1) where E is the eectric fied, B = µh is the magnetic fux, H is the magnetic fied, and J is the current density defined by Here σ is the eectric conductivity, J s is the source current density carried by some cois, and Ω denotes the c conducting region. The eddy current probem is a quasi-static approximation of Maxwe s equations at very ow frequency by negecting the dispacement currents in Ampere s aw (see [1]). The materia parameters coud be very compicated for eectric engineering appications. For exampe, grain-oriented stee aminations are widey used in iron cores and magnetic shieds of arge power transformers (see Figure 1 for a TEAM benchmark mode) [2]. They are very anisotropic and have mutipe scaes. The transformer 6 size coud be 10 times the thickness of coating fims over aminations. The magnetic permeabiity differs a ot in the roing direction and the other two orthogona directions. (2)

2 58 Tao Cui et a.: FEMAG: A High Performance Parae Finite Eement Toobox for Eectromagnetic Computations Furthermore, the energy oss in cois shoud aso be considered for compex exciting source. This necessitates huge number of eements in partitioning the domain. Fu three-dimensiona (3D) finite eement simuation is very difficut due to extensive unknowns. With the evoution of the computer technoogy, the massivey parae computing with ten thousands of even hundred thousands of CPU cores becomes one of the trends of the scientific computing. The main chaenge for simuating arge transformers is focused on deveoping efficient agorithms for soving the discrete probem and scaabe parae finite eement codes for arge supercomputers. The second issue concerns ow reguarity of the soution of Maxwe s equations. It is we-known that the soution of Maxwe's equations may have oca singuarities at corners and edges of the structure and materia interfaces. Numerica methods based on uniform meshes are inefficient in resoving the oca singuarities. The adaptive finite eement method (AFEM) based on the a posteriori error estimates has been successfuy and widey appied in many other areas [3-6], which provides a systematic way to achieve the optima computationa compexity by refining the mesh according to the oca a posteriori error estimator on the eements. Unfortunatey, parae impementation of the AFEM on distributed memory parae computers is very difficut because of the compexities of the mesh management and oad baance issues. Aso, highy efficient numerica methods for soving the inear system resuting from finite eement discretization are required. For faciitating impementing the AFEM, we have deveoped the toobox PHG, Parae Hierarchica Grid [7]. The motivation of this toobox is to support the research on AFEM agorithms and deveopment of AFEM codes. PHG deas with conforming tetrahedra meshes and uses bisection for adaptive mesh refinement and MPI for message passing. Using the idea of object oriented design, the detais of compex mesh management and paraeism are hidden from users. PHG provides supports for adaptive finite eement computations, such as finite eement bases (incuding the Nédééc edge eements for eectromagnetic computations) [8], numerica quadrature, and basic operations with finite eement functions. For buiding, assembing, and soving inear systems and eigenvaue probems resuting from finite eement discretization, an unified inear agebra modue for manipuating distributed sparse matrices stored in compressed sparse rows (CSR) and distributed vectors is provided,. Load baancing is achieved through mesh repartitioning and redistribution. FEMAG is deveoped based on PHG. It soves both time-dependent and time-harmonic eddy current probems and can dea with noninear and anisotropic materias. For arge-scae computing, the scaabiity of agebraic system sover pays the key roe in parae finite eement codes. An efficient sover shoud possess two properties: (1) its convergence rate shoud keep quasi-uniform as the number of eements increases; (2) its convergence rate shoud not degenerate when using massive CPU cores. In FEMAG, the Maxwe sover for the discrete probem is a GMRES agorithm with the auxiiary space preconditioning [9]. And the auxiiary probems are soved by the conjugate gradient method with the Boomer agebraic mutigrid preconditioning [10]. Figure 1. A magnetic shied mode (TEAM Probem 21 c -M1, The magnetic shied made of stee aminations). The most chaenging task is to compute the iron oss and eddy current density in grain-oriented stee aminations. An iron core may consist of hundreds of even more than one thousand of aminations, each of which is ony mm thick. But the thickness of the coating fim over the aminations is ony 2-5µm. Thus the amination stack has muti-scae sizes and the ratio of the argest scae to the smaest scae can amount to 10 6 (see Figure 1). Fu 3D finite eement modeing is extremey difficut due to extensive unknowns from meshing both aminations and the coating fim. There are very few papers on the computation of 3D eddy currents in the iterature. In FEMAG, stee aminations are treated by two approaches: 1. Compute 3D eddy current density with an approximate eddy current mode that omits the coating fim and meshes each amination respectivey [11-12]. This approach yieds accurate resuts but requires a fine mesh for the amination stack. 2. Compute 3D eddy current density with effective or homogenized conductivity and permeabiity [13]. This approach is very fast but ess accurate in computing iron oss in aminations than the previous one. But it is more favorabe in the simuation of the whoe of a power transformer. In this paper, we present the numerica resuts for a power transformer by using homogenized conductivity and permeabiity. In this paper, we present a numerica experiment on hp-adaptive finite eement method for time-harmonic eddy current probem. The h-adaptive finite eement method reduces the error by oca mesh refinements. It has been we-studied in the a posteriori error anaysis, mesh refinement agorithms, and optima compexity. Using the idea of error equidistribution, the h-adaptive method based on a posteriori error estimates coud yied a quasi-optima approximation with agebraic convergence rate

3 Internationa Journa of Energy and Power Engineering 2016; 5(1-1): η CN h p/d h where η h is the a posteriori error estimate, d is the spatia dimension, p is the poynomia degree, and N h is the number of degrees of freedom. However, due to the singuarity of the soution, the quasi-optimaity η h wi degenerate for higher-order finite eements since the constant C may bow up with increasingp. The hp-adaptive finite eement method reduces the error by both oca mesh refinement and oca improvement of poynomia degrees. It is more efficient than the pure h-adaptive or p-adaptive methods and coud reduce the error exponentiay. For exampe, the optima convergence rate of the hp-adaptive method is, 1/ 3, η hp Ce δn hp for two dimensiona eiptic probems [14], and is aso conjectured to be η hp Ce δn hp 1/5, for three dimensiona eiptic probems [15], where C, δ are positive constants independent of h and p, η is the a hp posteriori error estimate for the hp-adaptive method, and N hp is the number of degrees of freedom. But for soutions with edge singuarities, the meshes eading to the exponentia convergence must be obtained by anisotropic refinements, that is, by using "neede eements'' which are parae to the edges (see [15] for more comments). The impementation of the hp-adaptive method is very chaenging for higher dimensiona probems. In this paper, we present an hp-type a posteriori error estimate for the time-harmonic eddy current mode. Then based on the a posteriori error estimate, an hp-adaptive agorithm is proposed by the strategy of predicted error reduction. We impemented in FEMAG the parae hp-adaptive method on unstructured tetrahedra meshes. The numerica experiment is performed on TEAM Probem 21 a -2. The hp-adaptive method shows exponentia decay of the a posteriori error estimate and the numerica resuts agree we with experimenta data. The second numerica experiment is performed on TEAM Probem 21 b -MN which has a magnetic pate and a nonmagnetic pate. It shows that, on CPU cores, FEMAG has very good scaabiity for noninear eddy current probems. The ast numerica experiment is performed on a homemade singe-phase transformer. We use effective materia parameters for iron core and iron yokes. The reative error of the iron oss is ess than 10%. 2. The A-Formuation of Eddy Current Probems We start by the A-formuation of eddy current mode for aminated conductors. Let A be the magnetic vector potentia satisfying A =B. Then the A-formuation of eddy current mode reads (cf. e.g. [11]): σ A + ( ν A) =J s in Ω, t A n =0 on Γ, where Ω is the truncated domain with boundary Γ, n is the unit outer norma of Γ, and stands for the excited current or source current satisfying J s =0. In (1), σ is the conductivity, ν =diag(h 1 B 1, H 2 B 2, H 3 B 3 ) is the noninear and anisotropic reuctivity, and H =(H 1, H 2,H 3 ) denotes the magnetic fied and is determined by BH-curves and the magnetic fux B =(B 1, B 2,B 3 ). Now we introduce some function spaces used in this paper: L 2 (Ω) = { u: u(x) 2 dx < }, Ω H(cur,Ω) = { u L 2 (Ω): u L 2 (Ω)}, H 0 (cur,ω) = { u H(cur,Ω): u n =0 on Ω}. The weak formuation of (1) reads: Find A H 0 (cur,ω) such that, for a v H 0 (cur,ω), Let (3) σ A v +ν( A) ( v) Ω t = J s v (4) Ω 0 = t < t < < tn = T be a partition of the time 0 1 interva [ 0, T ] and τ n = tn tn 1 denote the n-th time step. Let I n be a tetrahedra trianguation of Ω at time t n such that I n forms a tetrahedra mesh of Ω. We define the c Ω c edge eement space as foows U(I n ) ={v H 0 (cur,ω): v T P k (T) 3 for a T I n }, (5) where P k (T) is the space of poynomias of order k > 0. The fuy discrete approximation to (4) reads: Given u 0 =0, find u n U(I n ), n 1 such that for a v U(I n ), where a(u n,v) = J Ω n v (6) 3. Numerica Experiments for FEMAG 3.1. Scaabiity for Noninear Eddy Current Probem In this subsection, we report the numerica experiment for the TEAM Probem 21 b [2]. The conducting region consists of a magnetic pate and a nonmagnetic pate (see Figure 2). The unit of dimensions in Figure 2 is miimeter. The thickness of J s

4 60 Tao Cui et a.: FEMAG: A High Performance Parae Finite Eement Toobox for Eectromagnetic Computations 6 the pate is 10mm. The conductivity is Siemens/Metre. The height of each coi is 12mm and the radiuses of the inner arc and the outer arc at four corners are 10mm and 45mm respectivey. The vertica distance between them is 24mm. The two cois carry the source currents of 3000 Ampere/Turn in opposite directions. The frequency of source currents is 50 Hertz. Tabe 1. Weak scaabiity and computationa time for soving the agebraic system. Number of CPU cores Number of unknowns (miion) Computationa time (S) Weak scaabiity % 1, % % % % Cacuated iron oss Tabe 2. Iron oss for TEAM Probem 21 b -MN Measured iron oss 3.2. Hp-Adaptive Finite Eement Method for Time-Harmonic Eddy Current Probem Figure 2. Geometric iustration for TEAM Probem 21 b -MN. The second numerica experiment is to test the hp-adaptive finite eement method for TEAM Probem 21 a -2(see[2]). This probem consists of a non-magnetic stee pate with two sits and two racetrack shaped cois (see Figure 3). The stee pate 6 has a conductivity of Siemens/Metre. The driving current for each coi is 3000 Ampere/Turn and has a frequency of 50 Hz. The driving currents for the two cois are in opposite directions. Since the materia parameters are inear, we consider the time-harmonic eddy current probem where i is the imaginary unit, ω is the anguar frequency, and µ is the magnetic permeabiity in the empty space. Figure 3. Magnetic fux: numerica data versus experimenta data. We carry out the computation on the super computer Tianhe-1A, Tianjin, China. We use the second-order edge eement method of the second famiy [8] to sove the probem. The argest number of CPU cores used is and the finest mesh contains 0.44 biions of degrees of freedom. Tabe 1 shows that the weak scaabiity is arger than 70%. It aso shows that, with CPU cores and 0.44 biions of unknowns, the soution time for the agebraic system is ony 11 minutes. Tabe 2 shows the measured iron oss versus the cacuated iron oss in the stee pates. They agree very we with each other. Figure 3 shows the experimenta vaues and the numerica vaues of the magnetic fux density. Figure 3. Geometric iustration for TEAM Probem 21 a -2. Let F(I h ) be the set of interior faces on the mesh I h. For any facef F(I h ) with F = 1 2,we denote the jump of a function v across F by [ v] F =v 1 v 2. For convenience in notation, we define the residua functions on

5 Internationa Journa of Energy and Power Engineering 2016; 5(1-1): each I h and F F(I h ) as foows r = µ (J s iωσa), R = µ(j s iωσa) A j F = µ [(J s iωσa) n] F, J F = [( A) n] F. Then the error indicator is defined on each eement as foows [16] η = h p ( r + R L 2 () L 2 ()) +1 h F ( j F 2 L + J 2 (F) F L 2 (F)). F It is used to refine the mesh adaptivey. The goba and maxima a posteriori error estimates are defined by 2 η hp = η I h 1/2 p F, η max = 1/2 max η. I h FEMAG uses the predicted error decrease strategy (PEDS) to refine the mesh. The agorithm assumes that the soution is ocay smooth and the optima convergence can be obtained by either oca h-refinement or oca p-refinement. Then on each eement marked for refinement, an error decrease factor λ is computed to judge which of the h-refinement and p-refinement shoud be performed. According to [16], λ can be defined heuristicay as foows where λ = p T p p T /2 T p T /3 T is the parent eement of. It may happen that, T = if the eement is not refined at the previous adaptive iteration. The PDES is presented as foows: Agorithm PDES. Given a toerance ε > 0 and an initia mesh I 0 and an initia distribution of poynomia degrees P 0. Set = 0 and θ (0,1 1 ). 1. Sove for the finite eement soution u hp U(I,P ). 2. Compute the oca error indicator η for a I, the goba error estimate η, and the maxima error estimate η max. 3. Whie η > ε do Iˆ = I : η > θ1η be the set of max eements marked for refinement. (2). Refine Î according to the predicted error decrease strategy, namey, (3). For any I ˆ, compute λ and η T. If (1). Let { } η λη, set T p p + 1; otherwise, refine using the bisection agorithm [17-18]. (4). Sove the finite eement soution u hp U(I,P ) based on the new mesh and the new distribution of poynomia degrees. (5). Compute the oca error indicator η for a I, the goba error estimate η, and the maxima error estimate η max. end whie. Figure 4. Eddy current distribution on a sice of the stee pate. Figure 5. Vaues of the magnetic fux density on two ines: numerica data versus experimenta data.

6 62 Tao Cui et a.: FEMAG: A High Performance Parae Finite Eement Toobox for Eectromagnetic Computations Figure 4 shows the eddy current distribution on a sice of the stee pate. The fow directions are ceary iustrated. Figure 5 shows the first component of the magnetic fux density B. The vaues are taken at some discrete points aong two ines (x,y,z):x = ,y =0.0 { } and (x,y,z):x = ,y =0.0 { }which correspond to the two ines {(x,y,z):x = ± ,y=0.0}in [2] respectivey. The numerica data agree we with the experimenta vaues. Figure 6 shows the reduction rates of the a posteriori error estimate by the h-adaptive method for p = 1, the PERS of the hp-adaptive method, and the Maximum Strategy of the hp-adaptive method used in [16]. We find that the hp-adaptive methods show great superiority over the h-adaptive method in reducing the error. It aso shows that both of the agorithms yied exponentia decay of the a posteriori error estimate: Figure 7 shows the hp-pair (I 47,P 47 ) by the PDES agorithm on the sice Σ 1 = {(x,y,z) Ω:x = }, where the number of degrees of freedom is 1,729,148. From the figures, we find roughy that the hp-adaptive method use 1. ower order eements near the boundary where the error is very sma, 2. ower order eements in the conducting domain where the soution varies rapidy, and 3. high order eements in the insuating region away from the boundary where the error is moderate. Since the soution varies rapidy in the conducting domain, I 47 dispays a fine mesh there in Figure 8. η hp Ce δn hp 1/5. Figure 6. Decreasing rate of the a posteriori error estimate: hp-adaptive FEM versus h-adaptive FEM. Figure 7. An adaptivey refined mesh and the distribution of poynomia degrees on one sice after 47 refinements. Figure 8. An adaptivey refined mesh after 47 refinements Simuation for a Singe-Phase Power Transformer The ast numerica experiment is carried out for a singe-phase power transformer. The magnetic conductors are the wa of the oi tank, the iron core, and two iron yokes. Moreover, the mode aso contains nonmagnetic conductors and cois which carry exciting source currents (see Figure 9). The computations are performed on the custer LSEC-III of the State ey Laboratory on Scientific and Engineering Computing, Chinese Academy of Sciences. We partition the computationa domain into a tetrahedra mesh with 1,381,600 eements. The poynomia degree of the finite eement space is set by k = 2. The tota number of

7 Internationa Journa of Energy and Power Engineering 2016; 5(1-1): degrees of freedom is 13,147,086. We use 128 CPU cores in the computation. At each time step, the reative residua of the Newton method is reduced to be ess than Tabe 3 shows the measured iron oss versus the cacuated iron oss in the conductors. The reative error is about 10%. This demonstrates that FEMAG has the capabiity to simuate compicated eectromagnetic devices. Tabe 3. Iron oss in the amination and the magnetic pate (W). Cacuated iron oss Measured iron oss Figure 9. The geometric iustration of a singe-phase transformer. 4. Concuding Remarks In this paper, we report the parae finite eement toobox, FEMAG (Fem for EectroMagnetics on Adaptive Grids), for simuating arge eectromagnetic devices. Three numerica experiments are proposed to demonstrate the competitive performance of FEMAG, in terms of the scaabiity of parae computation, the adaptivity for both mesh refinement and distribution adjustment of poynomia degrees, and the capabiity to dea with compex eectromagnetic devices. A numerica resuts are verified by experimenta data. The FEMAG is efficient for arge-scae simuation of eectromagnetic probems and has the potentia in practica appications. Acknowedgement The authors woud ike to thank Professor Zhiguang Cheng of R & D Center, Baobian Eectric Group, for his vauabe discussions on the paper. The first author was supported in part by Nationa 863 Project of China under the grant 2012AA01A309 and China NSF grant The second author was supported in part by China NSF grant and by the Fundamenta Research Funds for the Centra Universities RC17. The third author was supported in part by China NSF grants and , by the Funds for Creative Research Groups of China , and by the Nationa Magnetic Confinement Fusion Science Program 2015GB References [1] H. Ammari, A. Buffa, and J. Nedeec, A justification of eddy current mode for the Maxwe equations, SIAM J. App. Math., 60 (2000), pp [2] Z. Cheng, N. Takahashi, and B. Forghani, TEAM Probem 21 Famiy (V.2009), approved by the Internationa Compumag Society. [Onine] Avaiabe: [3] I. Babuska and W. C. Rheinbodt, Error estimates for adaptive finite eement computations, SIAM J. Numer. Ana., vo. 15, no. 4, pp , Aug [4] J. Chen, Z. Chen, T. Cui, and L.-B. Zhang, An adaptive finite eement method for the eddy current mode with circuit/fied coupings, SIAM J. Sci. Comput., vo. 32, no. 2, pp , Mar [5] W. Zheng, Z. Chen, and L. Wang, An adaptive finite eement method for the H ψ formuation of time-dependent eddy current probems, Numer. Math., 103 (2006), pp [6] Z. Chen, L. Wang, and W. Zheng, An adaptive mutieve method for time harmonic maxwe equations with singuarities, SIAM J. Sci. Comput., vo. 29, no. 1, pp , Jan [7] L.-B. Zhang, A parae agorithm for adaptive oca refinement of tetrahedra meshes using bisection, Numer. Math. Theory, Methods, App., vo. 2, no. 1, pp , 2009.

8 64 Tao Cui et a.: FEMAG: A High Performance Parae Finite Eement Toobox for Eectromagnetic Computations [8] J.N. Nédéec, A new famiy of mixed finite eements in R 3, Numer. Math., vo. 50, no. 1, pp , [9] R. Hiptmair and J. Xu, Auxiiary Space Preconditioning for Edge Eements, IEEE Trans. Magn., vo. 44, no.6, pp , [10] V. E. Henson and U. M. Yang, BoomerAMG: A parae agebraic mutigrid sover and preconditioner, App. Numer. Math., vo. 41, no. 1, pp , Apr [11] X. Jiang and W. Zheng, An efficient eddy current mode for noninear Maxwe equations with aminated conductors, SIAM J. App. Math., 72 (2012), pp [12] W. Zheng and Z. Cheng, An inner-constrained separation technique for 3D finite eement modeing of GO siicon stee aminations, IEEE Trans. Magn., 12 (2012), pp [14] W. Guo and I. Babuska, The h-p version of the finite eement method. Part 2. Genera resuts and appications, Comput. Mech. 1 (1986) [15] I. Babuska, B. Andersson, B. Guo, J.M. Meenk, and H.S. Oh, Finite eement method for soving probems with singuar soutions, J. Comput. App. Math. 74 (1996) [16] X. Jiang, L. Zhang, and W. Zheng, Adaptive hp-finite eement computations for time-harmonic Maxwe's equations, Comm. Comp. Phys., 13 (2013), pp [17] W. Mitche, Optima mutieve iterative methods for adaptive grids, SIAM J. Sci. Stat. Comput, 13 (1992), pp [18] I. ossaczky, A recursive approach to oca mesh refinement in two and three dimensions, J. Comput. App. Math. 55 (1994) [13] X. Jiang and W. Zheng, Homogenization of quasi-static Maxwe s equations, Mutiscae Modeing and Simuation: A SIAM Interdiscipinary Journa, 12(2014). pp